Perception decision-making method based on intelligent door lock interaction system

Abstract

The invention relates to the field of electric digital data processing, in particular to an intelligent door lock interaction system-based perception decision-making method, which comprises the following steps of: acquiring perception data, and labeling an intention label and an environment noise label corresponding to each perception data to obtain labeled perception data; constructing a perception decision model and training the perception decision model to obtain a trained perception decision model; new perception data are collected, voiceprint data, relative distance data between a person and a door lock and door lock state data in the new perception data are aligned to a unified reference according to timestamps and then input into the trained perception decision model, intention probability distribution is obtained, the intention with the highest probability is selected as a decision result, and a corresponding door lock control instruction is triggered. An existing perception decision-making method based on a door lock interaction system with intelligence has the problem that the user experience is poor. According to the method provided by the invention, the user experience can be better.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing, and more specifically to a perception and decision-making method for a door lock interaction system based on embodied intelligence. Background Technology

[0002] Since its emergence, electronic locks have been continuously updated and iterated with the development of technology. There have been card + password unlocking, fingerprint unlocking, multiple biometric recognition methods, and smart locks with cameras and other network devices. With the maturity of artificial intelligence technology, smart locks that combine artificial intelligence technology have been developed.

[0003] Smart locks incorporating artificial intelligence (AI) technology are essentially a form of embodied intelligence, hence also called embodied intelligence-based locks. The interaction system of embodied intelligence-based locks uses a perception-decision method to first perceive and then make decisions to serve the user. AI-integrated smart locks acquire images of the area in front of the door through cameras, detect footsteps through microphones, sense the approach of living beings through infrared sensors, detect tampering through pressure sensors, and provide feedback to the user through vibration and light rather than simply ringing a bell. Thus, AI-integrated smart locks possess physical presence and the ability to perceive and interact. Furthermore, AI-integrated smart locks also have learning and decision-making capabilities. They can learn user behavior, enabling functions such as automatically lighting up the screen when a user approaches the door and pushing information to the user upon detecting unusual behavior from strangers.

[0004] However, users typically interact with embodied intelligence-based door locks through voice, gestures, or contact. This process is affected by environmental noise, sensor sampling differences, and scene diversity, which can easily lead to problems such as inaccurate intent recognition, response delays, or false triggers. At the same time, existing perception decision-making methods mostly use single-modality or simple multimodal fusion methods, which cannot effectively solve the differences in sampling frequency, time alignment, and feature representation of different modal data. Conventional noise reduction and feature extraction methods are prone to losing key intent signals when removing environmental interference and lack the ability to finely decouple noise and intent features. In addition, the use of fixed-weight fusion methods makes it impossible to dynamically adjust modal contributions according to environmental conditions, resulting in a significant performance degradation in noisy environments or when sensors fail.

[0005] Therefore, existing perception and decision-making methods for door lock interaction systems based on embodied intelligence suffer from poor user experience. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a perception and decision-making method for a door lock interaction system based on embodied intelligence with a better user experience.

[0007] To address the aforementioned technical problems, the present invention provides a perception and decision-making method for a door lock interaction system based on embodied intelligence, comprising:

[0008] S1. Collect perception data, which includes voiceprint data, relative distance data between the person and the door lock, and door lock status data. Align the voiceprint data, relative distance data between the person and the door lock, and door lock status data to a unified benchmark according to the timestamp. Label each perception data with the corresponding intent label and environmental noise label to obtain labeled perception data.

[0009] S2. Construct a perception-decision model and train the perception-decision model to obtain a trained perception-decision model;

[0010] S201. Calculate the multimodal alignment matrix for each sample;

[0011] S202. Construct orthogonal constraint regularization terms;

[0012] S203. Calculate the sparse weighted feature matrix for each sample;

[0013] S204. Calculate the fusion feature matrix for each sample;

[0014] S205. Calculate the hidden state updated at the current time;

[0015] S206. Calculate the probability distribution of the final intention for each sample;

[0016] S207. Calculate the total loss function to obtain the perception decision model;

[0017] S208. Iteratively train and update the parameters of the perception-decision model to obtain the trained perception-decision model;

[0018] S3. Collect new sensing data, align the voiceprint data, relative distance data between the person and the door lock, and door lock status data in the new sensing data to a unified benchmark according to the timestamp, and then input them into the trained sensing decision model to obtain the intent probability distribution. Select the intent with the highest probability as the decision result and trigger the corresponding door lock control command.

[0019] As a further improvement of the present invention: the sensing data collected in S1 includes:

[0020] Sensing data from different real-world usage scenarios is collected by sensors installed in the door lock.

[0021] The sensing devices include a voiceprint sensor for acquiring voiceprint data, a distance sensor for acquiring relative distance data between a person and a door lock, and a motion sensor for acquiring door lock status data;

[0022] Real-world usage scenarios include indoors, outdoors, quiet environments, noisy environments, daytime, and nighttime.

[0023] As a further improvement of the present invention: step S201 includes:

[0024] S2011. Calculate the time offset compensation function for each mode;

[0025] S2012. Construct convolution kernels for each modality;

[0026] S2013. Calculate the multimodal alignment matrix of each sample based on the time offset compensation function of each modality and the convolution kernel of the m-th modality.

[0027] As a further improvement of the present invention: step S202 includes:

[0028] S2021. Calculate the environmental noise embedding vector of each sample based on the multimodal alignment matrix of each sample;

[0029] S2022. Calculate the intent feature embedding matrix of each sample based on the environmental noise embedding vector of each sample;

[0030] S2023. Construct orthogonal constraint regularization terms.

[0031] Preferably, step S2023 includes:

[0032] S20231. An orthogonality constraint is imposed by applying an orthogonality constraint threshold such that the square of the Frobenius norm of the product of the transpose of the environmental feature extraction weight matrix and the intention feature extraction weight matrix is ​​less than the orthogonality constraint threshold.

[0033] S20232. Construct orthogonal constraint regularization terms based on orthogonality constraints.

[0034] As a further improvement of the present invention: step S203 includes:

[0035] S2031. Calculate the attention score matrix for each sample;

[0036] S2032. Calculate the sparse gating matrix;

[0037] S2033. Calculate the sparse attention weight matrix for each sample based on the sparse gating matrix;

[0038] S2034. Calculate the sparse weighted feature matrix of each sample based on the sparse attention weight matrix of each sample.

[0039] As a further improvement of the present invention: step S204 includes:

[0040] S2041. Calculate the attention weights for each modality based on the sparse weighted feature matrix of each sample;

[0041] S2042. Calculate the feature vector of each modality projection based on the multimodal alignment matrix of each sample;

[0042] S2043. Calculate the fusion feature matrix of each sample based on the feature vectors of each modality projection and the attention weights of each modality.

[0043] As a further improvement of the present invention: step S205 includes:

[0044] S2051. Calculate the updated gate vector at the current time step;

[0045] S2052. Calculate the candidate hidden state at the current time;

[0046] S2053. Calculate the updated hidden state at the current time based on the updated gating vector and the candidate hidden state at the current time to update the spatiotemporal state.

[0047] As a further improvement of the present invention: step S206 includes:

[0048] S2061. Calculate the state transformation fusion feature matrix of each sample based on the hidden state updated at the current time;

[0049] S2062. Calculate the probability distribution of the final intention for each sample.

[0050] As a further improvement of the present invention: step S207 includes:

[0051] S2071. Calculate the intention classification loss;

[0052] S2072. Calculate the time series smoothing loss;

[0053] S2073. Calculate the modal bifurcation loss;

[0054] S2074. Calculate the total loss function based on intent classification loss, temporal smoothing loss, and modal divergence loss.

[0055] The beneficial effects of the present invention are as follows: The perception and decision-making method of the door lock interaction system based on embodied intelligence provided by the present invention has a better user experience.

[0056] First, this method employs a multimodal soft alignment approach based on a time offset compensation function and learnable convolutional kernels, maintaining temporal consistency between modalities at different sampling frequencies and avoiding information loss and feature distortion caused by hard alignment. Second, it utilizes a deentanglement coding mechanism for environmental noise and intent features, effectively separating noise and key signals through a dual-channel bottleneck structure and orthogonal constraints, thus improving the robustness of intent recognition. Third, it employs a dynamic sparse attention mechanism based on feature saliency, adaptively selecting key time segments, reducing computational redundancy while focusing on sudden intent features, improving real-time performance and accuracy. Finally, it constructs a fusion framework combining modal weight modulators and spatiotemporal dependency modeling, dynamically balancing multimodal contributions and capturing the joint dependencies of temporal and spatial relationships, thereby achieving context-adaptive intent recognition. Through these optimizations, this method achieves more accurate user intent recognition, with a low-latency, high-precision perception process that adapts to changing environments, providing a better user experience. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the principle of the present invention;

[0058] Figure 2 Dynamically align and compare box plots for multimodal time series data;

[0059] Figure 3 Experimental diagram showing the cumulative distribution of alignment errors;

[0060] Figure 4 Experimental diagram for analyzing the effects of dynamic sparse attention mechanism;

[0061] Figure 5 Experimental figures comparing the robustness of cross-modal fusion methods under different environments. Detailed Implementation

[0062] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0063] Definitions of relevant terms used in this invention:

[0064] (1) Embodied Artificial Intelligence (EAI) is an intelligent system that relies on physical entities to achieve intelligent growth through interaction with the environment. It is a form of intelligence that integrates perception, movement, and cognition.

[0065] like Figure 1 As shown, the perception and decision-making method for a door lock interaction system based on embodied intelligence provided by this invention includes:

[0066] S1. Collect perception data under different real-world usage scenarios through sensing devices installed in the door lock. The perception data is multimodal data, including voiceprint data, relative distance data between the person and the door lock, and door lock status data. Align the voiceprint data, relative distance data between the person and the door lock, and door lock status data to a unified benchmark according to timestamps. Manually label each perception data with the corresponding intent label and environmental noise label to obtain labeled perception data. Store the labeled perception data in a structured format to provide high-quality, fine-grained supervision signals.

[0067] The sensing devices include a voiceprint sensor for acquiring voiceprint data, a distance sensor for acquiring relative distance data between a person and a door lock, and a motion sensor for acquiring door lock status data;

[0068] The voiceprint sensor has a sampling frequency of 8kHz. The voiceprint data includes user voice commands and ambient background sounds. User voice commands include unlocking commands, deadbolting commands, etc. Ambient background sounds include wind and rain sounds, noisy human voices, range hood sounds, etc.

[0069] The distance sensor has a sampling frequency of 10Hz, and the relative distance data is a sequence of distance values ​​in meters.

[0070] Door lock status data includes the vibration or movement status of the door lock to identify knocking, touching, prying, etc.

[0071] Real-world usage scenarios include indoor, outdoor, quiet, noisy, daytime, and nighttime conditions; by collecting perception data from different real-world usage scenarios, we can obtain perception data under various environmental conditions to ensure the diversity and representativeness of the perception data.

[0072] Intent labels include unlock, deadbolt, password setting, and no operation;

[0073] Environmental noise labels include wind and rain noise, vehicle noise, and accidental vibrations;

[0074] The labeled perception data includes the perception data and its corresponding label files;

[0075] S2. Construct a perception-decision model and train the perception-decision model to obtain a trained perception-decision model;

[0076] S201. Calculate the multimodal alignment matrix of each sample based on the time offset compensation function of each modality and the convolution kernel of each modality;

[0077] S2011. Calculate the time offset compensation function for each mode;

[0078] Based on the deviation between the acquisition timestamp of each modality and the reference timestamp, the alignment tolerance control coefficient, and the reference timestamp, the time offset compensation function of each modality is calculated. The time offset compensation function of each modality is used to adjust the alignment weight of the modality data in the time dimension to avoid information leakage caused by hard alignment and to maintain the temporal consistency between modalities.

[0079] Time offset compensation function for the m-th mode The calculation formula is:

[0080] ,

[0081] In the formula, This represents the deviation between the acquisition timestamp of the m-th mode and the reference timestamp; m is the mode index. Indicates voiceprint modality, Indicates the distance sensor mode. Indicates the motion sensor mode; It is a natural constant; This is the alignment tolerance control coefficient, used to adjust the sensitivity of the alignment process to time deviations, and is preferably set to 1.0; This represents the absolute value operator; The mean time point of each modality time series is taken as the base timestamp;

[0082] An excessively large alignment tolerance control factor can lead to overly strict alignment, thus ignoring reasonable deviations. If the sensitivity is too small, the alignment will be loose, which can easily generate noise. Therefore, in door lock applications, to balance the sensitivity, the... Setting it to 1.0 ensures that the bias weight within 0.1 seconds decays to 0.9048 when calculating the time offset compensation function for each mode, thus maintaining robustness. For example, assuming a base timestamp... Seconds, the deviation between the acquisition timestamp and the reference timestamp of the voiceprint modality. Second, If set to 1.0, then the time offset compensation function for the first mode... .

[0083] As an exponential decay function, this term can smoothly handle time deviations and avoid edge information leakage caused by hard alignment of the sliding window. Combined with the alignment tolerance control coefficient, it can effectively mitigate these issues. Control decay rate, reference timestamp Using a reference point to ensure that each modality is aligned around the center can preserve the original temporal consistency, such as the causal relationship between speech and motion, while adaptively compensating for offsets and improving the accuracy of subsequent fusion.

[0084] S2012. Construct convolution kernels for each modality;

[0085] For each modality, a corresponding convolution kernel is set, and the convolution kernel parameters are optimized through backpropagation to adaptively extract and align multimodal features;

[0086] Define the convolution kernel of the m-th mode as The dimensions are:

[0087] ,

[0088] In the formula, 5 represents that the step size of the convolution kernel of the m-th mode in the time dimension is 5; Let m be the feature dimension of the m-th mode; Let be the number of output channels for all modalities, and let be the convolution kernel for all modalities. To set the same number of output channels, you can set... ;

[0089] S2013. Perform intermodal convolution alignment: Calculate the multimodal alignment matrix for each sample based on the time offset compensation function of each modality and the convolution kernel of the m-th modality;

[0090] The original data of each modality is convolved with the convolution kernels of their respective modalities, and a weighted time offset compensation function is applied to generate an aligned multimodal alignment matrix. The temporal information of voiceprint, distance and motion is fused to achieve soft alignment of multimodal data.

[0091] No. The multimodal alignment matrix of each sample fuses temporal information of voiceprint, distance, and motion. Multimodal alignment matrix of samples Represented as:

[0092] ,

[0093] In the formula, This represents a convolution operation with an output dimension of . ; This represents the total number of time steps. This represents the number of output channels for all modes. The convolution kernel for the m-th mode; Indicates the first The original data of the m-th mode of a sample; i is the sample index;

[0094] The term extracts modality-specific features through convolution operations, and a time-shift compensation function. Weighted adjustment of time sequence weights achieves soft alignment.

[0095] In one embodiment, the modalities of the raw data are in vector format, and the voiceprint modality is a one-dimensional audio waveform sampled at 8kHz. For example, one data point of the first modality is... The distance sensor mode is a sequence of distance values ​​sampled at 10Hz; one data point from the second mode is... The unit is meters, and so on for the third mode. Furthermore, a single data point may represent a specific event, such as... This represents data representing a single door lock interaction event.

[0096] The sensing data comes from different sensors with different sampling frequencies, such as an 8kHz sampling frequency for the voiceprint sensor and a 10Hz sampling frequency for the distance sensor. This leads to misalignment of the temporal data. Conventional processing methods use sliding window alignment, but this is prone to leakage of edge information, thereby compromising the temporal consistency between modalities and affecting the accuracy of data fusion. S2 overcomes the above problems by calculating a time offset compensation function and constructing a learnable convolutional kernel to perform convolutional alignment between modalities through inter-modal convolutional alignment.

[0097] To verify the performance advantages of dynamic alignment technology for multimodal time series data, sliding window alignment, fixed offset alignment, and our proposed technology (soft alignment) were compared. Figure 2 and Figure 3 .

[0098] Figure 2 This is a box-shaped diagram. Figure 2 The vertical axis represents the time alignment error in milliseconds, and the horizontal axis represents the three alignment methods. It can be seen that the median time alignment error of the sliding window alignment is about 85 milliseconds, the median time alignment error of the fixed offset alignment is about 65 milliseconds, while the median time alignment error of this technology (soft alignment) is only about 22 milliseconds. Figure 3 This is a cumulative distribution function plot, used to reveal the distribution pattern of errors. Figure 3 The horizontal axis represents the time alignment error in milliseconds, and the vertical axis represents the cumulative probability. It can be seen that the curve of this technique (soft alignment) reaches a high cumulative probability first, while other methods require more than 80 milliseconds. In summary, this technique, through soft alignment achieved by a time offset compensation function and learnable convolutional kernels, effectively solves the timing inconsistency problem caused by differences in sampling frequencies among multiple sensors, avoids information leakage caused by hard alignment in conventional methods, and lays a precise timing foundation for subsequent multimodal fusion.

[0099] S202. Calculate the environmental noise embedding vector of each sample based on the multimodal alignment matrix of each sample, calculate the intention feature embedding matrix of each sample based on the environmental noise embedding vector of each sample, and construct the orthogonal constraint regularization term;

[0100] S2021. Calculate the environmental noise embedding vector of each sample based on the multimodal alignment matrix of each sample;

[0101] The environmental noise embedding vector of each sample is obtained by processing the multimodal alignment matrix, environmental feature extraction weight matrix, and environmental feature extraction bias vector of each sample through the hyperbolic tangent activation function. The environmental noise embedding vector of each sample represents the low-dimensional features of environmental noise, providing a basis for noise separation and reducing the impact of noise on intent recognition.

[0102] No. The formula for calculating the environmental noise embedding vector of a sample is:

[0103] ,

[0104] In the formula, Indicates the first The environmental noise embedding vector of each sample, such as noise features like wind and rain sounds, and accidental vibrations; The hyperbolic tangent activation function is used to restrict the output to... Within the range; The weight matrix is ​​extracted for environmental features and consists of trainable parameters; Bias vectors are extracted for environmental features; these are trainable parameters.

[0105] S2022. Calculate the intent feature embedding matrix of each sample based on the environmental noise embedding vector of each sample, so as to initially extract pure intent features and improve the robustness of intent recognition;

[0106] The broadcast environmental noise embedding vector is subtracted from the multimodal alignment matrix of each sample, a linear transformation of the intention feature extraction weight matrix is ​​applied, and the intention feature embedding matrix of each sample is obtained by processing with a linear rectified activation function.

[0107] No. The dimension of the intent feature embedding matrix for each sample is... , The feature dimension of the intent feature embedding matrix; the first The intent feature embedding matrix of each sample Represented as:

[0108] ,

[0109] In the formula, It is a linear rectified activation function that maintains sparse activation through nonlinear transformation; The weight matrix is ​​extracted for the intent features; it is a trainable parameter. This represents an element-wise subtraction operation, used to remove environmental noise components from the original data; For broadcast operation; Let be the environmental noise embedding vector after broadcasting, representing the embedding vector of environmental noise. Extended to multimodal alignment matrices via broadcast operations. Same dimensions;

[0110] S023. Construct orthogonal constraint regularization terms;

[0111] S20231. By applying an orthogonality constraint that the square of the Frobenius norm of the product of the transpose of the environmental feature extraction weight matrix and the intention feature extraction weight matrix is ​​less than the orthogonality constraint threshold, the environmental noise space and the intention feature space are geometrically orthogonal, which promotes feature deentanglement and enhances the anti-interference ability of the perception decision model.

[0112] Orthogonality constraints are expressed as:

[0113] ,

[0114] In the formula, The square of the Frobenius norm is used to measure the correlation between two matrices. This represents the transpose of the environmental feature extraction weight matrix; This is the orthogonality constraint threshold, used to force two weight matrices to be approximately orthogonal; it is preferably set to 0.01.

[0115] S20232. Construct orthogonal constraint regularization terms based on orthogonality constraints. Optimize by gradient descent to satisfy ;

[0116] Orthogonal constraint regularization term The calculation formula is:

[0117] ,

[0118] In the formula, The weight hyperparameter for the orthogonal constraint regularization term is preferably set to 0.1; This is a function to find the maximum value.

[0119] Aligned multimodal data still contains environmental noise, such as wind and rain sounds or falsely triggered vibrations. Conventional denoising methods often lose key intent features when removing this noise, leading to a decline in intent recognition performance. S202 achieves deentanglement of environmental noise features and intent features in the latent space by constructing a dual-channel bottleneck encoder structure.

[0120] S203. Calculate the attention score matrix for each sample, calculate the sparse gating matrix, calculate the sparse attention weight matrix for each sample, and calculate the sparse weighted feature matrix for each sample.

[0121] S2031. Calculate the attention score matrix for each sample;

[0122] The intent feature embedding matrix of each sample is projected onto the query space through the query projection matrix to obtain the query projection matrix, and projected onto the key space through the key projection matrix to obtain the key projection matrix. The attention score matrix of each sample is calculated based on the query projection matrix and the key projection matrix. The attention score matrix of each sample is used to reflect the correlation between time steps, so as to provide a basis for sparse attention and reduce computational redundancy.

[0123] No. Attention score matrix of each sample The calculation formula is:

[0124] ,

[0125] In the formula, To query the projection matrix, the first trainable weight matrix and the intent feature embedding matrix are used. Multiplying them together allows the intent to be embedded and projected into the query space; The key projection matrix is ​​obtained by combining the second trainable weight matrix with the intent feature embedding matrix. Multiplication yields the result, enabling the embedding of the intent into the projection onto the key space; Key projection matrix Transpose of; For the projection dimension, too large a value increases the computational load, while too small a value reduces the expressive power; a value of 64 is preferred.

[0126] S2032. Calculate the sparse gating matrix;

[0127] By comparing the L2 norm of the embedding vector of intent features at each time step of each sample with the dynamic threshold coefficient of the product of the maximum norm of all time steps, a binary gating value is generated, forming a sparse gating matrix, which filters key time steps, focuses on key intent segments, and reduces computational complexity.

[0128] The sparse gated matrix is ​​represented as The element in its t-th row and j-th column is , The gate value from time step t to time step j; the gate value from time step t to time step j Represented as:

[0129] ,

[0130] In the formula, t is the time step index; j is the time step index that is different from t; 1 indicates to keep, and 0 indicates to mask, which is used to filter key time steps; Represents the L2 norm; For the first The first sample The intent feature embedding vector at time step is the first... The intent feature embedding matrix of each sample The Row vectors; This is a dynamic threshold coefficient used to control the sparsity level, and is preferably set to 0.5; This represents the maximum value among all time steps j.

[0131] The L2 norm is used to calculate the significance of features. The item serves as a dynamic threshold, adaptively filtering high-energy time steps, such as voice command peaks, which can effectively focus on command segments and avoid redundant calculations for non-critical steps by conventional attention.

[0132] S2033. Calculate the sparse attention weight matrix for each sample based on the sparse gating matrix;

[0133] The attention score matrix and the sparse gating matrix are subjected to Hadamard product operation, and the row-wise normalization is performed by the Softmax function to calculate the sparse attention weight matrix, ensuring that only important time steps are focused on, thus improving efficiency.

[0134] No. The sparse attention weight matrix of the n samples is used to weight the key time steps, the i-th The sparse attention weight matrix for each sample The calculation formula is:

[0135] ,

[0136] In the formula, The Softmax function represents the normalization process performed row by row, ensuring that the sum of the attention weights at each time step is 1. This indicates the Hadamard product operation; It is a sparse gated matrix, representing a binary shielding matrix;

[0137] S2034. Calculate the sparse weighted feature matrix of each sample based on the sparse attention weight matrix of each sample;

[0138] Based on the sparse attention weight matrix, value projection matrix and intent feature embedding matrix of each sample, a sparse weighted feature matrix of each sample is generated to enhance the representation of intent-related information and improve recognition accuracy.

[0139] No. The sparse weighted feature matrix representation of the n samples utilizes the features enhanced by sparse attention. Sparse weighted feature matrix of each sample The calculation formula is:

[0140] ,

[0141] In the formula, The value projection matrix is ​​a trainable parameter used to embed intent features into the matrix. Projected onto the value space;

[0142] Door lock intents are characterized by short-term bursts; for example, the unlocking voice command lasts only 0.5 seconds. Conventional self-attention mechanisms need to calculate all time steps, resulting in a large amount of computational redundancy and making it difficult to focus on key intent segments. S203 achieves dynamic threshold filtering of key time steps through a sparse gating attention mechanism based on feature saliency.

[0143] The experiment simulated a door lock interaction scenario, and the effect of the dynamic sparse attention mechanism was analyzed. Figure 4 This demonstrates the working principle and effect of the dynamic sparse attention mechanism. Figure 4 In Figures (a), (b), and (c), the horizontal axis represents time, in seconds. In Figure (a), the vertical axis represents sensor readings (dimensionless), showing how these readings change over time; the wavy lines represent the periods of voice commands, lasting approximately 3-5 seconds. In Figure (b), the vertical axis represents feature significance (dimensionless). In Figure (c), the vertical axis represents the gating value, a binary value where 0 corresponds to masking and 1 to retaining. Figure 4 As can be seen, during the voice command period, the feature saliency is significantly higher than the dynamic threshold (τ=0.5), and all gating values ​​are preserved. The green background area and the gating activation area are highly consistent. This verifies that feature saliency calculation based on the L2 norm can accurately capture key segments of intent, avoid redundant calculations for non-critical noise periods, and significantly improve the system's real-time performance.

[0144] S204. Calculate the fusion feature matrix of each sample based on the feature vectors of each modality projection and the attention weights of each modality, so as to achieve cross-modal feature collaborative fusion;

[0145] S2041. Calculate the attention weights for each modality based on the sparse weighted feature matrix of each sample;

[0146] The sparse weighted feature matrix of each sample is averaged and pooled to obtain a vector. The vector is then projected onto the attention weight matrix and activated by hyperbolic tangent. The vector is then dot-producted with the attention vector of each modality and normalized by exponentiation to obtain the attention weight of each modality, so as to adapt to environmental changes and optimize the balance of multimodal contributions.

[0147] The attention weight of the m-th mode is a scalar, reflecting the mode's contribution. For the first mode, the attention weight of the m-th mode in a quiet environment is close to 1. The calculation formula is:

[0148] ,

[0149] In the formula, It is a natural exponential function; for Transpose of; For the first The attention vector of a modality is a trainable parameter used to generate the first modality's attention vector. Modal attention score; The attention weight matrix is ​​a trainable parameter used to project the sparse weighted feature matrix of each sample into the hidden space. This is an average pooling operation; To distinguish it from the modal index of m; For the first The attention vector of a modality is a trainable parameter used to generate the first modality's attention vector. Modal attention score; for Transpose of;

[0150] Sparse weighted feature matrix of each sample Aggregate along the time step dimension;

[0151] S2042. Calculate the feature vector of each modality projection based on the multimodal alignment matrix of each sample;

[0152] The feature sub-tensors corresponding to each mode are extracted from the multimodal alignment matrix, and linear transformation is performed through the mode-specific projection matrix to obtain the feature vectors of each mode projection. The features of each mode are then projected onto the common space to maintain mode specificity.

[0153] The eigenvectors of the m-th modal projection unify the data into a common space through projection. The calculation formula is:

[0154] ,

[0155] In the formula, The modality-specific projection matrix is ​​a trainable parameter used to perform nonlinear mapping on the original data. Indicates extracting the first The eigenvalue tensor corresponding to the mode;

[0156] S2043. Calculate the fusion feature matrix of each sample based on the feature vectors of each modality projection and the attention weights of each modality;

[0157] The fusion feature matrix of each sample is obtained by summing the product of the feature vector of each modality projection and the corresponding attention weight of each modality, so as to realize context-adaptive multimodal integration and improve recognition performance.

[0158] No. The fusion feature matrix of each sample integrates multimodal information. The fusion feature matrix of each sample The calculation formula is:

[0159] ;

[0160] The contribution of different modalities to intent recognition dynamically changes with environmental conditions. For example, voiceprint modality dominates in quiet environments while motion modality dominates in noisy environments. Conventional fixed-weight fusion methods cannot adapt to this dynamic change, leading to a decline in recognition performance. S204 achieves context-adaptive multimodal feature fusion by employing a modality weight modulator.

[0161] S205. Calculate the updated hidden state at the current time based on the updated gating vector and the candidate hidden state at the current time;

[0162] S2051. Calculate the updated gate vector at the current time step;

[0163] The current time-fused feature vector, the previous time-fused hidden state, and the velocity vector are concatenated, and then linearly transformed through the gating weight matrix and processed by the Sigmoid activation function to calculate the current time-fused update gating vector, which is used to control the historical state update ratio, so as to achieve adaptive fusion of spatiotemporal information and enhance temporal dependence.

[0164] The updated gate vector at the current moment is the first... Update the gating vector at each time step , The calculation formula is:

[0165] ,

[0166] In the formula, Use the Sigmoid activation function; The gated weight matrix is ​​a trainable parameter used to learn spatiotemporal dependency rules. For the first The first sample The fusion feature vector at each time step is the fusion feature matrix of each sample. In the Slicing at time steps, i.e., fusing feature vectors It is the fusion feature matrix of each sample. The Row vectors; This represents a vector concatenation operation; For the first The hidden state at a time step represents the hidden state at the previous time step. It is initialized to a zero vector at time to pass time-dependent information; The velocity vector is calculated based on the velocity value derived from distance sensor data.

[0167] In practical implementation, the velocity vector Encode user movement speed to enhance spatiotemporal modeling, such as increasing the weight of voice intent when accelerating as the user approaches a door lock;

[0168] velocity vector The calculation formula is:

[0169] ,

[0170] In the formula, The velocity vector and weight vector are trainable parameters; It is a speed value calculated from distance sensor data, and it is a scalar. The velocity vector bias vector is a trainable parameter;

[0171] Velocity value calculated from distance sensor data The calculation formula is:

[0172] ,

[0173] In the formula, For the first Distance in time is a scalar quantity; For the first Distance in time; The sampling interval;

[0174] S2052. Calculate the candidate hidden state at the current time;

[0175] The current time-fusing feature vector is concatenated with the position code, and then linearly transformed through the state transformation weight matrix. After processing by the hyperbolic tangent activation function, the candidate hidden state of the current time-fusing time is generated to encode the current spatiotemporal features, so as to combine position information and capture spatial dependencies.

[0176] The candidate hidden state at the current moment is the nth Candidate hidden states of time step , The calculation formula is:

[0177] ,

[0178] In the formula, The state transition weight matrix is ​​a trainable parameter that fuses feature and location information. The location is encoded based on the distance values ​​collected by the distance sensor.

[0179] In practical implementation, position encoding Encode the relative position of the user and the door lock to capture spatial dependencies, such as increased unlocking intent when the distance is less than 1 meter;

[0180] Location coding The calculation formula is:

[0181] ,

[0182] In the formula, The position-encoding weight vector consists of trainable parameters; For the first Distance in time is a scalar quantity; The position-encoding bias vector is a trainable parameter;

[0183] S2053. Calculate the updated hidden state at the current time based on the updated gating vector and the candidate hidden state at the current time to update the spatiotemporal state;

[0184] Based on the updated gating vector at the current moment and the hidden state at the previous moment, the hidden state is updated to obtain the updated hidden state at the current moment, realizing the adaptive update of the spatiotemporal state and improving intent modeling;

[0185] The hidden state updated at the current moment integrates temporal and spatial information; the hidden state updated at the current moment is the [number]th [time unit]. Hidden state of time step update , The calculation formula is:

[0186] ,

[0187] In the formula, Represents the update gate vector at the current time. A vector of all 1s with the same dimension;

[0188] Intent recognition needs to consider both temporal dynamic changes and spatial relationships. For example, when a user issues a voice command near a door lock, conventional recurrent neural networks struggle to effectively fuse spatiotemporal information and cannot adaptively adjust information transmission weights. The S205 overcomes these problems by employing a gating spatiotemporal unit to achieve adaptive fusion of spatiotemporal states.

[0189] S206. Calculate the final intent probability distribution of each sample based on the fusion feature matrix of each sample's state transformation;

[0190] S2061. Calculate the state transformation fusion feature matrix of each sample based on the hidden state updated at the current time to achieve state transformation and transition fusion;

[0191] The hidden state matrix is ​​averaged and pooled to obtain a vector. This vector is then linearly transformed by the state transition weight matrix and added to the product of the intent transition matrix and the previous predicted probability distribution. After processing by the linear rectified activation function, the fusion feature matrix of the state transition for each sample is obtained, ensuring that the output probability conforms to the actual intent transition logic and improving decoding accuracy.

[0192] No. The fusion feature matrix of sample state transformation The calculation formula is:

[0193] ,

[0194] In the formula, is the state transition weight matrix, which are trainable parameters that map the hidden state to the intent space; For average pooling operation, For the hidden state matrix Pooling into vectors; The hidden state matrix represents the global spatiotemporal features across all time steps. The intent transition matrix encodes the transition constraints between intents; For the first The predicted probability distribution of each sample, i.e., the previous predicted probability distribution, provides the transition prior. Initialize to a zero vector;

[0195] Hidden state matrix at all time steps Represented as:

[0196] ,

[0197] In the formula, For the first The hidden state of time step updates; This represents the total number of time steps. express Transpose of;

[0198] In practice, the diagonal of the intention transition matrix is ​​set to zero to prevent self-repetition, while the off-diagonal elements are updated via gradient descent.

[0199] Define the intention transition matrix The Line number The elements of the column are , Characterization from the first Intent to transfer to the The weighting parameter of the intention, i.e., from the first Intent to transfer to the The intention is to bias the weights, but the actual matrix is ​​set to zero diagonally to avoid self-transition; Intended transition matrix row index, , Indicates unlocking, Indicates deadbolt, Indicates password setting, Indicates no operation. Intended transition matrix column index, ; Indicates unlocking, Indicates deadbolt, Indicates password setting, 4 indicates no operation; based on this, the element in the 1st row and 2nd column of the intended transfer matrix is... This indicates the weight of the tendency to switch from locking to unlocking;

[0200] Intent transition matrix Represented as:

[0201] ;

[0202] S2062. Calculate the probability distribution of the final intention for each sample;

[0203] The final intent probability distribution of each sample is obtained by linearly transforming the fusion feature matrix of each sample's state transformation through the intent weight matrix, so as to provide the final classification output and support decision-making.

[0204] No. The dimension of the final intent probability distribution for each sample (equivalent to the total number of intent categories) is: , Total number of intent categories;

[0205] No. The probability distribution of the final intent of each sample The calculation formula is:

[0206] ,

[0207] In the formula, The intention weight matrix is ​​a trainable parameter that maps fused features to class probabilities.

[0208] Conventional Softmax output layers ignore the transition constraints between intents, such as not immediately performing a locking operation after unlocking. This causes the prediction results to not conform to the intent transition logic in the actual scenario, reducing recognition accuracy and practicality. The S206 employs a state-conditional decoding mechanism to ensure that the output probability conforms to the actual intent transition constraints.

[0209] S207. Calculate the total loss function based on intent classification loss, temporal smoothing loss, and modal divergence loss to obtain the perception decision model;

[0210] S2071. Calculate the intention classification loss;

[0211] Based on the one-hot encoding of the final intent probability distribution and the true intent label, combined with the weight coefficients calculated based on the class frequency, the weighted cross-entropy loss is calculated to obtain the intent classification loss, which improves the recognition rate of a few intents and enhances the classification accuracy.

[0212] Intent classification loss is used to improve the recognition ability of a minority of classes based on cross-entropy loss. The calculation formula is:

[0213] ,

[0214] In the formula, The total number of samples, i.e., the total number of door lock interaction events in the training set; Indexed by category; This represents the total number of intent categories, such as unlocking, deadbolting, and password setting. For the first The weight coefficients of class intent are calculated based on the frequency of that class in the training set, and are expressed as follows; For the first The one-hot encoding of the true intent label of the nth sample, if the nth sample... The true intention of each sample is ,but Otherwise, it is 0; It is a logarithmic function, with the default base being the natural constant; The function is softmax, ensuring that the sum of the output probabilities is 1; For the first The probability distribution of the final intent of the nth sample dimension, It is the probability distribution of the final intention. The first vector The element, i.e., the th element The probability value of the class;

[0215] No. Weight coefficient of class intent The calculation formula is:

[0216] ,

[0217] In the formula, For the first The frequency of class intents is used to mitigate class imbalance.

[0218] Intended classification loss It is an improvement based on cross-entropy loss, designed to address the issue of intention class imbalance in door lock scenarios, such as the "lock" intention appearing less frequently than "unlock". Conventional cross-entropy loss assigns equal weight to all classes, but some classes may be ignored. This is addressed by adjusting the weighting coefficients... To improve the recognition rate, the minority class is given higher weight, but overcompensation is avoided by using the square root.

[0219] Multiple coefficient Based on the training set Frequency of class intentions The calculation is obtained by first statistically analyzing the data in the training set. The number of samples in a class determines the percentage of samples in that class, i.e., the percentage of samples in the first class. Frequency of class intentions ,calculate This assigns higher weights to low-frequency categories, alleviating the problem of category imbalance.

[0220] S2072. Calculate the time series smoothing loss;

[0221] The mean of the squared L2 norm of the hidden state of each sample at adjacent time steps is calculated to obtain the temporal smoothing loss, which reduces prediction fluctuations and improves temporal consistency.

[0222] Temporal smoothing loss is used to encourage smooth changes in hidden states between adjacent time steps, avoiding drastic fluctuations in intent prediction and improving the stability of door lock interactions; Temporal smoothing loss The calculation formula is:

[0223] ,

[0224] In the formula, For the first The first sample The hidden state is updated at each time step, and the first step is performed on a single sample. Hidden state of time step update The calculation yielded the result; For the first The first sample The hidden state of time step updates.

[0225] Temporal smoothing loss pass The penalty is applied to changes in the hidden state of adjacent time steps to ensure a smooth transition in intent prediction over time, which is consistent with the continuity of user intent in door lock operations, such as the gradual process from "approaching" to "unlocking".

[0226] S2073. Calculate the modal bifurcation loss;

[0227] The mean of the negative values ​​of the entropy of the attention weights of each modality for each sample is calculated to obtain the modal divergence loss, which avoids over-reliance on a single modality and enhances the robustness of the system.

[0228] Modal divergence loss is used to encourage perception-decision models to make balanced use of information from various modalities, avoid over-reliance on a single modality, and improve the robustness of the system in varying environments such as noisy environments or sensor failures; Modal divergence loss The calculation formula is:

[0229] ,

[0230] In the formula, For the first Attention weights for the m-th modality of a sample; attention weights for the m-th modality of a single sample. The calculation yielded the result;

[0231] Modal divergence loss, by maximizing the entropy of attention weights, prompts the perceptual decision model to fairly allocate attention to each modality, preventing the neglect of other modalities under specific environmental conditions, such as over-reliance on voiceprints when quiet, and enhancing generalization ability.

[0232] S2074. Calculate the total loss function based on intent classification loss, temporal smoothing loss, and modal divergence loss;

[0233] The total loss function is obtained by calculating the product of the intention classification loss, the temporal smoothing loss weight hyperparameter and the temporal smoothing loss, the modal divergence loss weight hyperparameter and the modal divergence loss, and the sum of the orthogonal constraint regularization term, ensuring that the perception decision model works collaboratively in terms of classification, smoothness, balance and deentanglement.

[0234] The total loss function is used to guide the training of the perceptual decision model. The calculation formula is:

[0235] ,

[0236] In the formula, The weight hyperparameter for time series smoothing loss is used to control the importance of time series smoothness, and is preferably set to 0.3; The modal divergence loss weight hyperparameter is used to control the importance of modal balance, and is preferably set to 0.3;

[0237] Door lock intent recognition requires high accuracy and robustness, avoiding environmental noise interference while maintaining temporal consistency and modal balance. Conventional loss functions only focus on classification accuracy, ignoring temporal smoothness, modal divergence, and orthogonal constraints related to feature disentanglement in door lock scenarios. This makes the perception decision model susceptible to noise and results in large prediction fluctuations. S207 overcomes these problems by designing a composite total loss function and comprehensively optimizing the parameters of the perception decision model. The total loss function is jointly optimized through multiple components to ensure that the perception decision model works collaboratively in intent classification, temporal smoothness, modal balance, and feature disentanglement, thereby improving the perception decision performance of the door lock interaction system.

[0238] S208. Iteratively train and update the parameters of the perception-decision model to obtain the trained perception-decision model;

[0239] The perception decision model is trained using a labeled multimodal dataset (i.e., a collection of labeled perception data). The training process is based on the gradient descent optimization algorithm. The perception decision model output and total loss function are calculated through forward propagation, and the perception decision model parameters are updated through backpropagation.

[0240] During training, the Adam optimizer is used to calculate gradients and adjust the parameters of the perceptual decision model to minimize the total loss;

[0241] Iterative training continues, and the stopping conditions include the validation set loss no longer decreasing for several consecutive iterations, the validation accuracy reaching a predetermined threshold (e.g., above 95%), or reaching the maximum number of iterations (e.g., 1000 times).

[0242] During training, an early stopping strategy is used to prevent overfitting, and the parameters of the perception-decision model with the best performance are saved periodically to ensure the convergence and generalization ability of the perception-decision model.

[0243] To evaluate the adaptability of cross-modal feature collaborative fusion technology under different environmental conditions, the robustness of our technology (adaptive fusion), conventional methods (fixed weights), single-modal (voiceprint), and single-modal (motion) methods under different environments was compared and analyzed. Figure 5 . Figure 5The horizontal axis represents environmental conditions, including five scenarios: quiet indoor environment, noisy outdoor environment, windy and rainy weather, nighttime environment, and sensor failure. The vertical axis represents the system robustness index, expressed as a percentage. It can be seen that single-modality (voiceprint) almost fails in sensor failure scenarios, and conventional methods (fixed weights) fluctuate significantly with environmental changes. Single-modality (motion) performs poorly in quiet indoor scenarios. However, the system robustness index of our proposed technology (adaptive fusion) is the highest in all environments, indicating that our technology (adaptive fusion) has stable performance. Even in challenging scenarios such as sensor failure (simulating single sensor failure), nighttime environments (low light affecting vision), and windy and rainy weather (noise interference), our technology (adaptive fusion) maintains usable performance by dynamically adjusting the voiceprint / motion modality weights. This proves that the modal weight modulator can adaptively adjust multimodal contributions according to environmental changes, solving the environmental adaptability defects of conventional fixed-weight fusion.

[0244] S3. Collect new sensing data, align the voiceprint data, relative distance data between the person and the door lock, and door lock status data in the new sensing data to a unified benchmark according to the timestamp, and then input them into the trained sensing decision model to obtain the intent probability distribution. Select the intent with the highest probability as the decision result and trigger the corresponding door lock control command.

Claims

1. A perception and decision-making method for a door lock interaction system based on embodied intelligence, characterized in that, include: S1. Collect perception data, which includes voiceprint data, relative distance data between the person and the door lock, and door lock status data. Align the voiceprint data, relative distance data between the person and the door lock, and door lock status data to a unified benchmark according to the timestamp. Label each perception data with the corresponding intent label and environmental noise label to obtain labeled perception data. S2. Construct a perception-decision model and train the perception-decision model to obtain a trained perception-decision model; S201. Calculate the multimodal alignment matrix for each sample; S202. Construct orthogonal constraint regularization terms; S203. Calculate the sparse weighted feature matrix for each sample; S204. Calculate the fusion feature matrix for each sample; S205. Calculate the hidden state updated at the current time; S206. Calculate the probability distribution of the final intention for each sample; S207. Calculate the total loss function to obtain the perception decision model; S208. Iteratively train and update the parameters of the perception-decision model to obtain the trained perception-decision model; S3. Collect new sensing data, align the voiceprint data, relative distance data between the person and the door lock, and door lock status data in the new sensing data to a unified benchmark according to the timestamp, and then input them into the trained sensing decision model to obtain the intent probability distribution. Select the intent with the highest probability as the decision result and trigger the corresponding door lock control command.

2. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The sensing data collected in S1 includes: Sensing data from different real-world usage scenarios is collected by sensors installed in the door lock. The sensing devices include a voiceprint sensor for acquiring voiceprint data, a distance sensor for acquiring relative distance data between a person and a door lock, and a motion sensor for acquiring door lock status data; Real-world usage scenarios include indoors, outdoors, quiet environments, noisy environments, daytime, and nighttime.

3. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The steps in S201 include: S2011. Calculate the time offset compensation function for each mode; S2012. Construct convolution kernels for each modality; S2013. Calculate the multimodal alignment matrix of each sample based on the time offset compensation function of each modality and the convolution kernel of the m-th modality.

4. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The steps in S202 include: S2021. Calculate the environmental noise embedding vector of each sample based on the multimodal alignment matrix of each sample; S2022. Calculate the intent feature embedding matrix of each sample based on the environmental noise embedding vector of each sample; S2023. Construct orthogonal constraint regularization terms.

5. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 4, characterized in that, The steps in S2023 include: S20231. An orthogonality constraint is imposed by applying an orthogonality constraint threshold such that the square of the Frobenius norm of the product of the transpose of the environmental feature extraction weight matrix and the intention feature extraction weight matrix is ​​less than the orthogonality constraint threshold. S20232. Construct orthogonal constraint regularization terms based on orthogonality constraints.

6. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The steps in S203 include: S2031. Calculate the attention score matrix for each sample; S2032. Calculate the sparse gating matrix; S2033. Calculate the sparse attention weight matrix for each sample based on the sparse gating matrix; S2034. Calculate the sparse weighted feature matrix of each sample based on the sparse attention weight matrix of each sample.

7. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The steps in S204 include: S2041. Calculate the attention weights for each modality based on the sparse weighted feature matrix of each sample; S2042. Calculate the feature vector of each modality projection based on the multimodal alignment matrix of each sample; S2043. Calculate the fusion feature matrix of each sample based on the feature vectors of each modality projection and the attention weights of each modality.

8. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The steps in S205 include: S2051. Calculate the updated gate vector at the current time step; S2052. Calculate the candidate hidden state at the current time; S2053. Calculate the updated hidden state at the current time based on the updated gating vector and the candidate hidden state at the current time to update the spatiotemporal state.

9. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, Step S206 includes: S2061. Calculate the state transformation fusion feature matrix of each sample based on the hidden state updated at the current time; S2062. Calculate the probability distribution of the final intention for each sample.

10. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The steps in S207 include: S2071. Calculate the intention classification loss; S2072. Calculate the time series smoothing loss; S2073. Calculate the modal bifurcation loss; S2074. Calculate the total loss function based on intent classification loss, temporal smoothing loss, and modal divergence loss.

Images