Embodied intelligence-based door lock interaction system perception decision method

Abstract

The present application relates to the field of electric digital data processing, and in particular to a method for sensing and decision-making of a door lock interaction system based on embodied intelligence, which comprises collecting sensing data, labeling an intention label and an environmental noise label corresponding to each sensing data to obtain labeled sensing data; constructing a sensing and decision-making model and training the sensing and decision-making model to obtain a trained sensing and decision-making model; collecting new sensing data, inputting the voiceprint data, the relative distance data between the person and the door lock, and the door lock state data in the new sensing data into the trained sensing and decision-making model after aligning them to a unified reference according to a time stamp to obtain an intention probability distribution, selecting the intention with the highest probability as a decision result and triggering a corresponding door lock control instruction. The existing sensing and decision-making method of the door lock interaction system based on embodied intelligence has the problem of poor user experience. The method provided by the present application can provide good user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electric digital data processing, in particular to a perception decision method of a door lock interaction system based on embodied intelligence. BACKGROUND

[0002] After the emergence of electronic locks, with the development of science and technology, it is constantly updated and iterated, there are card + password unlocking form, fingerprint unlocking form, multiple biological recognition coexistence form, and smart lock form with camera and other devices and networking. After the maturity of artificial intelligence technology, the smart lock combined with artificial intelligence technology has been developed.

[0003] The smart lock combined with artificial intelligence technology is actually a kind of embodied intelligence, so it is also called a door lock based on embodied intelligence. The interaction system of the door lock based on embodied intelligence realizes the service for users by sensing and decision making first. The smart lock combined with artificial intelligence technology obtains the image in front of the door through the camera, obtains the footstep sound through the microphone, senses whether there is a living body close by through the infrared sensor, senses whether the door lock is pried through the pressure sensor, and gives feedback to the user through vibration and light instead of simply ringing. In this way, the smart lock combined with artificial intelligence technology has the ability of entity, sensing and interaction. At the same time, the smart lock combined with artificial intelligence technology also has learning ability and decision making ability. It can learn user behavior, so as to realize the functions of automatically lighting the screen in advance when the user walks in front of the door, pushing information to the user when finding abnormal behavior of strangers, etc.

[0004] However, users usually interact with the door lock based on embodied intelligence through voice, action or contact, etc. This process is affected by environmental noise, sensor sampling differences and scene diversity, which can easily lead to inaccurate intent recognition, response delay or false triggering, etc. At the same time, the existing perception decision method mostly uses single mode or simple multi-modal fusion method, which cannot effectively solve the differences in sampling frequency, time alignment and feature expression of different modal data. Using conventional noise reduction and feature extraction methods, the key signals of intent are easily lost when removing environmental interference, and there is a lack of fine decoupling ability of noise and intent features. In addition, the fixed weight fusion method is used, so the modal contribution cannot be dynamically adjusted according to the environmental conditions, resulting in a significant performance decline in noisy environments or sensor failure conditions.

[0005] Therefore, the existing perception decision method of the door lock interaction system based on embodied intelligence has the problem of poor user experience. SUMMARY

[0006] The technical problem to be solved by the present application is to overcome the shortcomings of the prior art and provide a perception decision method of a door lock interaction system based on embodied intelligence with good user experience.

[0007] To solve the above technical problems, the embodiment of the application provides a body-aware intelligent door lock interaction system perception decision method, which comprises the following steps:

[0008] S1. Collecting perception data, wherein the perception data comprises voiceprint data, relative distance data between a person and a door lock, and door lock state data, aligning the voiceprint data, the relative distance data between the person and the door lock, and the door lock state data to a unified reference according to a time stamp, labeling an intention label and an environmental noise label corresponding to each perception data, and obtaining labeled perception data;

[0009] S2. Constructing a perception decision model and training the perception decision model to obtain a trained perception decision model;

[0010] S201. Calculating a multi-modal alignment matrix of each sample;

[0011] S202. Constructing an orthogonal constraint regularization term;

[0012] S203. Calculating a sparse weighted feature matrix of each sample;

[0013] S204. Calculating a fusion feature matrix of each sample;

[0014] S205. Calculating an updated hidden state at a current time;

[0015] S206. Calculating a final intention probability distribution of each sample;

[0016] S207. Calculating a total loss function to obtain the perception decision model;

[0017] S208. Iteratively training and updating parameters of the perception decision model to obtain the trained perception decision model;

[0018] S3. Collecting new perception data, inputting voiceprint data, relative distance data between a person and a door lock, and door lock state data in the new perception data into the trained perception decision model after aligning the voiceprint data, the relative distance data between the person and the door lock, and the door lock state data to a unified reference according to a time stamp, obtaining an intention probability distribution, selecting an intention with the highest probability as a decision result, and triggering a corresponding door lock control instruction.

[0019] As a further improvement of the application, the perception data collected in S1 comprises:

[0020] The perception data in different real use scenarios is collected by a sensing device arranged in the door lock;

[0021] The sensing device comprises a voiceprint sensor for obtaining voiceprint data, a distance sensor for obtaining relative distance data between a person and a door lock, and a motion sensor for obtaining door lock state data;

[0022] The real use scenarios comprise indoor, outdoor, quiet, noisy, daytime, and night.

[0023] As a further improvement of the application: the step of S201 comprises:

[0024] S2011. Calculate the time offset compensation function of each modality;

[0025] S2012. Construct the convolution kernel of each modality;

[0026] S2013. Calculate the multi-modal alignment matrix of each sample based on the time offset compensation function of each modality and the convolution kernel of the m-th modality.

[0027] As a further improvement of the application: the step of S202 comprises:

[0028] S2021. Calculate the environmental noise embedding vector of each sample based on the multi-modal alignment matrix of each sample;

[0029] S2022. Calculate the intent feature embedding matrix of each sample based on the environmental noise embedding vector of each sample;

[0030] S2023. Construct an orthogonal constraint regularization term.

[0031] Preferably, the step of S2023 comprises:

[0032] S20231. Apply an orthogonal constraint by imposing that the Frobenius norm square of the product of the environmental feature extraction weight matrix transpose and the intent feature extraction weight matrix is less than the orthogonality constraint threshold;

[0033] S20232. Based on the orthogonal constraint, construct an orthogonal constraint regularization term.

[0034] As a further improvement of the application: the step of S203 comprises:

[0035] S2031. Calculate the attention score matrix of each sample;

[0036] S2032. Calculate the sparse gating matrix;

[0037] S2033. Calculate the sparse attention weight matrix of each sample based on the sparse gating matrix;

[0038] S2034. Calculate the sparse weighted feature matrix of each sample based on the sparse attention weight matrix of each sample.

[0039] As a further improvement of the application: the step of S204 comprises:

[0040] S2041. Calculate the attention weight of each modality based on the sparse weighted feature matrix of each sample;

[0041] S2042. Calculate the eigenvector of each modality projection based on the multi-modal alignment matrix of each sample;

[0042] S2043. Calculate the fusion feature matrix of each sample based on the eigenvector of each modality projection and the attention weight of each modality.

[0043] As a further improvement of the application, the steps of S205 include:

[0044] S2051. Calculate the update gate vector of the current time;

[0045] S2052. Calculate the candidate hidden state of the current time;

[0046] S2053. Calculate the updated hidden state of the current time based on the update gate vector of the current time and the candidate hidden state of the current time to update the space-time state.

[0047] As a further improvement of the application, the steps of S206 include:

[0048] S2061. Calculate the state transformation fusion feature matrix of each sample based on the updated hidden state of the current time;

[0049] S2062. Calculate the final intention probability distribution of each sample.

[0050] As a further improvement of the application, the steps of S207 include:

[0051] S2071. Calculate the intention classification loss;

[0052] S2072. Calculate the timing smoothing loss;

[0053] S2073. Calculate the modality divergence loss;

[0054] S2074. Calculate the total loss function based on the intention classification loss, the timing smoothing loss and the modality divergence loss.

[0055] The beneficial effects of the application are as follows: the embodiment of the application provides a body-aware intelligent door lock interaction system perception decision method, which has good user experience.

[0056] Firstly, the method uses a multi-modal soft alignment method based on a time offset compensation function and a learnable convolution kernel, which can maintain the temporal consistency between modalities under different sampling frequencies, avoiding information loss and feature distortion caused by hard alignment; secondly, the method adopts an environmental noise and intention feature disentangled coding mechanism, effectively separates noise and key signals through a double-channel bottleneck structure and orthogonal constraint, and improves the robustness of intention recognition; thirdly, the method adopts a dynamic sparse attention mechanism based on feature saliency, adaptively selects key time segments, reduces computational redundancy while focusing on burst intention features, and improves real-time performance and accuracy; finally, the method constructs a fusion framework combining modal weight modulator and spatiotemporal dependence modeling, which can dynamically balance the contributions of multi-modalities and capture the joint dependence relationship of time and space, thereby realizing context-adaptive intention recognition. Through these optimizations, the method is more accurate in identifying the user's intention, and the perception process can be low-latency, high-precision, adaptive to changing environments, and provide better user experience. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a principle block diagram of the present application;

[0058] Figure 2 is a dynamic alignment comparison box plot of multi-modal time series data;

[0059] Figure 3 is an aligned error cumulative distribution experiment graph;

[0060] Figure 4 is a dynamic sparse attention mechanism effect analysis experiment graph;

[0061] Figure 5 is a cross-modal fusion method robustness comparison experiment graph under different environments. DETAILED DESCRIPTION

[0062] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0063] Definitions of related terms involved in the present application:

[0064] (1) Embodied Artificial Intelligence, abbreviated as "EAI", is an intelligent system that relies on physical entities to achieve intelligent growth through interaction with the environment, and is an intelligent form that integrates perception, movement and cognition.

[0065] As shown in Figure 1 , the embodied intelligent door lock interaction system perception decision method provided by the present application comprises:

[0066] S1. Collect perception data in different real use scenarios through a sensing device arranged in a door lock, the perception data being multi-modal data, the perception data including voiceprint data, relative distance data between a person and the door lock, and door lock state data, align the voiceprint data, the relative distance data between the person and the door lock, and the door lock state data to a unified reference according to timestamps, manually label an intention label and an environmental noise label corresponding to each perception data, and obtain labeled perception data, and store the labeled perception data in a structured format to provide high-quality and fine-grained supervision signals;

[0067] The sensing device includes a voiceprint sensor for obtaining voiceprint data, a distance sensor for obtaining relative distance data between a person and the door lock, and a motion sensor for obtaining door lock state data;

[0068] The sampling frequency of the voiceprint sensor is 8 kHz, and the voiceprint data includes user voice instructions and environmental background sounds, the user voice instructions including unlock commands, lock commands, etc., and the environmental background sounds including wind and rain sounds, noisy human voices, and extractor fan sounds, etc.;

[0069] The sampling frequency of the distance sensor is 10 Hz, and the relative distance data is a distance value sequence with a unit of meters;

[0070] The door lock state data is a vibration state or a motion state of the door lock to identify knocking, touching, and lock picking, etc.;

[0071] The real use scenarios include indoor, outdoor, quiet, noisy, daytime, and nighttime, etc.; the perception data in different real use scenarios is collected to obtain perception data under various environmental conditions to ensure the diversity and representativeness of the perception data;

[0072] The intention label includes unlocking, locking, password setting, and no operation;

[0073] The environmental noise label includes wind and rain sounds, vehicle noises, and false triggering vibrations;

[0074] The labeled perception data includes the perception data and a label file corresponding thereto;

[0075] S2. Construct a perception decision model and train the perception decision model to obtain a trained perception decision model;

[0076] S201. Calculate a multi-modal alignment matrix of each sample based on a time offset compensation function of each modality and a convolution kernel of each modality;

[0077] S2011. Calculate the time offset compensation function of each modality;

[0078] The time offset compensation function of each modality is calculated based on the deviation of the acquisition timestamp of each modality from the reference timestamp, the alignment tolerance control coefficient, and the reference timestamp. The time offset compensation function of each modality is used to adjust the alignment weight of the modality data in the time dimension by weighting, so as to avoid information leakage caused by hard alignment and retain the temporal consistency between modalities.

[0079] The time offset compensation function of the mth modality The calculation formula is:

[0080]

[0081] In the formula, The deviation of the acquisition timestamp of the mth modality from the reference timestamp; m is the modality index, The voiceprint modality, The distance sensor modality, The motion sensor modality; The natural constant; The alignment tolerance control coefficient, which is used to adjust the sensitivity of the alignment process to the time deviation, is preferably set to 1.0; The absolute value operator; The reference timestamp, which is the average time point of the time sequence of each modality;

[0082] If the alignment tolerance control coefficient is too large, the alignment will be too strict and reasonable deviations will be ignored, If it is too small, the alignment will be loose and noise will be easily generated, so in the door lock application scenario, the sensitivity is balanced by setting to 1.0, which ensures that the weight of the deviation within 0.1 seconds is attenuated to 0.9048 when calculating the time offset compensation function of each modality, so as to balance the robustness. For example, assuming that the reference timestamp is 0.1 seconds, the deviation of the acquisition timestamp of the voiceprint modality from the reference timestamp is 0.05 seconds, and is set to 1.0, then the time offset compensation function of the first modality is .

[0083] The term as an exponential attenuation function can smooth the time deviation and avoid edge information leakage caused by hard alignment of the sliding window. Combined with the alignment tolerance control coefficient controlling the attenuation rate, and the reference timestamp as the reference point to ensure that each modality is aligned around the center, the original temporal consistency can be retained, such as the causal relationship between speech and motion, while the offset is adaptively compensated, improving the accuracy of subsequent fusion.

[0084] S2012. Constructing the convolution kernel of each modality;

[0085] ​For each modality, a corresponding convolution kernel is set, and the convolution kernel parameters are optimized through backpropagation to adaptively extract and align multimodal features;

[0086] Define the convolution kernel of the m-th mode as The dimensions are:

[0087] ,

[0088] In the formula, 5 represents that the step size of the convolution kernel of the m-th mode in the time dimension is 5; Let m be the feature dimension of the m-th mode; Let be the number of output channels for all modalities, and let be the convolution kernel for all modalities. To set the same number of output channels, you can set... ;

[0089] S2013. Perform intermodal convolution alignment: Calculate the multimodal alignment matrix for each sample based on the time offset compensation function of each modality and the convolution kernel of the m-th modality;

[0090] The original data of each modality is convolved with the convolution kernels of its corresponding modalities, and a weighted time offset compensation function is applied to generate an aligned multimodal alignment matrix. The temporal information of voiceprint, distance and motion is fused to achieve soft alignment of multimodal data.

[0091] No. The multimodal alignment matrix of each sample fuses temporal information of voiceprint, distance, and motion. Multimodal alignment matrix of samples Represented as:

[0092] ,

[0093] In the formula, This represents a convolution operation with an output dimension of . ; This represents the total number of time steps. This represents the number of output channels for all modes. The convolution kernel for the m-th mode; Indicates the first The original data of the m-th mode of a sample; i is the sample index;

[0094] The term extracts modality-specific features through convolution operations, and a time-shift compensation function. Weighted adjustment of time sequence weights achieves soft alignment.

[0095] In one embodiment, the modalities of the raw data are in vector format, and the voiceprint modality is a one-dimensional audio waveform sampled at 8kHz. For example, one data point of the first modality is... , the distance sensor modality is a distance value sequence sampled at 10Hz; a piece of data of the second modality is , in meters, and the third modality is similar, in addition, a piece of data can represent a specific event, such as , data representing a door lock interaction event.

[0096] The perception data come from different sensors and the sampling frequencies of these sensors are different, such as the voiceprint sensor is sampled at 8kHz and the distance sensor is sampled at 10Hz, resulting in time series data misalignment. The conventional processing method uses a sliding window to align, but it is easy to cause edge information leakage, thereby destroying the time series consistency between modalities and affecting the accuracy of data fusion. S2 performs inter-modality convolution alignment by calculating a time offset compensation function and constructing a learnable convolution kernel, to realize soft alignment of multi-modal data and overcome the above problems.

[0097] To verify the performance advantage of the multi-modal time series data dynamic alignment technology, sliding window alignment, fixed offset alignment and the present technology (soft alignment) are compared, and Figure 2 and Figure 3 are obtained.

[0098] Figure 2 is a box plot, Figure 2 in which the vertical coordinate is the time alignment error, in milliseconds, and the horizontal coordinate is the three alignment methods. It can be seen that the median time alignment error of the sliding window alignment is about 85 milliseconds, the median time alignment error of the fixed offset alignment is about 65 milliseconds, and the median time alignment error of the present technology (soft alignment) is only about 22 milliseconds. Figure 3 is a cumulative distribution function graph, used to reveal the error distribution law, Figure 3 in which the horizontal coordinate is the time alignment error, in milliseconds, and the vertical coordinate is the cumulative probability. It can be seen that the curve of the present technology (soft alignment) reaches a high cumulative probability first, and the other methods take more than 80 milliseconds. In summary, the soft alignment realized by the time offset compensation function and the learnable convolution kernel of the present technology can effectively solve the problem of time series inconsistency caused by the difference in sampling frequency of multiple sensors, avoid information leakage caused by hard alignment of conventional methods, and lay a precise time series foundation for subsequent multi-modal fusion.

[0099] S202. Calculate the environmental noise embedding vector of each sample based on the multi-modal alignment matrix of each sample, calculate the intent feature embedding matrix of each sample based on the environmental noise embedding vector of each sample, and construct an orthogonal constraint regularization term;

[0100] S2021. Calculate the environmental noise embedding vector of each sample based on the multi-modal alignment matrix of each sample;

[0101] The environmental noise embedding vector of each sample is obtained by processing the multimodal alignment matrix, environmental feature extraction weight matrix, and environmental feature extraction bias vector of each sample through the hyperbolic tangent activation function. The environmental noise embedding vector of each sample represents the low-dimensional features of environmental noise, providing a basis for noise separation and reducing the impact of noise on intent recognition.

[0102] No. The formula for calculating the environmental noise embedding vector of a sample is:

[0103] ,

[0104] In the formula, Indicates the first The environmental noise embedding vector of each sample, such as noise features like wind and rain sounds, and accidental vibrations; The hyperbolic tangent activation function is used to restrict the output to... Within the range; The weight matrix is ​​extracted for environmental features and consists of trainable parameters; Bias vectors are extracted for environmental features; these are trainable parameters.

[0105] S2022. Calculate the intent feature embedding matrix of each sample based on the environmental noise embedding vector of each sample, so as to initially extract pure intent features and improve the robustness of intent recognition;

[0106] The broadcast environmental noise embedding vector is subtracted from the multimodal alignment matrix of each sample, a linear transformation of the intention feature extraction weight matrix is ​​applied, and the intention feature embedding matrix of each sample is obtained by processing with a linear rectified activation function.

[0107] No. The dimension of the intent feature embedding matrix for each sample is... , The feature dimension of the intent feature embedding matrix; the first The intent feature embedding matrix of each sample Represented as:

[0108] ,

[0109] In the formula, It is a linear rectified activation function that maintains sparse activation through nonlinear transformation; The weight matrix is ​​extracted for the intent features; it is a trainable parameter. This represents an element-wise subtraction operation, used to remove environmental noise components from the original data; For broadcast operation; Let be the environmental noise embedding vector after broadcasting, representing the embedding vector of environmental noise. Extended to multimodal alignment matrices via broadcast operations. the same dimension;

[0110] S2023. Constructing an orthogonal constraint regularization term;

[0111] S20231. Realizing the geometric orthogonality between the environmental noise space and the intent feature space by applying the orthogonal constraint that the square of the Frobenius norm of the product of the transpose of the environmental feature extraction weight matrix and the intent feature extraction weight matrix is less than the orthogonality constraint threshold, promoting feature disentanglement, promoting feature disentanglement, and enhancing the anti-interference ability of the perception decision model;

[0112] The orthogonal constraint is represented as:

[0113] ,

[0114] In the formula, represents the square of the Frobenius norm, which is used to measure the correlation between two matrices; represents the transpose of the environmental feature extraction weight matrix; is the orthogonality constraint threshold, which is used to force the two weight matrices to approach an orthogonal relationship, and is preferably set to 0.01.

[0115] S20232. Based on the orthogonal constraint, an orthogonal constraint regularization term is constructed , and the optimization of satisfies by gradient descent.

[0116] Orthogonal constraint regularization term The calculation formula of the orthogonal constraint regularization term is:

[0117] ,

[0118] In the formula, is the weight hyperparameter of the orthogonal constraint regularization term, and is preferably set to 0.1; is the maximum value function;

[0119] The aligned multi-modal data still contains environmental noise, such as wind and rain sounds or false triggering vibrations. When removing these noises, conventional noise reduction methods often lose key intent features, resulting in a decline in intent recognition performance. S202 realizes the disentanglement of environmental noise features and intent features in the hidden space by constructing a double-channel bottleneck encoder structure.

[0120] S203. Calculate the attention score matrix of each sample, calculate the sparse gating matrix, calculate the sparse attention weight matrix of each sample, and calculate the sparse weighted feature matrix of each sample;

[0121] S2031. Calculate the attention score matrix of each sample;

[0122] The intent feature embedding matrix of each sample is projected onto the query space through the query projection matrix to obtain the query projection matrix, and projected onto the key space through the key projection matrix to obtain the key projection matrix. The attention score matrix of each sample is calculated based on the query projection matrix and the key projection matrix. The attention score matrix of each sample is used to reflect the correlation between time steps, so as to provide a basis for sparse attention and reduce computational redundancy.

[0123] No. Attention score matrix of each sample The calculation formula is:

[0124] ,

[0125] In the formula, To query the projection matrix, the first trainable weight matrix and the intent feature embedding matrix are used. Multiplying them together allows the intent to be embedded and projected into the query space; The key projection matrix is ​​obtained by combining the second trainable weight matrix with the intent feature embedding matrix. Multiplication yields the result, enabling the embedding of the intent into the projection onto the key space; Key projection matrix Transpose of; For the projection dimension, too large a value increases the computational load, while too small a value reduces the expressive power; a value of 64 is preferred.

[0126] S2032. Calculate the sparse gating matrix;

[0127] By comparing the L2 norm of the embedding vector of intent features at each time step of each sample with the dynamic threshold coefficient of the product of the maximum norm of all time steps, a binary gating value is generated, forming a sparse gating matrix, which filters key time steps, focuses on key intent segments, and reduces computational complexity.

[0128] The sparse gated matrix is ​​represented as The element in its t-th row and j-th column is , The gate value from time step t to time step j; the gate value from time step t to time step j Represented as:

[0129] ,

[0130] In the formula, t is the time step index; j is the time step index that is different from t; 1 indicates to keep, and 0 indicates to mask, which is used to filter key time steps; Represents the L2 norm; For the first The first sample The intent feature embedding vector at time step is the first... The intent feature embedding matrix of the sample The first row vector is a dynamic threshold coefficient for controlling the sparsity degree, which is preferably set to 0.5; represents the maximum value in all j time steps.

[0131] The term adopts the calculation mode of the L2 norm, quantifies the feature saliency, The term is a dynamic threshold value, which adaptively filters high-energy time steps such as speech instruction peaks, can effectively focus on instruction fragments, and avoid redundant calculation of non-key steps by conventional attention;

[0132] S2033. Calculate the sparse attention weight matrix of each sample based on the sparse gating matrix;

[0133] The attention score matrix is subjected to Hadamard product operation with the sparse gating matrix, and the sparse attention weight matrix is calculated by row normalization through the Softmax function, so as to ensure that only important time steps are focused on and the efficiency is improved;

[0134] The sparse attention weight matrix of the first sample is used to weight the key time steps, and the sparse attention weight matrix of the first sample is used to weight the key time steps The calculation formula of the sparse attention weight matrix of the first

[0135] ,

[0136] In the formula, is a Softmax function, which represents row normalization processing to ensure that the sum of attention weights of each time step is 1; represents Hadamard product operation; is a sparse gating matrix, which represents a binary masking matrix;

[0137] S2034. Calculate the sparse weighted feature matrix of each sample based on the sparse attention weight matrix of each sample;

[0138] Based on the sparse attention weight matrix of each sample, the value projection matrix, and the intent feature embedding matrix of each sample, the sparse weighted feature matrix of each sample is generated, the representation of the intent related information is enhanced, and the recognition accuracy is improved;

[0139] The sparse weighted feature matrix of the first sample represents the feature enhanced by sparse attention, and the sparse weighted feature matrix of the first sample represents the feature enhanced by sparse attention The calculation formula of the sparse weighted feature matrix of the first

[0140] ,

[0141] wherein, is a value projection matrix, is a trainable parameter, the value projection matrix is used for embedding the intent feature matrix into a value space;

[0142] The door lock intent has a short-time burst characteristic, for example, the voice instruction for unlocking only lasts for 0.5 seconds. The conventional self-attention mechanism needs to be calculated for all time steps, which produces a large amount of calculation redundancy and is difficult to focus on the key intent segment. S203 realizes dynamic threshold filtering of the key time step through the sparse gating attention mechanism based on feature saliency.

[0143] The experimental simulation door lock interaction scene is used to analyze the effect of the dynamic sparse attention mechanism, and the following results are obtained Figure 4 , which shows the working principle and effect of the dynamic sparse attention mechanism. Figure 4 The abscissa of the (a) graph, the (b) graph and the (c) graph in the figure is time, in seconds; the ordinate of the (a) graph is the sensor reading (dimensionless), which shows the change of the sensor reading with time, and the inflection of the dotted line represents the voice instruction period, which lasts for about 3-5 seconds; the ordinate of the (b) graph is the feature saliency (dimensionless); the ordinate of the (c) graph is the gating value, which is a binary value, 0 for shielding and 1 for retaining. From Figure 4 it can be seen that during the voice instruction period, the feature saliency is obviously higher than the dynamic threshold (τ=0.5), and the gating value is all retained, and the green background area is highly consistent with the gating activation area. This verifies that the feature saliency calculation based on the L2 norm can accurately capture the key segment of the intent, avoid redundant calculation on the non-key noise period, and significantly improve the real-time performance of the system;

[0144] S204. Calculate the fusion feature matrix of each sample based on the feature vector projected by each modality and the attention weight of each modality, to realize cross-modal feature collaborative fusion;

[0145] S2041. Calculate the attention weight of each modality based on the sparse weighted feature matrix of each sample;

[0146] Average pooling is performed on the sparse weighted feature matrix of each sample to obtain a vector, which is projected through the attention weight matrix and the hyperbolic tangent activation, and then dot product with the attention vector of each modality, and then exponent and normalization are performed to obtain the attention weight of each modality, so as to adapt to environmental changes and optimize the balance of multi-modal contribution;

[0147] The attention weight of the m-th modality is a scalar, which reflects the contribution degree of the modality. For the first modality, the attention weight of the m-th modality in a quiet environment is close to 1; the attention weight of the m-th modality The calculation formula of the attention weight of the m-th modality is:

[0148] ,

[0149] In the formula, It is a natural exponential function; for transpose; For the first The attention vector of a modality is a trainable parameter used to generate the first modality's attention vector. Modal attention score; The attention weight matrix is ​​a trainable parameter used to project the sparse weighted feature matrix of each sample into the hidden space. This is an average pooling operation; To distinguish it from the modal index of m; For the first The attention vector of a modality is a trainable parameter used to generate the first modality's attention vector. Modal attention score; for transpose;

[0150] Sparse weighted feature matrix of each sample Aggregate along the time step dimension;

[0151] S2042. Calculate the feature vector of each modality projection based on the multimodal alignment matrix of each sample;

[0152] The feature sub-tensors corresponding to each mode are extracted from the multimodal alignment matrix, and linear transformation is performed through the mode-specific projection matrix to obtain the feature vectors of each mode projection. The features of each mode are then projected onto the common space to maintain mode specificity.

[0153] The eigenvectors of the m-th modal projection unify the data into a common space through projection. The calculation formula is:

[0154] ,

[0155] In the formula, The modality-specific projection matrix is ​​a trainable parameter used to perform nonlinear mapping on the original data. Indicates extracting the first The eigenvalue tensor corresponding to the mode;

[0156] S2043. Calculate the fusion feature matrix of each sample based on the feature vectors of each modality projection and the attention weights of each modality;

[0157] The fusion feature matrix of each sample is obtained by summing the product of the feature vector of each modality projection and the corresponding attention weight of each modality, so as to realize context-adaptive multimodal integration and improve recognition performance.

[0158] No. The fusion feature matrix of each sample integrates multimodal information. The fusion feature matrix of each sample The calculation formula is:

[0159] ;

[0160] The contribution of different modalities to intent recognition dynamically changes with environmental conditions. For example, voiceprint modality dominates in quiet environments while motion modality dominates in noisy environments. Conventional fixed-weight fusion methods cannot adapt to this dynamic change, leading to a decline in recognition performance. S204 achieves context-adaptive multimodal feature fusion by employing a modality weight modulator.

[0161] S205. Calculate the updated hidden state at the current time based on the updated gating vector and the candidate hidden state at the current time;

[0162] S2051. Calculate the updated gate vector at the current time step;

[0163] The current time-fused feature vector, the previous time-fused hidden state, and the velocity vector are concatenated, and then linearly transformed through the gating weight matrix and processed by the Sigmoid activation function to calculate the current time-fused update gating vector, which is used to control the historical state update ratio, so as to achieve adaptive fusion of spatiotemporal information and enhance temporal dependence.

[0164] The updated gate vector at the current moment is the first... Update the gating vector at each time step , The calculation formula is:

[0165] ,

[0166] In the formula, Use the Sigmoid activation function; The gated weight matrix is ​​a trainable parameter used to learn spatiotemporal dependency rules. For the first The first sample The fusion feature vector at each time step is the fusion feature matrix of each sample. In the Slicing at time steps, i.e., fusing feature vectors It is the fusion feature matrix of each sample. The Row vectors; This represents a vector concatenation operation; For the first The hidden state at a time step represents the hidden state at the previous time step. It is initialized to a zero vector at time to pass time-dependent information; is a velocity vector, calculated based on the distance sensor data;

[0167] In a specific implementation, the velocity vector encodes the user's moving speed, enhances the spatio-temporal modeling, such as increasing the voice intention weight when accelerating near the door lock;

[0168] is a velocity vector, calculated based on the distance sensor data; The calculation formula of the velocity vector

[0169] ,

[0170] In the formula, is a velocity vector weight vector, which is a trainable parameter; is a velocity value calculated from the distance sensor data, which is a scalar; is a velocity vector bias vector, which is a trainable parameter;

[0171] The velocity value calculated from the distance sensor data The calculation formula of the velocity value

[0172] ,

[0173] In the formula, is the distance at the i-th moment, which is a scalar; is the distance at the i-th moment; is the sampling interval; S2052. Calculate the candidate hidden state at the current moment;

[0174] The current moment fusion feature vector and the position encoding are spliced, linearly transformed through the state transformation weight matrix, and then processed through the hyperbolic tangent activation function to generate the candidate hidden state at the current moment, which is used to encode the current spatio-temporal feature to combine the position information and capture the spatial dependency;

[0175] The candidate hidden state at the current moment is the candidate hidden state at the i-th time step ,

[0176] The calculation formula of the candidate hidden state at the i-th time step is , The calculation formula of the candidate hidden state at the i-th time step

[0177] ,

[0178] In the formula, is a state transformation weight matrix, which is a trainable parameter, and fuses the feature and the position information; is a position encoding, calculated based on the distance value collected by the distance sensor.

[0179] In a specific implementation, the position encoding Encode the relative position of the user and the door lock to capture spatial dependencies, such as increased unlocking intent when the distance is less than 1 meter;

[0180] Location coding The calculation formula is:

[0181] ,

[0182] In the formula, The position-encoding weight vector consists of trainable parameters; For the first Distance in time is a scalar; The position-encoding bias vector is a trainable parameter;

[0183] S2053. Calculate the updated hidden state at the current time based on the updated gating vector and the candidate hidden state at the current time to update the spatiotemporal state;

[0184] Based on the updated gating vector at the current moment and the hidden state at the previous moment, the hidden state is updated to obtain the updated hidden state at the current moment, realizing the adaptive update of the spatiotemporal state and improving intent modeling;

[0185] The hidden state updated at the current moment integrates temporal and spatial information; the hidden state updated at the current moment is the [number]th [time unit]. Hidden state of time step update , The calculation formula is:

[0186] ,

[0187] In the formula, Represents the update gate vector at the current time. A vector of all 1s with the same dimension;

[0188] Intent recognition needs to consider both temporal dynamic changes and spatial relationships. For example, when a user issues a voice command near a door lock, conventional recurrent neural networks struggle to effectively fuse spatiotemporal information and cannot adaptively adjust information transmission weights. The S205 overcomes these problems by employing a gating spatiotemporal unit to achieve adaptive fusion of spatiotemporal states.

[0189] S206. Calculate the final intent probability distribution of each sample based on the fusion feature matrix of each sample's state transformation;

[0190] S2061. Calculate the state transformation fusion feature matrix of each sample based on the hidden state updated at the current time to achieve state transformation and transition fusion;

[0191] The hidden state matrix is average-pooled to obtain a vector, which is linearly transformed by a state transformation weight matrix, added to the product of an intention transition matrix and a previous predicted probability distribution, and processed by a linear rectification activation function to obtain a sample state transformation fusion feature matrix, ensuring that the output probability conforms to the actual intention transition logic and improving decoding accuracy.

[0192] The th sample state transformation fusion feature matrix is calculated as follows:

[0193] ,

[0194] In the formula, is a state transformation weight matrix, which is a trainable parameter for mapping hidden states to an intention space; is an average pooling operation, the hidden state matrix is pooled into a vector; is a hidden state matrix at all time steps, representing global spatio-temporal features; is an intention transition matrix, encoding the transition constraint relationship between intentions; is the predicted probability distribution of the th sample, i.e., the previous predicted probability distribution, providing transition prior, initialized as a zero vector;

[0195] The hidden state matrix at all time steps is represented as:

[0196] ,

[0197] In the formula, is the hidden state updated at the th time step; is the total number of time steps; denotes the transpose of ;

[0198] In specific implementations, the diagonal line of the intention transition matrix is set to zero to prevent self-repetition, and the non-diagonal elements are updated by gradient descent;

[0199] The element in the th row and the th column of the intention transition matrix is defined as , representing the weight parameter for transitioning from the th intention to the th intention, i.e., from the th intention to the th intention.The tendency weight of the intention, but the actual matrix is diagonally zero to avoid self-transition; The intention transition matrix is indexed by the row, , represents unlocking, represents anti-locking, represents password setting, represents no operation, The intention transition matrix is indexed by the column, ; represents unlocking, represents anti-locking, represents password setting, 4 represents no operation; based on this, the element of the 1st row and the 2nd column of the intention transition matrix represents the tendency weight of anti-locking transition to unlocking;

[0200] The intention transition matrix represents as follows:

[0201] ;

[0202] S2062. Calculate the final intention probability distribution of each sample;

[0203] The final intention probability distribution of each sample is obtained by linear transformation of the sample state transformation fusion feature matrix through the intention weight matrix, to provide the final classification output and support decision-making;

[0204] The dimension of the final intention probability distribution of the 1st sample (equivalent to the total number of intention categories) is , wherein the total number of intention categories is ;

[0205] The calculation formula of the final intention probability distribution of the 1st sample is as follows:

[0206] ,

[0207] In the formula, is the intention weight matrix, which is a trainable parameter, and maps the fusion feature to the category probability;

[0208] The conventional Softmax output layer ignores the transition constraint relationship between intentions, such as not performing an anti-locking operation immediately after unlocking, which leads to a predicted result that does not conform to the intention transition logic in the actual scene, and reduces the recognition accuracy and practicability. S206 adopts a state-condition decoding mechanism to ensure that the output probability conforms to the actual intention transition constraint.

[0209] ​S207. Calculate the total loss function based on intent classification loss, temporal smoothing loss, and modal divergence loss to obtain the perception decision model;

[0210] S2071. Calculate the intention classification loss;

[0211] Based on the one-hot encoding of the final intent probability distribution and the true intent label, combined with the weight coefficients calculated based on the class frequency, the weighted cross-entropy loss is calculated to obtain the intent classification loss, which improves the recognition rate of a few intents and enhances the classification accuracy.

[0212] Intent classification loss is used to improve the recognition ability of a minority of classes based on cross-entropy loss. The calculation formula is:

[0213] ,

[0214] In the formula, The total number of samples, i.e., the total number of door lock interaction events in the training set; Indexed by category; This represents the total number of intent categories, such as unlocking, deadbolting, and password setting. For the first The weight coefficients of class intent are calculated based on the frequency of that class in the training set, and are expressed as follows; For the first The one-hot encoding of the true intent label of the nth sample, if the nth sample... The true intention of each sample is ,but Otherwise, it is 0; It is a logarithmic function, with the default base being the natural constant; The function is softmax, ensuring that the sum of the output probabilities is 1; For the first The probability distribution of the final intent of the nth sample dimension, It is the probability distribution of the final intention. The first vector The element, i.e., the th element The probability value of the class;

[0215] No. Weight coefficient of class intent The calculation formula is:

[0216] ,

[0217] In the formula, For the first The frequency of class intents is used to mitigate class imbalance;

[0218] Intended classification loss It is an improvement based on cross-entropy loss, designed to address the issue of intention class imbalance in door lock scenarios, such as the "lock" intention appearing less frequently than "unlock". Conventional cross-entropy loss assigns equal weight to all classes, but some classes may be ignored. This is addressed by adjusting the weighting coefficients... To improve the recognition rate, give higher weight to minority classes, but avoid overcompensation by using the square root.

[0219] Multiple coefficient Based on the training set Frequency of class intentions The calculation is obtained by first statistically analyzing the data in the training set. The number of samples in a class determines the percentage of samples in that class, i.e., the percentage of samples in the first class. Frequency of class intentions ,calculate This assigns higher weights to low-frequency categories, alleviating the problem of category imbalance.

[0220] S2072. Calculate the time series smoothing loss;

[0221] The mean of the squared L2 norm of the hidden state of each sample at adjacent time steps is calculated to obtain the temporal smoothing loss, which reduces prediction fluctuations and improves temporal consistency.

[0222] Temporal smoothing loss is used to encourage smooth changes in hidden states between adjacent time steps, avoiding drastic fluctuations in intent prediction and improving the stability of door lock interactions; Temporal smoothing loss The calculation formula is:

[0223] ,

[0224] In the formula, For the first The first sample The hidden state is updated at each time step, and the first step is performed on a single sample. Hidden state of time step update The calculation yielded the result; For the first The first sample The hidden state of time step updates.

[0225] Temporal smoothing loss pass The penalty is applied to changes in the hidden state of adjacent time steps to ensure a smooth transition in intent prediction over time, which is consistent with the continuity of user intent in door lock operations, such as the gradual process from "approaching" to "unlocking".

[0226] S2073. Calculate the modal bifurcation loss;

[0227] The negative value mean of the entropy of the attention weight of each sample of each modality is calculated to obtain the modality divergence loss, which avoids over-reliance on a single modality and enhances system robustness.

[0228] The modality divergence loss is used to encourage the perception decision model to balance the use of information of each modality, avoid over-reliance on a single modality, and improve the robustness of the system in a variable environment such as noisy sound or sensor failure. The modality divergence loss The calculation formula of the modality divergence loss is as follows:

[0229] ,

[0230] In the formula, is the attention weight of the m-th modality of the n-th sample, and the calculation of the attention weight of the m-th modality of a single sample is obtained. The modality divergence loss maximizes the entropy of the attention weight, encourages the perception decision model to fairly distribute attention to each modality, prevents the phenomenon of over-reliance on voiceprint in a quiet environment, and enhances the generalization ability.

[0231] S2074. Calculate the total loss function based on the intent classification loss, the timing smoothing loss, and the modality divergence loss.

[0232] The total loss function is obtained by calculating the product of the intent classification loss and the timing smoothing loss weight hyperparameter, the product of the modality divergence loss and the modality divergence loss weight hyperparameter, and the sum of the orthogonal constraint regular terms, which ensures that the perception decision model works cooperatively in classification, smoothing, balance, and disentanglement.

[0233] The total loss function is used to guide the training of the perception decision model, and the calculation formula of the total loss function

[0234] is as follows:

[0235] , In the formula,

[0236] is the timing smoothing loss weight hyperparameter, which is used to control the importance of timing smoothing, and is preferably set to 0.3; is the modality divergence loss weight hyperparameter, which is used to control the importance of modality balance, and is preferably set to 0.3;

[0237] ​​The door lock intention recognition needs to ensure high accuracy and robustness, avoid environmental noise interference, and maintain timing consistency and modal balance. The conventional loss function only focuses on classification accuracy, ignores the timing smoothness, modal divergence, and orthogonal constraints of feature disentanglement in the door lock scene, resulting in that the perception decision model is easily affected by noise and has large prediction fluctuations. S207 overcomes the above problems by designing a composite total loss function and comprehensively optimizing the perception decision model parameters. The total loss function is optimized by multiple components to ensure that the perception decision model works collaboratively in intention classification, timing smoothing, modal balance, and feature disentanglement, and improves the perception decision performance of the door lock interaction system.

[0238] S208. Iterative training and parameter updating of the perception decision model are performed to obtain a trained perception decision model;

[0239] The perception decision model is trained using the labeled multi-modal data set (i.e., a set of labeled perception data). The training process is based on the gradient descent optimization algorithm, which calculates the perception decision model output and the total loss function through forward propagation, and updates the perception decision model parameters through back propagation;

[0240] During the training process, the Adam optimizer is used to calculate the gradient and adjust the perception decision model parameters to minimize the total loss;

[0241] The iterative training continues until the stopping conditions are met, including that the validation set loss does not decrease continuously for multiple iterations, the validation accuracy reaches a predetermined threshold (e.g., above 95%), or the maximum number of iterations (e.g., 1000) is reached;

[0242] The early stopping strategy is used during the training process to prevent overfitting, and the perception decision model parameters with the best validation performance are saved regularly to ensure the convergence and generalization ability of the perception decision model.

[0243] To evaluate the adaptability of the cross-modal feature collaborative fusion technology under different environmental conditions, the robustness of the technology (adaptive fusion), the conventional method (fixed weight), the single modal (voiceprint), and the single modal (motion) under different environments are compared, and the following results are obtained Figure 5 . Figure 5The abscissa of the figure is the environmental condition, including indoor quiet, outdoor noisy, rainy and windy weather, night environment and sensor failure, a total of five scenarios, and the ordinate is the system robustness index, with unit of %. It can be seen that the single mode (voiceprint) is almost ineffective in the sensor failure scenario, the conventional method (fixed weight) also fluctuates greatly when the environment changes, and the single mode (motion) performs poorly in the indoor quiet scenario. However, the system robustness index of the present technology (adaptive fusion) is the highest in all environments, which shows that the present technology (adaptive fusion) has stable performance. Even in challenging scenarios such as sensor failure scenario (simulating single sensor failure), night environment (low light affecting vision) and rainy and windy weather (noise interference), the present technology (adaptive fusion) still maintains usable performance by dynamically adjusting the voiceprint / motion mode weight. This proves that the mode weight modulator can adaptively adjust the multi-modal contribution according to the environmental changes, solving the environmental adaptability defects of the conventional fixed weight fusion.

[0244] S3. Collect new perception data, input the voiceprint data, the relative distance data between the person and the door lock, and the door lock state data in the trained perception decision model after aligning them to the unified reference according to the time stamp, obtain the intention probability distribution, select the highest probability intention as the decision result and trigger the corresponding door lock control instruction.

Claims

1. A perception and decision-making method for a door lock interaction system based on embodied intelligence, characterized in that, include: S1. Collect perception data, which includes voiceprint data, relative distance data between the person and the door lock, and door lock status data. Align the voiceprint data, relative distance data between the person and the door lock, and door lock status data to a unified benchmark according to the timestamp. Label each perception data with the corresponding intent label and environmental noise label to obtain labeled perception data. S2. Construct a perception-decision model and train the perception-decision model to obtain a trained perception-decision model; S201. Calculate the multimodal alignment matrix for each sample; S202. Construct orthogonal constraint regularization terms; S203. Calculate the sparse weighted feature matrix for each sample; S204. Calculate the fusion feature matrix for each sample; S205. Calculate the hidden state updated at the current time; S2051. Calculate the updated gate vector at the current time step; The updated gate vector at the current moment is the first... Update the gating vector at each time step , The calculation formula is: ， In the formula, Use the Sigmoid activation function; Let be the gated weight matrix, which consists of trainable parameters; For the first The first sample The fused feature vector at each time step; This represents a vector concatenation operation; For the first The hidden state at a time step represents the hidden state at the previous time step. Initialize to a zero vector; It is the velocity vector; S2052. Calculate the candidate hidden state at the current time; The candidate hidden state at the current moment is the nth Candidate hidden states of time step , The calculation formula is: ， In the formula, is the state transition weight matrix, which consists of trainable parameters; For position encoding; S2053. Calculate the updated hidden state at the current time based on the updated gating vector and the candidate hidden state at the current time to update the spatiotemporal state; The hidden state updated at the current moment is the th Hidden state of time step update , The calculation formula is: ， In the formula, Represents the update gate vector at the current time. A vector of all 1s with the same dimension; S206. Calculate the probability distribution of the final intention for each sample; S207. Calculate the total loss function to obtain the perception decision model; S208. Iteratively train and update the parameters of the perception-decision model to obtain the trained perception-decision model; S3. Collect new sensing data, align the voiceprint data, relative distance data between the person and the door lock, and door lock status data in the new sensing data to a unified benchmark according to the timestamp, and then input them into the trained sensing decision model to obtain the intent probability distribution. Select the intent with the highest probability as the decision result and trigger the corresponding door lock control command.

2. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The sensing data collected in S1 includes: Sensing data from different real-world usage scenarios is collected by sensors installed in the door lock. The sensing devices include a voiceprint sensor for acquiring voiceprint data, a distance sensor for acquiring relative distance data between a person and a door lock, and a motion sensor for acquiring door lock status data; Real-world usage scenarios include indoors, outdoors, quiet environments, noisy environments, daytime, and nighttime.

3. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The steps in S201 include: S2011. Calculate the time offset compensation function for each mode; S2012. Construct convolution kernels for each modality; S2013. Calculate the multimodal alignment matrix of each sample based on the time offset compensation function of each modality and the convolution kernel of the m-th modality.

4. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The steps in S202 include: S2021. Calculate the environmental noise embedding vector of each sample based on the multimodal alignment matrix of each sample; S2022. Calculate the intent feature embedding matrix of each sample based on the environmental noise embedding vector of each sample; S2023. Construct orthogonal constraint regularization terms.

5. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 4, characterized in that, The steps in S2023 include: S20231. An orthogonality constraint is imposed by applying an orthogonality constraint threshold such that the square of the Frobenius norm of the product of the transpose of the environmental feature extraction weight matrix and the intention feature extraction weight matrix is ​​less than the orthogonality constraint threshold. S20232. Construct orthogonal constraint regularization terms based on orthogonality constraints.

6. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The steps in S203 include: S2031. Calculate the attention score matrix for each sample; S2032. Calculate the sparse gating matrix; S2033. Calculate the sparse attention weight matrix for each sample based on the sparse gating matrix; S2034. Calculate the sparse weighted feature matrix of each sample based on the sparse attention weight matrix of each sample.

7. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The steps in S204 include: S2041. Calculate the attention weights for each modality based on the sparse weighted feature matrix of each sample; S2042. Calculate the feature vector of each modality projection based on the multimodal alignment matrix of each sample; S2043. Calculate the fusion feature matrix of each sample based on the feature vectors of each modality projection and the attention weights of each modality.

8. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The steps in S206 include: S2061. Calculate the state transformation fusion feature matrix of each sample based on the hidden state updated at the current time; No. The fusion feature matrix of sample state transformation The calculation formula is: ， In the formula, It is a linear rectification activation function; This is the state transition weight matrix; This is an average pooling operation; The hidden state matrix for all time steps; The intended transition matrix; For the first The predicted probability distribution of each sample. Initialize to a zero vector; Hidden state matrix at all time steps Represented as: ， In the formula, For the first The hidden state of time step updates; This represents the total number of time steps. express transpose; Define the intention transition matrix The Line number The elements of the column are , Characterization from the first Intent to transfer to the The weighting parameter of the intention, i.e., from the first Intent to transfer to the The intention is to bias the weights, but the actual matrix is ​​set to zero diagonally to avoid self-transition; Intended transition matrix row index, , Indicates unlocking, Indicates deadbolt, Indicates password setting, Indicates no operation. Intended transition matrix column index, , Indicates unlocking, Indicates deadbolt, Indicates password setting, 4 indicates no operation; based on this, the element in the 1st row and 2nd column of the intended transfer matrix is... This indicates the weight of the tendency to switch from locking to unlocking; Intent transition matrix Represented as: ； S2062. Calculate the probability distribution of the final intention for each sample; No. The dimension of the final intent probability distribution for each sample (equivalent to the total number of intent categories) is: , Total number of intent categories; No. The probability distribution of the final intent of each sample The calculation formula is: ， In the formula, Let be the intention weight matrix, and be the trainable parameters.

9. The perception and decision-making method for a door lock interaction system based on embodied intelligence according to claim 1, characterized in that, The steps in S207 include: S2071. Calculate the intention classification loss; S2072. Calculate the time series smoothing loss; S2073. Calculate the modal bifurcation loss; Modal divergence loss The calculation formula is: ， In the formula, is the total number of samples; i is the sample index; m is the modality index; For the first Attention weights for the m-th modality of a sample; attention weights for the m-th modality of a single sample. The calculation yielded the result; S2074. Calculate the total loss function based on intent classification loss, temporal smoothing loss, and modal divergence loss.

Images