Non-contact intelligent household electrical appliance control system and method based on posture recognition

Abstract

The invention provides a non-contact intelligent household electrical appliance control system and method based on posture recognition. The system comprises an acquisition module, a feature extraction module, a feature fusion module, a boundary positioning module, a recognition and classification module and a control module. The acquisition module is used for acquiring continuous CSI signal data of indoor WiFi where the intelligent household electrical appliance to be controlled is located and preprocessing the continuous CSI signal data; the feature extraction module is used for constructing a multi-modal feature extraction model based on the preprocessed continuous CSI signal data and extracting multi-modal features; the feature fusion module is used for fusing the multi-modal features based on Transform to obtain fused features; the boundary positioning module is used for positioning the target action boundary of the operator based on the fusion features; the identification and classification module is used for identifying and classifying the target action based on the positioned target action boundary; and the control module is used for realizing non-contact intelligent household electrical appliance control based on the classified target actions.

Description

Technical Field

[0001] The present invention belongs to the technical field of contactless gesture recognition, and in particular relates to a contactless intelligent home appliance control system and method based on gesture recognition. Background Art

[0002] Smart building systems are rapidly evolving towards contactless interaction. Traditional control methods based on cameras, wearable devices, or sensors face bottlenecks such as privacy breaches, high hardware deployment costs, and fragmented user experiences. Building non-intrusive, natural, and highly robust human-computer interaction systems has become a core requirement for smart city development. WiFi, with its widespread availability and non-intrusive nature, provides an ideal medium for seamless interaction between users and building systems, offering a new path to addressing this challenge. WiFi Channel State Information (CSI) can capture subtle changes in WiFi signal propagation, enabling high-resolution perception of physical space. By capturing the perturbations caused by human motion on electromagnetic wave multipath propagation, it can transform everyday spaces into natural "non-sensing interaction fields." Human motion can also be passively detected by analyzing the signal perturbations caused by the human body.

[0003] However, although existing research methods for unimodal gesture recognition based on CSI signals have achieved promising results, they still face inherent limitations due to the complexity of human motion. These methods typically focus on a single feature modality, such as amplitude, phase, or frequency domain features, which may not fully capture the diversity and dynamic characteristics of human motion: amplitude-based features are extremely susceptible to noise and interference caused by multipath effects, phase-based features are very sensitive to small movements of the device, and frequency domain features may have difficulty distinguishing human motion from environmental factors.

[0004] Previous gesture recognition methods implicitly assume that actions have clear start and end boundaries. Simply put, these studies set a 0-3 second period of inactivity, followed by a specific action, followed by another brief period of inactivity. By capturing these pre-set action segments and extracting temporal features, they then use deep learning models to optimize classification accuracy. However, this assumption conflicts with real-world application scenarios. User behavior is essentially an unconscious, continuous flow of movements, and CSI data exhibits continuous fluctuations, lacking clear action demarcation markers and containing a large amount of excessive movements and other noise.

[0005] As can be seen, current methods for object pose recognition based on CSI signals typically rely on incomplete single-modal feature representations and focus on identifying isolated actions in laboratory environments, making them difficult to adapt to the complexity of continuous motion streams in real-world scenarios. Accurately identifying human pose from continuous signal streams by integrating multiple feature modalities, such as amplitude, phase, and frequency domains, to comprehensively capture human pose representations while overcoming the reliance on feature extraction based on fixed segmentation, remains a pressing technical challenge. Summary of the Invention

[0006] To solve the problems existing in the prior art, the present invention provides a contactless smart home appliance control system and method based on gesture recognition, aiming to integrate multiple feature modes such as amplitude, phase and frequency domain, comprehensively capture the human body gesture representation, overcome the dependence on feature extraction of fixed segment cutting, accurately identify human gesture from continuous signal streams, and thus build a non-invasive, natural and highly robust smart home control system.

[0007] To achieve the above-mentioned object, the present invention provides the following solution: a contactless intelligent home appliance control system based on gesture recognition, the system comprising: an acquisition module, a feature extraction module, a feature fusion module, a boundary positioning module, a recognition and classification module, and a control module;

[0008] The acquisition module is used to collect and pre-process the continuous CSI signal data of the indoor WiFi where the smart home appliance to be controlled is located;

[0009] The feature extraction module is used to construct a multimodal feature extraction model based on the preprocessed continuous CSI signal data to extract multimodal features;

[0010] The feature fusion module is used to fuse the multimodal features based on Transformer to obtain fused features;

[0011] The boundary positioning module is used to locate the target action boundary of the operator's continuous action flow for controlling the smart home appliance based on the fusion feature;

[0012] The recognition and classification module is used to recognize and classify the target action based on the located target action boundary;

[0013] The control module is used to realize contactless control of smart home appliances based on classified target actions.

[0014] Preferably, in the feature extraction module, a multimodal feature extraction model is constructed based on the preprocessed continuous CSI signal data to extract multimodal features, including:

[0015] The spatial features in the multimodal features are extracted by CNN:

[0016]

[0017] in, Represents the spatial feature, W s represents the spatial weight matrix, and H represents the input features;

[0018] The long time series features in the multimodal features are extracted through the bidirectional LSTM structure:

[0019]

[0020] Among them, H lstm Represents long time series features, is the forward hidden state of LSTM at time step t, is the backward hidden state of LSTM at time step t;

[0021] The multi-scale dynamic features in the multi-modal features are extracted by TCN:

[0022] H tcn =ReLU(Y (l) +F attn (H))

[0023] Among them, H tcn Represents multi-scale dynamic features, ReLU represents the activation function, Y represents dilated convolution, l represents the number of convolution layers, and F attn (·) represents the attention function.

[0024] Preferably, the feature extraction module constructs a multimodal feature extraction model based on the preprocessed continuous CSI signal data to extract multimodal features, and further comprises: performing adaptive model collaborative training on the multimodal feature extraction model;

[0025] Based on the dynamic Dropout mechanism of feature energy norm, the multimodal feature extraction model is dynamically regularized and modal balanced:

[0026]

[0027] Among them, E (m) is the information intensity of each modal output feature, H (m) is the characteristic energy value of mode m, ||H (m) || F is the Frobenius norm, is the value of the i-th sample and j-th feature dimension in the feature matrix of modality m, γ (m) is the dynamic discard rate, α is the baseline discard rate, ∑ n E (n) is the sum of all modal energy values, n is the number of modes;

[0028] Based on the gradient reversal layer GRL and the domain discriminator E, the multimodal feature extraction model is subjected to domain adaptive adversarial training:

[0029]

[0030] Among them, θ f is the feature extraction parameter, θd are the parameters of the domain discriminator, is the classification loss, is the domain discrimination loss, δ is the trade-off factor, and GRL(f(x)) represents the application of the gradient reversal layer to the feature f(x).

[0031] Preferably, in the feature fusion module, the multimodal features are fused based on Transformer to obtain fused features, including: using a dual-headed attention mechanism to guide the fusion of the multimodal features to obtain fused features:

[0032] The head 1 subspace in the dual-head attention mechanism is used to obtain the dependency relationship between the spatial features and the long-term temporal features, and generate the head 1 attention weight matrix by calculating the dot product similarity between the query vector projected by the CNN and the key vector projected by the LSTM;

[0033] The head 2 subspace in the dual-head attention mechanism is used to analyze the relationship between the multi-scale dynamic features and the global context, identify the dependency between the action mutation moment and the long-term action context by interacting the query vector projected by the TCN and the global key vector, and generate the head 2 attention weight matrix;

[0034] Based on the first attention weight matrix and the second attention weight matrix, obtain the fusion feature:

[0035] F fusion =LayerNorm(F+Dropout(W O ·Concat(Attention1,Attention2)))

[0036] Among them, F fusion represents the fusion feature, F represents the concatenation of multimodal features, W O represents the learnable matrix, Attention1 and Attention2 represent the head 1 attention weight matrix and head 2 attention weight matrix respectively.

[0037] Preferably, in the boundary positioning module, positioning the target action boundary of the operator based on the fusion feature includes: positioning the target action boundary of the operator based on a main path-residual path double-branch structure;

[0038] The main path uses a 3×1 convolution kernel to capture global action patterns:

[0039] h main =LeakyReLU(BatchNorm(Conv1D 3×1 (h lstm )))

[0040] Among them, h main Represents the main path output feature, LeakyReLU represents the activation function, h lstm Represents the output features of bidirectional LSTM;

[0041] The residual path cascades 5×1 and 3×1 convolution kernels to model short-term mutations and local detail features respectively:

[0042] h res =LeakyReLU(BatchNorm(Conv1D 5×1 (Conv1D 3×1 (h lstm ))))

[0043] Among them, h res Represents the residual path output feature;

[0044] The interaction of the dual-path outputs is achieved through element-wise product, and finally normalized along the time dimension through the Softmax function:

[0045]

[0046] in, is the attention weight;

[0047] The weighted time series features are:

[0048]

[0049] Among them, h focused is the weighted time series feature, and T' is the total number of time steps.

[0050] Preferably, the boundary positioning module locates the target action boundary of the operator based on the fusion feature, and further comprises: combining the gradient feature with the main path output feature h through a gated fusion mechanism. main To perform the fusion:

[0051] g=σ(W g [h main ;h grad ]+b g )

[0052] Among them, g represents the gate value, σ is the sigmoid function, W g is the learnable weight matrix, b g is the bias term, h grad Represents the gradient feature.

[0053] Preferably, in the recognition and classification module, the target action is recognized and classified based on the located target action boundary, including: inputting the fusion feature into the fully connected layer, and mapping it to the specific posture category space through the classification decision layer to realize the recognition and classification of the target action.

[0054] The present invention also provides a gesture recognition-based contactless smart home appliance control method, which is implemented by applying the aforementioned gesture recognition-based contactless smart home appliance control system. The method includes:

[0055] S1. Collect and pre-process the continuous CSI signal data of the indoor WiFi where the smart home appliance to be controlled is located;

[0056] S2. Based on the preprocessed continuous CSI signal data, a multimodal feature extraction model is constructed to extract multimodal features;

[0057] S3. Fusing the multimodal features based on Transformer to obtain fused features;

[0058] S4. Locating the target action boundary for controlling the smart home appliance in the operator's continuous action flow based on the fusion features;

[0059] S5. Identify and classify the target action based on the located target action boundary;

[0060] S6. Based on classified target actions, contactless smart home appliance control is achieved.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. This paper uses a bidirectional LSTM network combined with a main path-residual path dual-branch structure to capture global patterns and local mutation features, achieving precise positioning of action boundaries;

[0063] 2. The present invention extracts high-frequency transient features from the signal through a time difference operator and combines it with a gated fusion mechanism to dynamically balance global temporal coherence and local mutation sensitivity, effectively suppressing the interference of transition actions.

[0064] 3. The present invention designs a multimodal feature extraction model that can extract features for gesture recognition from three aspects: spatial correlation between subcarriers, long-range motion continuity, and temporal dynamics. This reduces the problem of decreased gesture recognition accuracy due to insufficient feature extraction.

[0065] 4. This paper designs a dynamic dropout mechanism based on the feature energy norm and a domain adaptive adversarial training framework, which balances the contribution of multimodal features, eliminates the impact of different scenarios on feature distribution, and improves the generalization ability of the model;

[0066] 5. The present invention designs a Transformer-based feature fusion method, which can project heterogeneous features into a unified semantic space to eliminate the modal gap, and then use a multi-head attention mechanism to dynamically weight while retaining the original features, thereby improving the representation ability of the features and increasing the accuracy of posture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0068] Figure 1 Schematic diagram of a contactless smart home appliance control system based on gesture recognition according to an embodiment of the present invention;

[0069] Figure 2 Schematic diagram of a multimodal feature extraction model framework according to an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of a main path-residual path dual-branch structure framework according to an embodiment of the present invention;

[0071] Figure 4 Schematic diagram of the adversarial training framework for domain adaptation according to an embodiment of the present invention;

[0072] Figure 5 This is a schematic diagram of the fully connected layer network structure according to an embodiment of the present invention;

[0073] Figure 6 This is a schematic diagram of the framework of an intelligent building control system based on digital twins according to an embodiment of the present invention;

[0074] Figure 7 Generate a flow chart for the instructions of the embodiment of the present invention;

[0075] Figure 8 Schematic diagram of the physical experiment scene and the virtual experiment scene according to an embodiment of the present invention, wherein a) is the physical experiment scene and b) is the virtual experiment scene. DETAILED DESCRIPTION

[0076] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0077] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0078] Example 1

[0079] like Figure 1 As shown, the present invention provides a contactless intelligent home appliance control system based on gesture recognition, comprising:

[0080] The acquisition module is used to collect and pre-process the continuous CSI signal data of the indoor WiFi where the smart home appliance to be controlled is located;

[0081] The feature extraction module is used to build a multimodal feature extraction model based on the preprocessed continuous CSI signal data and extract multimodal features;

[0082] Feature fusion module, used to fuse multimodal features based on Transformer to obtain fused features;

[0083] The boundary positioning module is used to locate the boundary of the target action for controlling smart appliances in the operator's continuous action flow based on the fusion features;

[0084] Recognition and classification module, used to identify and classify target actions based on the located target action boundaries;

[0085] The control module is used to realize contactless control of smart home appliances based on classified target actions.

[0086] Specifically:

[0087] In the continuous CSI signal data collected from indoor WiFi, because WiFi is susceptible to environmental noise and inherent device interference, the original amplitude contains a large amount of noise unrelated to human behavior, which can negatively impact behavior recognition. Furthermore, data preprocessing is necessary to facilitate subsequent work. To this end, this paper employs a cascaded noise filtering and dynamic dimensionality reduction method to isolate stable human behavior features from complex signals.

[0088] The Butterworth filter is widely used in the field of signal processing due to its smooth frequency response curve. In order to eliminate high-frequency noise, a Butterworth low-pass filter is used to perform bidirectional zero-phase filtering on the data based on the signal spectrum characteristics. To ensure that the filter can effectively remove noise, an 8th-order filter is set, and the normalized cutoff frequency is selected as 0.5. It effectively suppresses high-frequency noise and retains the low-frequency characteristic components (0.1-5Hz) related to human motion. The filtering process is carried out along the time axis, so that the timing information of each signal channel is smoothed, thereby improving the accuracy of subsequent feature extraction. The transfer function of the filter is:

[0089]

[0090] Among them, s c is the cutoff frequency, U is the filter order, is the transfer function, and s is the complex frequency variable. Bidirectional filtering eliminates phase distortion through two filtering operations, forward and reverse, ensuring phase consistency of the timing signal.

[0091] In WiFi preprocessing, the signal typically has a large number of channels, and each time step contains high-dimensional data. However, some channels contribute only a limited amount to behavioral features. Directly using high-dimensional data can lead to the "curse of dimensionality" problem due to strong correlation between subcarriers. Therefore, the PCA algorithm is used to reduce the dimensionality of the valid data segments.

[0092] For each 30-dimensional subcarrier data in a 2-second time window, first calculate its covariance matrix S:

[0093]

[0094] Among them, μ is the mean vector of each subcarrier signal, p is the total number of samples in the current time window, and x i is the i-th sample data, T represents the transpose. The eigenvalue λ is obtained by singular value decomposition i and the corresponding eigenvector v i , select the first k = 10 principal components in descending order of eigenvalue, and the cumulative contribution rate is as follows:

[0095]

[0096] The low-dimensional feature after projection X'=XV k , X represents the original high-dimensional data matrix after preprocessing, V k v iThe principal component matrix constructed compresses the feature dimensions while retaining the main information. To further enhance robustness, a local PCA strategy based on sliding windows is proposed: the dimensionality reduction operation is performed independently on each time window, rather than using a global projection matrix, thereby adapting to the differences in feature distributions of different action modes.

[0097] WiFi CSI signals are collected continuously in the form of time series, and different behaviors change on different time scales. Secondly, the initiation and termination of actions are often accompanied by sudden changes in limb acceleration, which easily introduces high-frequency noise. It is necessary to intercept the middle segment data when the action is stable to improve the signal-to-noise ratio. Therefore, in the task of behavior recognition, it is necessary to reasonably divide these long time series data into time windows to reduce the computational complexity of the model, allowing the model to process and classify each time segment to better predict human behavior. The present invention proposes a three-level time series slicing strategy, which achieves precise alignment of action features and expansion of sample capacity through coarse-grained segmentation, effective segment extraction and fine-grained enhancement operations.

[0098] (1) Coarse-grained segmentation: The continuous signal is segmented into 5-second data segments using non-overlapping windows. Based on a sampling frequency of 50 Hz, each segment contains 250 time points. The total data length of 9,000 points (corresponding to 3 minutes) can be segmented into 36 segments. This operation is intended to eliminate transient noise during the device initialization phase and provide data units of uniform length for subsequent processing.

[0099] (2) Extraction of valid segments: The middle 2-second high-confidence interval (50 to 150 time points) is extracted from each 5-second data segment. This interval avoids the fuzzy boundary area between the start and end of the action to ensure signal stability. The dimension of the extracted valid data is 30 × 100 (number of subcarriers × time points), which directly corresponds to the typical duration of human body movements.

[0100] (3) Fine-grained sliding window: A sliding window with a window length of 120 points and a step size of 3000 points is applied to 2 seconds of valid data to generate samples. Since the valid data is only 100 points, the actual code generates non-overlapping samples through full window coverage (window length 100 points, step size 50 points), and finally reshapes the data into a three-dimensional tensor (number of samples × time series length × feature dimension) to adapt to the input requirements of the model.

[0101] The behavioral representation of CSI signal data can be deconstructed into three orthogonal dimensions: space, time series, and dynamics. Since a single network cannot fully capture the multi-dimensional features, the present invention constructs a multimodal feature extraction model, such as Figure 2As shown in the figure, CNN and LSTM work together to capture the spatial local correlation, long-term temporal dependence, and multi-scale dynamic patterns of the signal. This model, through the complementary use of heterogeneous modalities, addresses the inadequate representation of complex human behavior by traditional single models, achieving refined modeling of key representations of human behavior.

[0102] CSI signals exhibit strong spatial correlation, resulting in similar fluctuation patterns due to the multipath effect between adjacent subcarriers. A spatial feature extraction method, using a three-stage convolutional module in conjunction with a spatial attention mechanism, can extract spatial features from complex CSI data. Therefore, this paper employs a CNN to extract spatial features from CSI signal data.

[0103] The original CSI signal data is pre-processed and input into the convolution module, and its dimension is Where B is the number of samples, D is the feature dimension, and L is the time series length. The first layer of convolution uses a 3×1 kernel to slide along the subcarrier dimension to capture the local spatial correlation between adjacent subcarriers. When the arm is waving, adjacent subcarriers (such as No. 5, 6, and 7) may show similar energy fluctuations due to the multipath superposition effect. The 3×1 convolution extracts such local patterns through a weight sharing mechanism and outputs a 32-channel feature map. In order to alleviate the gradient vanishing problem, the present invention adopts the LeakyReLU (Leaky Rectified Linear Unit) activation function:

[0104]

[0105] Where * is the convolution operation, b d is the bias term of convolution, C (v) is the output of the v-th convolution layer, is the convolution kernel parameter of the vth layer, where v = 1 and H represents the input feature.

[0106] Subsequently, one-dimensional max pooling with a stride of 2 compresses the feature map size to L / 2, preserving significant response areas while suppressing high-frequency noise. The essence of the pooling operation is to improve the model's robustness to small signal shifts through downsampling. For example, the same hand-waving gesture may experience slight timing shifts due to differences in user height. Max pooling ensures that the maximum response of key features within a local window is preserved.

[0107] The second convolutional layer expands the number of channels to 64, abstracting higher-order spatial patterns by increasing the depth of the feature map. At this stage, each channel in the feature map corresponds to a specific subcarrier correlation pattern. This means that some channels may encode features characterized by the synchronous rise of adjacent subcarriers, while others may respond to alternating fluctuations between even and odd subcarriers. After the second pooling, the feature map length is reduced to L / 4, at which point high-level semantic features are gradually highlighted.

[0108] The third convolution layer further increases the number of channels to 128 and introduces a spatial attention module on the output features. This module dynamically generates subcarrier weights through a two-way aggregation strategy: first, the mean pooling M is calculated along the channel dimension. arg With maximum pooling M max , capturing the statistical characteristics of the subcarrier in the channel dimension:

[0109]

[0110] Where C is the number of channels, c, n, and l are variables of C, N, and L respectively, N is the number of subcarriers, and L is the time sequence length. The two are concatenated and input into a 7×1 convolutional layer, and the spatial weight matrix is generated by Sigmoid activation.

[0111] W s =σ(W attn *[M avg ;M max ])

[0112] Among them, M avg It is mean pooling, which captures the global statistical characteristics of the channel dimension and provides basic information for the subsequent generation of spatial attention weights; W attn is the convolution kernel parameter, and the extracted spatial features are finally output

[0113]

[0114] This process enables the model to autonomously enhance subcarriers that are sensitive to motion and suppress subcarriers that are severely interfered by environmental noise.

[0115] LSTM is a classic recurrent neural network that solves the vanishing gradient problem of traditional RNNs through a gating mechanism, effectively modeling long-term dependencies in time series. Its core concept is to selectively retain historical information and integrate current input through the updating and propagation of cell states. This makes it suitable for modeling the temporal evolution of continuous actions in human action recognition. However, traditional LSTM faces two major bottlenecks in WiFi action recognition tasks. First, its one-way propagation structure can only capture forward temporal dependencies, making it difficult to model backward dependencies of actions (such as the signs of imbalance before a fall). Second, its ability to capture the instantaneous features of sudden actions is insufficient due to the delayed response of the gating mechanism to high-frequency changes.

[0116] To solve the above problems, this paper improves Bi-LSTM and realizes efficient modeling of long-range dependencies of CSI signals. First, a bidirectional LSTM structure is adopted to capture the long-term dependencies of actions through forward and backward hidden state propagation. tis the input of LSTM at time step t, and the hidden state of LSTM is then updated:

[0117]

[0118] in, is the forward hidden state of LSTM at time step t, is the forward hidden state of LSTM at time step t-1,

[0119] is the backward hidden state of LSTM at time step t, is the backward hidden state of the LSTM at time step t-1. 128 hidden units are set in each direction, and the output dimension is 256. This design can simultaneously capture the forward causality and backward correlation of actions, such as the body's micro-movements that precede standing up. Secondly, by stacking two LSTM layers, the network's receptive field is expanded to 120 time steps, effectively modeling the complete evolution of the action cycle.

[0120] Finally, the output is the concatenation of the two-way hidden states, that is, the long time series feature h lstm :

[0121]

[0122] Although CNN and LSTM have covered spatial locality and long-term temporal dependencies respectively, in actual home scenarios, human behavior often exhibits cross-scale mixed characteristics. In the time series behavior recognition task, transient actions and continuous actions exhibit different characteristics in the time domain, and in the home environment, sudden interference may occur due to reasons such as the running of domestic pets. To this end, the present invention introduces TCN, and through the collaborative design of dilated convolution and residual connection, it constructs a multi-scale temporal perception pyramid to achieve hierarchical analysis of dynamic patterns of actions. Finally, parallel convolution calculations are used to reduce the inference consumption time to meet the low-latency requirements of home appliance control.

[0123] Traditional TCNs expand the receptive field by stacking dilated convolutional layers, but their fixed dilation coefficient strategy makes it difficult to adapt to the non-uniform time scale of human behavior. This paper proposes a progressive multi-scale TCN using three layers of dilated convolutions with dilation coefficients of 1, 2, and 4, respectively. The definition of dilated convolution is as follows:

[0124]

[0125] Among them, Y (l) [t] is the value of the output of the dilated convolution of the lth layer at time point t, which represents the features of the current layer calculated by the dilated convolution, d l =2 l-1 is the expansion coefficient of the lth layer, Z=3 is the convolution kernel size, The time series data input to the lth layer of dilated convolution. By stacking three layers of dilated convolution, i.e., d1 = 1, d2 = 2, d3 = 3, the model receptive field coverage is expanded to:

[0126]

[0127] Here, R is the model's receptive field, representing the range of time steps in the input sequence that influence a particular output unit. This design enables the network to abstract dynamic features at different scales layer by layer. The first layer captures local subtle changes, the second layer perceives mid-span patterns, and the third layer models the complete action cycle. By progressively expanding the receptive field, the network is able to gradually abstract dynamic features from detailed to global.

[0128] To address the problem of environmental noise interference, a temporal attention module is embedded in the residual path. The feature contribution is dynamically calibrated through 1×1 convolution. The final output features of TCN are used as multi-scale dynamic features:

[0129] H tcn =ReLU(Y (l) +F attn (H))

[0130] Among them, F attn (·) is the attention function, which suppresses the time frames dominated by noise such as children or pets running in the home environment; Y (l) is the output value of the l-th layer dilated convolution.

[0131] For the extracted spatial features, long-term features, and multi-scale dynamic features, traditional methods of fixed-weight splicing or simple weighted averaging are difficult to adapt to dynamic scene changes. To cope with the diversity of user behavior and the complexity of environmental interference, this paper uses a lightweight feature fusion method based on a lightweight Transformer encoder. This method achieves dynamic synergy between multimodal features through two core stages: semantic space mapping and attention-guided fusion.

[0132] In multimodal action recognition tasks, features from different modalities often carry complementary but heterogeneous semantic information. Spatial features extracted by CNNs focus on local correlations between subcarriers, temporal states captured by LSTMs reflect action continuity, and dynamic patterns captured by TCNs reveal fine-grained variations. These features are inherently in different semantic spaces—that is, feature representation systems with different mathematical distributions. Direct feature concatenation or weighted averaging can result in a semantic gap, where the numerical scales and distribution patterns of features from different modalities differ significantly. For example, the L2 norm of CNN features can be an order of magnitude higher than that of LSTM features, resulting in the dominant modality masking other information when simply fused. Furthermore, correlations between different features may be overlooked. For example, the starting moment of a wave (the focus of the LSTM) may be potentially correlated with the energy fluctuations of a specific subcarrier (the response area of the CNN). However, traditional methods cannot automatically capture such cross-modal patterns.

[0133] To solve the above problems, the present invention introduces the semantic space projection technology. The semantic space projection technology can project the features of different modalities into a unified low-dimensional latent space. The features in this space have the following characteristics:

[0134] (1) The distributions of all modal features are normalized to the same scale;

[0135] (2) Cross-modal semantic associations can be directly measured through vector operations;

[0136] (3) The attention mechanism can effectively capture the interaction patterns between modalities.

[0137] Assume that the output features of CNN, LSTM and TCN are Use H m Represents the output feature of the mth modal. First, feature standardization and L2 normalization are performed as follows:

[0138]

[0139] in, is the normalized feature, is the feature after L2 normalization, μ m ,ρ m is the mean and standard deviation of each modal feature. This operation eliminates the scale difference between modalities and lays the foundation for subsequent projection. In order to establish the foundation for cross-modal interaction, a learnable projection matrix is designed. Map the normalized features to a 100-dimensional shared semantic space:

[0140]

[0141] Among them, Q, K and V represent query, key and value respectively.

[0142] Note that G Q , G K , G V It is automatically learned through the Transformer's multi-head attention mechanism.

[0143] To capture diverse intermodal association patterns, a dual-headed attention mechanism is employed within a shared subspace to model different types of feature associations. The core of multi-headed attention is to decompose the semantic space into multiple subspaces, learning complementary interaction patterns. Ultimately, multi-view information is integrated to enhance feature representation capabilities.

[0144] First, the 100-dimensional shared semantic space is evenly divided into two 50-dimensional subspaces. Each subspace corresponds to an attention head, focusing on different types of feature interactions:

[0145] Head 1 Subspace - Spatial-Temporal Complementarity: This subspace focuses on modeling the relationship between the spatial features extracted by the CNN and the temporal dependencies captured by the LSTM. Specifically, an attention weight matrix is generated by calculating the dot product similarity between the query vector (from the CNN projection) and the key vector (from the LSTM projection). This weight reflects the changing importance of different subcarrier regions during the action cycle. For example, in a "wave" action, the energy of certain subcarriers fluctuates regularly within the arm swing cycle. The attention weights in the head 1 subspace can automatically capture this spatial-temporal coupling pattern.

[0146] Head 2 Subspace - Dynamic Global Collaboration: This subspace focuses on analyzing the relationship between the multi-scale dynamic features captured by the TCN and the global context. By interacting the query vector projected by the TCN with a globally oriented key vector (which integrates global features from all modalities), attention weights can identify the dependencies between sudden action moments (such as the moment of stepping) and the long-term action context. For example, in a fall detection task, Head 2 can correlate the dramatic dynamic changes at the moment of a fall with the characteristics of the equilibrium state before the fall, thereby enhancing the model's sensitivity to abnormal movements.

[0147] To avoid introducing too many parameters for multi-head attention, all attention heads share the same set G Q , G K , G V Projection matrix. In this design, a single linear transformation layer generates the projection results for all heads, which are then divided into blocks through dimension reshaping and transposition operations. This block division allows for shared computing resources and reduces memory usage.

[0148] The attention weight of each head is calculated by scaled dot product as follows:

[0149]

[0150] Among them, a is the number of attention heads, s k =50 is the dimension of each head, scaling factor It is used to prevent the dot product value from being too large and causing gradient saturation. After the dual-head output is spliced, it is passed through the learnable matrix Linearly transform to the original dimension, the matrix is automatically optimized by back propagation without manual setting. Then it is added to the residual connection feature to obtain the fusion feature:

[0151] F fusion =LayerNorm(F+Dropout(W O ·Concat(Attention1,Attention2)))

[0152] Here, F is the concatenation of the original multimodal features. This method adds the original concatenated features to the attention output, retaining the original discriminant features without attention weighting and avoiding the loss of key information.

[0153] Long-term CSI signals contain both discrete and continuous gesture distributions. In real-world continuous action recognition, user actions often exhibit a fluid, undefined nature, lacking clear boundaries. There's no clear "demarcation line" to indicate when an action begins and ends. Traditional methods, such as using a fixed-length "time window" to capture signals, have significant drawbacks. If the window falls right in the middle of an action, recognition errors can occur.

[0154] The purpose of continuous motion is to make it applicable to real-world scenarios. Previous research has assumed single actions, but in reality, human activity is not a series of intermittent movements; it is often continuous. For example, when someone walks from the bedroom to the living room at night to watch TV, their posture changes from walking to sitting on the sofa, and then turning on the TV. Assuming that the person turns on the TV with a single hand wave, the motion flow is walk-sit-wave. In this process, only the single hand wave gesture, which is the target action of turning on the TV, is significant. Previous research used fixed interval segmentation for the signal, such as walking for 7 seconds, sitting for 2 seconds, and waving for 2 seconds. This segmentation interval is 2 seconds. This process breaks the 2 seconds of waving into segments consisting of 1 second of sitting and 1 second of waving, as well as 1 second of waving and other 1 seconds of segments, which can interfere with recognition. Another possible issue is that the model may have incorrectly weighted the attention, assigning weight to irrelevant actions, leading to recognition errors. In the past, if you wanted to improve accuracy, you had to sit still for a while and then wave your hand to achieve accurate recognition. However, in this process, stillness is an ideal state. Even if you don't move, there will be interference in the surroundings, causing the CSI signal to fluctuate and affecting recognition.

[0155] To solve this problem, the present invention imitates the human intuition of observing actions - judging the boundaries of the target action through the association between the past and the present. First, the model needs to continuously observe the entire signal stream. Through the bidirectional LSTM network, forward learning can be achieved from the first frame to the current frame to understand "how the action has developed to the current state", and reverse learning can be achieved from the last frame back to the current frame to predict "what may happen next", so as to comprehensively judge whether the current action is at the transition point. The model will automatically divide the action interval, that is, find the boundary of "walk-sit down" and the boundary of "sit down-waving", so as to avoid posture confusion and ensure accurate recognition of the target posture, thereby turning on the TV.

[0156] Specifically, in order to enhance the spatiotemporal joint representation of the target posture, a main path-residual path dual-branch structure is constructed, such as Figure 3 As shown. The main path uses a 3×1 convolution kernel to capture the global action pattern:

[0157] h main =LeakyReLU(BatchNorm(Conv1D 3×1 (h lstm )))

[0158] The output feature h of the bidirectional LSTM lstm After enhancing numerical stability through batch normalization and LeakyReLU activation function, we get h main Subsequently, residual temporal attention is introduced into the bidirectional LSTM backend, and parallel main path and residual path convolutional networks are designed.

[0159] The residual path cascades 5×1 and 3×1 convolution kernels to model short-term mutations and local detail features respectively. The former expands the receptive field to capture short-term and violent fluctuations of the initial energy surge, such as the standing up action, while the latter refines the local details and finally outputs the feature h res :

[0160] h res =LeakyReLU(BatchNorm(Conv1D 5×1 (Conv1D 3×1 (h lstm ))))

[0161] The interaction of the dual-path outputs is achieved through element-wise product, and finally normalized along the time dimension through the Softmax function:

[0162]

[0163] The attention weight The global pattern of the main path provides contextual constraints for local mutations in the residual path, preventing isolated noise points from being misidentified as target gestures. For example, in a continuous hand-waving motion, the main path identifies the periodic swinging rhythm, while the residual path detects the acceleration change at the start of each wave. The product of the two significantly increases the attention weight near the peak of the cycle and automatically decays in transition regions (such as the gap between waves). The product increases significantly only when the main path detects a change in the global motion pattern and the residual path captures a local mutation, thus achieving automatic localization of the motion boundary.

[0164] Finally, the weighted time series features are as follows:

[0165]

[0166] Among them, h focused is the weighted time series feature, and T' is the total number of time steps, that is, the length of the input sequence.

[0167] While TCN also processes time series signals, its core goal is to extract the complete evolutionary pattern of an action through dilated convolution covering multi-scale time windows. RTA, on the other hand, focuses on enhancing keyframes at the 0.1-second level, creating a complementary effect. TCN identifies the complete action cycle from a sudden event, while RTA emphasizes the instantaneous onset of a sudden event and suppresses interfering signals from that state. This allows TCN to filter persistent noise when a sudden event occurs, while RTA can suppress transient interfering frames to reduce false trigger rates.

[0168] In continuous motion recognition, the essential characteristics of the target posture are often reflected in transient abrupt changes in signal amplitude. For example, when a person transitions from standing to sitting, the CSI signal experiences dramatic millisecond-level fluctuations on specific subcarriers. This abrupt change can be easily masked by the gentle changes during the transition phase in the original signal. To reduce the impact of transitional motion on posture recognition, this paper proposes a method for suppressing the interference of transitional motion.

[0169] First, the gradient changes at adjacent moments are extracted through the time difference operator. The difference values at adjacent moments are as follows:

[0170] Δ t =Y t -Y t-1

[0171] in, is the CSI signal vector at time t, The CSI signal vector at time t-1 is calculated by computing the difference between adjacent time steps. This amplifies transient changes during action transitions while suppressing low-frequency fluctuations during smooth actions or transitions. The essence of the differential operation is a discretized first-order gradient approximation, which effectively extracts high-frequency transient features from the signal.

[0172] The feature h output by the residual path res Mapped to the differential space, amplifying the sudden change component at the action transition moment. Then a 3×1 convolution is used to transform the gradient feature h grad After local pattern abstraction, the enhanced gradient-sensitive features are as follows:

[0173] h grad =Conv1D 3×1 (Δ t (h res ))

[0174] in, To perform a time-dependent differential operation on the residual features, a 3×1 convolution is used to extract local patterns from the differentiated features. Its narrow time window design (covering three consecutive time steps) is optimized for short-term motion changes (such as a sitting posture change within 0.2 seconds). The convolution kernel weights are learned through backpropagation to adaptively capture gradient patterns associated with the target pose.

[0175] In order to achieve a dynamic balance between global temporal coherence and local mutation sensitivity, a gated fusion mechanism is designed to combine the gradient features with the global representation h of the main path. main Through the fully connected layer, the concatenated feature vector Mapped to the gated space, the gate value g∈(0,1) output by the Sigmoid function essentially quantifies the significance level of the signal mutation at the current moment. The calculation method of g is as follows:

[0176] g=σ(W g [h main ;h grad ]+b g )

[0177] Where W g is a learnable weight matrix that maps concatenated features to latent space, b g is the bias term, and σ is the sigmoid function. When a sharp gradient change is detected, g tends to 0, and the system prioritizes h grad Decisions are made based on local, detailed features. During the steady motion phase, g increases close to 1, relying on the global temporal pattern of the main path to maintain motion continuity. This dynamic feature selection mechanism enables the model to both sensitively capture the boundary features of the target pose and avoid misjudgments caused by subtle jitter during transitions, forming a closed-loop optimization process of "gradient-aware triggering and context-adaptive."

[0178] To balance the contribution differences between multiple modalities, a dynamic Dropout mechanism based on the feature energy norm is designed. During the training process, the Frobenius norm of each modal output feature is calculated in real time to quantify the information strength of each modal output feature:

[0179]

[0180] in is the output feature matrix of mode m, is the value of the i-th sample and j-th feature dimension in the feature matrix of modality m. ||H (m) || F is the Frobenius norm, which calculates the square root of the sum of the squares of all elements of the characteristic matrix and represents the characteristic energy of mode m. (m) is the characteristic energy value of mode m, reflecting its contribution to the current task. After obtaining the characteristic energy value, the Dropout rate of each mode is dynamically adjusted according to the energy ratio:

[0181]

[0182] where γ (m) is the dynamic drop rate, which is used to adjust the retention probability of the modality during training. α is the baseline drop rate, which controls the overall drop strength. n E (n) is the sum of all modal energies, and n is the number of modes. This strategy allows the model to evenly explore the potential of each mode in the early stages of training, avoiding premature convergence to the dominant mode. In the later stages of training, it reduces the discard rate of high-energy modes and retains their discriminative information.

[0183] To eliminate the impact of different scenarios on feature distribution, a gradient reversal layer (GRL) and a domain discriminator E are introduced. GRL is a core component in adversarial distribution alignment. Its function is to force the model to learn scenario-independent feature representations by reversing the sign of the gradient during backpropagation. In adversarial training, the domain discriminator E, through a simple two-layer lightweight MLP, acts as an "adversary" in adversarial training, forcing the feature extractor to generate scenario-independent discriminative features. The goal of the feature extractor is to generate features that prevent the domain discriminator from distinguishing between scenarios. Figure 4 We present an adversarial training framework for domain adaptation.

[0184] The domain discriminator E receives the multimodal fusion feature F fusion , predicting the scene to which the sample belongs (such as laboratory or home environment), while the feature extractor confuses scene discrimination through adversarial training, forcing the modal features to approach the same distribution:

[0185]

[0186] Among them, θ f is the feature extraction parameter, θ d are the parameters of the domain discriminator, is the classification loss, Domain discrimination loss, δ is a trade-off factor, GRL(f(x)) means applying a gradient reversal layer to the feature f(x). The feature extractor needs to minimize Ensure mission performance while maximizing Through GRL gradient reversal, the feature scene is made independent. The goal of the domain discriminator is to minimize Accurately distinguish scenes.

[0187] Specifically, the gradient reversal layer directly passes the input features during forward propagation without any numerical modification. During backward propagation, the GRL reverses the sign of the gradients generated by the domain discriminator and passes these reversed gradients to the feature extractor. When the domain discriminator attempts to adjust its parameters based on the gradients to better distinguish scenes, the feature extractor receives the opposite gradient, forcing it to adjust its parameters to generate features that confuse the domain discriminator.

[0188] After completing the multimodal feature extraction, fusion and positioning of the operator's target action boundary, it is necessary to map it to the specific posture category space through the classification decision layer to achieve posture classification.

[0189] As the core component of the classifier, the fully connected layer can achieve feature dimension compression and category judgment learning. The fully connected layer network structure is as follows Figure 5 As shown in Figure 1. The fused multidimensional features are first fed into a fully connected layer for spatial compression, mapping the 128-dimensional feature vector to a lower-dimensional space, ultimately generating a 10-dimensional output vector. During training, these log-probabilities are compared with the true labels using a loss function, calculating the difference and updating the model parameters accordingly.

[0190] During inference, the model applies the same process to new CSI data, selecting the class with the highest confidence score as the prediction. Specifically, when the user performs the "sit down" action, the corresponding node's activation value is significantly higher than that of other nodes. Because the confidence score range is unconstrained, direct comparison can lead to misjudgment due to scale differences. Therefore, the Softmax function is introduced for probability normalization. This converts the raw scores into a probability distribution such that the sum of all class probabilities is 1. The class with the highest probability is the prediction.

[0191] This paper designs a set of virtual-reality mapping intelligent home appliance control simulation system. By reproducing the real building space in a 1:1 digital twin environment, the posture recognition results in the physical space are mapped to the control logic of the virtual device in real time, realizing a human body posture-based intelligent home appliance control method. The system consists of three core modules: signal acquisition module, posture recognition module and instruction mapping module. Figure 6 shown.

[0192] The signal acquisition module consists of a commercial Wi-Fi transmitter and an Intel 5300 network card as a receiver, forming a sensing area. This sensing area acquires data by monitoring the dynamic fluctuations of the Wi-Fi signal generated by users in real time. The CSI Tool extracts CSI data from this data for subsequent gesture recognition analysis. The signal acquisition module ensures the system's real-time and accurate perception of environmental changes, providing the data required for the entire gesture recognition-based contactless intelligent building control system.

[0193] The posture recognition module analyzes collected CSI data and identifies the specific posture of the human body. After the signal acquisition module provides CSI data, the posture recognition module uses a posture recognition model that integrates multimodal features to process and classify the data, providing a basis for control instructions for smart home devices.

[0194] The command mapping module is responsible for converting the classification results of the gesture recognition module into specific device control commands. Through this module, the system can automatically adjust the electrical devices in the smart building, such as lights and curtains, according to the user's gestures.

[0195] The signal preprocessing method and the gesture recognition method have been introduced in detail in the first embodiment. The second embodiment mainly introduces the design and implementation of the instruction mapping method.

[0196] (1) Posture-command mapping design

[0197] The core challenge in designing gesture-command mapping for intelligent building control systems lies in efficiently and reliably translating human movements into device control commands, while ensuring that gesture selections have strong semantic connections to device functions, allowing users to naturally control them based on their daily experience. In this paper, gesture-command mapping adheres to the principle of minimal cognitive load, making control logic intuitive and establishing a deterministic correspondence between action IDs and device operations.

[0198] To ensure that the system can still quickly and accurately execute user intent in a multi-gesture input environment, we assign a unique action ID to each action and establish a one-to-one deterministic correspondence with the specific device operation, as shown in Table 1.

[0199] Table 1 Posture-command mapping table

[0200]

[0201] When a user makes a gesture, the system quickly retrieves and triggers the corresponding command based on the action ID, avoiding ambiguity and misoperation. The natural connection between action and function is a core design principle. By analyzing the semantic connection between daily behavior and device operation, gestures that closely match functional logic are selected (such as a horizontal wave simulating the opening and closing of curtains), allowing users to operate intuitively without learning.

[0202] To further ensure system reliability and safety in complex scenarios, emergency commands (such as clapping or waving to brake) are given the highest priority, interrupting all lower-level operations and restoring the device to a safe state. When the user manually initiates control signals through traditional buttons or touchscreens, the system temporarily blocks gesture input to prevent conflicts caused by unintended movements. Regular gesture commands are only executed in the pre-set priority order without manual input or emergency interruption.

[0203] (2) Equipment status feedback

[0204] The device status feedback module establishes a real-time data link with the physical device via the IoT communication protocol. When the gesture recognition module outputs a valid action type, the system initiates an asynchronous status query request to the target device (e.g., a master lighting fixture). The device response message contains its current on / off state and a health status code corresponding to normal, warning, or fault, and is returned to the control system via a lightweight data protocol. The status data is cached in an in-memory database and timestamped to verify timeliness. If the device does not respond within a preset timeout threshold of 500 milliseconds, the system initiates an auxiliary polling mechanism, resending the query request at fixed intervals of 200 milliseconds until the maximum number of retries is reached, or a valid status is obtained. After consecutive failures, the device is marked as offline, and the generation of related control instructions is suspended to prevent the accumulation of invalid operations. This process ensures the reliability of status feedback through active push and passive polling while maintaining low communication overhead.

[0205] (3) Instruction generation

[0206] The control command generation module generates operation commands based on a static mapping table and the device's real-time status. Upon receiving gesture recognition results, the system queries predefined command mapping rules to determine the target device and basic operation type. The system then reads the device's current on / off state from the status cache: if the device is off, an "on" command is generated; if the device is on, an "off" command is generated. The generated command is encapsulated into a standard control message containing a unique device identifier, operation type, request ID, and timestamp, for example, {"device_id":"main_light","operation":"turn_on","req_id":"a1b2c3","timestamp":1625097600}. The message uses a CRC-16 checksum to ensure transmission integrity and inserts a unique request identifier to support subsequent execution tracking. If the target device is marked as offline or faulty, the command generation process automatically terminates and an exception log is recorded.

[0207] The device status feedback module and the instruction generation module achieve loosely coupled interaction through a shared state cache. Status updates trigger cache refreshes, ensuring that instruction generation is always based on the latest device status. Control instructions are asynchronously sent to the target device through the message queue RabbitMQ, and the system synchronously listens to the confirmation response topic (such as $ACK / main_light). If the device returns a successful response within the preset time, the instruction is marked as executed; if it times out or returns an error code, it is marked as failed and recorded in the audit log. The complete instruction generation process is as follows: Figure 7 As shown, this design separates state feedback from command generation, ensuring real-time system performance while avoiding latency jitter caused by resource competition. The entire process strictly adheres to the unidirectional data flow of "trigger-query-generate-execute," forming a highly deterministic, low-complexity control chain suitable for standardized scenarios requiring strict mitigation of environmental interference and dynamic policies.

[0208] Building and maintaining smart home control systems in the physical world requires significant investment in hardware procurement, installation, commissioning, and ongoing maintenance. Furthermore, the performance of physical devices is limited by hardware, making it difficult to meet the demands for flexible adjustment and expansion. Therefore, a digital twin platform was chosen to provide a contactless control method. This platform accurately replicates various scenarios and control tasks in the physical world without relying on physical devices, enabling more efficient and flexible system simulation and optimization.

[0209] This paper uses the Unity3D engine as the primary digital twin simulation platform, building a virtual experimental platform through the Unity3D engine to simulate devices and action responses in physical space. As the primary simulation platform, Unity3D's powerful graphics rendering and visualization capabilities give it significant advantages in three-dimensional modeling and virtual environment creation. In addition, the rich scripting API provided by Unity and its support for integration with third-party programs enable a high degree of coupling between control commands and device model actions.

[0210] This invention utilizes an innovative virtual-reality mapping mechanism to connect real-world motion recognition with control systems in a virtual environment. Through precise gesture recognition technology, when a user performs actions in the physical world, such as "waving to turn on a light," the system instantly captures these movements and generates corresponding control commands. These control signals are transmitted in real time to the digital twin platform via the Internet of Things (IoT) protocol, driving the virtual device model to respond in the same way as the actual device, thereby fully simulating the operating logic and scenario reactions of the smart home system.

[0211] Experimental scenario such as Figure 8 As shown, Figure 8 a) and b) in the figure correspond to the physical experimental scene and the virtual experimental scene, respectively. When the user waves one hand in the physical environment, the main light in the virtual scene will light up immediately, accurately responding to the user's action. Furthermore, when the user performs a series of continuous actions, including "standing up - walking - waving", the recognition model accurately identifies the target posture, and the main lighting in the virtual environment goes out. This process demonstrates the efficiency of the digital twin platform in capturing and analyzing user actions, as well as the real-time response of the virtual environment to user behavior. Through this virtual-reality interaction, the system can seamlessly simulate and execute smart home control tasks, providing an immersive operating experience.

[0212] In summary, the present invention utilizes a bidirectional LSTM network combined with a main path-residual path dual-branch structure to capture global patterns and local mutation features, thereby achieving precise positioning of action boundaries. High-frequency transient features in the signal are extracted through a temporal difference operator, and a gated fusion mechanism is used to dynamically balance global temporal coherence and local mutation sensitivity, thereby effectively suppressing the interference of transitional actions. The present invention designs a multimodal feature extraction model, which can extract features for posture recognition from three aspects: spatial correlation between subcarriers, long-range action continuity, and temporal dynamics, thereby reducing the problem of decreased posture recognition accuracy due to insufficient feature extraction. A dynamic Dropout mechanism based on feature energy norm and a domain adaptive adversarial training framework are designed to balance the contribution of multimodal features, eliminate the influence of different scenarios on feature distribution, and improve the generalization ability of the model. A Transformer-based feature fusion method is designed, which can project heterogeneous features into a unified semantic space to eliminate the modal gap, and then dynamically weight them while retaining the original features using a multi-head attention mechanism to improve the representation ability of the features, thereby increasing the accuracy of posture recognition.

[0213] Example 2

[0214] The present invention further provides a contactless smart home appliance control method based on gesture recognition, which implements the contactless smart home appliance control system based on gesture recognition described in the first embodiment, including:

[0215] S1. Collect and pre-process the continuous CSI signal data of the indoor WiFi where the smart home appliance to be controlled is located;

[0216] S2. Based on the preprocessed continuous CSI signal data, a multimodal feature extraction model is constructed to extract multimodal features;

[0217] S3. Fusing the multimodal features based on Transformer to obtain fused features;

[0218] S4. Locating the target action boundary for controlling the smart home appliance in the operator's continuous action flow based on the fusion features;

[0219] S5. Identify and classify the target action based on the located target action boundary;

[0220] S6. Based on classified target actions, contactless smart home appliance control is achieved.

[0221] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A contactless intelligent home appliance control system based on gesture recognition, characterized in that: The system includes: an acquisition module, a feature extraction module, a feature fusion module, a boundary positioning module, an identification and classification module, and a control module; The acquisition module is used to collect and pre-process the continuous CSI signal data of the indoor WiFi where the smart home appliance to be controlled is located; The feature extraction module is used to construct a multimodal feature extraction model based on the preprocessed continuous CSI signal data to extract multimodal features; The feature fusion module is used to fuse the multimodal features based on Transformer to obtain fused features; The boundary positioning module is used to locate the target action boundary of the operator's continuous action flow for controlling the smart home appliance based on the fusion feature; The recognition and classification module is used to recognize and classify the target action based on the located target action boundary; The control module is used to realize contactless control of smart home appliances based on classified target actions.

2. The contactless intelligent home appliance control system based on gesture recognition according to claim 1 is characterized in that: In the feature extraction module, a multimodal feature extraction model is constructed based on the preprocessed continuous CSI signal data to extract multimodal features, including: The spatial features in the multimodal features are extracted by CNN: in, Represents the spatial feature, W s represents the spatial weight matrix, and H represents the input features; The long time series features in the multimodal features are extracted through the bidirectional LSTM structure: Among them, H lstm Represents long time series features, is the forward hidden state of LSTM at time step t, is the backward hidden state of LSTM at time step t; The multi-scale dynamic features in the multi-modal features are extracted by TCN: A tcn =ReLU(Y (l) +F attn (H)) Among them, H tcn Represents multi-scale dynamic features, ReLU represents the activation function, Y represents dilated convolution, l represents the number of convolution layers, and F attn (·) represents the attention function.

3. The contactless intelligent home appliance control system based on gesture recognition according to claim 2 is characterized in that: In the feature extraction module, a multimodal feature extraction model is constructed based on the preprocessed continuous CSI signal data to extract multimodal features, and further includes: performing adaptive model collaborative training on the multimodal feature extraction model; Based on the dynamic Dropout mechanism of feature energy norm, the multimodal feature extraction model is dynamically regularized and modal balanced: Among them, E (m) is the information intensity of each modal output feature, H (m) is the characteristic energy value of mode m, ||H (m) || F is the Frobenius norm, is the value of the i-th sample and j-th feature dimension in the feature matrix of modality m, γ (m) is the dynamic discard rate, α is the baseline discard rate, ∑ n E (n) is the sum of all modal energy values, n is the number of modes; Based on the gradient reversal layer GRL and the domain discriminator E, the multimodal feature extraction model is subjected to domain adaptive adversarial training: Among them, θ f is the feature extraction parameter, θ d are the parameters of the domain discriminator, is the classification loss, is the domain discrimination loss, δ is the trade-off factor, and GRL(f(x)) represents the application of the gradient reversal layer to the feature f(x).

4. The contactless intelligent home appliance control system based on gesture recognition according to claim 3 is characterized in that: In the feature fusion module, the multimodal features are fused based on Transformer to obtain fused features, including: using a dual-headed attention mechanism to guide the fusion of the multimodal features to obtain fused features: The head 1 subspace in the dual-head attention mechanism is used to obtain the dependency relationship between the spatial features and the long-term temporal features, and generate the head 1 attention weight matrix by calculating the dot product similarity between the query vector projected by the CNN and the key vector projected by the LSTM; The head 2 subspace in the dual-head attention mechanism is used to analyze the relationship between the multi-scale dynamic features and the global context, identify the dependency between the action mutation moment and the long-term action context by interacting the query vector projected by the TCN and the global key vector, and generate the head 2 attention weight matrix; Based on the first attention weight matrix and the second attention weight matrix, obtain the fusion feature: F fusion =LayerNorm(F+Dropout(W O ·Concat(Attention1,Attention2))) Among them, F fusion represents the fusion feature, F represents the concatenation of multimodal features, W O represents the learnable matrix, Attention1 and Attention2 represent the head 1 attention weight matrix and head 2 attention weight matrix respectively.

5. The contactless intelligent home appliance control system based on gesture recognition according to claim 1, characterized in that: In the boundary positioning module, based on the fusion feature, the boundary of the operator's target action is positioned, including: based on a main path-residual path double branch structure, the boundary of the operator's target action is positioned; The main path uses a 3×1 convolution kernel to capture global action patterns: h main =LeakyReLU(BatchNorm(Conv1D 3×1 (h lstm ))) Among them, h main Represents the main path output feature, LeakyReLU represents the activation function, h lstm Represents the output features of bidirectional LSTM; The residual path cascades 5×1 and 3×1 convolution kernels to model short-term mutations and local detail features respectively: h res =LeakyReLU(BatchNorm(Conv1D 5×1 (Conv1D 3×1 (h lstm )))) Among them, h res Represents the residual path output feature; The interaction of the dual-path outputs is achieved through element-wise product, and finally normalized along the time dimension through the Softmax function: in, is the attention weight; The weighted time series features are: Among them, h focused is the weighted time series feature, and T' is the total number of time steps.

6. The contactless intelligent home appliance control system based on gesture recognition according to claim 5, characterized in that: In the boundary positioning module, based on the fusion feature, the target action boundary of the operator is positioned, and further includes: combining the gradient feature with the main path output feature h through a gated fusion mechanism. main To perform the fusion: g=σ(W g [h main ;h grad ]+b g ) Among them, g represents the gate value, σ is the sigmoid function, W g is the learnable weight matrix, b g is the bias term, h grad Represents the gradient feature.

7. The contactless intelligent home appliance control system based on gesture recognition according to claim 1, characterized in that: In the recognition and classification module, the target action is recognized and classified based on the located target action boundary, including: inputting the fusion feature into the fully connected layer, and mapping it to the specific posture category space through the classification decision layer to realize the recognition and classification of the target action.

8. A method for controlling a contactless smart home appliance based on gesture recognition, implemented by using the contactless smart home appliance control system based on gesture recognition according to any one of claims 1 to 7, characterized in that: The method comprises: S1. Collect and pre-process the continuous CSI signal data of the indoor WiFi where the smart home appliance to be controlled is located; S2. Based on the preprocessed continuous CSI signal data, a multimodal feature extraction model is constructed to extract multimodal features; S3. Fusing the multimodal features based on Transformer to obtain fused features; S4. Locating the target action boundary for controlling the smart home appliance in the operator's continuous action flow based on the fusion features; S5. Identify and classify the target action based on the located target action boundary; S6. Based on classified target actions, contactless smart home appliance control is achieved.

Images