A gesture recognition method based on WIFI-CSI multi-scale feature fusion
By using a gesture recognition model that integrates multi-scale features, the shortcomings of the existing WiFi-CSI single neural network in complex scenarios are addressed, achieving high-precision and lightweight gesture recognition, and improving recognition accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2026-03-31
AI Technical Summary
Existing single neural network architectures based on WiFi-CSI cannot meet the actual needs of gesture recognition tasks in resource-constrained and complex scenarios. They cannot effectively capture multi-scale features and have a large number of parameters.
A gesture recognition model employing multi-scale feature fusion is proposed, comprising a feature extraction module and a gesture recognition module. It utilizes multi-scale feature fusion modules 1 and 2, a depth-pooling module, and a gated recurrent unit module, combined with lightweight CBAM and depthwise separable convolution, and optimizes model parameters through gradient descent backpropagation.
It improves the accuracy and robustness of gesture recognition, reduces computational complexity and the number of parameters, and enhances the model's generalization ability and training efficiency.
Smart Images
Figure CN119939406B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of gesture recognition technology in the field of wireless sensing, and specifically relates to a gesture recognition method based on WIFI-CSI multi-scale feature fusion. Background Technology
[0002] In recent years, with the rapid development of integrated communication and sensing, more and more researchers have devoted themselves to the field of wireless sensing. Gesture recognition is a hot topic in the field of wireless sensing, which is crucial to the development of new human-computer interaction modes and brings tremendous changes to human life. WiFi sensing technology is an emerging technology. In-depth research on gesture recognition using WiFi sensing technology can detect more subtle changes in human activity through the multipath effect of signal propagation. These changes in activity are manifested as changes in channel state information (CSI).
[0003] However, most deep learning methods utilizing WiFi-CSI signals employ a single-architecture neural network. Such single-architecture networks can only capture a single feature and have a large number of parameters, which cannot meet the practical application requirements for gesture recognition tasks in resource-constrained and complex scenarios. Therefore, there is an urgent need for a lightweight gesture recognition method based on multi-scale feature fusion of WiFi-CSI. Summary of the Invention
[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a gesture recognition model that can identify the types of gesture activities and has the advantages of high accuracy, lightweight design, and strong robustness.
[0005] Another objective of this invention is to provide a training method for a gesture recognition model.
[0006] Another objective of this invention is to provide a gesture recognition method based on WIFI-CSI multi-scale feature fusion.
[0007] The present invention is achieved through the following technical solution.
[0008] A gesture recognition model includes a feature extraction module and a gesture recognition module. The feature extraction module comprises a multi-scale feature fusion module 1, a multi-scale feature fusion module 2, a depth-pooling module, and a gated recurrent unit module. The input to the multi-scale feature fusion module 1 is a sample x. input1 The output is a feature vector X1. The multi-scale feature fusion module 1 is used to extract each sample x. input1The multi-scale feature fusion module 2 takes feature vector X1 as input and outputs feature vector X2 as output. The multi-scale feature fusion module 2 is used to extract features from feature vector X1 and perform multi-scale feature fusion. The deep pooling module takes feature vector X2 as input and outputs feature vector X3 as output. The deep pooling module is used to extract features from feature vector X2 and perform dimensionality reduction. The gated recurrent unit module takes feature vector X3 as input and outputs feature vector H as output. The gated recurrent unit module is used to extract the temporal features of feature vector X3 to obtain feature vector H.
[0009] The input to the gesture recognition module is the feature vector H, and the output is the predicted gesture activity category.
[0010] Each sample is three-dimensional data consisting of the velocity spectrum in the human body coordinate system along the time frame dimension.
[0011] In the above technical solution, the multi-scale feature fusion module 1 is a fusion structure, which includes an upper branch structure, a lower branch structure, and a summing module. The upper branch structure includes a dilated convolution module, a first batch normalization layer, a PReLU activation function, and a first CBAM connected in sequence. The lower branch structure includes a first depthwise separable convolution module, a second batch normalization layer, a ReLU activation function, and a second CBAM connected in sequence. The summing module is used to sum the outputs of the first CBAM and the second CBAM element by element to obtain the feature vector.
[0012] In the above technical solution, the multi-scale feature fusion module 2 includes: a filling module and the fusion structure. The filling module is used to perform a filling operation with padding=1. The filling module is connected to the dilated convolution module in the fusion structure. The filling module and the first depth-separable convolution module of the multi-scale feature fusion module 2 simultaneously receive the feature vector X1.
[0013] In the above technical solution, the depth-pooling module includes: a second depthwise separable convolutional module, a padding module, a batch normalization layer, a ReLU activation function, a max pooling layer and a first fully connected layer connected in sequence. The padding module is used to perform padding=1 padding operation. The feature vector X2 is used as the input of the second depthwise separable convolutional module and the first fully connected layer outputs the feature vector X3.
[0014] In the above technical solution, the second depth-separable convolution module includes: two-dimensional depth convolution and two-dimensional point convolution.
[0015] In the above technical solution, the first fully connected layer includes: a first linear transformation layer, a ReLU function and a first Dropout layer, a second linear transformation layer and a ReLU function connected in sequence.
[0016] In the above technical solution, the gesture recognition module includes: a second Dropout layer, a second fully connected layer, and a normalized Softmax function connected in sequence.
[0017] A method for training a gesture recognition model includes: treating each sample in the training set as sample x. input1 The labels are input into the gesture recognition model for training, and the predicted gesture activity category for each sample is obtained. The gesture recognition model is equipped with a loss function calculation module, which uses the cross-entropy loss function to calculate the loss value. The gesture recognition model uses the gradient descent backpropagation method to optimize its parameters until the maximum number of iterations is reached, and the training ends, resulting in a trained gesture recognition model.
[0018] A gesture recognition method based on WIFI-CSI multi-scale feature fusion includes: inputting samples from the test set into a trained gesture recognition model to identify gesture activity categories, and obtaining the predicted gesture activity category for each sample.
[0019] Compared with the prior art, the beneficial effects of the present invention are:
[0020] 1. The gesture recognition model of the present invention introduces a lightweight CBAM, which enables the gesture recognition model to extract key information of channels and space, reduce unnecessary features, and enhance the representation ability of the gesture recognition model of the present invention.
[0021] 2. The gesture recognition model of the present invention introduces an element-by-element (corresponding elements) addition method to fuse features, thereby increasing the robustness of the gesture recognition model and improving its generalization ability.
[0022] 3. The gesture recognition model of the present invention introduces a parallel network structure to capture features at different scales. As can be seen from the floating-point operations (Flops) and the number of parameters (Params), the training efficiency of the gesture recognition model of the present invention is improved.
[0023] 4. The gesture recognition method of the present invention has a high accuracy rate in recognizing complex gesture activities, and enhances the robustness of the gesture recognition model while maintaining a lightweight design. Attached Figure Description
[0024] Figure 1 A framework diagram of the gesture recognition model for the invention;
[0025] Figure 2 A schematic diagram of the fusion structure in the gesture recognition model of the invention;
[0026] Figure 3 This is a graph showing the prediction results of the gesture recognition model of the present invention on the test set. Detailed Implementation
[0027] The following describes in detail a gesture recognition method based on WIFI-CSI multi-scale feature fusion according to the present invention, with reference to the accompanying drawings and embodiments.
[0028] Example 1
[0029] like Figure 1 As shown, a gesture recognition model includes a feature extraction module and a gesture recognition module. The feature extraction module includes a multi-scale feature fusion module 1 (D-CBAM-1), a multi-scale feature fusion module 2 (D-CBAM-2), a depth-pooling module, and a gated recurrent unit module (GRU, Shuai, G., Yuefei, H., Shuo, Z., Jingcheng, H., Guangqian, W., Meixin, Z., & Qingsheng, L. (2020) Short-term runoff prediction with GRU and LSTM networks without requiring time step optimization during sample generation, Journal of Hydrology, 589.). The input of the multi-scale feature fusion module 1 (D-CBAM-1) is a sample x. input1 The output is a feature vector X1. The multi-scale feature fusion module 1 is used to extract each sample x. input1 The system extracts features from feature vector X1 and performs multi-scale feature fusion on it. The multi-scale feature fusion module 2 (D-CBAM-2) takes feature vector X1 as input and outputs feature vector X2 as output. It extracts features from feature vector X1 and performs multi-scale feature fusion on it. The deep pooling module (DS-Conv Block) takes feature vector X2 as input and outputs feature vector X3 as output. It extracts features from feature vector X2 and performs dimensionality reduction. The gated recurrent unit module takes feature vector X3 as input and outputs feature vector H as output. It extracts the temporal features of feature vector X3 to obtain feature vector H.
[0030] The input to the gesture recognition module is the feature vector H, and the output is the predicted gesture activity category.
[0031] Each sample is a three-dimensional data consisting of a velocity spectrum (BVP) in the human body coordinate system (which is two-dimensional data) in the time frame dimension.
[0032] Example 2
[0033] A gesture recognition model, based on embodiment 1, wherein the multi-scale feature fusion module 1 is as follows: Figure 2 The fusion structure shown includes an upper branch structure, a lower branch structure, and a summing module. The upper branch structure includes: a dilated convolutional module (Yali, P., Lu, Z., Shigang, L., Xiaojun, W., Yu, Z., & Xili, W. (2019) Dilated Residual Networks with Symmetric Skip Connection for Image Denoising., Neurocomputing, 345: 67-76.) connected in sequence, a first batch normalization layer, a PReLU activation function, and a first CBAM (Sanghyun, W., Jongchan, P., Joon-Young, L., & In So, K. (2018) CBAM: Convolutional Block Attention Module., European Conference on Computer Science). Vision,abs / 1807.06521:3-19.), the lower branch structure includes: a first depthwise separable convolutional module (M.Humayun,K.,Md.Ali,H.,&Wonjae,S.(2022)CSI-DeepNet:A Lightweight DeepConvolutional Neural Network Based Hand Gesture Recognition System Using Wi-Fi CSI Signal,IEEE Access,10:114787-114801.), a second batch normalization layer, a ReLU activation function and a second CBAM, the summing module is used to sum the output of the first CBAM and the output of the second CBAM element by element to obtain the feature vector.
[0034] The dilated convolutional module and the first depthwise separable convolutional module of the multi-scale feature fusion module 1 simultaneously receive sample x. input1 Sample x input1 Simultaneously, the input is fed into the dilated convolution module and the first depthwise separable convolution module of the multi-scale feature fusion module 1, and the first CBAM outputs the feature vector x. d1 The second CBAM outputs the feature vector x. ds1 The summation module adds the feature vector x d1 and eigenvector x ds1Element-wise addition is performed to obtain the feature vector X1; the dilated convolution module of the multi-scale feature fusion module 1 has a kernel size of 6×6, a stride size of 2, and a number of kernels of 8; the first depth-separable convolution module of the multi-scale feature fusion module 1 includes: a two-dimensional depth convolution module and a two-dimensional point convolution module. The kernel size of the two-dimensional depth convolution module is 6×6, the stride size is 2, and the number of kernels of 8. The kernel size of the two-dimensional point convolution module is 1×1, the stride size is 1, and the number of kernels of 8.
[0035] Multi-scale feature fusion module 2 includes: a filling module and, for example, a filling module and, Figure 2 The fusion structure shown includes a padding module for performing a padding operation with padding=1. This padding module is connected to the dilated convolutional module in the fusion structure. Both the padding module and the first depthwise separable convolutional module of the multi-scale feature fusion module 2 receive the feature vector X1. The feature vector X1 serves as the input to both the padding module and the first depthwise separable convolutional module, and the first CBAM of the multi-scale feature fusion module 2 outputs the feature vector x. d2 The second CBAM output feature vector x from the multi-scale feature fusion module 2 ds2 The summation module will add the feature vector x d2 and eigenvector x ds2 Element-wise addition is performed to obtain the feature vector X2. The dilated convolution module of the multi-scale feature fusion module 2 has a kernel size of 3×3, a stride size of 1, dilation = 2, and a number of kernels of 16. The first depthwise separable convolution module of the multi-scale feature fusion module 2 includes: two-dimensional depthwise convolution and two-dimensional pointwise convolution. The kernel size of the two-dimensional depthwise convolution is 3×3, the stride size is 1, and the number of kernels is 16. The kernel size of the two-dimensional pointwise convolution is 1×1, the stride size is 1, and the number of kernels is 16.
[0036] The deep pooling module comprises: a second depthwise separable convolutional module, a padding module, a batch normalization layer, a ReLU activation function, a max pooling layer, and a first fully connected layer (FC1), connected in sequence. The padding module performs a padding operation with padding = 1. The feature vector X2 serves as the input to the second depthwise separable convolutional module, and the first fully connected layer (FC1) outputs the feature vector X3. The second depthwise separable convolutional module includes: a 2D depthwise convolution and a 2D pointwise convolution. The 2D depthwise convolution has 16 kernels, a kernel size of 3×3, and a stride of 1. The 2D pointwise convolution has 16 kernels, a kernel size of 1×1, and a stride of 1. The first fully connected layer (FC1) comprises: a first linear transformation layer, a ReLU function and a first Dropout layer, a second linear transformation layer and a ReLU function, connected in sequence. The dropout rate of the first Dropout layer is 0.5.
[0037] The gesture recognition module includes: a second Dropout layer, a second fully connected layer (FC2), and a normalized Softmax function connected in sequence (the normalized Softmax function outputs the probability of each gesture activity category, and the gesture activity category with the highest probability is taken as the predicted gesture activity category). The second fully connected layer (FC2) maps the high-dimensional feature vector to the category space, avoiding redundant calculations and unnecessary computational overhead.
[0038] Example 3
[0039] Widar3.0 data set (from: Yue, Z., Yi, Z., Kun, Q., Guidong, Z., Yunhao, L., Chenshu, W., & Zheng, Y. (2021) Widar3.0: Zero-Effort Cross-Domain Gesture Recognition with Wi-Fi, IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:8671-8688.) is a WiFi-CSI-based gesture activity dataset, consisting of 43,000 original samples divided into 22 gesture activity categories. Data from 7 gesture activity categories in the Widar dataset, totaling 27,600 samples, is selected as dataset D. The 7 gesture activity categories are "push and pull", "sweep", "clap", "swipe", "draw N", "draw Z" and "draw O". Each sample is a three-dimensional data composed of the velocity spectrum (BVP) in human coordinate system (which is two-dimensional data) in the time frame dimension. The dimensions of each sample in dataset D are 22×20×20, where 22 represents the time frame of the sample, and the length and width of the velocity spectrum (BVP) in human coordinate system are 20. Each sample contains a label, which is the gesture activity category of the sample (ground truth).
[0040] The samples in dataset D are divided into training and test sets in an 8:2 ratio, and the gesture recognition module makes predictions in 7 gesture activity categories.
[0041] Example 4
[0042] A method for training a gesture recognition model, comprising:
[0043] Each sample in the training set of Example 3 is taken as sample x. input1The labels are input into the gesture recognition model of Example 2 for training to obtain the predicted gesture activity category for each sample. A batch of 16 samples is used for training. The gesture recognition model includes a loss function calculation module, which uses the cross-entropy loss function (Bing, L., Wei, C., Wei, W., Le, Z., Zhenghua, C., & Min, W. (2021) Two-Stream Convolution Augmented Transformer for Human Activity Recognition, AAAI Conference on Artificial Intelligence, 35.1:286-293.) to calculate the loss value. The gesture recognition model uses gradient descent backpropagation (Lin, W., Yi, Z., & Tao, C. (2015) Back Propagation Neural Network with Adaptive Differential Evolution Algorithm for Time Series Forecasting, Expert Systems with (Applications, 42.2:855-863.) Optimize its parameters until the maximum number of iterations epcohs = 200 is reached, then end the training and obtain the trained gesture recognition model.
[0044] Example 5
[0045] A gesture recognition method based on WIFI-CSI multi-scale feature fusion includes: inputting samples from the test set into the gesture recognition model trained in Example 4 to identify the gesture activity category, and obtaining the predicted gesture activity category for each sample.
[0046] Example 6
[0047] A gesture recognition method based on WIFI-CSI multi-scale feature fusion is basically the same as that in Example 5, except that: in the gesture recognition model used in this example, neither the multi-scale feature fusion module 1 nor the multi-scale feature fusion module 2 has an upper branch structure or an addition module, and the multi-scale feature fusion module 2 does not have a filling module.
[0048] Example 7
[0049] A gesture recognition method based on WIFI-CSI multi-scale feature fusion is basically the same as that in Example 5, except that the gesture recognition model used in this example does not have a lower branch structure or an addition module in the multi-scale feature fusion module 1 and the multi-scale feature fusion module 2.
[0050] Example 8
[0051] A gesture recognition method based on WIFI-CSI multi-scale feature fusion is basically the same as that in Example 5, except that the gesture recognition model used in this example does not have a first CBAM and a second CBAM in the multi-scale feature fusion module 1 and the multi-scale feature fusion module 2.
[0052] The recognition accuracy of the gesture recognition methods in Examples 5-8 is compared, as shown in Table 1. Table 1 shows that the gesture recognition method of the present invention has the highest accuracy in recognizing gesture activities on the test set, at 84.30 ± 0.31%. Therefore, the gesture recognition method of the present invention has the best gesture activity recognition effect. Figure 3 As shown, the gesture recognition method of the present invention has an accuracy rate of over 80% for each gesture activity category in the test set.
[0053] Table 1
[0054] Example Accuracy (%) Example 6 83.44±1.13 Example 7 71.19±0.60 Example 8 84.22±0.68 Example 5 84.30±0.31
[0055] Example 9
[0056] We used MLP, CNN-5, CNN+GRU, LSTM, GRU, ViT, and ResNet18 (Jianfei, Y., Xinyan, C., Han, Z., Chris Xiaoxuan, L., Dazhuo, W., Sumei, S., & Lihua, X. (2023) SenseFi: A Library and Benchmark on Deep-Learning-Empowered WiFi Human Sensing, PATTERNS, 4.3:100703-100703.) to identify the gesture activity category of the test set and obtain the gesture activity category of each sample.
[0057] Using the accuracy (Acc) to evaluate predictive ability, the floating-point operations (Flops) to evaluate computational complexity, and the number of parameters (Params) to measure GPU memory requirements, Example 5 is compared with MLP, CNN-5, CNN+GRU, LSTM, GRU, ViT, and ResNet18. The comprehensive evaluation results are shown in Table 2.
[0058] Table 2
[0059]
[0060]
[0061] As shown in Table 2, the gesture recognition method of the present invention has an accuracy of 84.30 ± 0.31% when the number of parameters is small and the computational complexity is low.
[0062] The present invention has been described above by way of example. It should be noted that any simple modifications, alterations or other equivalent substitutions that can be made by those skilled in the art without creative effort without departing from the core of the present invention fall within the protection scope of the present invention.
Claims
1. A gesture recognition model, characterized in that, The input of the gesture recognition module is a feature vector H, and the output is a predicted gesture activity category; The feature extraction module and the gesture recognition module, wherein the feature extraction module comprises: a multi-scale feature fusion module 1, a multi-scale feature fusion module 2, a depth-pooling module and a gated recurrent unit module, the input of the multi-scale feature fusion module 1 is the sample , and the output is a feature vector , the multi-scale feature fusion module 1 is used for extracting features in each sample and performing multi-scale feature fusion; the input of the multi-scale feature fusion module 2 is the feature vector , and the output is a feature vector , the multi-scale feature fusion module 2 is used for extracting features in the feature vector and performing multi-scale feature fusion; the input of the depth-pooling module is the feature vector , and the output is a feature vector , the depth-pooling module is used for extracting features of the feature vector and performing dimension reduction; the input of the gated recurrent unit module is the feature vector , and the output is a feature vector H, the gated recurrent unit module is used for extracting time features of the feature vector , and obtaining the feature vector H; Each sample is three-dimensional data composed of a velocity spectrum in a human body coordinate system in a time frame dimension; The multiscale feature fusion module 1 is a fusion structure; The fusion structure includes an upper branch structure, a lower branch structure, and a summation module, the upper branch structure includes a hollow convolution module, a first batch normalization layer, a PReLU activation function, and a first CBAM connected in sequence, the lower branch structure includes a first depth separable convolution module, a second batch normalization layer, a ReLU activation function, and a second CBAM connected in sequence, and the summation module is used for element-wise summation of the output of the first CBAM and the output of the second CBAM to obtain a feature vector; The second depth separable convolution module includes a two-dimensional depth convolution and a two-dimensional point convolution. The multi-scale feature fusion module 2 comprises a padding module and the fusion structure, the padding module is used for padding operation of padding=1, the padding module is connected with the hollow convolution module in the fusion structure in the multi-scale feature fusion module 2, and the padding module and the first depth separable convolution module in the fusion structure in the multi-scale feature fusion module 2 simultaneously receive the feature vector .
2. The gesture recognition model of claim 1, wherein, The depth-pooling module comprises, in sequence, a second depth separable convolution module, a padding module, a batch normalization layer, a ReLU activation function, a maximum pooling layer and a first full connection layer. The padding module is configured to perform a padding operation with padding=1. The feature vector The feature vector output by the first full connection layer is input into the second depth separable convolution module. .
3. The gesture recognition model of claim 2, wherein, The first full connection layer includes a first linear transformation layer, a ReLU function, and a first Dropout layer connected in sequence, a second linear transformation layer, and a ReLU function.
4. The gesture recognition model of claim 2, wherein, The gesture recognition module includes a second Dropout layer, a second full connection layer, and a normalized Softmax function connected in sequence.
5. The gesture recognition model of claim 1, wherein, The input of the gesture recognition module is a feature vector H, and the output is a predicted gesture activity category; 6. A method for training a gesture recognition model, the method comprising: Each sample is three-dimensional data composed of a velocity spectrum in a human body coordinate system in a time frame dimension; each sample in the training set as a sample and the label into the gesture recognition model of any one of claims 1-5 for training to obtain a predicted gesture activity class of each sample, wherein the gesture recognition model is provided with a loss function calculation module, the loss function calculation module uses a cross-entropy loss function to calculate a loss value, the gesture recognition model optimizes its parameters using a gradient descent back propagation method until a maximum number of iterations is reached to end the training and obtain a trained gesture recognition model.
7. A gesture recognition method based on WIFI-CSI multi-scale feature fusion, comprising: The multiscale feature fusion module 1 is a fusion structure; The fusion structure includes an upper branch structure, a lower branch structure, and a summation module, the upper branch structure includes a hollow convolution module, a first batch normalization layer, a PReLU activation function, and a first CBAM connected in sequence, the lower branch structure includes a first depth separable convolution module, a second batch normalization layer, a ReLU activation function, and a second CBAM connected in sequence, and the summation module is used for element-wise summation of the output of the first CBAM and the output of the second CBAM to obtain a feature vector; The second depth separable convolution module includes a two-dimensional depth convolution and a two-dimensional point convolution. The first full connection layer includes a first linear transformation layer, a ReLU function, and a first Dropout layer connected in sequence, a second linear transformation layer, and a ReLU function. The gesture recognition module includes a second Dropout layer, a second full connection layer, and a normalized Softmax function connected in sequence. The input of the gesture recognition module is a feature vector H, and the output is a predicted gesture activity category; Each sample is three-dimensional data composed of a velocity spectrum in a human body coordinate system in a time frame dimension; The multiscale feature fusion module 1 is a fusion structure; The fusion structure includes an upper branch structure, a lower branch structure, and a summation module, the upper branch structure includes a hollow convolution module, a first batch normalization layer, a PReLU activation function, and a first CBAM connected in sequence, the lower branch structure includes a first depth separable convolution module, a second batch normalization layer, a ReLU activation function