A joint deep learning method for gesture recognition and position classification

Abstract

The present invention provides a joint deep learning method for gesture recognition and position classification, comprising the following steps: S1: inputting CSI samples into a one-dimensional convolution in a multi-task shrinking residual network module for downsampling and normalization; S2: inputting the processed CSI samples into a residual structure for feature matrix addition and soft thresholding; S3: inputting the extracted common feature information into the gesture and position output layers; S4: performing gesture spatial feature extraction; S5: performing a reshape operation; S6: inputting the gesture spatial feature information into an LSTM network to extract temporal features; S7: the position output layer directly outputs the classification result; S8: fusing the gesture spatiotemporal features extracted by the feature extraction auxiliary module with their shared feature vectors to output the gesture classification result. The present invention fuses feature parameters in the output layer of the gesture recognition branch task of the feature extraction auxiliary module and the multi-task shrinking residual network module, thereby improving the evaluation index of gesture recognition and the generalization ability of its network model.

Description

Technical Field

[0001] The present invention relates to the technical field of feature extraction and recognition, and in particular to a joint deep learning method for gesture recognition and position classification. Background Art

[0002] Currently, the different subcarrier amplitudes based on Wi-Fi device channel state information (CSI) are widely used for human feature extraction tasks such as person identification, trajectory prediction, gesture recognition, and location classification. Unlike camera-based person recognition, Wi-Fi devices do not need to consider issues such as privacy, facial obstruction, insufficient lighting, and high computational complexity. As a result, Wi-Fi technology has become a hot topic in the field of human feature extraction research.

[0003] Currently, the main approaches for extracting Wi-Fi-based personal features include CNN, RNN, and GAN network structures. CNNs primarily consist of convolutional layers, activation layers, normalization layers, and pooling layers. Through multiple convolutional and pooling layers, the network extracts feature vectors, which are then input into a fully connected layer to output classification results. RNNs primarily consist of input, hidden, and output layers. They memorize previous information and use it to influence the output of subsequent nodes. GANs primarily consist of generation, discrimination, and adversarial modules. The network outputs the corresponding results through a game of learning between the generator and the discriminator.

[0004] These algorithms employ different deep learning networks to extract CSI feature information in both temporal and spatial dimensions, then output corresponding classification results at the decoding layer. However, WiFi data collection is susceptible to environmental interference, which can easily generate multipath noise, hindering feature extraction and negatively impacting classification results. Furthermore, small CSI datasets collected by WiFi are prone to overfitting due to high model complexity. Furthermore, current research on WiFi feature information extraction primarily focuses on single classification tasks: location classification and gesture recognition, with limited application of WiFi feature information extraction to multi-task networks. In real-world work and life, people often require multiple application functions to address their needs, and single-task networks cannot meet these requirements. CSI dataset labels based on multi-task networks typically include two or more labels, and the features of these labels overlap, hindering the extraction of identification features. Classification accuracy is often compromised when performing single- or multi-label classification because the CSI label features contain features from other labels. Summary of the Invention

[0005] To solve the technical problems existing in the prior art, the present invention provides a joint deep learning method for gesture recognition and position classification. By fusing feature parameters at the output layer of the gesture recognition branch task using an MFCs-LSTM feature extraction auxiliary module and a multi-task shrinkage residual network module, the evaluation index of gesture recognition is improved, and the generalization ability of its network model is enhanced.

[0006] The present invention is implemented by the following technical solution: a gesture recognition and position classification combined deep learning method, comprising the following steps:

[0007] S1: Input the CSI sample into the one-dimensional convolution of the multi-task contraction residual network module to downsample the CSI sample and then perform normalization on it;

[0008] S2: Input the processed CSI samples into several residual structures with contraction sub-networks to perform feature matrix addition and soft thresholding operations;

[0009] S3, inputting the extracted common feature information into the gesture output layer and the position output layer simultaneously;

[0010] S4, the CSI samples are input in parallel into two FCNs of different scales with SE modules in the feature extraction auxiliary module to extract gesture spatial features;

[0011] S5, reshape the output gesture spatial features;

[0012] S6. Input the gesture spatial feature information after the reshape operation into the LSTM network to extract its temporal features;

[0013] S7, the position output layer directly outputs its classification results;

[0014] S8, the gesture output layer fuses the gesture spatiotemporal features extracted by the feature extraction auxiliary module with its shared feature vector through feature enhancement, and finally outputs the enhanced gesture classification results.

[0015] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0016] 1. The present invention inserts soft thresholding into the residual module and uses the gradient descent algorithm to automatically identify and eliminate multipath noise features, thereby reducing the interference of environmental noise on personnel feature extraction. In addition, it also solves the problem of overfitting of training results caused by a small number of samples by reducing the number of network layers.

[0017] 2. The present invention integrates feature parameters through the output layer of the gesture recognition branch task of the feature extraction auxiliary module MFCs-LSTM and the multi-task shrinkage residual network module, thereby improving the evaluation index of gesture recognition and enhancing the generalization ability of its network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is the AFERSM-Net model architecture diagram of the present invention;

[0019] Figure 2 This is a schematic diagram of the location and gesture collection scenario;

[0020] Figure 3 (a) is a diagram of gesture categories and sample numbers;

[0021] Figure 3 (b) is a schematic diagram of location categories and sample numbers;

[0022] Figure 4 This is the architecture diagram of the feature extraction auxiliary module MFCs-LSTM;

[0023] Figure 5 is a flow chart of the method of the present invention;

[0024] Figure 6 (a) is a schematic diagram of the gesture confusion matrix;

[0025] Figure 6 (b) is a schematic diagram of the position confusion matrix. DETAILED DESCRIPTION

[0026] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0027] Example

[0028] like Figure 1 As shown, this embodiment provides a joint deep learning method for gesture recognition and position classification, and provides an AFERSM-Net model (Auxiliary Feature Extraction based Residual Shrinkage Multi-tasking Network) for gesture recognition and position classification of one-dimensional multivariate time series.

[0029] Specifically, in this embodiment, a CSI dataset with gesture and location tags is used, and the experimental scenario is as follows: Figure 2As shown in the figure, the CSI dataset involves 6 types of hand movements, namely, a user's hand gestures at 16 positions, including up, down, left, right, circle, and cross. At each position, each activity is repeated 15 times. After removing bad samples, a total of 1394 sample data were collected. Figure 3 (a) Figure 3 As shown in (b); each sample data contains 52 subcarrier signals, 1116 samples are used as training set data, and then 278 samples with the same conditions as the training set are uniformly selected from all samples as test set data.

[0030] Specifically, in this embodiment, the AFERSM-Net model includes: a feature extraction auxiliary module MFCs-LSTM (Multiscale Fully Convolutional SE-LSTM) and a multi-task shrinkage residual network module (ResidualShrinkage Multi-tasking Network); wherein the feature extraction auxiliary module MFCs-LSTM is used to train unique gesture weight parameters to enhance the feature information of gestures. Its network structure is as follows: Figure 4 As shown in Figure 2, the multi-task contraction residual network module implements two branch tasks by sharing weight parameters.

[0031] like Figure 5 As shown, in this embodiment, a gesture recognition and position classification combined deep learning method includes the following steps:

[0032] S1. Input the CSI sample into the Residual Shrinkage Multi-tasking Network module, perform a one-dimensional convolution of size 7 to downsample the CSI sample, and then perform normalization on it.

[0033] S2: Input the processed CSI samples into three residual modules with contraction sub-networks to perform feature matrix addition and soft thresholding operations;

[0034] S3, inputting the extracted common feature information into the gesture output layer and the position output layer simultaneously;

[0035] S4, the CSI samples are input in parallel into two FCNs of different scales with SE modules in the feature extraction auxiliary module to extract gesture spatial features;

[0036] S5, reshape the output gesture spatial features;

[0037] S6. Input the gesture spatial feature information after the reshape operation into the LSTM network to extract its temporal features;

[0038] S7, the position output layer directly outputs its classification results;

[0039] S8, the gesture output layer fuses the gesture spatiotemporal features extracted by the feature extraction auxiliary module with its shared feature vector through feature enhancement, and finally outputs the enhanced gesture classification results.

[0040] Specifically, in this embodiment, the specific process in step S2 is as follows:

[0041] S21. To address the phenomenon that the multipath noise generated by the environment causes signal attenuation on the amplitude of the received signal, a contraction sub-network is used to perform wavelet decomposition on each sub-carrier signal, transforming it into an area where near-zero numbers are unimportant. Then, a soft threshold is applied to convert the near-zero features to zero, and finally, wavelet reconstruction is performed. The soft thresholding can be expressed as:

[0042]

[0043] Among them, x is the input feature, y is the output feature, and τ is the threshold, which is a positive parameter;

[0044] S22, the soft threshold is integrated into a trainable module through several neural network layers, and is automatically learned using a gradient descent algorithm. Figure 3 As shown;

[0045] S23. The automatically determined soft threshold is inserted into the residual module of 1DResNet[1,1,1] as a nonlinear transformation layer. The soft threshold module, as a subnetwork of the residual network, absolutizes the input features and performs a global pooling operation using GAP to obtain a one-dimensional vector.

[0046] S24. Input the one-dimensional vector into the FC network, then perform BN and RELU operations, and finally input the FC network to obtain a scaling parameter. The scaling parameter is scaled to the range of (0, 1) through the Sigmoid function, which is expressed as:

[0047]

[0048] Where z is the output of the second layer FC network in the residual module, and α is the corresponding scaling parameter;

[0049] S25. Multiply the scaling parameter α by the average value of |x| to obtain a threshold value, which is expressed as:

[0050] τ=α·average|x i,j,c |

[0051] τ=α·average|xi,j,c|

[0052] Among them, τ is the threshold, i, j, and c are the indicators of the width, height, and channel of the input feature x, respectively. The threshold can be kept within a reasonable range so that the output of the soft threshold is not all zero.

[0053] Specifically, in this embodiment, the specific process of step S4 is as follows:

[0054] S41, by establishing the feature extraction auxiliary module MFCs-LSTM to extract gesture features separately, reduce the influence of position-independent features, set the number of input channels to 52, the feature extraction auxiliary module MFCs-LSTM is as follows Figure 4 As shown in the figure, the two FCNs networks of different scales both contain three convolution blocks, each of which contains a convolution layer with the number of channels of (128, 256, 128) respectively. The convolution kernel sizes of the first FCNs network are (7, 5, 3) respectively, and the convolution kernel sizes of the second FCN network are (5, 3, 1) respectively. The first two convolution blocks contain two SE modules, and the last two convolution blocks contain two average pooling layers. The FCNs network uses two FCN modules of different scales to capture the differences in the duration of each gesture, and then selectively focuses on the spatial features of the gesture and suppresses the position-independent features through the SE module. Finally, the spatial features of the gesture are input into the LSTM network to extract its temporal features.

[0055] S42. The feature extraction auxiliary module MFCs-LSTM inputs the 52 subcarriers in the CSI sample as multivariate features into two FCNs networks. After the CSI sample undergoes feature extraction, compression, extraction, and average pooling operations in three convolution blocks, it outputs two three-dimensional tensors of size [64, 128, 192]. The two three-dimensional tensors are concatenated at dim=1 to obtain a three-dimensional tensor of size [64, 256, 192]. Finally, dim=1 and dim=2 are converted through the transposition function to obtain a three-dimensional tensor of size [64, 192, 256]. The CSI sample obtains a three-dimensional output tensor after extracting the spatial dimension features through the MFCNs network. The three-dimensional output tensor is input into an LSTM network with a feature dimension of 256, a hidden layer dimension of 15, a recurrent neural network layer number of 2, and an output category number of 2. Through LSTM network training, the gesture spatial features independent of the position state are finally obtained.

[0056] Specifically, in this embodiment, the specific process of step S8 is as follows:

[0057] S81. During the CSI sample collection phase, gesture amplitude varies more complexly than position and is affected by different locations, making its features more difficult to extract and difficult for the network to learn. To address this issue, we use the feature extraction auxiliary module MFCs-LSTM to enhance the gesture branch task. The CSI samples are input into the shared layer of the multi-task contraction residual network module to obtain n multi-channel one-dimensional vectors of length 6.

[0058] S82, the one-dimensional vector undergoes a 3*1 1DCONV convolution operation and then performs BN, ReLU, and average pooling operations to output the gesture vector feature;

[0059] S83, the output gesture vector feature is converted into a two-dimensional vector with a column size of 512 through a reshape operation;

[0060] S84. The CSI samples are passed through the feature extraction auxiliary module MFCs-LSTM, which outputs n one-dimensional vectors of length 6. The obtained one-dimensional vectors are reshaped to be converted into two-dimensional vectors with a column size of 512.

[0061] S85. Concatenate the two two-dimensional vectors by column to generate a two-dimensional vector with a column size of 518. Connect the generated two-dimensional vector to the fully connected layer and output the gesture category. While paying attention to position classification, fuse the gesture feature parameters extracted by the feature extraction auxiliary module MFCs-LSTM with the gesture classification task output.

[0062] In this embodiment, the experimental result classification evaluation indicators used include: confusion matrix, accuracy, precision, recall, specificity, and F1 scores. Figure 6 (a) and Figure 6 Figure (b) shows the confusion matrices of the proposed network for gesture recognition and position classification. As can be seen, the proposed network achieved an accuracy rate of 97.84% for gesture recognition and 98.92% for position classification. Gesture recognition prediction errors were primarily concentrated in downward and rightward gestures, while position classification prediction errors were primarily concentrated in positions 7, 8, and 12. Experiments have demonstrated that the proposed network achieves excellent results in both gesture recognition and position classification.

[0063] Furthermore, the precision, recall, specificity, and F1 scores are calculated based on the confusion matrix. The results are shown in Tables 1 and 2 below:

[0064]

[0065] Table 1

[0066]

[0067] Table 2

[0068] Table 1 shows that all gesture recognition evaluation metrics achieved good results. The precision of the cross gesture was 1.0, the recall of the up gesture was 1.0, the specificity of the cross gesture was 1.0, and the F1 score of the up gesture was 0.989. The evaluation metrics for the circle gesture were slightly lower than those for the up and cross gestures. Table 2 shows that the evaluation metrics for position classification outperformed those for gesture recognition. With the exception of position 4, where the accuracy was relatively low, all evaluation metrics for each position were 1.0 or close to 1.0. Tables 1 and 2 indicate that the present invention achieved excellent performance in both gesture recognition and position classification tasks.

[0069] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A joint deep learning method for gesture recognition and position classification, characterized in that: The following steps are involved: S1: Input the CSI sample into the one-dimensional convolution of the multi-task contraction residual network module to downsample the CSI sample and then perform normalization on it; S2: Input the processed CSI samples into the residual structure of several contraction sub-networks to perform feature matrix addition and soft thresholding operations; S3, inputting the extracted common feature information into the gesture output layer and the position output layer simultaneously; S4, the CSI samples are input in parallel into two FCNs of different scales with SE modules in the feature extraction auxiliary module to extract gesture spatial features; S5, reshape the output gesture spatial features; S6. Input the gesture spatial feature information after the reshape operation into the LSTM network to extract its temporal features; S7, the position output layer directly outputs its classification results; S8, the gesture output layer fuses the gesture spatiotemporal features extracted by the feature extraction auxiliary module with its shared feature vector through feature enhancement, and finally outputs the enhanced gesture classification result; The specific process of step S4 is as follows: S41. Establish a feature extraction auxiliary module MFCs-LSTM to extract gesture features separately and set the number of input channels. The two FCNs networks of different scales contain three convolution blocks, each of which contains a convolution layer with channel numbers of (128, 256, 128) respectively. The convolution kernel sizes of the first FCNs network are (7, 5, 3) respectively, and the convolution kernel sizes of the second FCN network are (5, 3, 1) respectively. The first two convolution blocks contain two SE modules, and the last two convolution blocks contain two average pooling layers. The FCNs network uses two FCN modules of different scales to capture the differences in the duration of each gesture, and then uses the SE module to selectively focus on the spatial features of the gesture and suppress position-independent features. Finally, the spatial features of the gesture are input into the LSTM network to extract its temporal features. S42. The feature extraction auxiliary module MFCs-LSTM inputs the subcarriers in the CSI samples as multivariate features into two FCNs networks. After the CSI samples undergo feature extraction, compression, extraction, and average pooling operations in three convolution blocks, they output two three-dimensional tensors of size [64, 128, 192]. The two three-dimensional tensors are concatenated at dim=1 to obtain a three-dimensional tensor of size [64, 256, 192]. Finally, dim=1 and dim=2 are converted through the transposition function to obtain a three-dimensional tensor of size [64, 192, 256]. After the CSI samples extract the features of the spatial dimension through the MFCNs network, a three-dimensional output tensor is obtained. The three-dimensional output tensor is input into an LSTM network with a feature dimension of 256, a hidden layer dimension of 15, a recurrent neural network layer number of 2, and an output category number of 2. Through LSTM network training, the gesture spatial features independent of the position state are finally obtained.

2. The method for joint deep learning of gesture recognition and position classification according to claim 1, characterized in that: The CSI sample acquisition process is as follows: using a CSI dataset with gesture and position labels, the CSI dataset involves several hand movements, including gestures up, down, left, right, circle, and cross at multiple locations by a user; at each location, each activity is repeated several times, and a number of sample data are collected; each sample data contains several subcarrier signals, and several samples are used as training set data. Then, several samples with the same conditions as the training set are uniformly selected from all samples as test set data.

3. The method for joint deep learning of gesture recognition and position classification according to claim 2, characterized in that: The specific process in step S2 is as follows: S21. Use the shrinkage sub-network to perform wavelet decomposition on each sub-carrier signal and transform it into a near-zero digital area. Then apply soft thresholding to convert the near-zero features to zero, and finally reconstruct it with wavelet. The soft thresholding is expressed as: Among them, x is the input feature, y is the output feature, and τ is the threshold, which is a positive parameter; S22, the soft threshold is assembled into a training module through several neural network layers, and is automatically learned using a gradient descent algorithm; S23. The automatically determined soft threshold is inserted into the residual module of 1DResNet[1,1,1] as a nonlinear transformation layer. The soft threshold module, as a subnetwork of the residual network, absolutizes the input features and performs a global pooling operation using GAP to obtain a one-dimensional vector. S24. Input the one-dimensional vector into the FC network, then perform BN and RELU operations, and finally input the FC network to obtain a scaling parameter. The scaling parameter is scaled to the range of (0, 1) through the Sigmoid function, which is expressed as: Where z is the output of the second layer FC network in the residual module, and α is the corresponding scaling parameter; S25. Multiply the scaling parameter α by the average value of |x| to obtain a threshold value, which is expressed as: τ=α·average|x i,j,c | Among them, τ is the threshold, i, j, and c are the indicators of the width, height, and channel of the input feature x respectively.

4. The method for joint deep learning of gesture recognition and position classification according to claim 2, characterized in that: The specific process of step S8 is as follows: S81. Input the CSI sample into the shared layer of the multi-task contraction residual network module to obtain n multi-channel one-dimensional vectors of length 6; S82, the one-dimensional vector undergoes a 3*1 1DCONV convolution operation and then performs BN, ReLU, and average pooling operations to output the gesture vector feature; S83, the output gesture vector feature is converted into a two-dimensional vector through a reshape operation; S84. The CSI samples are passed through the feature extraction auxiliary module MFCs-LSTM, which outputs n one-dimensional vectors of length 6. The obtained one-dimensional vectors are reshaped to be converted into two-dimensional vectors. S85. Concatenate the two two-dimensional vectors by column to generate a new two-dimensional vector. Connect the generated two-dimensional vector to the fully connected layer to output the gesture category. While paying attention to position classification, fuse the gesture feature parameters extracted by the feature extraction auxiliary module MFCs-LSTM with the gesture classification task output.

Images