A method and system for recognizing indoor user gestures and positions based on WIFI awareness

Abstract

The application discloses a WIFI sensing-based indoor user gesture and position recognition method and system. Gesture actions and the position information of the actor are contained in the high-dimensional Wifi data which has been preprocessed. The application adopts the architecture of a double-flow 2D neural network and uses the double-flow 2D neural network as a shared layer to complete the feature extraction function for two recognition tasks, i.e. a gesture recognition task and a position recognition task. After the feature fusion step, a 3D neural network is used for further feature extraction, and a special loss function is designed in the training stage to complete the convergence process of the multi-task learning of the two tasks. Experiments prove that the two tasks can both show good recognition effects under the same neural network framework.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication and artificial intelligence recognition technology, specifically relating to a method and system for recognizing indoor user gestures and locations based on WIFI sensing. Background Technology

[0002] In recent years, with the development of artificial intelligence technologies, indoor perception tasks have been greatly promoted and developed. Indoor perception tasks often refer to the tasks of distinguishing and judging people's activities indoors, and important sub-tasks include gesture recognition and user location recognition (localization).

[0003] Currently, image-based recognition methods have relatively mature solutions to the aforementioned problems in IoT smart homes. However, image-based solutions are often limited in practical applications due to concerns about user privacy and factors such as lighting conditions.

[0004] With the advancement of wireless communication technology, methods based on radio frequency signals are gradually entering the research field due to their superior privacy protection, such as millimeter wave, RFID technology and various wearable devices. Among them, millimeter wave technology and RFID technology are often not suitable for popularization in ordinary households due to the high cost of their supporting equipment. Some smart sensing wearable devices, such as wristbands, are also limited in their application because they must be worn by the user. As a wireless device popular in households, WIFI signal has excellent properties and can be used in the field of indoor sensing. In recent years, it has gradually attracted the attention of some researchers and has been extensively studied. In the field of gesture recognition, for example, the team of Liu Yunhao of Tsinghua University obtained Doppler velocity from WiFi signals, extracted higher-level gesture motion information, and combined it with deep learning methods to realize the gesture recognition system Widar3.0[1]. Reference [2] used the same dataset as Reference [1], and the authors proposed a deep learning method based on 3D convolutional neural network for gesture recognition and user recognition. For the current positioning methods, fingerprint positioning based on WiFi is mature and has approached the commercial stage.

[0005] However, the aforementioned work has not adequately explored the correlation between gesture recognition and location recognition tasks. In practical solutions, these tasks are often treated as two separate systems, failing to make good use of software and hardware resources. Currently, there is no single solution that can accomplish both tasks. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for recognizing indoor user gestures and locations based on WIFI sensing, so as to solve the problem of insufficient correlation between solutions of indoor sensing subtasks, which leads to the duplication of hardware and software resources.

[0007] To achieve the above objectives, the present invention employs the following technical solution:

[0008] A method for recognizing indoor user gestures and location based on Wi-Fi sensing, comprising the following steps:

[0009] Step 1: Receive the disturbed Wi-Fi signal to obtain a 3D video stream dataset, where the disturbance is caused by gesture changes; after preprocessing and normalizing the 3D video stream dataset, obtain a gesture information dataset.

[0010] Step 2: After transposing the gesture information dataset, two datasets D1 and D2 are obtained. After processing the datasets D1 and D2 by a 2D convolutional neural network, two high-dimensional feature maps are obtained.

[0011] Step 3: Fuse the two high-dimensional features into independent channels to obtain fused features. After further feature extraction using a 3D convolutional neural network, obtain the probability of the gesture change type and the probability of the indoor location.

[0012] Step 4: Output the type of gesture change and the indoor location.

[0013] A further improvement of the present invention is that:

[0014] Preferably, in step 1, the preprocessing involves zero-filling the three-dimensional video stream dataset.

[0015] Preferably, in step 1, the normalization process involves constraining the preprocessed dataset to the interval [0,1] to obtain the gesture information dataset.

[0016] Preferably, in step 2, the gesture information dataset is N_t-XY, dataset D1 is in the form of X-N_t-Y, and dataset D2 is in the form of Y-N_t-X; where X is the Doppler component on the X-axis, Y is the magnitude of the Doppler component on the Y-axis, and N_t is a set constant, greater than or equal to 36.

[0017] Preferably, in step 2, the 2D convolutional neural network includes a 2D convolutional layer, a batch sample normalization 2DBatchNorm layer, a ReLU activation function layer, and a pooling layer; the 2D convolutional neural network includes two branches, which are used to process dataset D1 and dataset D2 respectively.

[0018] Preferably, the 2D convolutional layer is used to multiply and sum the data in the dataset using different convolutional kernels to obtain feature map U1 and feature map U2;

[0019] The 2DBatchNorm layer for batch sample normalization is used to standardize the data.

[0020] The ReLU activation function layer is used to process the normalized data using ReLU functions;

[0021] The pooling layer downsamples from feature maps U1 and U2, reducing the dimensionality of the feature matrix to obtain a dimensionless matrix. Feature map U1 and dimension are Feature map U2.

[0022] Preferably, in step 3, the two high-order feature maps are transposed and then superimposed to obtain a result of size [size missing]. Feature map; with a size of The feature maps are processed by a 3D neural network.

[0023] Preferably, the 3D neural network includes two 3D convolutional layers, a ReLU layer, a batch 3D layer, a 3D pooling layer, and a fully connected layer;

[0024] Two 3D convolutional layers are used for... The data in the feature maps are multiplied in three dimensions and then added together to obtain feature maps U5 and U6.

[0025] The fully connected layer, which is a softmax layer, is used to predict the gesture type and location.

[0026] Preferably, both the 2D convolutional neural network and the 3D convolutional neural network are obtained by training with gradient descent using Loss.

[0027] An indoor user gesture and location recognition system based on Wi-Fi sensing, comprising:

[0028] The input module is used to receive disturbed Wi-Fi signals and obtain a three-dimensional video stream dataset, wherein the disturbance is caused by gesture changes; the three-dimensional video stream dataset is preprocessed and normalized to obtain a gesture information dataset.

[0029] The 2D extraction module is used to transpose the gesture information dataset to obtain two formats, dataset D1 and dataset D2. After the 2D convolutional neural network processes dataset D1 and dataset D2, it obtains two high-dimensional feature maps.

[0030] The 3D extraction module is used to fuse two high-dimensional features in independent channels to obtain fused features. After the fused features are further extracted through a 3D convolutional neural network, the probability of the gesture change type and the probability of the indoor location are obtained.

[0031] The output module is used to output the type of gesture change and the indoor location.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] This invention discloses a joint recognition method for indoor gestures and location based on Wi-Fi sensing. This method addresses two classic sub-tasks of indoor sensing by proposing a joint recognition approach based on Doppler frequency shift data of Wi-Fi signals. Based on 3D and 2D convolutional neural network structures, a low-complexity dual-stream parallel neural network recognition framework is designed, and multi-task learning techniques are employed to balance the losses of the two sub-tasks, achieving good performance. This method has advantages over existing independent sensing schemes in terms of practicality and resource utilization. Based on the Doppler changes of Wi-Fi signals during user gestures, a low-complexity dual-stream parallel neural network framework is designed for user gesture and location recognition, achieving better practicality and system integration.

[0034] This invention discloses a method and system for recognizing indoor user gestures and locations based on Wi-Fi sensing. The gestures and the location information of the person performing them are contained in pre-processed high-dimensional Wi-Fi data. This invention employs a dual-stream 2D ​​neural network architecture as a shared layer for two recognition tasks: gesture recognition and location recognition, to complete feature extraction. After feature fusion, a 3D neural network is used for further feature extraction. During the training phase, a special loss function is designed to facilitate the convergence of the multi-task learning process for both tasks. Experimental results demonstrate that both tasks, within the same neural network framework, exhibit good recognition performance.

[0035] The purpose of this invention is to address the shortcomings of existing solutions for indoor wireless sensing tasks by proposing a method for joint recognition of gestures and location based on Wi-Fi signals. Existing methods rarely explore the correlation between location recognition (positioning) and gesture recognition schemes, often using two independent schemes for indoor sensing sub-tasks. This method utilizes high-level synthetic feature data (BVP, body-coordinate velocity profile) that records the Doppler frequency shift changes of Wi-Fi signals, employing a dual-stream parallel 2D convolutional neural network feature extraction architecture and feature fusion structure. Furthermore, compared to the best-performing pure 3D convolutional neural network architecture, this network uses a low-complexity dual-stream 2D ​​neural network instead of a 3D neural network. Moreover, based on multi-task learning techniques, this method can simultaneously achieve two indoor sensing tasks: location recognition and gesture recognition. This method also demonstrates high accuracy in sensing different gestures and user locations under variations in individual user characteristics, user orientation, and experimental environment, exhibiting high algorithm robustness. Attached Figure Description

[0036] Figure 1 System flowchart;

[0037] Figure 2 This is the scenario for experimental data acquisition;

[0038] Figure 3 This shows the changes in loss during the dual-sensory task training phase.

[0039] Figure 4 The accuracy of gesture recognition and location recognition. Detailed Implementation

[0040] The present invention will now be described in further detail with reference to the accompanying drawings:

[0041] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. The terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, unless otherwise explicitly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection or a detachable connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0042] The basic principle of this invention is as follows: In a typical enclosed indoor environment, a signal undergoes multiple paths and changing channels during its transmission and reception. The multiple paths are caused by reflections from the enclosed environment, while the changing channels are primarily caused by regular changes in the user's hand movements. Only the portion of the reflection path change corresponding to the user's hand movements corresponds to a gesture, denoted as a gesture signal. Furthermore, the overall pattern of signal change in the received signal is related to the position of the receiver and transmitter, as well as the user's position relative to the transmitter. Other factors affecting this pattern include individual differences between users and indoor environmental factors. Therefore, deep learning methods can be used to extract the features of the received signal changing with human gestures and the user's position relative to the transmitter and receiver, thereby achieving the purpose of recognition.

[0043] This invention discloses a method for joint gesture recognition and location recognition based on WiFi signals, the method comprising the following steps:

[0044] S1. Receive the Wi-Fi signal disturbed by gesture changes, filter and video process the Wi-Fi signal to obtain the Doppler frequency shift change of the signal, and normalize it by transforming the world coordinates to user coordinates to obtain a three-dimensional video stream dataset; further, after preprocessing and normalization postprocessing, obtain a gesture information dataset of size N_t-XY.

[0045] S2. Transpose the gesture information dataset to obtain data D1 and data D2 in different formats. Extract two parallel high-dimensional features from data D1 and data D2 using a dual-stream 2D ​​convolutional feature extraction network.

[0046] S3. Fuse the two high-dimensional features into independent channels to obtain the fused features. Then, extract the fused features through a 3D neural network to obtain the probability that the input data is a certain gesture or a certain location indoors.

[0047] S4. Output the category of the gesture and the location of the gesture indoors.

[0048] This invention also discloses a joint gesture recognition and location recognition system based on WiFi signals, the system comprising:

[0049] The input module receives Wi-Fi signals disturbed by gesture changes, filters and processes the Wi-Fi signals to obtain the Doppler frequency shift of the recognized signals, and normalizes the data by transforming the world coordinates to user coordinates to obtain three-dimensional video stream data. After preprocessing and normalization of the data, gesture information data of size N_t-XY is obtained.

[0050] The 2D extraction module is used to transpose the gesture information dataset to obtain data D1 and data D2 in different formats. Two high-dimensional features are extracted in parallel from data D1 and data D2 through a dual-stream 2D ​​convolutional feature extraction network.

[0051] The 3D extraction module fuses two high-dimensional features through independent channels to obtain fused features. The fused features are then further extracted through a 3D neural network to obtain the probability that the input data represents a certain gesture or a certain location indoors.

[0052] The output module outputs the type of the gesture and its location indoors.

[0053] This invention discloses a method for acquiring a joint gesture recognition and location recognition system based on WiFi signals, comprising the following steps:

[0054] Preprocessing and normalization methods for S1 and BVP data;

[0055] The data preprocessing and normalization methods described in step S1 are designed for the input method of the neural network. The dataset used in this experiment involves user gestures that disturb the surrounding Wi-Fi signal. This disturbed Wi-Fi signal is received by the receiver, filtered, and processed using time and frequency analysis to obtain the Doppler frequency shift of the signal caused by gesture recognition. This is then normalized by transforming the world coordinates to user coordinates, resulting in a three-dimensional video stream dataset named the BVP (body-coordinate velocity profile) dataset. Each sample in this dataset corresponds to a gesture, and each data category contains not only gesture information but also the positional features of the person making the gesture. In other words, each data point in the dataset contains both gesture information and positional features.

[0056] The i-th data point from the BVP dataset is a data point of type T. i Data of size XY, T i X and Y represent the temporal dimension features, respectively, and the spatial dimension features, representing the time of the gesture and the position of the person making the gesture. Here, X is the Doppler component on the X-axis, and Y is the magnitude of the Doppler component on the Y-axis. For each BVP dataset, the magnitudes of X and Y are constants, and T... i The data ranges in size from 14 to 34. The preprocessing here is essentially a zero-padded process. Specifically, this is done in T... i The two-dimensional matrix data of XY is placed before T. i_e A matrix of all zeros, followed by T matrices. i_l There exist n zero matrices that satisfy the following formula:

[0057] T i +T i_e +T i_l =N_t(1)

[0058] Where N_t is a constant that is greater than or equal to 36. Data that has undergone zero-padding can be fed into the neural network in a uniform format.

[0059] Data normalization involves constraining the size of each element of the data to the interval [0, 1]. The specific method is as follows:

[0060]

[0061] Where d i ’ For the new element, d i Let i be the i-th element in the data.

[0062] After the above preprocessing of the BVP dataset, a gesture information dataset of size N_t-XY can be obtained. This dataset contains two-dimensional Doppler features of the signal transformation over a certain time period, which can be extracted by the feature extraction layer of a neural network for gesture classification. Simultaneously, positional information also affects the feature representation in the data, and can also be used to design a specific neural network to extract certain features and then make a judgment.

[0063] S2 and BVP extract parallel high-dimensional features through a dual-stream 2D ​​convolutional feature extraction network;

[0064] The described dual-stream 2D ​​convolutional network is a network that performs parallel feature extraction on the input 3D data. Specifically, the input N_t-XY data is transposed into X-N_t-Y data D1 and Y-N_t-X data D2. Data D1 and D2 are then fed into branch one and branch two of the 2D convolutional neural network, respectively.

[0065] Specifically, the 2D convolutional neural network branch one and branch two include 2D convolutional layers, batch sample normalization 2DBatchNorm layers, ReLU activation function layers, and pooling layers, respectively.

[0066] The 2D convolutional layer uses 16 convolutional kernels of different (2,2) sizes to multiply and then sum the data to obtain feature maps U1 and U2 (these feature maps U1 and U2 are continuously updated in the 2DBatchNorm layer, ReLU activation function layer, and pooling layer). The parameters in the convolutional kernels are all updated and learned during training using the gradient descent method of the neural network.

[0067] The batch sample normalization 2DBatchNorm layer is actually used to standardize the size range of the feature maps U1 and U2 obtained from the 2D convolutional layer. Its basic process is similar to formula (2). Its main purpose is to eliminate the size difference between data and prevent the neural network from overfitting.

[0068] The ReLU activation function layer is used to pass the data input after the BatchNorm layer (that is, the normalized feature maps U1 and U2) through the ReLU function, as shown in formula (3).

[0069]

[0070] Where x is the input function value.

[0071] The pooling layer utilizes max pooling to downsample local regions of the feature map matrix output by the ReLU activation function layer, reducing the dimensionality of the feature matrix. The input matrix is ​​first divided into several partitions. Within each partition, a max operation is used to obtain the local maximum feature. The max operation preserves the feature size while reducing the dimensionality of the processed data. The pooling size is (2, 2), with a stride of 2, thus reducing the size of the sampled data by half compared to the original data size.

[0072] In summary, data D1 and data D2, after passing through branch one and branch two of the convolutional neural network, respectively, yielded... and Feature maps U1 and U2.

[0073] S3. Parallel high-dimensional feature fusion and feature re-extraction methods;

[0074] High-dimensional feature fusion involves fusing data. The purpose of fusion is to combine the two parallel high-dimensional feature maps U1 and U2 for subsequent feature extraction. Specifically, the two different-dimensional features are fused using independent channels, with the number of channels of the previously obtained features used as the frame number of the input 3D data. Specifically, feature maps U1 and U2 are first transposed and expanded to a size of... Feature maps U3 and U4 are used, and the features are superimposed. Finally, the fused post-processed result is of size [size missing]. The feature map.

[0075] The main method for feature re-extraction is to use 3D neural networks to extract features from... The feature is then extracted again. The 3D neural network includes two 3D convolutional layers, a batch sample normalization 3DBatchNorm layer, a ReLU layer, and a fully connected layer.

[0076] The two 3D convolutional layers are actually composed of two layers with convolutional kernels of size (3, 3, 3), numbering 64 and 32 respectively, scanning in three dimensions with a stride of 1. The data is multiplied and then added in three different dimensions to obtain feature maps U5 and U6. The parameters in the 3D convolutional kernels are also updated and learned during training using the gradient descent method of the neural network.

[0077] The ReLU layer, the batch sample normalization 3DBatchNorm layer, and the corresponding layer in step 2 work on the same principle, performing an additional step of data expansion in three dimensions.

[0078] The batch sample normalization 2DBatchNorm layer actually standardizes the size range of the data in feature map U5 and feature map U6. Its basic process is similar to formula (2). Its main purpose is to eliminate the size difference between data and prevent the neural network from overfitting.

[0079] The ReLU activation function layer is used to process the data input after the BatchNorm layer through the ReLU function, as shown in formula (3).

[0080]

[0081] In summary, after processing through convolutional layers, batch normalization (3DBatchNorm) layers, and ReLU layers, the final data size is [data size value missing]. Here, this method specifies that N_t is 34 in size, and X = Y = 20. Therefore, the data size is (8, 32, 4, 4). The flattened data size is (4096, 1).

[0082] The fully connected layer is implemented using three layers of neurons. The first layer receives data from the ReLU activation function layer, with input and output sizes of (4096, 256), (256, 64), and (64, 5) for each layer, respectively. Dropout layers are placed between adjacent connected layers. Their main function is to randomly deactivate neurons in certain layers during training, based on probability, thus preventing the classification parameters from depending on any particularly large values ​​and avoiding overfitting. The output layer uses a softmax layer commonly used in neural networks. This layer outputs the probabilities of each predicted target and selects the maximum value as the final classification result. and in and It is a 5-dimensional vector, where each dimension represents the probability that the current input data is a certain gesture and a certain position.

[0083] Specifically, the output recognized by the softmax layer and This will be the output of the recognition system, representing the decoupled gesture and location information. Because and Each dimension represents the probability of a specific category for the gesture's action (push, pull, swipe, etc.) and location (point A, point B, etc.). Therefore, each dimension is used to determine... and The largest p-th dimension and the largest q-th dimension in the array, i.e. and This corresponds to the p-th gesture category and q-th position of this input.

[0084] S4. Training and classification methods for joint gesture recognition and location recognition.

[0085] In previous neural network designs, shared layers and separate output layers were designed for two different tasks: gesture recognition and location recognition. The significance of the shared layer lies in leveraging the correlation between the two recognition tasks to jointly train the shared layer parameters, thereby improving the recognition accuracy together. The separate output layers decouple the features extracted from the shared layer. This means that, on the one hand, in the shared layer, the common features between the two relatively independent but closely related tasks can be better extracted, thus simultaneously achieving performance gains in their respective recognition tasks. On the other hand, the separate output layers enable the two tasks to further extract high-dimensional, independent features, allowing the two neural network branches to complete the classification of their respective tasks.

[0086] During the neural network training phase, the cross-entropy loss function is used to represent the cross-entropy between the two classification tasks.

[0087]

[0088] To simultaneously update the loss function Loss1 for gesture recognition and Loss2 for location recognition, and to reduce the difference caused by the different convergence speeds of the two tasks, a geometric regularization method is used for overall loss balancing.

[0089]

[0090] When training with gradient descent to minimize loss, a threshold value can be set to determine whether the data has converged. When the loss is sufficiently small and stabilizes, the data can be considered well-trained.

[0091] The following analysis and explanation will be further illustrated with specific examples.

[0092] Example 1

[0093] See Figure 1 The system forms the framework for recognition. The original signal input BVP data undergoes a data preprocessing process, passing through 2D neural network branch one and branch two respectively. Then, features are extracted through the branches of the two-stream neural network. After feature fusion, further feature extraction for gesture recognition and position recognition is performed through 3D neural network branch one and branch two respectively. Through multi-task learning, classification results are obtained.

[0094] See Figure 2 The basic experimental conditions were set up, and the experimental data was BVP data. The locations of the Wi-Fi transmitter and receiver in the experiment are shown in the figure. (The last part, "loc," appears to be an unrelated fragment and is omitted from the translation.) i The five points (i = 1, 2, ..., 5) representing five different positions are one of the main contents of this recognition. At the same time, the recognized gestures include five types: pushing and pulling, swiping, clapping, drawing circles, and drawing zigzag.

[0095] See Figure 3 This study examines the changes in loss during the dual-sensory task training phase. A total of 18,000 datasets were used, with 50 training epochs and a learning rate of 0.01. The training, validation, and test sets were divided in a 7:2:1 ratio. Geometric regularization was used to balance the training loss for the two tasks. While the convergence times differed between the two tasks, the geometric averaging method kept the loss difference within a normal range of 0.5. The final prediction results on the test set were 88.56% accuracy for gesture recognition and 88.9% accuracy for location recognition.

[0096] See Figure 4 Figure (a) shows the accuracy of gesture recognition in multi-task learning. It can be seen that the recognition results have a good effect on the recognition of different gestures. Figure (b) shows that the neural network also has a high recognition probability for different positions.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for recognizing indoor user gestures and location based on WIFI sensing, characterized in that, Includes the following steps: Step 1: Receive the disturbed Wi-Fi signal to obtain a 3D video stream dataset, where the disturbance is caused by gesture changes; after preprocessing and normalizing the 3D video stream dataset, obtain a gesture information dataset. Step 2: After transposing the gesture information dataset, two datasets, D1 and D2, are obtained. A 2D convolutional neural network processes datasets D1 and D2 to obtain two high-dimensional feature maps. The gesture information dataset is... The dataset D1 is in the form of The dataset D2 is in the form of Where X represents the Doppler component on the X-axis, and Y represents the magnitude of the Doppler component on the Y-axis. The constant to be set is greater than or equal to 36; Step 3: Fuse the two high-dimensional features using independent channels to obtain fused features. Then, use a 3D convolutional neural network to further extract features, obtaining the probability of the gesture change type and the probability of the indoor location. Finally, transpose the two high-dimensional feature maps and superimpose them to obtain a result of size [size missing]. Feature map; with a size of The feature maps are processed through a 3D neural network; The 3D neural network includes two 3D convolutional layers, a ReLU layer, a batch 3D layer, a 3D pooling layer, and a fully connected layer; Two 3D convolutional layers are used for... The data in the feature maps are multiplied in three dimensions and then added together to obtain feature maps U5 and U6. The fully connected layer, using softmax, is used to predict gesture type and location; Step 4: Output the type of gesture change and the indoor location.

2. The method for recognizing indoor user gestures and location based on WIFI sensing according to claim 1, characterized in that, In step 1, the preprocessing involves zero-filling the 3D video stream dataset.

3. The method for recognizing indoor user gestures and location based on WIFI sensing according to claim 1, characterized in that, In step 1, the normalization process involves constraining the preprocessed dataset to the interval [0,1] to obtain the gesture information dataset.

4. The method for recognizing indoor user gestures and location based on WIFI sensing according to claim 1, characterized in that, In step 2, the 2D convolutional neural network includes 2D convolutional layers, batch sample normalization 2DBatchNorm layers, ReLU activation function layers, and pooling layers; the 2D convolutional neural network includes two branches, which are used to process dataset D1 and dataset D2 respectively.

5. The method for recognizing indoor user gestures and location based on WIFI sensing according to claim 4, characterized in that, 2D convolutional layers are used to multiply and sum the data in the dataset using different convolutional kernels to obtain feature maps U1 and U2. The 2DBatchNorm layer for batch sample normalization is used to standardize the data. The ReLU activation function layer is used to process the normalized data using ReLU functions; The pooling layer downsamples from feature maps U1 and U2, reducing the dimensionality of the feature matrix to obtain a dimensionless matrix. Feature map U1 and dimension are Feature map U2.

6. The method for recognizing indoor user gestures and location based on WIFI sensing according to claim 1, characterized in that, Both the 2D convolutional neural network and the 3D convolutional neural network are obtained through... The results were obtained after training using the gradient descent method.

7. A WIFI-sensing-based indoor user gesture and location recognition system for implementing the method of claim 1, characterized in that, include: An input module is used to receive disturbed Wi-Fi signals and obtain a three-dimensional video stream dataset, wherein the disturbance is caused by a change in gesture. The gesture information dataset is obtained by preprocessing and normalizing the 3D video stream dataset; The 2D extraction module is used to transpose the gesture information dataset to obtain two formats, dataset D1 and dataset D2. After processing dataset D1 and dataset D2 by the 2D convolutional neural network, two high-dimensional feature maps are obtained. The 3D extraction module is used to fuse two high-dimensional features in independent channels to obtain fused features. After the fused features are further extracted through a 3D convolutional neural network, the probability of the gesture change type and the probability of the indoor location are obtained. The output module is used to output the type of gesture change and the indoor location.

Images