A human action recognition method, system and device based on WiFi channel state information imaging and a readable storage medium
By converting WiFi channel state information into Gram difference field images and using a four-layer convolutional neural network for feature extraction, the problem of high model complexity in existing technologies is solved, and high-precision human activity recognition is achieved.
Patent Information
- Application Number
- CN202211413940.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-11
- Publication Date
- 2026-01-06
- Estimated Expiration
- 2042-11-11
AI Technical Summary
Existing methods for human activity recognition based on WiFi channel state information struggle to improve recognition accuracy while reducing model complexity. Traditional methods are limited by complex feature processing and model architecture.
WiFi channel state information is converted into Gram difference field images, and features are extracted through a four-layer convolutional neural network, including batch normalization, rectified linear unit layer, max pooling and adaptive average pooling layer, combined with a classifier with dropout layer and linear layer to achieve high-precision recognition.
It achieves higher activity recognition accuracy, while reducing model parameters by 90 times and floating-point operations by 20 times, resulting in lower model complexity and better generalization.
Smart Images
Figure CN115830705B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to human activity recognition, specifically to a method, system, device, and readable storage medium for human action recognition based on WiFi channel state information imaging. Background Technology
[0002] Human Activity Recognition (HAR) is a core driving force behind a wide range of applications, including smart homes, healthcare, security monitoring, and virtual reality. Traditional methods use cameras, wearable devices, and sonar as sensing modules to achieve HAR. While effective, these methods suffer from several drawbacks, including privacy concerns, the need for wearable sensors, and limitations in sensing range. Compared to vision-based and wearable device-based methods, HAR based on WiFi Channel State Information (CSI) offers greater privacy and security, eliminating the need for any specific devices on the target, making it a hot research topic. From CSI, we can extract features such as amplitude and frequency, mapping them to human activity to achieve activity recognition.
[0003] Various CSI-based activity recognition methods have been proposed by exploring the characteristics of machine learning and signal processing techniques. For example, feature preprocessing is performed first to filter out environmental noise, followed by machine learning (e.g., support vector machines or K-nearest neighbors) classification algorithms for activity recognition. While these methods have achieved some results, there is still room for improvement in recognition accuracy. With the maturity of deep learning technology, research has used deep learning methods to implement CSI-based human activity recognition. For example, deep learning networks such as convolutional neural networks (CNNs) and long short-term memory (LSTMs) have been used in CSI-based activity recognition. Although recognition accuracy has been improved to some extent, the performance of these methods largely depends on complex feature processing and model architecture. How to improve recognition accuracy while reducing model complexity remains a challenge in HAR research. Summary of the Invention
[0004] Purpose of the invention: In view of the defects and shortcomings of the existing technology, the purpose of this invention is to provide a method and system for human activity recognition based on WiFi-CSI imaging, so as to achieve higher activity recognition accuracy with lower model complexity.
[0005] Technical solution: A method for human motion recognition based on WiFi channel state information imaging, comprising the following steps:
[0006] Collect WiFi channel status information and extract the amplitude information of channel status changes over time;
[0007] Based on the amplitude information that changes over time, a Gram angular difference field matrix is constructed, and the Gram angular difference field matrix is converted into a Gram angular difference field image;
[0008] Gram angular difference field image data is input into a convolutional neural network model to extract image features and output recognition results. The convolutional neural network includes four convolutional layers, each followed by a batch normalization layer and a rectified linear unit layer. After the rectified linear unit layer, a max pooling layer is used to extract the main features of adjacent regions. After the fourth convolution, an adaptive average pooling layer is used to specify the output size of the feature vector. After the adaptive average pooling layer, there is a classifier with a dropout layer and a linear layer.
[0009] Furthermore, WiFi channel state information is collected, and the amplitude information of time-varying channel state information is extracted, including:
[0010] The data received by the receiving antenna is acquired within the sampling time T. The CSI value that varies with time at a given subcarrier frequency is called the CSI stream, denoted as [H]. 1×T Each element is represented by the complex number a+bi. Taking the modulus of the complex number yields the amplitude value at the corresponding time. The CSI amplitude information for any antenna and subcarrier is represented as follows:
[0011] Furthermore, constructing the Gram angle difference field matrix based on the time-varying amplitude information includes:
[0012] Using normalization methods, Scale to the 0-1 range;
[0013] Use the inverse trigonometric cosine function to calculate the CSI amplitude at timestamp t. Corresponding angle:
[0014] The amplitude relationship between any two original timestamps i and j can be represented by the difference between two angles of a sine function, as expressed by the formula:
[0015]
[0016] The generated Gram difference field matrix is represented as follows:
[0017]
[0018] Each element in the matrix represents the magnitude relationship between two timestamps.
[0019] Furthermore, the training process of a convolutional neural network is as follows:
[0020] Features are extracted from Gram difference field images using convolutional layers. For a point in the image, the origin of the convolution kernel is made to coincide with that point. Then, the point on the convolution kernel is multiplied with the corresponding point in the image. Finally, the products of these points are added together to obtain the convolution value of that point.
[0021] After the convolutional layer, calculate the average value of the batch. and variance The input x of the convolutional layer i Normalization calculation
[0022] A ReLU layer is used to perform a thresholding operation on each input element: any value less than zero is set to zero, while any value greater than zero remains unchanged;
[0023] Max pooling and adaptive average pooling are used to extract the maximum and average values of the rectangular region surrounding the tensor, respectively.
[0024] The loss value is calculated using the cross-entropy loss function, and the output of the fully connected layer is converted into a probability distribution of the expected output class. The class with the highest probability value is judged as the true label.
[0025] A human motion recognition system based on WiFi channel state information imaging includes:
[0026] The data acquisition module is used to collect WiFi channel status information and extract the amplitude information of channel status changes over time.
[0027] The data conversion module is used to construct a Gram angular difference field matrix based on the amplitude information that changes over time, and to convert the Gram angular difference field matrix into a Gram angular difference field image;
[0028] The action recognition module is used to input Gram angular difference field image data into a convolutional neural network model to extract image features and output recognition results. The convolutional neural network includes four convolutional layers, each followed by a batch normalization layer and a rectified linear unit layer. After the rectified linear unit layer, a max pooling layer is used to extract the main features of adjacent regions. After the fourth convolution, an adaptive average pooling layer is used to specify the output size of the feature vector. After the adaptive average pooling layer, there is a classifier with a dropout layer and a linear layer.
[0029] The present invention also provides an apparatus comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the human motion recognition method based on WiFi channel state information imaging as described above.
[0030] The present invention also provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the human motion recognition method based on WiFi channel state information imaging as described above.
[0031] Beneficial effects: (1) This invention converts WiFi channel state information into Gram difference field images, which expands the original channel state feature information and makes it more adaptable to the powerful feature extraction capability of 2D convolutional neural networks, resulting in high recognition accuracy. (2) This invention achieves high-precision recognition based on Gram difference field images using a convolutional neural network model with four convolutional layers, requiring only 6.5MB of parameters and 0.1GFLOPs of floating-point operations. Compared with classic deep learning models (ResNet50, VGG19, ShuffleNet), the number of parameters is reduced by 90 times and the required floating-point operations are reduced by 20 times. This invention achieves higher activity recognition accuracy with lower model complexity. Attached Figure Description
[0032] Figure 1 This is a schematic diagram of the experimental environment;
[0033] Figure 2 This is a flowchart of the overall process for human motion recognition based on WiFi channel state information imaging.
[0034] Figure 3 This provides the raw amplitude information of the CSI for a single subcarrier clapping motion.
[0035] Figure 4 The GADF image corresponding to the clapping action;
[0036] Figure 5 To verify the recognition accuracy of the experiment. Detailed Implementation
[0037] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0038] The experimental environment in this embodiment is as follows: The experimental environment is a laboratory with an area of 5m × 8m, containing tables, chairs, and other items. The laboratory layout is as follows. Figure 1As shown in the diagram. In the experiment, the router served as the transmitter of the WiFi signal, and the computer equipped with an Intel 5300 wireless network card was used as the receiver. The distance between the transmitter (i.e., the router) and the receiver (i.e., the computer) was 4 meters. The computer was equipped with the Ubuntu 14.04 operating system, the CSI acquisition tool CSITool, MATLAB, and Python. The WiFi signal transmission frequency was set to 2.4 GHz, the channel bandwidth was set to 20 MHz, and each antenna received CSI data from 30 subcarriers. The receiver used a 100 Hz sampling frequency to collect data, with each sample collected for 3 seconds. Each volunteer performed each action 20 times, for a total of 5 volunteers and 6 actions: pushing / pushing, sweeping, patting, sliding, drawing a Z-shape, and drawing an N-shape.
[0039] Reference Figure 2 The human motion recognition method based on WiFi channel state information imaging according to the present invention includes the following steps:
[0040] Step S1: Collect WiFi channel status information and extract the amplitude information of channel status changes over time.
[0041] In a wireless channel, the relationship between the received signal Y and the transmitted signal X is represented as follows:
[0042] Y = HX + N
[0043] Where H is the channel gain matrix and N is noise.
[0044] In this embodiment, by pinging the router gateway from a laptop, the Channel State Information (CSI) from the transmitter to the receiving antenna can be obtained using the CSITool within a sampling time T. This CSI information is represented by each value in matrix H. H is 3x30xT, where 3 represents the number of receiving antennas, 30 represents the number of subcarriers, and T is the sampling time. Given a pair of receiving antennas, the CSI values changing over time at a given subcarrier frequency are called the CSI stream, denoted as [H]. 1×T Each element is represented by the complex number a + bi, and taking the modulus of the complex number yields the amplitude value at the corresponding moment. The CSI amplitude information for any antenna and subcarrier is represented as follows: Figure 3 The image shows the raw amplitude information of a single subcarrier clapping motion CSI.
[0045] Step S2: Construct the Gram angular difference field matrix based on the amplitude information that changes over time, and convert the Gram angular difference field matrix into a Gram angular difference field image.
[0046] As CSI signals for various activities, changes over continuous periods are more characteristic than at a single point. From this perspective, this invention uses the Gram difference field matrix to describe the amplitude relationship between any two timestamps. In this process, the characteristics of the CSI time series are preserved, while the original data is expanded and interpreted. To fully utilize the powerful feature extraction capabilities of two-dimensional convolutional networks (CNNs) for images, the CSI is simultaneously converted into an image.
[0047] In n-dimensional Euclidean space, the matrix formed by the pairwise inner products of any k vectors is called the Gram matrix of those k vectors. The Gram matrix reflects the relationships between the vectors. If the vectors are unit vectors, then the Gram matrix can be represented as a cosine function, as shown below:
[0048]
[0049] Where θ i,j This represents the angle between vectors i and j.
[0050] Based on the Gram matrix principle, the min-max normalization method is used to... Scale to the 0-1 range.
[0051] Use the inverse trigonometric cosine function to find For the corresponding angle, the formula is as follows:
[0052]
[0053] The amplitude relationship between any two original timestamps i and j can be represented by the difference between two angles of a sine function, as expressed by the formula:
[0054]
[0055] The generated GADF matrix is represented as follows:
[0056]
[0057] Through this transformation, the one-dimensional CSI signal is represented as a two-dimensional matrix, where each element represents the amplitude relationship between two timestamps. To leverage the powerful feature extraction capabilities of CNNs, the GADF is plotted as a heatmap, displaying data differences through color and brightness variations. In this embodiment, the GADF matrix is used to generate an image using the phthon plotting Matplotlib library. Figure 4 The image shown is the GADF image corresponding to the clapping action. The greater the amplitude change, the darker the corresponding color.
[0058] Step S3: Input the Gram angular difference field image data into the convolutional neural network model to extract image features and output the recognition result.
[0059] In this embodiment, a CNN with four convolutional layers is constructed to extract multi-scale GAF image features. Each convolutional layer is followed by a batch normalization (BN) layer, which improves the network's convergence speed. After BN, a rectified linear unit (ReLU) layer is applied. Then, a max-pooling layer is used to extract the principal features of neighboring regions. An adaptive average pooling layer is used after the fourth convolution to specify the feature output size. Following the adaptive average pooling layer is a classifier with a dropout layer and a linear layer. To prevent overfitting, the dropout layer discards features learned from the network with a certain probability during network training. The linear layer performs a linear combination of the input data to determine the probability of the expected output class. The CNN structure parameters are shown in Table 1.
[0060] Table 1 CNN Structure Parameters
[0061]
[0062] The model training process is as follows:
[0063] (1) Features are extracted from the GADF image using convolutional layers. For a point in the image, the origin of the convolution kernel is aligned with that point. Then, the points on the convolution kernel are multiplied by the corresponding points in the image. Finally, the products of these points are summed to obtain the convolution value of that point. The calculation formula is as follows:
[0064]
[0065] Where S represents the output, G is the convolution kernel, and X is the input.
[0066] (2) After the convolutional layer, calculate the average value of the batch (number of input images). and variance The input x of the convolutional layer i Normalized calculation. The calculation formula is:
[0067]
[0068] When the mini-batch variance is small, ε improves numerical stability. To ensure that the original learned features are preserved after each data normalization, a scaling parameter δ and a translation parameter λ are introduced as follows:
[0069]
[0070] (3) Use a ReLU layer to perform a thresholding operation on each input element: set any value less than zero to zero, and leave any value greater than zero unchanged. This improves the nonlinearity of the decision function and the entire network without affecting the receptive field of the convolutional layers.
[0071] (4) In order to minimize the computational cost, the maximum value and average value of the rectangular region around the tensor are extracted using the max pooling layer and the adaptive average pooling layer, respectively.
[0072] (5) The loss value is calculated using the CrossEntropyLoss function. The output of the fully connected layer is converted into a probability distribution of the expected output class. The class corresponding to the highest probability value is determined as the true label. The ADAM algorithm is used as the optimizer. During backpropagation in deep learning, the optimizer guides the loss function parameters to be updated in the correct direction and by an appropriate size, so that the updated parameters keep the loss function value close to the global minimum.
[0073] like Figure 5 As shown, experimental results demonstrate that the GADF-based model achieves a recognition accuracy of 0.997 with only four convolutional layers. To verify the performance of the proposed method, it was compared with three classic deep learning models: ResNet50, VGG19, and ShuffleNet. The model of this invention slightly outperforms ShuffleNet in recognition accuracy and is comparable to ResNet and VGG19. However, in terms of model resource consumption, the model of this invention reduces the number of parameters by 90 times and the required floating-point operations (FLOPs) by 20 times.
[0074] This invention converts WiFi channel state information into Gram difference field images, which expands the original channel state feature information and makes it more suitable for the powerful feature extraction capabilities of 2D convolutional neural networks. The imaging method is simple and the model accuracy reaches the state-of-the-art results. At the same time, this invention considers the nature of the feature expression of the active signal itself, that is, the meaningful feature of different active signals is not in a single point but in the trend of amplitude change, making the model more generalizable.
[0075] The present invention also provides a human motion recognition system based on WiFi channel state information imaging, comprising:
[0076] The data acquisition module is used to collect WiFi channel status information and extract the amplitude information of channel status changes over time.
[0077] The data conversion module is used to construct a Gram angular difference field matrix based on the amplitude information that changes over time, and to convert the Gram angular difference field matrix into a Gram angular difference field image;
[0078] The action recognition module is used to input Gram angular difference field image data into a convolutional neural network model to extract image features.
[0079] The output recognition result is provided, wherein the convolutional neural network includes four convolutional layers, each followed by a batch normalization layer and a rectified linear unit layer. After the rectified linear unit layer, a max pooling layer is used to extract the main features of the adjacent regions. After the fourth convolution, an adaptive average pooling layer is used to specify the output size of the feature vector. After the adaptive average pooling layer, there is a classifier with a dropout layer and a linear layer.
[0080] According to an embodiment of the present invention, the data acquisition module includes:
[0081] The data acquisition unit is used to acquire the data received by the receiving antenna within the sampling time T;
[0082] The data processing unit is used to define the time-varying CSI values at a given subcarrier frequency as a CSI stream, denoted as [H]. 1×T Each element is represented by the complex number a+bi. Taking the modulus of the complex number yields the amplitude value at the corresponding time. The CSI amplitude information for any antenna and subcarrier is represented as follows:
[0083] According to an embodiment of the present invention, the data conversion module includes:
[0084] Scaling the cells, using a normalization method, to Scale to the 0-1 range;
[0085] The angle calculation unit uses the inverse trigonometric cosine function to calculate the CSI amplitude value at timestamp t. Corresponding angle
[0086] The amplitude extraction unit uses the difference between two angles of a sine function to represent the amplitude relationship between any two original timestamps i and j, as expressed by the formula:
[0087]
[0088] The matrix generating unit generates the Gram angle difference field matrix as follows:
[0089]
[0090] Each element in the matrix represents the magnitude relationship between two timestamps;
[0091] The image generation unit generates a Gram angular difference field image based on the Gram angular difference field matrix.
[0092] According to an embodiment of the present invention, the training process of the convolutional neural network model used in the action recognition module is as follows:
[0093] Features are extracted from Gram difference field images using convolutional layers. For a point in the image, the origin of the convolution kernel is made to coincide with that point. Then, the point on the convolution kernel is multiplied with the corresponding point in the image. Finally, the products of these points are added together to obtain the convolution value of that point.
[0094] After the convolutional layer, calculate the average value of the batch. and variance The input x of the convolutional layer i Normalization calculation
[0095] A ReLU layer is used to perform a thresholding operation on each input element: any value less than zero is set to zero, while any value greater than zero remains unchanged;
[0096] Max pooling and adaptive average pooling are used to extract the maximum and average values of the rectangular region surrounding the tensor, respectively.
[0097] The loss value is calculated using the cross-entropy loss function, and the output of the fully connected layer is converted into a probability distribution of the expected output class. The class with the highest probability value is judged as the true label.
[0098] During backpropagation in a convolutional neural network, the ADAM algorithm is used as an optimizer to guide the loss function parameters to be updated in the correct direction and by an appropriate size, so that the updated parameters keep the loss function value close to the global minimum.
[0099] It should be understood that the human motion recognition system based on WiFi channel state information imaging can implement all the technical solutions in the above method embodiments. The functions of each functional module can be specifically implemented according to the methods in the above method embodiments. The specific implementation process not mentioned in the system embodiments can be referred to the relevant descriptions in the above embodiments.
Claims
1. A method for human action recognition based on WiFi channel state information imaging, characterized in that, The method comprises the following steps: Collect WiFi channel state information and extract the amplitude information changing over time in the channel state information; Construct a Gram angle difference field matrix based on the amplitude information changing over time, and convert the Gram angle difference field matrix into a Gram angle difference field image; Input the Gram angle difference field image data into a convolutional neural network model to extract image features and output recognition results, wherein the convolutional neural network comprises four convolutional layers, each followed by a batch normalization layer and a rectified linear unit layer, and after the rectified linear unit layer, a maximum pooling layer is used to extract the main features of adjacent regions, and after the fourth convolution, an adaptive average pooling layer is used to specify the output size of the feature vector, and after the adaptive average pooling layer, a classifier with a Dropout layer and a linear layer is arranged; The step of collecting WiFi channel state information and extracting the amplitude information changing over time in the channel state information comprises: The data received by the receiving antennas is acquired in the sampling time T, and the CSI value varying with time at a given subcarrier frequency is called a CSI stream, denoted as [H] 1×T , each element of which is represented by a complex number a+bi, and the amplitude value at the corresponding moment is obtained by taking the modulus of the complex number, and the CSI amplitude information of any antenna and subcarrier is denoted as [H] ] 1×T ; The step of constructing a Gram angle difference field matrix based on the amplitude information changing over time comprises: Using a normalization method, [ ] 1×T Scale to the 0-1 range; Using inverse cosine function to find the corresponding angle of the CSI magnitude value at timestamp t t of the corresponding angle: ; The amplitude relationship corresponding to two time stamps i and j is represented by a sine function two-angle difference, and the formula is: The generation of the Gram angle difference field matrix is represented as: Each element in the matrix represents the amplitude relationship corresponding to two time stamps.
2. The method of claim 1, wherein, The training process of the convolutional neural network is as follows: The convolutional layer is used to extract features from the Gram angle difference field image. For a point on the image, the origin of the convolution kernel is made to coincide with the point, then the points on the convolution kernel are multiplied by the corresponding points on the image, and the products of these points are added to obtain the convolution value of the point; After the convolution layer, the average value of the batch is calculated k and the variance , the input x to the convolution layer is normalized i ; The ReLU layer performs a threshold operation on each input element: any value less than zero is set to zero, and any value greater than zero remains unchanged; The maximum pooling layer and the adaptive average pooling layer are used to extract the maximum value and the average value of the rectangular region around the tensor, respectively; The cross-entropy loss function is used to calculate the loss value, the output of the fully connected layer is converted into the probability distribution of the expected output class, and the class corresponding to the maximum probability value is judged as the real label.
3. The method of claim 2, wherein, During the back propagation of the convolutional neural network, the ADAM algorithm is used as the optimizer to guide the update of the loss function parameters in the correct direction, so that the updated parameters keep the loss function value close to the global minimum value.
4. A human action recognition system based on WiFi channel state information imaging, characterized in that, It comprises: A data collection module for collecting WiFi channel state information and extracting the amplitude information changing over time in the channel state information; A data conversion module for constructing a Gram angle difference field matrix based on the amplitude information changing over time and converting the Gram angle difference field matrix into a Gram angle difference field image; An action recognition module for inputting the Gram angle difference field image data into a convolutional neural network model to extract image features and output recognition results, wherein the convolutional neural network comprises four convolutional layers, each followed by a batch normalization layer and a rectified linear unit layer, and after the rectified linear unit layer, a maximum pooling layer is used to extract the main features of adjacent regions, and after the fourth convolution, an adaptive average pooling layer is used to specify the output size of the feature vector, and after the adaptive average pooling layer, a classifier with a Dropout layer and a linear layer is arranged; The data collection module comprises: A data acquisition unit is configured to acquire data received by the receiving antenna within a sampling time T; data processing unit for referring to the CSI values varying with time at a given subcarrier frequency as a CSI stream, denoted as [H] 1×T , each element of which is represented by a complex number a + bi, and the modulus of the complex number gives the amplitude value at the corresponding time instant, and the CSI amplitude information for any antenna and subcarrier is denoted as [A] ] 1×T ; The data conversion module comprises: scaling unit, using a normalization method, to scale [ ] 1×T to the 0-1 interval; An angle finding unit uses an inverse cosine function to find the corresponding angle of the CSI magnitude value at the timestamp t t ; An amplitude extraction unit is configured to represent the amplitude relationship corresponding to any two time stamps i and j by using a two-angle difference of a sine function, and the formula is represented as: A matrix generation unit is configured to generate a Gram angle difference field matrix, which is represented as: Each element in the matrix represents the amplitude relationship corresponding to two time stamps. An image generation unit is configured to generate a Gram angle difference field image according to the Gram angle difference field matrix.
5. An apparatus, comprising: comprise: one or more processors; memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs, when executed by the processors, implement the steps of the human action recognition method based on WiFi channel state information imaging according to any one of claims 1-3.
6. A readable storage medium, having stored thereon a computer program, characterized in that, The computer programs, when executed by the processors, implement the steps of the human action recognition method based on WiFi channel state information imaging according to any one of claims 1-3.