Human motion mode recognition method, system and equipment based on channel state information and medium

Abstract

The invention belongs to the technical field of human motion recognition, and discloses a human motion mode recognition method, system and device based on channel state information, and a medium, and the method comprises the steps: obtaining the channel state information at different moments under different human motion states; performing time synchronization processing and time frequency analysis on the channel state information to obtain preprocessed channel state information; inputting the preprocessed channel state information into a human body motion mode recognition model for classification prediction to obtain a human body motion mode recognition result; wherein the human body motion pattern recognition model is integrated in a cloud, and the human body motion pattern recognition model is constructed based on a convolutional neural network and a long and short term memory network. According to the technical scheme, high-precision action recognition is achieved, and a high recognition effect can be kept in complex through-wall and non-line-of-sight environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of human motion recognition, and particularly relates to a human motion pattern recognition method, system, device and medium based on channel state information. Background Art

[0002] Human motion pattern recognition, as a core technology in fields such as navigation and positioning, intelligent healthcare, and health monitoring, demonstrates broad application potential. Traditional motion pattern recognition methods usually rely on different signal sources, such as optical signals, electrical signals, etc., and perform pattern recognition by analyzing the obstacles and feature changes encountered by these signals during propagation. Early research mainly focused on human behavior analysis and action understanding. Usually, wearable sensors were used to collect acceleration data of various parts and input it into a machine learning model as a judgment basis. In addition, effective recognition of motion patterns can also be achieved by analyzing human physiological signs (such as electrocardiogram (ECG) and photoplethysmogram (PPG)). However, the wearable sensor method still faces many challenges in practical applications.

[0003] Firstly, most systems require users to wear dedicated sensor devices, which restricts the freedom of movement of users and causes certain inconvenience in wearing and operation. Secondly, the types of sensors used in traditional methods are relatively single, and it is difficult to monitor multi-dimensional motion characteristics in real time and accurately, thereby affecting the accuracy of recognition. Although some gait recognition methods have been proposed in recent years, such as millimeter-wave radar, 5G technology, and wearable devices, the popularity of these methods is relatively low, and the acquisition of multi-dimensional features still faces great technical difficulties. In addition, vision-based recognition methods can effectively capture human activities, but they have problems such as blind spots and high energy consumption.

[0004] In contrast, Wi-Fi CSI (Channel State Information), as a monitoring technology that does not require wearing devices, has become a key technology in human motion pattern recognition due to its advantages of wide coverage, low cost, and real-time data acquisition. Wi-Fi CSI can accurately monitor human motion without additional hardware devices, especially suitable for applications in complex environments.

[0005] However, the human motion pattern recognition method based on Wi-Fi CSI still faces the following main challenges: (1) The data of a single Wi-Fi receiver fluctuates greatly, has high noise, and is prone to losing potential key information, which affects the effectiveness of the data collection stage and further affects the accuracy of subsequent recognition; (2) Traditional machine learning models have limitations in feature extraction and are difficult to capture the long-term and short-term characteristics of data simultaneously, resulting in the model failing to deeply mine the spatio-temporal information in the data; (3) Most current methods are only applicable to simple in-place action recognition and do not consider the application requirements in complex environments (such as through-wall or non-line-of-sight environments); (4) There are biases such as phase shift and frequency shift in Wi-Fi CSI data, which affect the accuracy of the data and thus limit the performance of the recognition model. Summary of the Invention

[0006] The purpose of the present invention is to provide a human motion pattern recognition method, system, device and medium based on channel state information to solve the problems existing in the above-mentioned prior art.

[0007] To achieve the above purpose, the present invention provides a human motion pattern recognition method based on channel state information, including:

[0008] Obtain the channel state information at different times under different human motion states;

[0009] Perform time synchronization processing and time-frequency analysis on the channel state information to obtain the preprocessed channel state information;

[0010] Input the preprocessed channel state information into a human motion pattern recognition model for classification prediction to obtain a human motion pattern recognition result; wherein, the human motion pattern recognition model is integrated and set in the cloud, and the human motion pattern recognition model is constructed based on a convolutional neural network and a long short-term memory network.

[0011] Optionally, the obtaining of the channel state information at different times under different human motion states specifically includes:

[0012] Deploy a channel state information receiving array device within the human motion range, and collect the channel state information at different times under different human motion states within the signal coverage range through the channel state information receiving array device.

[0013] Optionally, the performing of time synchronization processing and time-frequency analysis on the channel state information specifically includes:

[0014] Perform time synchronization processing on the channel state information, calibrate the local clock and the clock obtained during communication, and obtain the time-synchronized channel state information;

[0015] Perform filtering processing and time interpolation processing on the channel state information after time synchronization to obtain the filtered channel state information;

[0016] Extract the frequency domain characteristics corresponding to the filtered channel state information based on the discrete-time Fourier transform;

[0017] Perform outlier detection on the time domain data of the channel state information based on the Hampel algorithm to obtain the processed time domain characteristics, and complete the preprocessing of the channel state information.

[0018] Optionally, the training process of the human motion pattern recognition model specifically includes:

[0019] Obtain training data, where the training data includes channel state training information and the corresponding human motion pattern recognition results. Among them, the human motion pattern recognition results include walking, running, jumping, standing, turning in circles, bending over, lifting legs, and clapping hands;

[0020] Construct an initial human motion pattern recognition model, input the training data into the initial human motion pattern recognition model for classification prediction, and perform training with the goal of minimizing the loss between the initial training results after classification prediction and the human motion pattern recognition results corresponding to the channel state training information, to obtain a trained human motion pattern recognition model.

[0021] Optionally, the processing process of the human motion pattern recognition model specifically includes:

[0022] Extract the spatial characteristics of the channel state information based on the convolutional neural network, input the extracted spatial characteristics into the long short-term memory network for time series modeling, and introduce an attention mechanism to perform weighted summation on the outputs of different time steps of the long short-term memory network to obtain a weighted feature vector, and input the weighted feature vector into the prediction layer to output the corresponding human motion pattern recognition result.

[0023] A human motion pattern recognition system based on channel state information includes:

[0024] A data acquisition module for obtaining channel state information at different times under different human motion states;

[0025] A preprocessing module for performing time synchronization processing and time-frequency analysis on the channel state information to obtain preprocessed channel state information;

[0026] A human motion pattern recognition module for inputting the preprocessed channel state information into a human motion pattern recognition model for classification prediction to obtain a human motion pattern recognition result; among them, the human motion pattern recognition model is integrated and set in the cloud, and the human motion pattern recognition model is constructed based on a convolutional neural network and a long short-term memory network.

[0027] An electronic device includes a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the described method for human motion pattern recognition based on channel state information.

[0028] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the described method for human motion pattern recognition based on channel state information.

[0029] The technical effects of the present invention are as follows:

[0030] Through the CNN-LSTM-Attention model and data processing, the present invention realizes efficient and accurate human motion pattern recognition. In the data preprocessing stage, through steps such as time synchronization, filtering and noise reduction, outlier detection and correction, the data quality is effectively improved, providing more accurate input for the model. The model combines the feature extraction of convolutional neural networks, the time series modeling of long short-term memory networks, and the key information focusing ability of the attention mechanism, and can extract the most discriminative features from complex Wi-Fi CSI data. The present invention realizes high-precision action recognition and can also maintain a high recognition effect in complex wall-penetrating and non-line-of-sight environments, further expanding the application potential of Wi-Fi CSI signals in human action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings without creative efforts based on these drawings.

[0032] The drawings constituting a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application. In the drawings:

[0033] Figure 1 It is a flowchart for realizing accurate motion pattern recognition through a CSI receiver provided by an embodiment of the present invention;

[0034] Figure 2 It is a comparison chart of the non-line-of-sight recognition accuracy rates between the C-L-A model and other classic models provided by an embodiment of the present invention;

[0035] Figure 3 It is a confusion matrix of the C-L-A model for different motion pattern recognitions provided by an embodiment of the present invention. Detailed implementation manners

[0036] The various exemplary implementation manners of the present invention will be described in detail below. This detailed description should not be considered as a limitation on the present invention, but rather as a more detailed description of certain aspects, features, and implementation schemes of the present invention.

[0037] It should be understood that the terms described in the present invention are only used to describe specific implementation manners and are not used to limit the present invention. Additionally, for the numerical ranges in the present invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Each intermediate value within any stated value or stated range, as well as each smaller range between any other stated value or intermediate value within the stated range, is also included in the present invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0038] Without departing from the scope or spirit of the present invention, various improvements and changes can be made to the specific implementation manners of the specification of the present invention, which are obvious to those skilled in the art. Other implementation manners obtained from the specification of the present invention are obvious to those skilled in the art. The specification and embodiments of this application are merely exemplary.

[0039] Regarding the terms "comprising", "including", "having", "containing", etc. used herein, they are all open-ended terms, meaning including but not limited to.

[0040] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The following will refer to the accompanying drawings and combine with embodiments to detail this application.

[0041] As Figure 1 - Figure 3 shown, in this embodiment, a human motion pattern recognition method based on channel state information is provided, including: obtaining channel state information at different moments under different human motion states; performing time synchronization processing and time-frequency analysis on the channel state information to obtain preprocessed channel state information; inputting the preprocessed channel state information into a human motion pattern recognition model for classification prediction to obtain a human motion pattern recognition result; wherein, the human motion pattern recognition model is integrated and set in the cloud, and the human motion pattern recognition model is constructed based on a convolutional neural network and a long short-term memory network.

[0042] This embodiment provides a CNN-LSTM-Attention human motion pattern recognition system based on WiFi Channel State Information (CSI). This system uses the CSI data of WiFi signals to capture the impact of human motion on the wireless channel, employs a Convolutional Neural Network (CNN) to extract the spatial features of the CSI time series, models the time-dependent relationship through a Long Short-Term Memory network (LSTM), and introduces an Attention mechanism to assign importance weights to different time steps, thereby improving the accuracy and robustness of motion recognition. Experimental results show that the system described in this embodiment can effectively recognize various human motion patterns in multiple complex environments such as Line of Sight (LOS) and Non-Line of Sight (NLOS), has the advantages of low deployment cost, strong real-time performance, and high adaptability, and is suitable for fields such as smart home, behavior monitoring, and security protection.

[0043] First, this embodiment designs and implements a 2x2 array Wi-Fi CSI receiver that can synchronously collect and merge all Wi-Fi channel data within the CSI collectable range, thereby significantly enhancing the stability and reliability of the data. Subsequently, this embodiment proposes a deep learning framework that combines a Convolutional Neural Network (CNN), a Long Short-Term Memory network (LSTM), and an Attention mechanism. This framework can effectively capture the short-term and long-term temporal features in the motion data, thereby improving the recognition accuracy and robustness. Through this method, this embodiment not only achieves high-precision action recognition in a standard environment but also can maintain a high recognition effect in complex wall-penetrating and non-line-of-sight environments, further expanding the application potential of Wi-Fi CSI signals in human action recognition.

[0044] This embodiment develops a 2×2 array receiver based on ESP32 and places it between the detected person and the WiFi router. This embodiment writes a serial port output program for decoding the WiFi CSI information of ESP32. By reading the change pattern and trend of the WiFi CSI information, features are extracted, and the extracted data is put into the designed CNN-LSTM-Attention model. Features are extracted by CNN, time series modeling is performed by LSTM, and then the Attention mechanism leads to changes in the weights of the aforementioned model to achieve accurate motion pattern recognition.

[0045] Specifically, a method for recognizing human motion patterns based on Wi-Fi CSI using CNN-LSTM-Attention provided by this embodiment includes the following steps:

[0046] S1: Deploy a Wi-Fi CSI receiving array device within the range of motion, and at the same time, move within the Wi-Fi signal coverage area to collect Wi-Fi CSI data at different times under different motion states.

[0047] S2: Clean the data by synchronizing the time of the Wi-Fi CSI receiving array and performing time-frequency analysis based on filtering and noise reduction, delete outliers, and extract and enhance the features of the data;

[0048] S3: Transmit the processed Wi-Fi CSI information to the cloud through the mobile communication network, and input it into the designed CNN-LSTM-Attention network to achieve real-time detection and recognition of various current human motion patterns;

[0049] Implementable, in step S1, it includes:

[0050] S1-1: The Wi-Fi CSI receiving device is a 2*2 array composed of Esp32. More stable and clear feature data can be obtained through the complementarity and comparison of data between arrays. The motion patterns include eight common actions such as walking, running, jumping, standing, turning in circles, bending over, lifting legs, and clapping hands;

[0051] After Wi-Fi CSI is transmitted through OFDM modulation, the communication of the channel state information can be expressed as:

[0052] R = HS + N (1)

[0053]

[0054] Among them, R is the received channel information vector, S is the transmitted channel information vector, H is the equivalent frequency response characteristic of the CSI transmission process, N is the noise signal, τ is the receiver time offset, β is the random phase, and Z is the measurement error.

[0055] Implementable, in step S2, it includes:

[0056] S2-1: In order to synchronize the data time of four ESP32 Wi-Fi CSI receivers, in this embodiment, the local clock of ESP32 and the clock obtained during Wi-Fi communication are calibrated to ensure data synchronization, which is convenient for subsequent interpolation or sorting processing. CSI data is affected by carrier frequency offset (CFO), sampling frequency offset (SFO), and error offset during analog-to-digital conversion. The noise filtering methods include mean filtering and time interpolation.

[0057]

[0058] Among them, x[n] is the received Wi-Fi CSI data sequence containing noise, and the data y[n] is obtained after mean filtering. At the same time, a smoother curve is obtained by cubic sequence interpolation.

[0059]

[0060] In the formula, S i (x) is the fitting curve of the i-th segment of data, and a third-order polynomial is used for fitting.

[0061] Implementable, it also includes:

[0062] S2-2: Considering the distinction of different action frequencies, through the discrete-time Fourier transform (DTFT), the frequency-domain feature extraction of the original data is completed.

[0063]

[0064] In the formula, S(n) is the sequence obtained from the fitting curve S i (x), and the obtained frequency-domain sequence S[k] is also used as data input to the prediction model.

[0065] Implementable, it also includes:

[0066] S2-3: For time-domain data, outlier detection is also required before inputting it into the prediction model to avoid incorrect prediction results caused by outliers due to measurement errors or data loss. The Hampel algorithm is used.

[0067] m = median(s[n-k], s[n-k+1], …, s[n+k]) (6)

[0068] After obtaining the median m of the sequence s[n], calculate the median of the absolute deviations between all points within the window and the median m:

[0069] MAD = median(|s[n]-m|, |s[n+1]-m|, …) (7)

[0070] For each data point s[n], calculate its standardized deviation:

[0071] z n = |s[n]-m| / MAD (8)

[0072] If z[n] exceeds the given threshold λ (set to 3), then s[n] is considered an outlier and is replaced by the median:

[0073] s[n] new = m (9)

[0074] The Hampel algorithm is more robust to outliers in the data because it relies on the median and the median absolute deviation, which are insensitive to outliers.

[0075] Implementable, in step S3, it includes:

[0076] S3-1: In the CNN-LSTM-Attention network structure, the CNN model obtains the output layer through convolution. Assume that the input matrix after arrangement is X extend , W is the convolution kernel, and b is the bias term.

[0077] Z = Conv2D(X extend , W) + b (10)

[0078] The output Z of the convolutional layer passes through an activation function (such as ReLU) to obtain the activation value:

[0079] A = ReLU(Z) (11)

[0080] Implementable, it also includes:

[0081] S3-2: In the CNN-LSTM-Attention network structure, the features extracted by the CNN are modeled for time series through the LSTM. The core of the LSTM is its three-part gating mechanism:

[0082] Forget Gate:

[0083] f t = σ(W f · [h t-1 , x t + b f ) (12)

[0084] Input Gate:

[0085] i t = σ(W i · [h t-1 , x t + b i ) (13)

[0086] Candidate Memory Cell:

[0087] C~ t = tanh(W C · [h t-1 , x t + b C ) (14)

[0088] Update memory cell (Cell State):

[0089] C t = f t ·C t-1 + i t ·C~ t (15)

[0090] Output Gate:

[0091] o t = σ(W o ·[h t-1 , x t + b o ) (16)

[0092] Final output (Hidden State):

[0093] h t = o t ·tanh(C t ) (17)

[0094] Implementable, further including:

[0095] S3-3: The output of LSTM in the CNN-LSTM-Attention network structure is weighted by the Attention mechanism for different time steps of the output.

[0096] Assume the hidden state output by LSTM is h t , and then calculate the weighted hidden state through the attention mechanism:

[0097] α t = softmax(w a ·h t ) (18)

[0098] where, w a is the learned weight vector, and α t is the attention weight corresponding to each time step t.

[0099] Perform weighted summation, and h att is the final output:

[0100]

[0101] The final output h att is sent to the subsequent layer for decision-making or prediction.

[0102] Figure 2 and Figure 3They are respectively the comparison chart of various indicators obtained by the C-L-A model provided in this embodiment and the remaining common models for action pattern recognition prediction, and the confusion matrix predicted by the C-L-A model. It can be seen from the figure that the human motion pattern recognition method based on Wi-Fi CSI and the C-L-A model proposed in this embodiment can obtain more stable and excellent prediction conclusions compared with other common models.

[0103] In summary, this embodiment proposes a system based on a 2x2 array WiFi CSI receiver, which can scan and merge all WiFi channel data within a range of 20 meters. Compared with a single WiFi receiver, the data stability is improved by 10%.

[0104] This embodiment designs a composite human motion pattern recognition framework that integrates CNN, LSTM, and Attention. Combining the advantages of different deep learning models, it has stronger generalization ability, data focusing ability, and accuracy compared with traditional CNN, LSTM, and Transformer models, and the recognition accuracy is close to 99%.

[0105] In addition to the line-of-sight environment, the system developed in this embodiment can also work effectively in a non-line-of-sight scenario where the signal passes through walls, and the action recognition accuracy reaches 95%.

[0106] Implementably, this embodiment also provides a human motion pattern recognition system based on channel state information, including:

[0107] A data acquisition module, configured to obtain channel state information at different times under different human motion states;

[0108] A preprocessing module, configured to perform time synchronization processing and time-frequency analysis on the channel state information to obtain preprocessed channel state information;

[0109] A human motion pattern recognition module, configured to input the preprocessed channel state information into a human motion pattern recognition model for classification prediction to obtain a human motion pattern recognition result; wherein, the human motion pattern recognition model is integrated and set in the cloud, and the human motion pattern recognition model is constructed based on a convolutional neural network and a long short-term memory network.

[0110] Implementably, this embodiment also provides an electronic device, including a memory and a processor, where the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned human motion pattern recognition method based on channel state information.

[0111] Implementable, this embodiment further provides a computer-readable storage medium storing a computer program, which when executed by a processor implements the method for human motion pattern recognition based on channel state information described above.

[0112] As described above, the above are only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A human motion pattern recognition method based on channel state information, characterized in that: include: Obtain channel state information at different times under different human motion states; Performing time synchronization processing and time-frequency analysis on the channel state information to obtain preprocessed channel state information; The preprocessed channel state information is input into a human motion pattern recognition model for classification prediction to obtain a human motion pattern recognition result; wherein the human motion pattern recognition model is integrated in a cloud setting, and the human motion pattern recognition model is constructed based on a convolutional neural network and a long short-term memory network.

2. The human body motion pattern recognition method based on channel state information according to claim 1, characterized in that: The obtaining of channel state information at different times under different human motion states specifically includes: A channel state information receiving array device is deployed within the range of human motion, and channel state information at different times under different human motion states within the signal coverage range is collected through the channel state information receiving array device.

3. The human body motion pattern recognition method based on channel state information according to claim 1, characterized in that: The performing time synchronization processing and time-frequency analysis on the channel state information specifically includes: Performing time synchronization processing on the channel state information, calibrating a local clock and a clock obtained during communication, and obtaining the channel state information after time synchronization; Performing filtering and time interpolation processing on the time-synchronized channel state information to obtain filtered channel state information; Extract frequency domain features corresponding to the filtered channel state information based on discrete time Fourier transform; Based on the Hampel algorithm, outlier detection is performed on the time domain data of the channel state information to obtain the processed time domain features and complete the preprocessing of the channel state information.

4. The human body motion pattern recognition method based on channel state information according to claim 1, characterized in that: The training process of the human motion pattern recognition model specifically includes: Acquire training data, wherein the training data includes channel state training information and corresponding human motion pattern recognition results, wherein the human motion pattern recognition results include walking, running, jumping, standing, turning in circles, bending over, lifting legs, and clapping; An initial human motion pattern recognition model is constructed, the training data is input into the initial human motion pattern recognition model for classification prediction, and training is performed with the goal of minimizing the loss between the initial training result after classification prediction and the human motion pattern recognition result corresponding to the channel state training information to obtain a trained human motion pattern recognition model.

5. The human body motion pattern recognition method based on channel state information according to claim 1, characterized in that: The processing process of the human motion pattern recognition model specifically includes: The spatial features of the channel state information are extracted based on the convolutional neural network, and the extracted spatial features are input into the long short-term memory network for time series modeling. The attention mechanism is introduced to perform weighted summation on the outputs of the long short-term memory network at different time steps to obtain a weighted feature vector. The weighted feature vector is input into the prediction layer, and the corresponding human motion pattern recognition result is output.

6. A human motion pattern recognition system based on channel state information, characterized in that: include: A data acquisition module is used to obtain channel state information at different times under different human motion states; A preprocessing module, used to perform time synchronization processing and time-frequency analysis on the channel state information to obtain preprocessed channel state information; The human motion pattern recognition module is used to input the preprocessed channel state information into the human motion pattern recognition model for classification prediction to obtain the human motion pattern recognition result; wherein, the human motion pattern recognition model is integrated in the cloud setting, and the human motion pattern recognition model is constructed based on a convolutional neural network and a long short-term memory network.

7. An electronic device, characterized in that: It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute a human motion pattern recognition method based on channel state information according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: A computer program is stored therein, and when the computer program is executed by a processor, a human motion pattern recognition method based on channel state information as described in any one of claims 1 to 5 is implemented.

Images