Human motion mode recognition method, device and system and storage medium

Abstract

The invention discloses a human motion mode recognition method, device and system and a storage medium, and the method comprises the steps: enabling a hand-held smart phone to move in a Wi-Fi signal coverage range, synchronously collecting Wi-Fi channel state information (CSI) and multi-source sensor data of a built-in accelerometer, a gyroscope and a barometer of the mobile phone; wherein nine types of common human body action data including walking, running, standing and jumping are included; performing preprocessing operations such as time synchronization, filtering and noise reduction, data fusion and dimension reduction on the acquired multi-source data; the processed multi-source data are input into a Fusion Transform network, the network captures a data time sequence relation through learnable position codes, and spatial-temporal features of multi-modal data are efficiently extracted by combining a multi-head self-attention mechanism and a feedforward neural network, so that the human body activity recognition precision is improved to 98%, the performance bottleneck of a single sensor is effectively overcome, and the recognition accuracy of human body activity is improved. The method has the advantages of low cost, wide coverage and no need of extra hardware support.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of human motion recognition, and particularly relates to a human motion pattern recognition method and device, system, and storage medium. BACKGROUND

[0002] Human activity recognition technology has important application value in the fields of smart home, health monitoring, motion tracking, etc. Traditional activity recognition methods mainly rely on wearable sensors, video monitoring, and photoelectric sensors, etc. Although these methods can accurately identify activity types, they often have the following problems: (1) Wearable sensors often limit the freedom of user activities, and the inconvenience of wearing and operating affects user experience; (2) Video monitoring methods are easily disturbed by factors such as occlusion and privacy protection, and have high energy consumption; (3) Recognition methods based on photoelectric sensors or other signal sources have certain accuracy, but in complex environments such as non-line-of-sight conditions, the accuracy and stability are poor.

[0003] In recent years, activity recognition based on Wi-Fi signals has gradually become a new non-contact technology. Wi-Fi CSI (Channel State Information) can monitor human activity by analyzing signal changes without the need to wear devices, and has the advantages of low cost, wide coverage, and no need for additional hardware support. However, relying solely on Wi-Fi CSI for activity recognition still faces some technical challenges: (1) Wi-Fi signals are easily disturbed by environmental interference such as reflection and multipath effect during transmission, resulting in large signal fluctuations; (2) Data timing synchronization and frequency offset problems of Wi-Fi CSI affect the stability of the data, thereby reducing the accuracy of recognition; (3) Wi-Fi CSI signals are difficult to capture some fine motion features, especially in the detection of complex motions or small amplitude motions.

[0004] In addition, the data of a single sensor often cannot fully describe the multi-dimensional features of human activity. For example, although accelerometers and gyroscopes can provide motion data in three-dimensional space, they can only capture static acceleration changes or rotation angles, making it difficult to fully recognize the subtle differences between some motions. Even the latest multi-sensor fusion technology still faces many challenges in sensor data synchronization, noise filtering, and multi-modal data fusion, resulting in insufficient accuracy and robustness in practical applications. SUMMARY

[0005] To solve the problems existing in the prior art, the present application provides a human motion pattern recognition method and device, system, and storage medium, which can effectively improve the recognition accuracy, overcome the limitations of a single sensor, and maintain efficient and real-time activity recognition performance in complex environments.

[0006] To achieve the above object, the present application provides the following scheme: A human motion pattern recognition method, comprising: S1: performing motion in a Wi-Fi signal coverage range through a handheld smart phone, and collecting multi-source sensor data of Wi-Fi CSI, accelerometer, gyroscope and barometer at different times; S2: pre-processing the multi-source sensor data, wherein the pre-processing includes time synchronization and filtering and noise reduction; S3: inputting the processed multi-source sensor data into a Fusion Transformer network to realize real-time detection and recognition of current human motion patterns.

[0007] Preferably, the multi-source sensor data includes nine common actions of walking, running, standing, jumping, bending, turning, lifting legs, clapping hands and sitting.

[0008] Preferably, the Fusion Transformer network structure uses a learnable classification mark to classify and recognize actions; and the Fusion Transformer network structure uses a learnable position encoding to capture the time sequence relationship of data.

[0009] The present application also provides a human motion pattern recognition device, comprising: A first processing module for performing motion in a Wi-Fi signal coverage range through a handheld smart phone, and collecting multi-source sensor data of Wi-Fi CSI, accelerometer, gyroscope and barometer at different times; A second processing module for pre-processing the multi-source sensor data, wherein the pre-processing includes time synchronization and filtering and noise reduction; A third processing module for inputting the processed multi-source sensor data into a Fusion Transformer network to realize real-time detection and recognition of current human motion patterns.

[0010] Preferably, the multi-source sensor data includes nine common actions of walking, running, standing, jumping, bending, turning, lifting legs, clapping hands and sitting.

[0011] Preferably, the Fusion Transformer network structure uses a learnable classification mark to classify and recognize actions; and the Fusion Transformer network structure uses a learnable position encoding to capture the time sequence relationship of data.

[0012] The application further provides a human motion pattern recognition system, comprising a memory and a processor, wherein the memory stores a computer program which is run by the processor, and the computer program performs the human motion pattern recognition method when run by the processor.

[0013] The application further provides a storage medium, wherein the storage medium stores a computer program which performs the human motion pattern recognition method when run.

[0014] The technical scheme of the application first designs and implements a 2x2 array Wi-Fi CSI receiver, which can synchronously collect and merge all Wi-Fi channel data within the CSI collectable range. The Wi-Fi CSI signal can realize non-intrusive accurate monitoring by analyzing the phase and amplitude changes of the signal. We use the smartphone sensor to collect rich motion information including acceleration, angular velocity and atmospheric pressure to supplement the subtle motion changes that the Wi-Fi signal cannot capture. In order to handle the time synchronization and feature extraction problems in multi-source data, the application uses interpolation algorithm and time-frequency analysis technology to ensure efficient fusion of sensor data. In addition, the deep learning model Fusion Transformer can adaptively extract spatial and temporal features in the data, and improve the recognition ability of the model for complex actions through multi-level feature fusion. This method significantly improves the accuracy and robustness of human activity recognition by fusing Wi-Fi channel state information (CSI) and multi-source sensor data of smartphones in edge devices and using the deep learning model Fusion Transformer. Compared with traditional methods, the application has stronger real-time and adaptability, and is particularly suitable for smart home, health monitoring and other application scenarios that require efficient and low-power human activity recognition. The following technical effects are achieved: 1. The application realizes more accurate human activity recognition by fusing Wi-Fi CSI signal and smartphone sensor data. Wi-Fi CSI signal can capture spatial features of human activity through phase and amplitude changes of the signal, while smartphone sensor supplements detailed information of motion. Through time synchronization algorithm, noise filtering and interpolation method, efficient fusion and processing of data from different sources are ensured, and the quality of data is improved, overcoming the limitations of single sensor.

[0015] 2. The application uses Fusion Transformer deep learning framework to capture spatial and temporal features in Wi-Fi CSI signal and sensor data simultaneously. The framework has strong feature extraction and fusion capability, and can effectively process and recognize complex activities. Through multi-level feature fusion and adaptive weighting mechanism, the accuracy of human activity recognition of Fusion Transformer reaches 98%.

[0016] 3. The application avoids dependence on cloud computing by processing sensor data on an edge computing platform, achieving low-latency, real-time human activity recognition. This solution not only reduces system power consumption, but also ensures the timeliness of data processing, providing an efficient solution for smart home and health monitoring applications. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the present application, the following briefly introduces the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0018] Figure 1 Flow chart of the human motion pattern recognition method of the embodiment of the present application; Figure 2 Fusion Transformer model network structure provided by the embodiment of the present application; Figure 3 Comparison chart of recognition accuracy of the Fusion Transformer model provided by the embodiment of the present application and other classic models; Figure 4 Confusion matrix of the Fusion Transformer model provided by the embodiment of the present application when identifying different motion patterns. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0020] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0021] Embodiment 1 As Figure 1As shown, this invention provides a method for human motion pattern recognition. A 2×2 array receiver based on an ESP32 microcontroller is developed and placed between the monitoring personnel and a Wi-Fi router. A serial port output program is written to decode the Wi-Fi CSI information from the ESP32. Real-time accelerometer, gyroscope, and barometer data are obtained by calling the Android sensor application interface built into a smartphone. Feature fusion and feature extraction are performed on the multimodal sensor data, and the extracted data is fed into the designed Fusion Transformer model. The Transformer encoder captures the temporal features of the data, and action classification is performed using learnable classification tokens (CLS tokens) to achieve accurate motion pattern recognition. Specifically, the method includes the following steps: S1: Deploy a Wi-Fi CSI receiver array and a smartphone sensor data acquisition system. The user moves within the Wi-Fi signal coverage area and collects data from Wi-Fi CSI, accelerometer, gyroscope, and barometer at different times. S2: Upload the collected data to the edge computing platform and preprocess the multi-source sensor data. The preprocessing includes time synchronization and filtering and noise reduction. S3: Input the processed multi-source sensor data into the Fusion Transformer network on the edge device to achieve real-time detection and recognition of various current human motion patterns.

[0022] As one embodiment of the present invention, step S1 includes: S1-1: Wi-Fi CSI data is acquired through a 2*2 Esp32 array. By complementing and comparing data between arrays, more stable and clearer feature data can be obtained. After Wi-Fi CSI is transmitted via OFDM modulation, a set of CSI values ​​can be obtained from each received data packet. Each set of CSI represents the amplitude and phase of an orthogonal frequency division multiplexing subcarrier. Channel state information communication can be represented as follows: ; in It is the CSI channel frequency response. It is the transmitted signal vector. It's a noise signal. It is the received signal vector. For the first CSI values ​​of each subcarrier, and They represent the first The amplitude and phase of each subcarrier.

[0023] S1-2: The smartphone multi-modal data is acceleration, angular velocity, and atmospheric pressure data, which is obtained by calling the Android sensor application programming interface (API) built into the smartphone. A three-axis accelerometer obtains linear acceleration data in the x, y, and z orthogonal directions (unit: m / s²) at a sampling frequency of 100 Hz, a three-axis gyroscope collects angular velocity information in the x, y, and z axes (unit: rad / s) at a frequency of 100 Hz, and a built-in barometer records the ambient air pressure value (unit: hPa) at a frequency of 1 Hz. The vertical height displacement is calculated by the rate of change of air pressure, ; wherein is the standard temperature, is the acceleration of gravity, is the initial air pressure, is the real-time air pressure.

[0024] S1-3: The motion patterns include walking, running, standing, jumping, bending, turning, lifting the leg, clapping, and sitting, which are nine common actions.

[0025] As an embodiment of the present application, in step S2, the following steps are included: S2-1: In order to achieve data time synchronization between the four ESP32 Wi-Fi CSI receivers and the smartphone sensor data, the clock obtained during multi-source sensor communication needs to be calibrated to ensure the accuracy and consistency of the data. During transmission, CSI data may be subject to packet loss, time delay, noise, and other problems. Resampling and bidirectional linear interpolation methods are used to fuse and filter noise of four-channel CSI data, ; wherein , is the resampling interval; is the interpolated CSI value, and are known data at adjacent time points, and are the time stamps of adjacent time points, is the time point that needs to be interpolated.

[0026] S2-2: Each data frame of CSI has 64*2 dimensions. To reduce the system calculation cost, principal component analysis (PCA) is used to reduce the dimension of CSI data, ; S2-3: Considering that the original multi-modal data has noise, it is necessary to perform outlier detection before inputting the prediction model, and discrete wavelet transform is used to filter and denoise the multi-modal data, soft threshold, db6 wavelet basis is selected for three-layer decomposition of the signal, ; wherein, is the third layer approximation coefficient, are the detail coefficients of each layer respectively, L is the sequence length, is the global threshold, is the soft threshold function, is the coefficient obtained after soft thresholding, is the signal after denoising, is the db6 wavelet basis function.

[0027] As an embodiment of the embodiment of the application, in the step S3, comprising: S3-1: The Fusion Transformer network structure uses a Transformer encoder, in which the multi-head self-attention mechanism and the feed-forward neural network enable the model to analyze the input data from multiple different angles, thereby effectively extracting key features in the data and processing complex relationships between different modal data. Let the dimension of the input data be , the number of heads be , and the dimension of each head be represented as: ; For the query matrix , the key matrix and the value matrix of the input data, linear transformation is performed respectively: ; wherein, , and are learnable weight matrices. For each head , the attention score is calculated respectively: ; are the query matrix, the key matrix and the value matrix of the first head respectively. Next, the outputs of all heads are spliced together, and the information of each head is integrated through a linear transformation layer: ; wherein, is a learnable weight matrix, is the attention matrix after splicing all heads.

[0028] S3-2: The Fusion Transformer network structure uses learnable classification tokens (CLS tokens) to classify and recognize actions. The CLS token vector is extracted from the encoder's output and passed through a classifier consisting of two fully connected layers and a ReLU activation function to output the classification result.

[0029] S3-3: The Fusion Transformer network structure uses learnable positional encoding to capture the temporal relationships of data. The positional encoding module uses a learnable parameter matrix initialized by Xavier and adds it to the input data during forward propagation, enabling the model to learn the inherent patterns of positional information in the data, thereby better capturing the changing characteristics of human activities over time.

[0030] Figure 2 This is the Fusion Transformer model provided in the embodiments of the present invention. Figure 3 and Figure 4 These figures show a comparison of various metrics obtained from motion pattern recognition prediction using the Fusion Transformer model provided in this invention and other common models, as well as the confusion matrix predicted by the Fusion Transformer model. As can be seen from the figures, the human motion pattern recognition method based on the multimodal sensor fusion-based Fusion Transformer model proposed in this invention can achieve more stable and superior prediction results compared to other common models.

[0031] Example 2 The present invention also provides a human motion pattern recognition device, comprising: The first processing module is used to collect multi-source sensor data at different times from Wi-Fi CSI, accelerometer, gyroscope and barometer by moving within the Wi-Fi signal coverage area using a handheld smartphone; The second processing module is used to preprocess the multi-source sensor data, and the preprocessing includes: time synchronization and filtering and noise reduction. The third processing module is used to input the processed multi-source sensor data into the Fusion Transformer network to achieve real-time detection and recognition of various current human motion patterns.

[0032] As one embodiment of the present invention, the multi-source sensor data includes nine common actions: walking, running, standing, jumping, bending over, turning, raising legs, clapping, and sitting.

[0033] As one embodiment of the present invention, the Fusion Transformer network structure uses learnable classification labels to classify and recognize actions; the Fusion Transformer network structure uses learnable positional encoding to capture the temporal relationship of data.

[0034] Example 3 The present invention also provides a human motion pattern recognition system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program performs a human motion pattern recognition method when executed by the processor.

[0035] Example 4 The present invention also provides a storage medium storing a computer program, which executes a human motion pattern recognition method when running.

[0036] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for recognizing human motion patterns, characterized in that, include: S1: By moving within the Wi-Fi signal coverage area using a handheld smartphone, multi-source sensor data from Wi-Fi CSI, accelerometer, gyroscope, and barometer are collected at different times; S2: Preprocess the multi-source sensor data, the preprocessing including: time synchronization and filtering and noise reduction; S3: Input the processed multi-source sensor data into the Fusion Transformer network to achieve real-time detection and recognition of various current human motion patterns.

2. The human motion pattern recognition method as described in claim 1, characterized in that, The multi-source sensor data includes nine common movements: walking, running, standing, jumping, bending over, turning, raising legs, clapping, and sitting.

3. The human motion pattern recognition method as described in claim 2, characterized in that, The Fusion Transformer network structure uses learnable classification labels to classify and recognize actions; the Fusion Transformer network structure uses learnable positional encoding to capture the temporal relationships of data.

4. A human motion pattern recognition device, characterized in that, include: The first processing module is used to collect multi-source sensor data at different times from Wi-Fi CSI, accelerometer, gyroscope and barometer by moving within the Wi-Fi signal coverage area using a handheld smartphone; The second processing module is used to preprocess the multi-source sensor data, and the preprocessing includes: time synchronization and filtering and noise reduction. The third processing module is used to input the processed multi-source sensor data into the Fusion Transformer network to achieve real-time detection and recognition of various current human motion patterns.

5. The human motion pattern recognition device as described in claim 4, characterized in that, The multi-source sensor data includes nine common movements: walking, running, standing, jumping, bending over, turning, raising legs, clapping, and sitting.

6. The human motion pattern recognition device as described in claim 5, characterized in that, The Fusion Transformer network structure uses learnable classification labels to classify and recognize actions; the Fusion Transformer network structure uses learnable positional encoding to capture the temporal relationships of data.

7. A human motion pattern recognition system, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executed by the processor, the computer program performing the human motion pattern recognition method as described in any one of claims 1-3 when executed by the processor.

8. A storage medium, characterized in that, The storage medium stores a computer program, which executes the human motion pattern recognition method as described in any one of claims 1-3 when it runs.

Images