A multi-modal human motion recognition system and method based on RFID passive sensing

Abstract

The application discloses a kind of multi-modal human action recognition system and method based on RFID passive sensing, belong to human action recognition technical field, the system includes RFID data acquisition module, data processing module, feature extraction module and human action recognition module;Through the RSSI data of receiving antenna under different actions is collected, and time-frequency chart is obtained using wavelet transform, constitute multi-modal data for human action recognition, by extracting the features corresponding to different actions is recognized;For the characteristics of one-dimensional signal action data and two-dimensional time-frequency chart action data, respectively select random forest and CNN-LSTM classification model to recognize action, and the recognition results of two kinds of models are decision fusion, output the final action category decision result, realize the passive sensing of human action.The privacy of the user is protected, the convenience of the system is greatly improved by the characteristics of passive sensing, and a new path is opened up for the health monitoring of homebound old people.

Description

Technical Field

[0001] This invention belongs to the field of human motion recognition technology, specifically relating to a multimodal human motion recognition system and method based on RFID passive sensing. Background Technology

[0002] Human motion recognition is a crucial technology essential for the development of fields such as elderly care, medical nursing, and fitness. It can prevent falls and other dangers for elderly people living at home, assist patients in correcting and rehabilitating their movements, and reduce the difficulty of care. Traditional human motion recognition methods primarily rely on machine vision and wearable sensors. Machine vision-based methods capture human actions using cameras and then use deep learning algorithms to classify the captured videos or images. Wearable sensor-based methods involve wearing miniature sensing devices such as accelerometers or inertial sensors to collect motion information such as acceleration and angle during human activity, extracting motion features, and then combining these features with classification algorithms to achieve motion classification.

[0003] However, machine vision-based methods are susceptible to limitations imposed by lighting conditions when collecting motion data, resulting in lower video image clarity and increasing the difficulty of motion recognition. Furthermore, using cameras to collect motion data can easily lead to user privacy leaks. Wearable sensor-based methods are limited by the high energy consumption caused by frequent information interaction in active environments, requiring regular device maintenance.

[0004] Due to its widespread availability and privacy-preserving characteristics, Radio Frequency Identification (RFID) technology has opened up new research avenues for human behavior recognition technology. Wireless signals not only play a crucial role in data transmission but can also be used for motion sensing. During propagation, wireless signals encounter obstacles or moving individuals, resulting in reflection, diffraction, and scattering. Signal strength changes under the influence of different movements. Compared to the drawbacks of the two aforementioned human behavior recognition methods, using RFID technology for passive human movement sensing has become a developing trend. Summary of the Invention

[0005] To address the limitations of existing machine vision-based methods, such as their susceptibility to ambient lighting and privacy breaches, and the limitation of wearable sensors to active environments, this invention proposes a multimodal human motion recognition system and method based on passive RFID sensing. This invention utilizes the influence of human motion on wireless signal propagation, attaching RFID tags to human movement nodes. Based on changes in the received signal strength index (RSSI) of the receiving antenna, features corresponding to different actions are extracted. Wavelet transform is applied to the collected RSSI motion data to obtain corresponding time-frequency maps. Different classification algorithms are selected to construct classification models based on the features of the one-dimensional RSSI data and the two-dimensional time-frequency maps. Finally, decision fusion of the two models achieves multimodal human motion recognition in an RFID environment.

[0006] This invention is achieved through the following technical solution:

[0007] A multimodal human motion recognition method based on RFID passive sensing specifically includes the following steps:

[0008] Step 1: Attach RFID tags to human movement nodes, synchronously collect RSSI data from the multi-tag channel receiving antennas during the actions performed, arrange the collected action data according to time series, and import them into the corresponding action category tables to create a one-dimensional signal action dataset for human action recognition.

[0009] Step 2: Use Gaussian filtering to denoise the acquired one-dimensional signal action dataset, filtering out interference values ​​with large errors to smooth the signal; then use deviation standardization to normalize the data, thereby improving the convergence speed and accuracy of the classification model.

[0010] Step 3: Extract features from the denoised and normalized one-dimensional signal action dataset. Use a sliding window to extract the time-domain features of the signal, including the mean, variance, and standard deviation. Perform wavelet analysis on the signal to extract the approximate and detail components, and finally construct the feature vector.

[0011] Step 4: Use wavelet transform to perform time-frequency analysis on the denoised and normalized one-dimensional signal action dataset to obtain the time-frequency map under each type of action, and create a two-dimensional time-frequency map action dataset for human action recognition, thus forming multimodal human action recognition data.

[0012] Step 5: For the feature vectors from Step 3, use the feature vectors as input to the random forest classification model for classification and recognition, and introduce a grid search algorithm to optimize the parameters and find the global optimum.

[0013] For the two-dimensional time-frequency map action dataset in step four, a CNN-LSTM network model is built for classification and recognition.

[0014] Step Six: Use the maximum probability criterion to fuse the recognition results of the two classification models. The maximum probability criterion and the formula for the final discrimination result are as follows:

[0015] P j (n)=max(P ij (n)), where i = 1, 2, j = 1, 2, 3, 4, 5, 6, 7, 8;

[0016] P j ′(n)=P j (n) / Σ j P j (n);

[0017] Where i and j represent the i-th classification model and the j-th action category, P ij (n) represents the probability value that the i-th classification model classifies the action as the j-th action category, P j (n) represents the maximum probability of classifying the j-th action in the i-th classification model, P j ’ (n) represents the probability value that the action belongs to the j-th action category after decision fusion.

[0018] Furthermore, the human motion nodes mentioned in step one include motion nodes of the arms, abdomen, and legs; the actions performed include eight categories of actions: standing, sitting, lying down, walking, running, jumping, standing up, and falling down.

[0019] Furthermore, the normalization process in step two involves performing a linear transformation on the original data to map the results to the 0-1 interval, thereby improving the convergence speed and accuracy of the model. The deviation standardization formula is as follows:

[0020]

[0021] Where x is the initial value of RSSI data, max is the maximum value of RSSI data, min is the minimum value of RSSI data, and x* is the normalized RSSI value.

[0022] Furthermore, the continuous wavelet transform formula used in step four is as follows:

[0023]

[0024] Where a and τ are the scaling factor and translation factor, respectively, f(t) is the signal to be wavelet transformed, and ψ a,τ (t) is the wavelet function, and W(a,τ) is the wavelet sequence after wavelet transform of the signal.

[0025] Furthermore, a hybrid attention mechanism is introduced into the CNN-LSTM network model in step five. The hybrid attention mechanism includes a spatial attention module (SAM) and a temporal attention module (TAM). The SAM module is placed after the convolutional layer, and the important features are given greater weight by allowing the model to learn the correlation between different features. The TAM module is added after the LSTM layer, allowing the model to mine discrimination clues from adjacent action data.

[0026] On the other hand, the present invention also provides a multimodal human motion recognition system based on RFID passive sensing, including an RFID data acquisition module, a data processing module, a feature extraction module, and a human motion recognition module. The RFID data acquisition module is used to collect RSSI motion data from the multi-tag channel receiving antenna under human motion. The data processing module is used to denoise and normalize the collected RSSI motion data. The feature extraction module is used to extract the time-domain and frequency-domain features of the processed RSSI motion data as feature vectors, and use wavelet transform to obtain a two-dimensional time-frequency graph motion dataset to form multimodal data for human motion recognition. The human motion recognition module consists of two classification models, random forest and CNN-LSTM, which are used to classify one-dimensional signal motion data and two-dimensional time-frequency graph motion data, respectively. The classification results of the two models are fused to output the final action category.

[0027] Furthermore, the RFID data acquisition module consists of an RFID tag, an external antenna, a reader / writer, and a multi-channel analysis system. The RFID tag is fixed at a human movement node, and the reader / writer is connected to the external antenna. The RFID tag is used to receive interrogation signals from the reader / writer and to send backscatter signals. The external antenna is used to receive and transmit radio frequency signals. The reader / writer is used to perform the reading and writing functions of the RFID tag and to communicate with the multi-channel analysis system. The multi-channel analysis system obtains the frequency response of different channels in different frequency bands by switching antenna paths, which is used for importing and exporting data.

[0028] Furthermore, the RFID tag is a passive tag that operates in the ultra-high frequency (UHF) band and adopts the ISO18000-6C (EPC C1 G2) standard. The tag has anti-collision performance.

[0029] Compared with the prior art, the advantages of the present invention are as follows:

[0030] This invention discloses a multimodal human motion recognition system and method based on RFID passive sensing. By collecting RSSI motion data from the receiving antenna during human motion and analyzing the signal characteristics under different motions, passive sensing for human motion recognition is achieved. Compared to wearable sensor methods, this significantly improves the system's convenience. Furthermore, from a multimodal data perspective, a one-dimensional signal dataset and a two-dimensional time-frequency graph dataset are created for human motion recognition, protecting user privacy while achieving motion recognition. For the one-dimensional signal dataset and the two-dimensional time-frequency graph dataset, two different classification models, random forest and CNN-LSTM, are selected respectively, and the classification results of the two models are fused to avoid the limitations of manual feature extraction. Attached Figure Description

[0031] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0032] Figure 1 This is a system block diagram of a multimodal human motion recognition system based on RFID passive sensing according to the present invention;

[0033] Figure 2 This is a flowchart illustrating a multimodal human motion recognition method based on RFID passive sensing according to the present invention.

[0034] Figure 3 This is a schematic diagram of the one-dimensional signal action dataset of the present invention;

[0035] Figure 4 This is a schematic diagram of a portion of the two-dimensional time-frequency graph action dataset of the present invention. Detailed Implementation

[0036] To clearly and completely describe the technical solution and its specific working process of the present invention, the specific embodiments of the present invention are as follows, in conjunction with the accompanying drawings:

[0037] Example 1

[0038] like Figure 1The diagram shown is a system block diagram of a multimodal human motion recognition system based on RFID passive sensing in this embodiment. The recognition system includes an RFID data acquisition module, a data processing module, a feature extraction module, and a human motion recognition module. The RFID data acquisition module is used to collect RSSI motion data from the multi-tag channel receiving antenna under human motion. The data processing module is used to denoise and normalize the collected RSSI motion data. The feature extraction module is used to extract the time-domain and frequency-domain features of the processed RSSI motion data, and use wavelet transform to obtain a two-dimensional time-frequency map motion dataset to form multimodal data for human motion recognition. The human motion recognition module consists of two classification models, random forest and CNN-LSTM, which are used to classify one-dimensional signal motion data and two-dimensional time-frequency map motion data, respectively. The classification results of the two models are fused to output the final action category.

[0039] In this embodiment, the RFID data acquisition module consists of an RFID tag, an external antenna, a reader / writer, and a multi-channel analysis system. The RFID tag is fixed at a human movement node and is used to receive interrogation signals from the reader / writer and to send backscatter signals. The external antenna is used to receive and transmit radio frequency signals. The reader / writer is used to perform the reading and writing functions of the RFID tag and to communicate with the multi-channel analysis system. The multi-channel analysis system obtains the frequency response of different channels in different frequency bands by switching antenna paths and is responsible for importing and exporting data.

[0040] In this embodiment, the RFID tag is a passive tag that operates in the ultra-high frequency (UHF) band and adopts the ISO18000-6C (EPC C1 G2) standard. The tag has anti-collision performance.

[0041] Example 2

[0042] like Figure 2 As shown, this embodiment provides a multimodal human motion recognition method based on RFID passive sensing, including the following steps:

[0043] Step 1: Place one transmitting antenna and three receiving antennas in the experimental space. Bind three RFID tags (Tag0, Tag1, Tag2) to the movement nodes of the human body's arms, abdomen, and legs, respectively. The three tags and the three receiving antennas form nine channels. The sampling rate of the entire system is 421 Sample / s, and the sampling rate allocated to a single channel is 46.8 Sample / s. The human body faces the antenna and makes eight different movement categories: standing, sitting, lying down, walking, running, jumping, standing up, and falling down. The RFID data acquisition module simultaneously collects RSSI data from the nine receiving antennas under the human body's movements for 10 seconds.

[0044] like Figure 3 As shown, RSSI data of the receiving antennas were collected for eight different actions. For each action category, 40 experiments were conducted, for a total of 320 experiments. Each experiment lasted 10 seconds, and each experiment collected an average of 470 data points. The entire one-dimensional signal action dataset contains 151,174 data points. Tag0, Tag1, and Tag2 represent three tags bound to the arm, abdomen, and leg movement nodes, respectively. T1 represents the transmitting antenna, and R1, R2, and R3 represent the three receiving antennas. Trial represents the 320 experiment numbers. Activity represents the eight action categories: standing, sitting, lying down, walking, running, jumping, standing up, and falling. Code represents the code corresponding to each action category, using numbers 0 to 7 to encode the data for each of the eight action categories.

[0045] Step two involves preprocessing the collected RSSI motion data. Gaussian filtering is used for noise reduction. After filtering, deviation standardization is applied for normalization, which involves linearly transforming the original data to map the results to the 0-1 interval, thereby improving the model's convergence speed and accuracy. The deviation standardization formula is as follows:

[0046]

[0047] Where x is the initial value of RSSI data, max is the maximum value of RSSI data, and min is the minimum value of RSSI data. * The normalized RSSI value;

[0048] Step three involves extracting the time-domain features of the signal using a sliding window, including the mean, variance, and standard deviation. Wavelet analysis is then used to obtain the energy features of the signal at each level, extracting the approximate and detail components to form a feature vector that serves as the input to the random forest classification model. Finally, a continuous wavelet transform (CWT) is performed on the preprocessed RSSI action data to obtain two-dimensional time-frequency plots for each action class. The CWT formula is as follows:

[0049]

[0050] Where a and τ are the scaling factor and translation factor, respectively, f(t) is the signal to be wavelet transformed, and ψ a，τ (t) is the wavelet function, and W(a,τ) is the wavelet sequence after wavelet transform of the signal.

[0051] like Figure 4As shown, when creating the two-dimensional time-frequency motion dataset, the sampling rate was set to 47Hz, the wavelet function used was the Morse wavelet, and the preprocessed RSSI motion data was subjected to continuous wavelet transform in the form of a sliding window. The sliding window size was 47 data points, the sliding step size was 1 second, and a total of 14,121 two-dimensional time-frequency images of the eight motions were generated. The width and height of the time-frequency images were both 224 pixels.

[0052] Step 4: Build a random forest model to classify and identify one-dimensional signal action data. Introduce a grid search algorithm to arrange and combine the possible values ​​of each parameter, list all possible combinations to generate a grid for continuous training of the model, and use ten-fold cross-validation to continuously change the training set and test set, thereby automatically adjusting the optimal parameters to generate the global optimum.

[0053] Step 5: Build a CNN-LSTM model to classify the two-dimensional time-spectrum action data. In order to improve the model's ability to extract important features and better learn the temporal correlation between action data, a hybrid attention mechanism is introduced, which includes a spatial attention module (SAM) and a temporal attention module (TAM). The SAM module is placed after the convolutional layer. By allowing the model to learn the correlation between different features, more weight is given to important features. The TAM module is added after the LSTM layer, allowing the model to mine discrimination clues from adjacent action data.

[0054] Step six: Use the maximum probability criterion to fuse the results of the two classification models, Random Forest and CNN-LSTM, and output the final action category judgment result; determine whether a dangerous event such as a fall has occurred based on the classification result, and if a dangerous event has occurred, send a warning message to the relevant personnel.

[0055] Applying the human motion recognition method described in this embodiment to perform motion recognition on the collected RSSI motion data, compared with the prior art, using the random forest classification model, the accuracy can be improved from 88.41% to 95.54%. It can be seen that this method can indeed improve the accuracy of human motion recognition, providing more options for people who are healthy and elderly and those who are active and fit.

[0056] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solution of the present invention, and these simple modifications all fall within the protection scope of the present invention.

[0057] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not describe the various possible combinations separately.

[0058] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed by the present invention.

Claims

1. A multi-modal human action recognition method based on RFID passive sensing, characterized in that, Specifically comprising the following steps: Step one: bind the RFID tag at the human motion node, synchronously collect the RSSI data of the multiple tag channel receiving antennas under the action, arrange the collected action data in time sequence, and respectively import into the corresponding action category table to prepare the one-dimensional signal action data set for human action recognition; Step two: adopt Gaussian filtering to denoise the collected one-dimensional signal action data set, filter out the interference values with large errors, and make the signal smooth; then adopt dispersion standardization to normalize the data, thereby improving the convergence speed and accuracy of the classification model; Step three: extract the features of the denoised and normalized one-dimensional signal action data set, adopt sliding window to extract the time domain features of the signal, including the mean, variance and standard deviation; extract the approximation component and detail component of the signal through wavelet analysis, and finally form the feature vector; Step four: use wavelet transform to perform time-frequency analysis on the denoised and normalized one-dimensional signal action data set, obtain the time-frequency graph under each action, prepare the two-dimensional time-frequency graph action data set for human action recognition, and thus form the multi-modal human action recognition data; Step five: for the feature vector of step three, take the feature vector as the input of the random forest classification model, perform classification recognition, and introduce the grid search algorithm to optimize the parameters and find the global optimal value; For the two-dimensional time-frequency graph action data set of step four, build a CNN-LSTM network model to perform classification recognition; Step six: adopt the maximum probability criterion to make decision fusion on the recognition results of the two classification models, and the maximum probability criterion and the final discrimination result formula are as follows: P j (n) = max(P ij (n)), where i = 1,2, j = 1,2,3,4,5,6,7,8. P j ’ (n) = P j (n) / ∑ j P j (n); wherein i and j are the i-th classification model and the j-th action category, P ij (n) is the probability value of the i-th classification model judging the action as the j-th action category, j (n) is the maximum value of the i-th classification model judging the j-th action category, j ’ (n) is the probability value of the action being the j-th action category after decision fusion.

2. The method of claim 1, wherein the method is based on RFID passive sensing. The human motion nodes in step one include the motion nodes of arms, abdomen and legs; the actions include eight action categories of standing, sitting, lying, walking, running, jumping, standing up and falling down.

3. The method of claim 1, wherein the method is based on RFID passive sensing. The normalization processing in step two is a linear transformation on the original data, which maps the results to the interval of 0-1, thereby improving the convergence speed and accuracy of the model; the dispersion standardization formula is as follows: Wherein, x is the initial value of the RSSI data, max is the maximum value of the RSSI data, min is the minimum value of the RSSI data, and x* is the normalized RSSI value.

4. The method of claim 1, wherein the method is based on RFID passive sensing. The continuous wavelet transform formula used in step four is as follows: where a and τ are scale and translation factors, respectively, f(t) is the signal being wavelet transformed, and ψ a,τ (t) is the wavelet function, and W(a, τ) is the wavelet sequence of the signal after wavelet transformation.

5. The method of claim 1, wherein the method is based on RFID passive sensing. In the CNN-LSTM network model in step five, a hybrid attention mechanism is introduced, which includes a spatial attention module (SAM) and a temporal attention module (TAM); the SAM module is placed after the convolution layer, which makes the model learn the relevance between different features and gives more weight to important features; the TAM module is added after the LSTM layer, which makes the model dig out the discrimination clues from the adjacent action data.

6. A multi-modal human motion recognition system based on RFID passive sensing, characterized in that, It includes an RFID data acquisition module, a data processing module, a feature extraction module and a human action recognition module; the RFID data acquisition module is used to collect the RSSI action data of the multiple tag channel receiving antennas under the action of the human body, and the data processing module is used to denoise and normalize the collected RSSI action data; The feature extraction module is configured to extract time domain features and frequency domain features of the processed RSSI motion data as feature vectors, and obtain a two-dimensional time-frequency graph motion data set by using wavelet transform to form a multi-modal data for human motion recognition. The human motion recognition module is composed of a random forest and a CNN-LSTM classification model, and is configured to classify one-dimensional signal motion data and two-dimensional time-frequency graph motion data respectively, and to fuse classification results of the two models to output a final decision of a motion category.

7. The multi-modal human action recognition system based on RFID passive sensing of claim 6, wherein, The RFID data acquisition module is composed of an RFID tag, an external antenna, a reader and a multi-channel analysis system, the RFID tag is fixed at a human motion node, the reader is connected with the external antenna, the RFID tag is configured to receive an interrogation signal of the reader and send a backscattering signal, the external antenna is configured to receive and transmit a radio frequency signal, the reader is configured to complete a read-write function of the RFID tag and communication with the multi-channel analysis system, and the multi-channel analysis system is configured to obtain frequency response conditions of different channels in different frequency bands by switching an antenna path, and is configured to import and export data.

8. The multi-modal human action recognition system based on RFID passive sensing of claim 7, wherein, The RFID tag is a passive tag, works in an ultra-high frequency band, adopts an ISO18000-6C (EPC C1 G2) standard as a protocol, and has a conflict prevention performance.

Images