Human action recognition method based on phase information and signal strength

By collecting and processing phase data and signal strength in an indoor environment, and using CNN deep neural network for identification, the wearing discomfort and contact restriction problems of gesture action recognition in the prior art are solved, and non-contact action without wearing a tag is achieved Identify and interact in real time.

CN113935373BActive Publication Date: 2024-11-19NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111182962.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-11
Publication Date
2024-11-19
Estimated Expiration
2041-10-11

AI Technical Summary

Technical Problem

The existing gesture action recognition technology has discomfort in wearing sensors, privacy issues and contact-based restrictions, making it difficult to achieve contactless real-time action recognition, which limits its application and development.

Method used

Using a human body motion recognition method based on phase information and signal strength, by collecting phase data and signal strength in an indoor environment, pre-processing and discrete wavelet transformation, constructing a matrix, using CNN deep neural network for action recognition model training, realizing non-contact Style action recognition.

Benefits of technology

It realizes contactless action recognition without wearing tags, reduces costs, has good applicability, realizes real-time action recognition and intelligent interaction, and is suitable for applications such as somatosensory games.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113935373B_ABST
    Figure CN113935373B_ABST
Patent Text Reader

Abstract

A human motion recognition method based on phase information and signal strength, the implementation steps of which include: collecting phase information and signal strength data of human motion in an indoor environment, extracting phase information and signal strength; performing preprocessing steps such as phase data unwrapping, smoothing filtering, data normalization, and signal strength data smoothing filtering on the phase information; segmenting the preprocessed signal, detecting the start and end points of the action, and dividing the segmented data segments into a training set and a test set; by inputting the training set into a deep neural network based on CNN to train a human motion recognition model, a human motion recognition model is obtained, and the model can be used to classify the collected phase and signal strength test set data to achieve the purpose of human motion recognition. The present invention uses a deep neural network based on CNN to automatically learn and select features to achieve recognition of seven different behavioral actions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of human perception and mobile behavior recognition, and to a recognition detection method for human motion recognition based on phase information and signal strength. Background Art

[0002] At present, many large shopping malls are equipped with smart screens, and smart interaction has become a new way of interaction. Especially under the influence of the COVID-19 virus, reducing contact can avoid many unnecessary risks. Motion recognition has attracted extensive research, such as the FootprintID system, which performs reliable indoor pedestrian recognition by using vibration sensors to detect vibrations caused by footsteps. However, it requires many sensors to cover the sensing area. DEEM is a motion detection that combines computer vision and RFID to identify people through motion and evaluate the effect of motion; the WiSee system uses Doppler shifts in narrowband extracted from broadband OFDM (orthogonal frequency division multiplexing) transmission to recognize nine different human gestures.

[0003] The gesture recognition methods that have been proposed so far are based on sensors that need to be worn, which can easily cause discomfort to the wearer, and based on computer vision, which can easily cause privacy issues. In addition, some gesture recognition methods are contact-based, which is less convenient than contactless gesture recognition, thus limiting its application and development. Summary of the invention

[0004] The purpose of the present invention is to provide a method for human motion recognition based on phase information and signal strength to achieve non-contact motion recognition. In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0005] The human motion recognition method based on phase information and signal strength specifically includes the following steps:

[0006] Step 1, collecting phase data and signal strength information of human motion in an indoor environment, and extracting phase information and signal strength;

[0007] Step 2, phase unwrapping the phase data collected in step 1 to remove the influence of phase jump, then smoothing and filtering the phase data and signal strength data, using discrete wavelet transform method to eliminate environmental noise and obtain clean phase and signal strength, and preprocessing step of normalizing the phase data;

[0008] Step 3, obtain the key information of the action from the signal preprocessed in step 2, construct two n-dimensional vectors of phase and signal strength, and form two n*m matrices Θ m×n and R m×n, corresponding to phase and RSSI, respectively, where m is the number of sampling points and n is the number of tags. Then, the two matrices are split along time to generate two sliding windows of length l, which are represented as l×n matrices,

[0009]

[0010] Then the starting and ending points of the action are detected, and the segmented data segments are divided into training and test sets;

[0011] Step 4, resample the data, calculate the length of the segmented data, and use the mean T of all sample lengths as the resampled length;

[0012] Step 5: Input the training set into the CNN-based deep neural network to train the action recognition model to obtain a human action recognition model. This model can be used to classify the collected phase data and signal strength test set data to achieve the purpose of human action recognition.

[0013] Furthermore, step 1 is specifically to connect a computer to an RFID reader, the reader is equipped with a directional gain antenna and a fixed frequency of 925 Hz, and the RFID tags are attached to a wooden board to form a 2*2 tag array to collect RFID signals and signal strengths of different human body movements, and extract phase information and signal strength changes caused by different human body movements. Phase is the channel information of the physical layer, reflecting the impact of mobile behavior on signal propagation.

[0014] Furthermore, the specific steps of step 2 are:

[0015] Step 2-1, unwrap the phase data from step 1:

[0016]

[0017] and Represent the phase readings of the i-th tag at time t+1 and t respectively;

[0018] Step 2-2, the phase number and signal strength data extracted in step 2-1 are smoothed by discrete wavelet transform (DWT), and 5-level db4 discrete wavelet transform (DWT) is applied to reduce noise, filter out outliers, and eliminate noise interference;

[0019] Step 2-3, normalize the phase information after denoising in step 2-2, that is,

[0020]

[0021] in represents the new phase after normalization, xi represents the old phase before normalization, argmaxx k and argmin x k They represent the maximum and minimum values ​​of all samples before the feature is processed, and N represents the number of phases in the phase sequence.

[0022] Furthermore, the specific steps of step 3 are:

[0023] Step 3-1, determine the start of the action by using the dynamic threshold method, first calculate the sum of the phase information of the lth sliding window and amplitude measurement where l and x k The window length and the data of the k-th data point are displayed respectively.

[0024] Step 3-2, the two coefficients A1 and F1 are added to the equation because A dif and F dif One of the pairs G m The influence of is always greater, so in order to balance their influence on Gm, A1 and F1 are used as follows:

[0025]

[0026] Where m represents the window number, and the local maximum in the G function above the empirical threshold indicates the boundary of each segment;

[0027] Step 3-3: Since the impact on signal strength and phase is simultaneous, the phase boundary divided in step 3-2 is used as the boundary of signal strength.

[0028] Furthermore, step 4: Since multiple labels are used and multiple samplings are performed at the same time, the length of the data collection samples may be inconsistent, so it is necessary to resample the data, calculate the duration of all actions, and then calculate their average value T, and then use the matlab function resample(phase,T,length), where length is the original length of the matrix and T is the length after resampling.

[0029] Furthermore, the CNN-based deep neural network in step 5 is divided into two parts, a single convolutional subnet for phase and signal strength, and a merged convolutional subnet for merging the outputs of the single convolutional subnets.

[0030] Furthermore, a CNN-based deep learning model is constructed using deep learning tools, and the phase information and signal strength training samples resampled in step 4 are input to explore the local interactions between various modes and tag arrays for feature extraction; the temporal and spatial relationships contained in the data matrix are utilized to learn the local relationships in adjacent sampling points and the interactions between the entire RFID tag array in the first layer; the low-level features are then fed into subsequent convolutional layers to learn high-level features; in addition, SAME padding is used so that the output has the same spatial size as the input; merging is applied along the temporal dimension in the first two layers to reduce the spatial size of the representation and control overfitting; then, the outputs of the various convolutional subnetworks are concatenated along the feature dimension and fed into the merged convolutional layer to learn cross-modal features; finally, the output is flattened and fed into the fully connected layer to obtain the classification prediction result of mobile behavior recognition.

[0031] Furthermore, in step 5, the human action recognition model consists of 6 parts: signal input layer, convolution layer, fusion layer, maximum pooling layer, flattening layer, fully connected layer, and Softmax classification layer. The functions of each layer are as follows:

[0032] Signal input layer: input preprocessed and resampled phase data and signal strength data;

[0033] Convolutional layer: extracts the features of different human body movements from the input phase data and signal strength data, and sends the extracted features to the next layer;

[0034] Fusion layer: receives features from two independent convolutions, connects them in series and fuses them to learn cross-features;

[0035] Max pooling layer: reduces the size of data input, speeds up calculations, and makes the extracted features more robust;

[0036] Flattening layer: flattens the data into a structure suitable for the neural network; converts multi-dimensional data into one-dimensional data for transition between convolutional layers and fully connected layers;

[0037] Fully connected layer: compresses the one-dimensional data of the flattened layer, weights the extracted features, and maps the learned features to the label space;

[0038] Softmax classification layer: The Softmax classification layer receives the feature vector obtained after the dimensionality reduction processing of the flattening layer and finally identifies different actions.

[0039] The beneficial effects of the present invention are:

[0040] The present invention is a detection method for mobile behavior recognition based on phase and signal strength. It does not need to attach the tag to the user's body or clothes, but only needs to attach the tag to a fixed position, which will not cause discomfort to the user. It only needs to use a single antenna reader and several simple RFID tags, and uses a deep neural network CNN model to recognize the action to achieve the purpose of intelligent interaction. It has the characteristics of low cost and good applicability. At present, many systems still use the dynamic time warping (DTW) method to compress and stretch the data and then match it, and select the one with the highest matching rate as their output, which has high latency. This paper adopts a deep learning method to recognize actions, realize real-time action recognition, achieve the purpose of real-time interaction, and better complete the action interaction of the dance machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 4 is a system schematic diagram of a human motion recognition method in an embodiment of the present invention.

[0042] Figure 2 It is an experimental scene diagram in an embodiment of the present invention.

[0043] Figure 3 It is a schematic diagram of the motion sensing game console in an embodiment of the present invention.

[0044] Figure 4 It is a neural network architecture based on CNN in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0046] A method for identifying mobile behavior based on phase information and signal strength, the implementation steps of which include: collecting human motion data in an indoor environment, extracting phase information and signal strength information; performing discrete wavelet transform (DTW) on the phase data and signal strength data to eliminate noise and obtain clean phase information and signal strength information; performing phase unwrapping on the phase data to solve the phase jump, and then normalizing the phase data; segmenting the pre-processed data to detect the starting point and the end point of the human motion, and cutting out the motion fragments; dividing the segmented data into a training set and a test set; inputting the training set into a deep neural network based on CNN to extract features, performing model training of human motion, obtaining a model for feature and motion recognition, and using the model to classify the divided test set to achieve the purpose of human motion recognition. The present invention uses a neural network based on CNN for feature extraction, realizes the recognition of different movements of a dance machine, and achieves the purpose of interaction.

[0047] A method for identifying mobile behavior based on phase information and signal strength, specifically comprising the following steps:

[0048] Step 1: Collect phase data and signal strength information of human motion in an indoor environment, and extract phase information and signal strength.

[0049] In step 1, specifically:

[0050] A personal computer with AMD R7-4800H and 16GB RAM is connected to a commercial RFID reader (ImpinjR420). The reader is equipped with a directional gain antenna and a fixed frequency of 925Hz. RFID tags are attached to a wooden board to form a 2*2 tag array to collect RFID signals of different human movements and extract the phase information and signal strength changes caused by different human movements. Phase is the channel information of the physical layer, reflecting the impact of mobile behavior on signal propagation.

[0051] Step 2: Due to environmental noise and hardware defects, the collected signal cannot be used directly for feature extraction. Therefore, the phase data collected in step 1 needs to be phase unwrapped to remove the influence of phase jump, and then the phase data and signal strength data are smoothed and filtered. The discrete wavelet transform (DWT) method is used to eliminate environmental noise and obtain clean phase and signal strength, and the phase data is normalized and other preprocessing steps are performed.

[0052] In step 2, specifically:

[0053] Step 2-1: Perform phase unwrapping on the phase data in step 1. Since the collected phase data is a periodic signal, there may be a phase jump from 0 to 2π, or from 2π to 0. If you want to clearly characterize the offset between the received signal and the transmitted signal through the phase, you must perform corresponding de-periodic processing. Therefore, you first need to perform phase unwrapping on the collected original phase information.

[0054]

[0055] Step 2-2: The phase data and signal strength data extracted in step 2-1 are smoothed by discrete wavelet transform (DWT) algorithm, and 5-level db4 discrete wavelet transform (DWT) is applied to reduce noise, filter out outliers, and eliminate noise interference.

[0056] Step 2-3: Normalize the phase information after denoising in step 2-2.

[0057]

[0058] in represents the new phase after normalization, x i represents the old phase before normalization, argminx k and argmin x kThey represent the maximum and minimum values ​​of all samples before the feature is processed, and N represents the number of phases in the phase sequence.

[0059] Step 3: Get the key information of the action from the signal preprocessed in step 2, construct two n-dimensional vectors of phase and signal strength, and form two n*m matrices Θ m×n and R m×n , corresponding to phase and RSSI, respectively, where m is the number of sampling points and n is the number of tags. Then, the two matrices are split along time to generate two sliding windows of length l, which can be represented as an l×n matrix:

[0060]

[0061] Then the starting and ending points of the action are detected, and the segmented data segments are divided into training and testing sets.

[0062] In step 3, specifically:

[0063] Step 3-1: Determine the start of the action through the dynamic threshold algorithm. First, calculate the sum of the phase information of the lth sliding window. and frequency measurement where l and x k The window length and the data of the k-th data point are displayed respectively.

[0064] Step 3-2: Adding two coefficients (A1 and F1) to the equation is because A dif and F dif One of the pairs G m The influence of is always greater, so in order to balance their influence on Gm, A1 and F1 are used as follows:

[0065]

[0066] Where m represents the window number and the local maximum in the G function above an empirical threshold indicates the boundary of each segment.

[0067] Step 3-3: Since the impact on signal strength and phase is simultaneous, the phase boundary divided in step 3-2 can be used as the signal strength boundary.

[0068] Step 4: Since multiple labels are used and multiple samplings are performed at the same time, the length of the data collection samples may be inconsistent, so it is necessary to resample the data, calculate the duration of all actions, and then calculate their average value T, and then use the matlab function resample(phase,T,length), where length is the original length of the matrix and T is the length after resampling.

[0069] Step 5: Input the training set into the CNN-based deep neural network for action recognition model training to obtain a human action recognition model. This model can be used to classify the collected phase data and signal strength data test set to achieve the purpose of human action recognition.

[0070] In step 5, the CNN-based deep neural network is divided into two parts, a single convolutional subnet for phase and signal strength, and a merged convolutional subnet for merging the outputs of the single convolutional subnet. After continuous learning, a human motion recognition model is obtained. This model can be used to classify the collected phase data and signal strength data test sets to achieve the purpose of human motion recognition, specifically:

[0071] A CNN-based deep learning model is constructed using deep learning tools, and the phase information and signal strength training samples resampled in step 4 are input to explore the local interactions between various modalities and tag arrays for feature extraction. Using the temporal and spatial relationships contained in the data matrix, in the first layer, local relationships in adjacent sampling points and interactions between the entire RFID tag array are learned. The low-level features are then fed into subsequent convolutional layers to learn high-level features. In addition, SAME padding is used so that the output has the same spatial size as the input. Merging is applied along the time dimension in the first two layers to reduce the spatial size of the representation and control overfitting. The outputs of the individual convolutional subnetworks are then concatenated along the feature dimension and fed into the merged convolutional layer to learn cross-modal features. Finally, the output is flattened and fed into the fully connected layer to obtain the classification prediction result for mobile behavior recognition. Among them, the model architecture established in this step is as follows Figure 1 shown.

[0072] The model consists of 6 parts: signal input layer, convolution layer, fusion layer, max pooling layer, flatten layer, fully connected layer, and softmax classification layer. The functions of each layer are as follows:

[0073] Signal input layer: Input pre-processed resampled phase data and signal strength data.

[0074] Convolutional layer: extracts the features of different human movements from the input phase data and RSSI data, and sends the extracted features to the next layer.

[0075] Merged Layer: Receives features from two independent convolutions, connects them in series, and fuses them to learn cross-features.

[0076] MaxPooling Layer: Reduces the size of data input, speeds up calculations, and makes the extracted features more robust.

[0077] Flatten Layer: Flattens the data into a structure suitable for the neural network, used in the input layer or between the convolution layer and the fully connected layer. Converts multi-dimensional data into one-dimensional data, used for the transition between the convolution layer and the fully connected layer.

[0078] Fully Connected Layers: compress the one-dimensional data of the flattened layer, weight the extracted features, and map the learned features to the label space.

[0079] Softmax Classification Layer: The Softmax classification layer receives the feature vector obtained after the dimensionality reduction processing of the flatten layer, and finally identifies different actions.

[0080] The present invention is a detection method for mobile behavior recognition based on phase and RSSI. It does not need to attach the tag to the user's body or clothes. It only needs to attach the tag to a fixed position, which will not cause discomfort to the user. It only needs to use a single antenna reader and a few simple RFID tags. It uses a deep neural network CNN model to recognize the action to achieve the purpose of interaction. It has the characteristics of low cost and good applicability. At present, many systems still use the dynamic time warping (DTW) method to compress and stretch the data and then match it, and select the one with the highest matching rate as their output, which has high latency. This paper adopts a deep learning method to recognize actions, realize real-time action recognition, achieve the purpose of real-time interaction, and better complete the action interaction of the somatosensory game console.

[0081] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A human motion recognition method based on phase information and signal strength, characterized in that: The method specifically comprises the following steps: Step 1, collecting phase data and signal strength information of human motion in an indoor environment, and extracting phase information and signal strength; Step 2, phase unwrapping the phase data collected in step 1 to remove the influence of phase jump, then smoothing and filtering the phase data and signal strength data, using discrete wavelet transform method to eliminate environmental noise and obtain clean phase and signal strength, and preprocessing step of normalizing the phase data; Step 2 The specific steps are: Step 2-1, unwrap the phase data from step 1: and Represent the phase readings of the i-th tag at time t+1 and t respectively; Step 2-2, the phase number and signal strength data extracted in step 2-1 are smoothed by discrete wavelet transform DWT, and 5-level db4 discrete wavelet transform DWT is applied to reduce noise, filter out abnormal values, and eliminate noise interference; Step 2-3, normalize the phase information after denoising in step 2-2, that is, in represents the new phase after normalization, x i represents the old phase before normalization, argmax x k and argmin x k They represent the maximum and minimum values ​​of all samples before the feature is processed, respectively, and N represents the number of phases in the phase sequence; Step 3, obtain the key information of the action from the signal preprocessed in step 2, construct two n-dimensional vectors of phase and signal strength, and form two n*m matrices Θ m×n and R m×n , corresponding to the phase and RSSI respectively, where m is the number of sampling points and n is the number of tags; then, the two matrices are split along time to generate two sliding windows of length l, which are represented as l×n matrices, Then the starting and ending points of the action are detected, and the segmented data segments are divided into training and test sets; Step 4, resample the data, calculate the length of the segmented data, and use the mean T of all sample lengths as the resampled length; Step 5: Input the training set into the CNN-based deep neural network to train the action recognition model to obtain a human action recognition model. The model is used to classify the collected phase data and signal strength test set data to achieve the purpose of human action recognition.

2. The method for human motion recognition based on phase information and signal strength according to claim 1, characterized in that: Specifically, step 1 uses a computer to connect to an RFID reader equipped with a directional gain antenna and a fixed frequency of 925 Hz. RFID tags are attached to a wooden board to form a 2*2 tag array to collect RFID signals and signal strengths of different human movements, and extract phase information and signal strength changes caused by different human movements; phase is the channel information of the physical layer, which reflects the impact of mobile behavior on signal propagation.

3. The method for human motion recognition based on phase information and signal strength according to claim 1, characterized in that: Step 3 The specific steps are: Step 3-1, determine the start of the action by using the dynamic threshold method, first calculate the sum of the phase information of the lth sliding window and amplitude measurements where l and x k Display the window length and the data of the kth data point respectively; Step 3-2, the two coefficients A1 and F1 are added to the equation because A dif and F dif One of the pairs G m The influence of is always greater, so in order to balance their influence on Gm, A1 and F1 are used as follows: Where m represents the window number, and the local maximum in the G function above the empirical threshold indicates the boundary of each segment; Step 3-3: Since the impact on signal strength and phase is simultaneous, the phase boundary divided in step 3-2 is used as the boundary of signal strength.

4. The method for human motion recognition based on phase information and signal strength according to claim 1, characterized in that: Step 4: Since multiple labels are used and multiple samplings are performed at the same time, the length of the data collection samples may be inconsistent, so it is necessary to resample the data, calculate the duration of all actions, and then calculate their average value T, and then use the matlab function resample(phase,T,length), where length is the original length of the matrix and T is the length after resampling.

5. The method for human motion recognition based on phase information and signal strength according to claim 1, characterized in that: The CNN-based deep neural network in step 5 is divided into two parts, a single convolutional subnet for phase and signal strength, and a merged convolutional subnet for merging the outputs of the single convolutional subnets.

6. The method for human motion recognition based on phase information and signal strength according to claim 5, characterized in that: A CNN-based deep learning model is constructed using deep learning tools, and the phase information and signal strength training samples resampled in step 4 are input to explore the local interactions between various modalities and label arrays for feature extraction; Using the temporal and spatial relationships contained in the data matrix, in the first layer, local relationships among adjacent sampling points and interactions between the entire RFID tag array are learned; low-level features are then fed into subsequent convolutional layers to learn high-level features; In addition, SAME padding is used so that the output has the same spatial size as the input; merging is applied along the time dimension in the first two layers to reduce the spatial size of the representation and control overfitting; then, the outputs of the individual convolutional subnetworks are concatenated along the feature dimension and fed into the merged convolutional layer to learn cross-modal features; finally, the output is flattened and fed into the fully connected layer to obtain the classification prediction result for mobile behavior recognition.

7. The method for human motion recognition based on phase information and signal strength according to claim 1, characterized in that: In step 5, the human action recognition model consists of 6 parts: signal input layer, convolution layer, fusion layer, maximum pooling layer, flattening layer, fully connected layer, and Softmax classification layer. The functions of each layer are as follows: Signal input layer: input preprocessed and resampled phase data and signal strength data; Convolutional layer: extracts the features of different human body movements from the input phase data and signal strength data, and sends the extracted features to the next layer; Fusion layer: receives features from two independent convolutions, connects them in series and fuses them to learn cross-features; Max pooling layer: reduces the size of data input, speeds up calculations, and makes the extracted features more robust; Flattening layer: flattens the data into a structure suitable for the neural network; converts multi-dimensional data into one-dimensional data for transition between convolutional layers and fully connected layers; Fully connected layer: compresses the one-dimensional data of the flattened layer, weights the extracted features, and maps the learned features to the label space; Softmax classification layer: The Softmax classification layer receives the feature vector obtained after the dimensionality reduction processing of the flattening layer and finally identifies different actions.