A gesture recognition method and device based on low-precision ultra-wideband signal

Through the gesture recognition method of extracting and processing low-precision ultra-wideband signals, the deep learning model is used for feature extraction and classification, which solves the problem that low-precision ultra-wideband signal chips cannot achieve fine-grained gesture recognition, and achieves high accuracy on low-cost hardware gesture recognition.

CN118567490BActive Publication Date: 2024-11-29ZHEJIANG UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411045375.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-29
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

Low-precision ultra-wideband signal chips cannot obtain signal amplitude and phase information, resulting in the inability to achieve fine-grained gesture recognition.

Method used

A gesture recognition method based on low-precision ultra-wideband signal is designed, and the ultra-wideband signal transmitted by the transmitter is received through the receiver, RF information is extracted, clutter interference is removed, data is processed using background subtraction algorithm, and the processing is done after The data input deep learning model is used for feature extraction and classification.

Benefits of technology

It realizes gesture recognition using low-cost ultra-wideband chips, reduces deployment costs, improves recognition accuracy, and accurately recognizes gesture actions in different environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118567490B_ABST
    Figure CN118567490B_ABST
Patent Text Reader

Abstract

The present invention discloses a gesture recognition method and device based on low-precision ultra-wideband signals. The method uses low-cost hardware equipment and saves deployment costs. When collecting ultra-wideband signals containing human gesture action information, a plurality of channel impulse response data points caused by a first path index are selected, and the integer part of the first path index is used to preliminarily align the channel impulse response data points. Multiple frames of data received in one gesture action collection are vertically spliced ​​into a time domain diagram, so that the effect of assembling a time domain diagram that can reflect the changing trend of objects at different distances with the environment is achieved by using fewer transmission signals. A background subtraction algorithm is used, and a subdivided background reference curve is used as a reference to subtract the channel impulse response data of the corresponding subdivided interval in the gesture sample data to obtain an aligned time domain diagram, thereby solving the interference of data offset in the low-precision ultra-wideband signal.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of human gesture recognition, and in particular to a gesture recognition method and device based on low-precision ultra-wideband signals. Background Art

[0002] Gesture recognition is a technology that enables human-computer interaction by capturing and analyzing human hand movements or gestures. It does not require physical contact, and users can control devices or input commands simply through natural gestures. Gesture recognition has a wide range of applications, from smartphones and tablets to virtual reality devices, smart homes, and car control systems. Through gesture recognition, users can achieve more intuitive and convenient operations. There are many ways to implement gesture recognition, among which wireless sensing is a common implementation method.

[0003] Wireless sensing is a technology that uses wireless signals to sense and detect the environment. By transmitting wireless signals and receiving and analyzing the reflected signals, it can sense environmental changes. Wireless sensing technology can be used in many fields such as object tracking, multi-person detection, and vital sign detection. Compared with traditional sensor technology, wireless sensing has the advantages of not being restricted by light and line of sight, and has better security and concealment. Gesture recognition based on wireless sensing is to use wireless sensing to realize gesture recognition applications. Specifically, gesture detection and recognition is realized by analyzing the impact of gestures on wireless signals. Commonly used wireless signals include WiFi, millimeter waves, and ultra-wideband signals.

[0004] Ultra-wideband signals refer to wireless signals with a bandwidth greater than 500 MHz or 20% of its center carrier frequency. Since they transmit pulses at very low power, they pose no risk to human health and can be used in daily life. In recent years, ultra-wideband signal technology has been widely used due to its robustness in harsh environments, centimeter-level accurate ranging, low power consumption, and good object penetration. Ultra-wideband signal technology was initially used mainly for ranging and positioning, and then gradually expanded to wireless sensing fields such as medicine, multi-person detection, gesture recognition, imaging, tumor detection, and people counting.

[0005] Although there have been many studies and applications on gesture recognition based on ultra-wideband signals, the existing work is based on high-cost ultra-wideband signal radars, such as Novelda X4M300 and TDSR P440 hardware devices. These ultra-wideband signal radars can output raw signal samples in the time domain to facilitate researchers to process signals, including signal amplitude and signal phase information. However, the price of such ultra-wideband signal radars is generally more than 1,000 yuan, which is not suitable for smartphones, smart watches or daily use by users. Currently, most of the ultra-wideband signal chips equipped in smartphones are commercial low-priced chips, such as DW1000, which are priced at around 100 yuan and have higher market universality. However, the detection granularity of such ultra-wideband signal chips is relatively low, and it is impossible to directly obtain the original information such as signal amplitude and phase. Therefore, such low-precision ultra-wideband signal chips are currently only used for ranging and positioning, and cannot realize wireless sensing applications with higher precision and granularity such as gesture recognition. Summary of the invention

[0006] In view of the defects of the prior art, the technical problem to be solved by the present invention is to overcome the problem that low-precision ultra-wideband signal chips cannot realize fine-grained gesture recognition due to the difficulty in obtaining original information and the low accuracy of collected data. A method and device for gesture recognition based on low-precision ultra-wideband signals are provided, which can obtain and process low-precision ultra-wideband signals using low-cost ultra-wideband chips.

[0007] To achieve the above objectives, the present invention designs an independent signal acquisition device and an independent gesture recognition and detection method, which realizes the recognition and detection of human hand movements in the environment by transmitting, collecting and analyzing low-precision ultra-wideband signals.

[0008] The present invention is achieved through the following technical solutions:

[0009] The present invention discloses a gesture recognition method based on low-precision ultra-wideband signals, comprising:

[0010] A receiver is used to receive an ultra-wideband signal including human gesture information transmitted by a transmitter at a specific rate;

[0011] Extract the radio frequency information from the ultra-wideband signal of the received signal and convert it into a time domain diagram;

[0012] By using the normalization and filtering methods, the clutter interference in the time domain image is removed to obtain the time domain image without clutter interference;

[0013] Using the background subtraction algorithm, the time domain image without clutter interference is processed to remove the interference of data offset and obtain the aligned time domain image;

[0014] The aligned time domain image is used as the input of the deep learning model, and the gesture-related features are extracted using the deep learning model;

[0015] The classifier is used to identify and classify gesture-related features.

[0016] As a further improvement, the ultra-wideband signal including human gesture information obtained by the receiver of the present invention includes: signal strength, total number of preamble codes, a first path index including an integer part and a decimal part, and several channel impulse response data points caused by the first path index.

[0017] As a further improvement, the present invention extracts the radio frequency information in the ultra-wideband signal of the received signal and converts it into a time domain diagram. The specific method is to use the integer part of the first path index to preliminarily align the channel impulse response data points, and vertically splice the multiple frames of data received in a gesture action collection into a time domain diagram, whose horizontal axis represents fast time and the vertical axis represents slow time, reflecting the changing trend of objects at different distances with the environment.

[0018] As a further improvement, the normalization of the present invention is to firstly use the total number of leading codes to normalize a frame of data of the time domain diagram, and then normalize the time domain diagram as a whole to obtain a normalized time domain diagram.

[0019] As a further improvement, the filtering method of the present invention includes a high-pass filter and a low-pass filter, and performs filtering processing on the normalized time domain image to obtain a time domain image without clutter interference.

[0020] As a further improvement, the specific steps of the background subtraction algorithm of the present invention are as follows:

[0021] 1) The horizontal axis direction in the time domain diagram is a number of aligned large intervals. According to the transmission principle of the ultra-wideband chip, each large interval is divided into 64 sub-intervals. For a number of channel impulse response data points collected in each frame of data, the decimal part of the first path index is used to place them in a corresponding sub-interval;

[0022] 2) For the reference data collected from the static background, several channel impulse response data points of each frame of data are placed in the corresponding subdivision intervals. For the channel impulse response data in each subdivision interval, a linear averaging method is used to obtain an average value as the reference value of the subdivision interval. If there are some empty subdivision intervals, that is, no corresponding value falls into the subdivision interval, the linear interpolation method is used to process it, and finally a subdivision background reference curve is obtained;

[0023] 3) For each gesture sample that only contains a small number of frames, several channel impulse response data points of each frame data are subtracted from the reference values ​​of the corresponding subdivision intervals in the background reference curve to obtain the gesture data after background subtraction, and realign them to obtain the aligned time domain image.

[0024] As a further improvement, the deep learning model of the present invention is a combination of a convolutional neural network and a long short-term memory network. First, a one-dimensional convolutional neural network is used to extract features in the fast time direction of the time domain graph, and then the extracted features are input into the long short-term memory network to obtain features in the slow time direction, thereby obtaining two-dimensional gesture features. Finally, two fully connected layers are used as classifiers to predict the gesture category based on the two-dimensional gesture features.

[0025] The present invention also discloses a gesture recognition device based on a low-precision ultra-wideband signal, comprising:

[0026] A data receiving module, used for receiving, by means of a receiver, an ultra-wideband signal including human gesture information transmitted by a transmitter at a specific rate;

[0027] The first data processing module is used to extract the radio frequency information in the received signal and convert it into a time domain diagram;

[0028] The second data processing module is used to remove clutter interference in the time domain image by using normalization and filtering methods;

[0029] The third data processing module is used to remove the interference of data offset in the time domain image by using a background subtraction algorithm and obtain an aligned time domain image;

[0030] The deep learning module is used to use the aligned time domain image as the input of the deep learning model, extract gesture-related features using the deep learning model, and use the classifier to identify and classify the gesture features.

[0031] The beneficial effects of the present invention are as follows:

[0032] The present invention can be deployed in any space using commercial low-cost chips and realize gesture recognition using low-precision ultra-wideband signals. Compared with existing gesture recognition methods, the present invention has very low requirements on the accuracy of ultra-wideband signals, and the entire invention can use low-cost hardware equipment, saving deployment costs.

[0033] When collecting ultra-wideband signals including human gesture information, the present invention selects a number of channel impulse response data points caused by a first path index, and uses the integer part of the first path index to preliminarily align the channel impulse response data points, and vertically splices multiple frames of data received in a gesture collection into a time domain diagram, thereby achieving the effect of assembling a time domain diagram that can reflect the changing trend of objects at different distances with the environment using fewer transmission signals.

[0034] In the background subtraction algorithm of the present invention, each large interval in the time domain diagram is divided into 64 subdivided intervals, and the decimal part of the first path index is used to place the channel impulse response data points of the static environment into the corresponding subdivided intervals, and then the linear averaging method and linear interpolation method are used to process the subdivided intervals to successfully obtain a subdivided background reference curve.

[0035] The present invention utilizes a background subtraction algorithm and uses a subdivided background reference curve as a reference to subtract the channel impulse response data of the corresponding subdivided interval in the gesture sample data to obtain an aligned time domain diagram, thereby solving the data offset problem caused by too few frames of gesture sample data and further solving the interference of data offset in low-precision ultra-wideband signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of using an ultra-wideband signal transmitter and receiver to sense the environment and recognize gestures;

[0037] Figure 2 is a flow chart of the signal background subtraction algorithm of the present invention;

[0038] Figure 3 It is a flow chart of the feature extraction part of the present invention. DETAILED DESCRIPTION

[0039] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and results of the present invention.

[0040] Figure 1It is a flowchart for using an ultra-wideband signal transmitter and receiver to sense the environment and recognize gestures. The present invention discloses a gesture recognition method and device based on a low-precision ultra-wideband signal, using a receiver to receive an ultra-wideband signal containing human gesture information transmitted by a transmitter at a specific rate; extracting the radio frequency information in the ultra-wideband signal of the received signal and converting it into a time domain diagram; using normalization and filtering methods to remove clutter interference in the time domain diagram to obtain a time domain diagram without clutter interference; using a background subtraction algorithm to process the time domain diagram without clutter interference, remove the interference of data offset, and obtain an aligned time domain diagram; using the aligned time domain diagram as the input of a deep learning model, using the deep learning model to extract gesture-related features; using a classifier to identify and classify gesture-related features.

[0041] The specific implementation method of the present invention is as follows:

[0042] Step 1: Set up a pair of ultra-wideband signal transmitters and receivers to transmit data at a specific rate in space, where the transmission rate is 6.8Mbps and one data packet is transmitted every 20 milliseconds.

[0043] Step 2: The ultra-wideband signal including human gesture information obtained by the receiver includes: signal strength, total number of preamble codes, first path index including integer part and decimal part, and 15 channel impulse response data points caused by the first path index.

[0044] Step 3: For a static environment, collect 1000 frames of data as a background reference. Connect the receiver to a laptop computer, use software to extract the RF information in the received signal, and use the integer part of the first path index to preliminarily align the 15 channel impulse response data points. Splice the multiple frames of data received in a gesture action collection vertically into a time domain diagram, with the horizontal axis representing fast time and the vertical axis representing slow time, reflecting the changing trend of objects at different distances with the environment.

[0045] Step 4: Normalize and remove clutter from the background reference data. The normalization operation is to first normalize a frame of the time domain image using the total number of preamble codes, and then normalize the time domain image as a whole to obtain a normalized time domain image. Clutter removal is to filter the normalized time domain image using a low-pass filter and a high-pass filter respectively to obtain a time domain image without clutter interference. The frequency of the high-pass filter is set to 7Hz, and the frequency of the low-pass filter is set to 0.05Hz.

[0046] Step 5: For 6 different gestures, 30 sample data are collected for each gesture as training, where each gesture lasts for 3 seconds and 150 frames of sample data are collected. For each sample data, the receiver is connected to a laptop, and the software is used to extract the RF information in the received signal, and the integer part of the first path index is used to preliminarily align the 15 channel impulse response data points, and the multiple frames of data received in a gesture collection are vertically spliced ​​into a time domain diagram, with the horizontal axis representing fast time and the vertical axis representing slow time, reflecting the changing trend of objects at different distances with the environment.

[0047] Step 6: Normalize and remove clutter from the gesture sample data. The normalization operation is to first normalize a frame of the time domain image using the total number of preamble codes, and then normalize the time domain image as a whole to obtain a normalized time domain image. Clutter removal is to filter the normalized time domain image using a low-pass filter and a high-pass filter respectively to obtain a time domain image without clutter interference. The frequency of the high-pass filter is set to 7Hz, and the frequency of the low-pass filter is set to 0.05Hz.

[0048] Step 7: Use background subtraction algorithm to process gesture sample data. Figure 2 This is a flowchart of the background subtraction algorithm. The specific steps are as follows:

[0049] 1) In the time domain diagram, the horizontal axis direction is 15 aligned large intervals. According to the transmission principle of the ultra-wideband chip, each large interval can be divided into 64 subdivided intervals, that is, there are a total of 15 * 64 subdivided intervals. For each frame of data, the 15 channel impulse response data points collected are placed in a corresponding subdivided interval using the decimal part of the first path index;

[0050] 2) For the 1000 frames of reference data collected under static background, the 15 channel impulse response data points of each frame of data are placed in the corresponding subdivision intervals. For the channel impulse response data in each subdivision interval, a linear averaging method is used to obtain an average value as the reference value of the subdivision interval. If there are some empty subdivision intervals, that is, no corresponding value falls into the subdivision interval, the linear interpolation method is used to process it, and finally a subdivision background reference curve containing 15*64 data points is obtained;

[0051] 3) For each gesture sample that only contains 150 frames of data, the 15 channel impulse response data points of each frame of data are subtracted from the reference values ​​of the corresponding subdivision intervals in the background reference curve to obtain the gesture data after background subtraction, and then realigned to 150*15 to obtain the aligned time domain image.

[0052] Step 8: For the gesture time domain image after background subtraction, use deep learning to extract its gesture features. Figure 3 This is the flow chart of the feature extraction module. The selected deep learning model is a combination of convolutional neural network and long short-term memory network. First, a one-dimensional convolutional neural network is used to extract features in the fast time direction of the time domain image, and then the extracted features are input into the long short-term memory network to obtain features in the slow time direction, thereby obtaining two-dimensional gesture features.

[0053] Step 9: Use two fully connected layers as classifiers to predict the corresponding gesture category based on the input two-dimensional gesture features. Use the collected 6*30 sample data as training to obtain a trained prediction model.

[0054] Step 10: In the application, directly put the collected gesture data into the above-mentioned trained model for prediction, and the category corresponding to the gesture can be obtained to realize gesture recognition.

[0055] In order to verify the reliability of the gesture recognition algorithm of the present invention, additional experiments were carried out for verification. In the experiment, a set of static environment data and 6*30 sets of gesture action data were collected in 10 different environments, including 6 different gesture actions, each gesture action was repeated 30 times. For the data collected in each environment, 25 sets of data for each of the 6 gestures were used as training data, and 5 sets of data for each of the 6 gestures were used as test data, and the average accuracy rate could reach 92%.

[0056] The present invention also discloses a gesture recognition device based on a low-precision ultra-wideband signal, the recognition device comprising:

[0057] A data receiving module, used for receiving, by means of a receiver, an ultra-wideband signal including human gesture information transmitted by a transmitter at a specific rate;

[0058] The first data processing module is used to extract the radio frequency information in the received signal and convert it into a time domain diagram;

[0059] The second data processing module is used to remove clutter interference in the time domain image by using normalization and filtering methods;

[0060] The third data processing module is used to remove the interference of data offset in the time domain image by using a background subtraction algorithm and obtain an aligned time domain image;

[0061] The deep learning module is used to use the aligned time domain image as the input of the deep learning model, extract gesture-related features using the deep learning model, and use the classifier to identify and classify the gesture features.

[0062] The above is not a limitation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several changes, modifications, additions or substitutions can be made without departing from the essential scope of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A gesture recognition method based on low-precision ultra-wideband signals, It is characterized in that A receiver is used to receive an ultra-wideband signal including human gesture information transmitted by a transmitter at a specific rate; the ultra-wideband signal including human gesture information obtained by the receiver includes: signal strength, total number of preamble codes, a first path index including an integer part and a decimal part, and a plurality of channel impulse response data points caused by the first path index; Extracting the radio frequency information from the ultra-wideband signal of the received signal and converting it into a time domain diagram; the specific method of extracting the radio frequency information from the ultra-wideband signal of the received signal and converting it into a time domain diagram is to use the integer part of the first path index to preliminarily align the channel impulse response data points, and vertically splice the multiple frames of data received in a gesture action collection into a time domain diagram, the horizontal axis of which represents fast time and the vertical axis represents slow time, reflecting the changing trend of objects at different distances with the environment; By using the normalization and filtering methods, the clutter interference in the time domain image is removed to obtain the time domain image without clutter interference; The background subtraction algorithm is used to process the time domain image without clutter interference, remove the interference of data offset, and obtain the aligned time domain image; the specific steps of the background subtraction algorithm are as follows: 1) The horizontal axis direction in the time domain diagram is a number of aligned large intervals. According to the transmission principle of the ultra-wideband chip, each large interval is divided into 64 sub-intervals. For a number of channel impulse response data points collected in each frame of data, the decimal part of the first path index is used to place them in a corresponding sub-interval; 2) For the reference data collected from the static background, several channel impulse response data points of each frame of data are placed in the corresponding subdivision intervals. For the channel impulse response data in each subdivision interval, a linear averaging method is used to obtain an average value as the reference value of the subdivision interval. If there are some empty subdivision intervals, that is, no corresponding value falls into the subdivision interval, the linear interpolation method is used to process it, and finally a subdivision background reference curve is obtained; 3) For each gesture sample that only contains a small number of frames, several channel impulse response data points of each frame data are subtracted from the reference values ​​of the corresponding subdivision intervals in the background reference curve to obtain the gesture data after background subtraction, and realign them to obtain the aligned time domain image; The aligned time domain image is used as the input of the deep learning model, and the gesture-related features are extracted using the deep learning model; The classifier is used to identify and classify gesture-related features.

2. The gesture recognition method based on low-precision ultra-wideband signal according to claim 1, It is characterized in that The normalization is to firstly perform normalization processing on a frame of data of the time domain diagram using the total number of leading codes, and then perform normalization processing on the time domain diagram as a whole to obtain a normalized time domain diagram.

3. The gesture recognition method based on low-precision ultra-wideband signal according to claim 2, It is characterized in that The filtering method includes a high-pass filter and a low-pass filter, and performs filtering processing on the normalized time domain image to obtain a time domain image without clutter interference.

4. The gesture recognition method based on low-precision ultra-wideband signal according to claim 1, It is characterized in that The deep learning model is a combination of a convolutional neural network and a long short-term memory network. First, a one-dimensional convolutional neural network is used to extract features in the fast time direction of the time domain image. The extracted features are then input into the long short-term memory network to obtain features in the slow time direction, and two-dimensional gesture features are obtained. Finally, two fully connected layers are used as classifiers to predict gesture categories based on the two-dimensional gesture features.

5. A gesture recognition device based on low-precision ultra-wideband signals, It is characterized in that include: The data receiving module is used to use a receiver to receive an ultra-wideband signal including human gesture information transmitted by a transmitter at a specific rate; the ultra-wideband signal including human gesture information obtained by the receiver includes: signal strength, total number of preamble codes, a first path index including an integer part and a decimal part, and a number of channel impulse response data points caused by the first path index; The first data processing module is used to extract the radio frequency information from the received signal and convert it into a time domain diagram; the specific method of extracting the radio frequency information from the ultra-wideband signal of the received signal and converting it into a time domain diagram is to use the integer part of the first path index to preliminarily align the channel impulse response data points, and vertically splice the multiple frames of data received in a gesture action collection into a time domain diagram, the horizontal axis represents fast time, and the vertical axis represents slow time, reflecting the change trend of objects at different distances with the environment; The second data processing module is used to remove clutter interference in the time domain image by using normalization and filtering methods; The third data processing module is used to use a background subtraction algorithm to remove the interference of data offset in the time domain image and obtain an aligned time domain image; the specific steps of the background subtraction algorithm are as follows: 1) The horizontal axis direction in the time domain diagram is a number of aligned large intervals. According to the transmission principle of the ultra-wideband chip, each large interval is divided into 64 sub-intervals. For a number of channel impulse response data points collected in each frame of data, the decimal part of the first path index is used to place them in a corresponding sub-interval; 2) For the reference data collected from the static background, several channel impulse response data points of each frame of data are placed in the corresponding subdivision intervals. For the channel impulse response data in each subdivision interval, a linear averaging method is used to obtain an average value as the reference value of the subdivision interval. If there are some empty subdivision intervals, that is, no corresponding value falls into the subdivision interval, the linear interpolation method is used to process it, and finally a subdivision background reference curve is obtained; 3) For each gesture sample that only contains a small number of frames, several channel impulse response data points of each frame data are subtracted from the reference values ​​of the corresponding subdivision intervals in the background reference curve to obtain the gesture data after background subtraction, and realign them to obtain the aligned time domain image; The deep learning module is used to use the aligned time domain image as the input of the deep learning model, extract gesture-related features using the deep learning model, and use the classifier to identify and classify the gesture features.

Citation Information

Patent Citations

  • Gesture recognition detection method based on time-frequency spectrum and distance Doppler spectrum

    CN110309690A

  • Gesture recognition method and device, mobile terminal and computer readable storage medium

    CN115412104A