Gesture recognition method based on multi-feature image fusion of FMCW radar

Abstract

The present invention relates to the field of radar-based gesture recognition technology, and discloses a gesture recognition method based on multi-feature image fusion of frequency modulated continuous wave radar, comprising the following steps: step 1, real-time collection of gesture data at the radar end, and pre-processing of the gesture data to obtain the intermediate frequency signal of the gesture data; step 2, characteristic parameter calculation of the intermediate frequency signal to obtain continuous motion parameters of the gesture; step 3, filtering irrelevant objects and extracting data segments of interest from the continuous data segments of the motion parameters to obtain data segments of interest containing gesture information; step 4, constructing a gesture feature image based on the data segments of interest, and inputting the gesture feature image into a shallow convolutional neural network for real-time recognition and classification. The present invention can generate three feature images of gestures in real time, and perform real-time recognition and classification of gestures based on the feature images.

Description

Technical Field

[0001] The present invention relates to the technical field of gesture recognition based on radar, and in particular to a gesture recognition method based on multi-feature image fusion of frequency modulated continuous wave radar. Background Art

[0002] With the development of sensor and machine learning technology, gesture recognition technology has become a hot research direction in the field of human-computer interaction due to its convenience, rich meaning and easy-to-understand characteristics. Compared with the currently widely used gesture recognition methods based on optical images, radar-based gesture recognition methods are not easily affected by environmental factors such as ambient lighting, delays, haze, etc., require a small amount of data to be processed, and will not leak the privacy information of the person whose gesture is collected. This makes radar-based gesture recognition methods a research hotspot in the field of gesture recognition. At present, the mainstream method of radar gesture recognition research at home and abroad is to obtain the distance, speed, angle and other data of the gesture by de-skewing, fast Fourier transform, coherent accumulation and other processing of the radar echo signal, and then input these data as input samples into various convolutional neural networks with different structures for classification and recognition. However, there are the following problems in current research:

[0003] The current mainstream radar gesture recognition method does not achieve real-time collection, processing and recognition of gesture data. Summary of the invention

[0004] The present invention provides a gesture recognition method based on multi-feature image fusion of frequency modulated continuous wave radar, which can realize real-time collection, processing and recognition of gesture data for the radar gesture recognition method.

[0005] The present invention is achieved through the following technical solutions:

[0006] A gesture recognition method based on multi-feature image fusion of frequency modulated continuous wave radar includes the following steps:

[0007] Step 1: collecting gesture data in real time at the radar end, and preprocessing the gesture data to obtain an intermediate frequency signal of the gesture data;

[0008] Step 2: Calculate characteristic parameters of the intermediate frequency signal to obtain continuous motion parameters of the gesture;

[0009] Step 3: filtering out irrelevant objects and extracting interesting data segments from the continuous data segments of the motion parameters to obtain interesting data segments containing gesture information;

[0010] Step 4: construct a gesture feature image based on the data segment of interest, and input the gesture feature image into a shallow convolutional neural network for real-time recognition and classification.

[0011] As an optimization, in step 1, the specific steps of collecting gesture data in real time and preprocessing the real-time collected gesture data to obtain an intermediate frequency signal are:

[0012] Step 1.1, the RF module at the radar end generates a transmission signal which is sequentially transmitted through the frequency multiplier, the power amplifier and the transmitting antenna to send out a frequency modulated continuous wave radar signal;

[0013] Step 1.2, the frequency modulated continuous wave radar signal is reflected by the waving gesture to generate an echo signal, and the echo signal passes through the receiving antenna and the low noise amplifier in sequence to reach the mixer;

[0014] Step 1.3: The echo signal and the transmission signal are mixed through a mixer to obtain an intermediate frequency signal.

[0015] As an optimization, the method further includes step 1.4, wherein the intermediate frequency signal is converted into a digital signal of the gesture data by an analog-to-digital conversion module.

[0016] As an optimization, in step 1.3, the specific formula for obtaining the intermediate frequency signal by the echo signal and the transmission signal through the mixer is:

[0017]

[0018] Where B is the effective frequency modulation bandwidth of the radar, T c is the frequency modulation period, R0 is the distance between the hand and the radar when the signal is transmitted, v is the speed of the hand, t is the time required for the transmitted signal to be transmitted to the hand, c is the speed of light, is the frequency shift of the echo signal.

[0019] As an optimization, in step 2, the continuous motion parameters of the gesture include a distance parameter from the radar to the gesture subject, a speed parameter of the gesture, and a horizontal angle parameter of the gesture.

[0020] As an optimization, the calculation formula of the distance parameter from the radar to the gesture subject is:

[0021]

[0022] Where B is the effective frequency modulation bandwidth of the radar, c is the speed of light, and f IF is the intermediate frequency signal, T is the frequency modulation period;

[0023] The calculation formula of the speed parameter of the gesture is:

[0024]

[0025] Where λ is the wavelength of the transmitted signal, T c is the frequency modulation period, f FFT is the Doppler shift, is the frequency shift of the echo signal.

[0026] As an optimization, in step 3, the specific process of filtering out irrelevant objects from the continuous data segments of the motion parameters is as follows:

[0027] Step 3.1, define a gesture with N frames of data, each frame of data includes the number of objects monitored by the radar, the distance parameter from the radar to the gesture subject, the speed parameter of the gesture and the horizontal angle parameter of the gesture, where N is a positive integer;

[0028] Step 3.2, dynamic distance detection is performed on the continuous motion parameters, and the length of the speed detection window is set to N frames. The maximum speed in the speed detection window corresponds to the subject distance value where the gesture subject is located;

[0029] Step 3.3: Take the first 2 cm to the last 5 cm of the subject distance value as the dynamic distance threshold.

[0030] As an optimization, in step 3, the specific steps for extracting the data segment of interest are:

[0031] Step 3.4, performing an absolute value operation on all speed data in the speed detection window;

[0032] Step 3.5, searching for a data segment in which the absolute value of the speed in the speed detection window increases from 0 to a maximum value and then decreases to 0, and the maximum value is greater than 0.7;

[0033] Step 3.6: The data segments corresponding to N / 2 frames before and after the time point where the maximum value of the data segment obtained in step 3.5 is located are the data segments of interest containing the motion gesture information.

[0034] As an optimization, the specific steps of step 4, constructing a gesture feature image according to the data segment of interest, are:

[0035] Step 4.1, the distance parameter from the radar to the gesture subject, the speed parameter of the gesture, and the horizontal angle parameter of the gesture in each frame of data in the data segment of interest are spliced ​​in the frame time sequence to obtain a distance-time diagram, a speed-time diagram, and a horizontal angle-time diagram respectively;

[0036] Step 4.2: normalize, grayscale and scale the distance-time graph, speed-time graph and horizontal angle-time graph to obtain a 64*64 gesture feature image.

[0037] As an optimization, in step 4, the shallow convolutional neural network includes an input layer, a convolutional layer, a pooling layer, two fully connected layers and a softmax layer.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] The present invention is based on an irrelevant object filtering method of a continuous data stream, an interesting data segment extraction method and a shallow convolutional neural network structure. It can perform real-time data collection and processing on a pre-designed set of six gesture actions including push forward, pull back, left swing, right swing, up swing and down swing, generate three feature images of gestures in real time, and perform real-time recognition and classification of gestures according to the feature images. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0041] Figure 1 A system block diagram of a radar gesture recognition system used in a gesture recognition method based on multi-feature image fusion of a frequency modulated continuous wave radar according to the present invention;

[0042] Figure 2 This is the workflow diagram of the radar gesture recognition system;

[0043] Figure 3 Produce schematic diagram for intermediate frequency signal;

[0044] Figure 4 This is a schematic diagram of waving;

[0045] Figure 5 is the unprocessed speed data graph;

[0046] Figure 6 For Figure 5 Speed ​​data graph for absolute value operation of speed;

[0047] Figure 7 Three characteristic images of right hand waving;

[0048] Figure 8 This is a structural diagram of a shallow convolutional neural network. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0050] Example

[0051] A gesture recognition method based on multi-feature image fusion of frequency modulated continuous wave radar includes the following steps:

[0052] The block diagram of the radar gesture recognition system is as follows: Figure 1 As shown, data acquisition and preprocessing of intermediate frequency signals are completed on the radar side, and subsequent data processing, feature extraction, and gesture classification are completed on the PC side.

[0053] The system workflow diagram is as follows: Figure 2 As shown in the figure, it is divided into 6 parts, namely, firstly, gesture data is collected to obtain the original data of radar echo signal; then the radar echo original data is preprocessed to obtain intermediate frequency signal; then characteristic parameters are calculated, and the motion parameters of gesture are obtained by applying fast Fourier transform, MUSIC and other algorithms; then, irrelevant objects are filtered out and data segments of interest are extracted from continuous motion parameter data segments to obtain data segments of interest containing gesture information; then, gesture feature images are constructed according to the calculated gesture motion parameters; finally, the feature images are input into the trained convolutional neural network for recognition and classification.

[0054] The specific steps are as follows:

[0055] Step 1: Collect gesture data in real time at the radar end, and pre-process the gesture data to obtain an intermediate frequency signal of the gesture data.

[0056] The specific steps of collecting gesture data in real time and preprocessing the collected gesture data in real time to obtain an intermediate frequency signal are as follows:

[0057] Step 1.1: The RF module at the radar end generates a transmission signal which is then sent out through a frequency multiplier, a power amplifier and a transmitting antenna.

[0058] Step 1.2: The frequency modulated continuous wave radar signal is reflected by the waving gesture to generate an echo signal, and the echo signal passes through the receiving antenna and the low noise amplifier in sequence to reach the mixer.

[0059] Step 1.3: The echo signal and the transmission signal are mixed through a mixer to obtain an intermediate frequency signal.

[0060] Step 1.4: The intermediate frequency signal is converted into a digital signal of gesture data by an analog-to-digital conversion module.

[0061] The modulation method of the frequency modulated continuous wave radar signal is usually a sawtooth wave. The signal transmitted by the radar is reflected by an object. After a time delay, the radar receiving antenna receives the echo signal and mixes the echo signal with the transmitted signal. The overlapping part will generate an intermediate frequency signal with a constant frequency, such as Figure 3 shown.

[0062] The specific formula for obtaining the intermediate frequency signal by the echo signal and the transmission signal through the mixer is:

[0063]

[0064] Where B is the effective frequency modulation bandwidth of the radar, T c is the frequency modulation period, R0 is the distance between the hand and the radar when the signal is transmitted, v is the speed of the hand, t is the time required for the transmitted signal to be transmitted to the hand, c is the speed of light, is the frequency shift of the echo signal.

[0065] Step 2: Calculate characteristic parameters of the intermediate frequency signal to obtain continuous motion parameters of the gesture; the motion parameters include distance parameters from the radar to the gesture subject, speed parameters of the gesture, and horizontal angle parameters of the gesture.

[0066] Specifically, the calculation formula of the distance parameter from the radar to the gesture subject is:

[0067]

[0068] Where B is the effective frequency modulation bandwidth of the radar, c is the speed of light, and f IF is the intermediate frequency signal, T is the frequency modulation period;

[0069] The calculation formula of the speed parameter of the gesture is:

[0070]

[0071] Where λ is the wavelength of the transmitted signal, T c is the frequency modulation period, f FFT is the Doppler shift, is the frequency shift of the echo signal;

[0072] As for the horizontal angle parameter of the gesture, it can be obtained by the MUSIC algorithm, which will not be described here.

[0073] Step 3: Filter out irrelevant objects and extract interesting data segments from the continuous data segments of the motion parameters to obtain interesting data segments containing gesture information.

[0074] Specifically, step 3.1, first define a gesture with N frames of data, each frame of data includes the number of objects monitored by the radar, the distance parameter from the radar to the gesture subject, the speed parameter of the gesture and the horizontal angle parameter of the gesture, where N is a positive integer;

[0075] Here N can be obtained as follows:

[0076] The radar is set to return 16 frames of data per second. Each frame of data contains parameters such as the number of objects monitored by the radar, distance, speed, and horizontal angle. The continuous data stream returned by the radar is parsed, and the average duration of the gesture is taken as 2.25 seconds, that is, every 36 frames of data represent a gesture, and N is 36. Of course, N can also take other values ​​by referring to the previous method, which will not be repeated here. In the following embodiments, N is 36.

[0077] like Figure 4 The figure shows a schematic diagram of the gesture and the radar when the gesture subject waves his hand. There is no obstacle between the palm and the radar. According to the relationship between angular velocity and linear velocity, the front end of the palm has the largest velocity.

[0078] Step 3.2, dynamic distance detection is performed on the continuous motion parameters, and the length of the speed detection window is set to 36 frames. The maximum speed in the speed detection window corresponds to the subject distance value where the gesture subject is located;

[0079] Step 3.3, take the first 2 cm to the last 5 cm of the subject distance value as the dynamic distance threshold, so that the interference of the radar module itself, the interference of the arm and the interference of other large objects in the experimental environment can be quickly filtered out.

[0080] In the present invention, the front refers to the direction approaching the radar module, and the rear refers to the direction away from the radar module; because in actual scenarios, only the motion parameters of the hand need to be detected, and the object with the maximum speed detected is the front end of the hand. Taking the front 2 cm and the rear 5 cm is to extract part of the data containing only the hand motion parameters. The interference of other objects is outside this distance range, so the interference can be filtered out. Here, 2 cm and 5 cm are the best values ​​obtained by the technicians of the present invention through hard judgment, and can be appropriately adjusted in actual applications.

[0081] In order to extract the data segment of interest after filtering out the interference, the continuous speed data needs to be detected. Figure 5 It is the original speed data segment after filtering out interference.

[0082] Step 3.4, perform absolute value operation on all speed data in the speed detection window; that is, first convert all speed data into absolute values, such as Figure 6 As shown, every 36 frames are still used as a detection window.

[0083] Performing absolute value operations can make the speed value more intuitively present a trend of first increasing and then decreasing, making detection easier.

[0084] Step 3.5, search for a data segment in the speed detection window where the absolute value of the speed increases from 0 to a maximum value and then decreases to 0, and the maximum value is greater than 0.7; 0.7 is the minimum value of the gesture movement speed obtained, and gestures below this speed will be ignored.

[0085] Step 3.6, the data segments corresponding to N / 2 frames (i.e., 18 frames each) before and after the time point where the maximum value in the data segment obtained in step 3.5 is located are the data segments of interest containing the motion gesture information.

[0086] Step 4: construct a gesture feature image based on the data segment of interest, and a plurality of the gesture feature images constitute a gesture dataset, and input the gesture dataset into a shallow convolutional neural network for recognition and classification.

[0087] Specifically, step 4.1, the distance parameter from the radar to the gesture subject, the speed parameter of the gesture, and the horizontal angle parameter of the gesture in each frame of data in the data segment of interest are spliced ​​in frame time sequence to obtain a distance-time diagram, a speed-time diagram, and a horizontal angle-time diagram respectively;

[0088] Step 4.2: normalize, grayscale and scale the distance-time graph, speed-time graph and horizontal angle-time graph to obtain a 64*64 gesture feature image, such as Figure 7 As shown (distance-time diagram, speed-time diagram, horizontal angle-time diagram are located at Figure 7 left, center, and right).

[0089] Several experimenters repeatedly make preset gestures such as push forward, pull back, swing left, swing right, swing up, and swing down within the radar field of view. After processing, no less than 9,000 feature images are obtained to form a gesture data set. The convolutional neural network is trained using the gesture data set. After the training is completed, the expected recognition accuracy can reach more than 95%. The trained convolutional neural network can be used as a classifier for gesture recognition and classification. Then the gesture feature map obtained in step 4.2 is input into the trained convolutional neural network for real-time recognition.

[0090] In step 4 of the present invention, the shallow convolutional neural network includes an input layer, a convolutional layer, a pooling layer, two fully connected layers and a softmax layer.

[0091] That is, the present invention uses a shallow convolutional neural network with only 6 layers, and the shallow network structure is as follows Figure 8As shown. The three feature images obtained by processing are input into the convolutional neural network in real time, with a size of 64×64 and a number of channels of 3; 20 convolution kernels of size 5×5×3 and a step size of 1 are used to extract features to obtain the first data, and the first data obtained after convolution is 60×60×20; ReLU is used as the excitation function to process the first data to obtain the second data, and the second data after ReLU is 60×60×20; the second data is pooled using max pool to obtain the third data, with a pooling kernel size of 2×2 and a step size of 2, and the third data after pooling is 30×30×20; the third data enters the first fully connected layer and the second fully connected layer in turn, with the first fully connected layer using 100 neurons and the second fully connected layer using 4 neurons, and finally connected to the softmax layer.

[0092] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A gesture recognition method based on multi-feature image fusion of frequency modulated continuous wave radar, characterized in that: The steps include: Step 1: collecting gesture data in real time at the radar end, and preprocessing the gesture data to obtain an intermediate frequency signal of the gesture data; Step 2: Calculate characteristic parameters of the intermediate frequency signal to obtain continuous motion parameters of the gesture; Step 3: filtering out irrelevant objects and extracting interesting data segments from the continuous data segments of the motion parameters to obtain interesting data segments containing gesture information; The specific process of filtering out irrelevant objects from the continuous data segments of the motion parameters is as follows: Step 3.1, define a gesture with N frames of data, each frame of data includes the number of objects monitored by the radar, the distance parameter from the radar to the gesture subject, the speed parameter of the gesture and the horizontal angle parameter of the gesture, where N is a positive integer; Step 3.2, dynamic distance detection is performed on the continuous motion parameters, and the length of the speed detection window is set to N frames. The maximum speed in the speed detection window corresponds to the subject distance value where the gesture subject is located; Step 3.3, taking the first 2 cm to the last 5 cm of the subject distance value as the dynamic distance threshold; The specific steps to extract the data segments of interest are: Step 3.4, performing an absolute value operation on all speed data in the speed detection window; Step 3.5, searching for a data segment in which the absolute value of the speed in the speed detection window increases from 0 to a maximum value and then decreases to 0, and the maximum value is greater than 0.7; Step 3.6: Take the data segment corresponding to each N / 2 frames before and after the time point where the maximum value of the data segment obtained in step 3.5 is located, which is the data segment of interest containing the motion gesture information. Step 4: construct a gesture feature image based on the data segment of interest, and input the gesture feature image into a shallow convolutional neural network for real-time recognition and classification.

2. The method for gesture recognition based on multi-feature image fusion of frequency modulated continuous wave radar according to claim 1, characterized in that: In step 1, the specific steps of collecting gesture data in real time and preprocessing the real-time collected gesture data to obtain an intermediate frequency signal are: Step 1.1, the RF module at the radar end generates a transmission signal which is sequentially transmitted through the frequency multiplier, the power amplifier and the transmitting antenna to send out a frequency modulated continuous wave radar signal; Step 1.2, the frequency modulated continuous wave radar signal is reflected by the waving gesture to generate an echo signal, and the echo signal passes through the receiving antenna and the low noise amplifier in sequence to reach the mixer; Step 1.3: The echo signal and the transmission signal are mixed through a mixer to obtain an intermediate frequency signal.

3. The method for hand gesture recognition based on multi-feature image fusion of frequency modulated continuous wave radar according to claim 2, characterized in that: The method also includes step 1.4, wherein the intermediate frequency signal is converted into a digital signal of gesture data by an analog-to-digital conversion module.

4. The method for gesture recognition based on multi-feature image fusion of FMCW radar according to claim 2 is characterized in that: In step 1.3, the specific formula for obtaining the intermediate frequency signal by the echo signal and the transmission signal through the mixer is: Where B is the effective frequency modulation bandwidth of the radar, T c is the frequency modulation period, R0 is the distance between the hand and the radar when the signal is transmitted, v is the speed of the hand, t is the time required for the transmitted signal to be transmitted to the hand, c is the speed of light, is the frequency shift of the echo signal.

5. The method for hand gesture recognition based on multi-feature image fusion of frequency modulated continuous wave radar according to claim 4, characterized in that: In step 2, the continuous motion parameters of the gesture include a distance parameter from the radar to the gesture subject, a speed parameter of the gesture, and a horizontal angle parameter of the gesture.

6. The method for hand gesture recognition based on multi-feature image fusion of FMCW radar according to claim 5, characterized in that: The calculation formula of the distance parameter from the radar to the gesture subject is: Where B is the effective frequency modulation bandwidth of the radar, c is the speed of light, and f IF is the intermediate frequency signal, T is the frequency modulation period; The calculation formula of the speed parameter of the gesture is: Where λ is the wavelength of the transmitted signal, T c is the frequency modulation period, f FFT is the Doppler shift, is the frequency shift of the echo signal.

7. The method for hand gesture recognition based on multi-feature image fusion of FMCW radar according to claim 1, characterized in that: Step 4: The specific steps of constructing a gesture feature image according to the data segment of interest are as follows: Step 4.1, the distance parameter from the radar to the gesture subject, the speed parameter of the gesture, and the horizontal angle parameter of the gesture in each frame of data in the data segment of interest are spliced ​​in the frame time sequence to obtain a distance-time diagram, a speed-time diagram, and a horizontal angle-time diagram respectively; Step 4.2: normalize, grayscale and scale the distance-time graph, speed-time graph and horizontal angle-time graph to obtain a 64*64 gesture feature image.

8. The method for hand gesture recognition based on multi-feature image fusion of frequency modulated continuous wave radar according to claim 1, characterized in that: In step 4, the shallow convolutional neural network includes an input layer, a convolutional layer, a pooling layer, two fully connected layers and a softmax layer.

Images