Method for gesture recognition by millimeter wave radar based on envelope data feature extraction

Abstract

The application relates to a millimeter wave radar gesture recognition method based on envelope data feature extraction, belonging to the technical field of millimeter wave radar application. Millimeter wave radar is used to collect data of different gestures to form envelope data, and the data is packed and saved in txt or mat format for subsequent processing; all packed data is read by Matlab, feature quantities of all data are directly calculated, and feature quantity scatter diagrams are drawn according to all permutations and combinations of the feature quantities; three feature quantities with the largest scatter point separation of different gestures are selected from the scatter diagrams, that is, the scatter points of different shapes are most obviously separated, and the feature quantities are used for gesture discrimination; interval division is carried out, and different intervals of each feature quantity correspond to different gesture determination. The application uses a single-transmitting single-receiving narrow pulse radar to collect gesture data and uses the method to perform gesture recognition and classification, so that a higher gesture recognition accuracy is realized by using a simpler algorithm and lower calculation complexity.

Description

Technical Field

[0001] This invention belongs to the technical field of millimeter-wave radar applications, and particularly relates to a millimeter-wave radar gesture recognition and classification problem based on multi-feature extraction and analysis. Background Technology

[0002] With technological advancements, millimeter-wave radar is increasingly used in daily life. Take nursing home caregiver monitoring as an example: traditional caregiver monitoring systems are often complex and bulky, requiring skilled operators. In contrast, millimeter-wave radar-based health radar systems only require a palm-sized radar placed in a specific location for continuous monitoring. They can also incorporate many other functions, such as heart rate and respiration detection, and even gesture recognition to enable further functionality (such as calling for caregivers). Millimeter-wave radar-based health radar systems require no additional equipment from operators, are unaffected by ambient light, have extremely low power consumption, high reliability, and can operate 24 / 7. Furthermore, high-frequency radar signals have high penetration, offering unparalleled advantages over low-frequency signal-based systems. Moreover, millimeter-wave radars are typically small and easy to integrate, allowing them to be embedded into other devices, greatly expanding their application scenarios. For large-scale scenarios like caregiver monitoring, radar offers a wide detection range and is less susceptible to interference from shielding objects; for smaller scenarios like elderly health monitoring and gesture recognition, radar offers high accuracy and strong anti-interference capabilities. Furthermore, such millimeter-wave health radar can be integrated into any mobile device (phone) or made into a discrete, externally connectable extension, making its application scenarios very flexible.

[0003] Among these, gesture recognition algorithms are relatively the most complex. Current gesture recognition algorithms include two-dimensional and three-dimensional ones. Two-dimensional gesture recognition algorithms mainly involve image processing technology and use computer vision to classify gestures; while three-dimensional gesture recognition includes depth information in addition to two-dimensional data, and millimeter-wave radar can easily obtain three-dimensional echo data of gestures.

[0004] Traditional gesture recognition algorithms are generally based on deep learning neural network methods and typically use multi-channel radar systems with multiple transmitters and receivers. They generally use linear frequency modulated (LFM) signals for transmission and reception, and first preprocess the received data, such as filtering, Fast Fourier Transform (FFT), and Short-Time Fourier Transform (STFT). Then, based on a large amount of data as training samples, a neural network (such as a convolutional neural network, CNN) is built to analyze and process the data, thereby achieving the gesture classification task. This classification method has high accuracy and good robustness, but it requires a large amount of test data as training samples, and the training time varies greatly depending on the depth and complexity of the network, resulting in high learning costs and computational complexity. Therefore, this method is not suitable for situations where the computing power of an integrated system is limited. Summary of the Invention

[0005] To address the shortcomings of the aforementioned neural network-based gesture recognition algorithms, this invention proposes a millimeter-wave radar gesture recognition method based on envelope data feature extraction. This method uses a single-transmitter, single-receiver ultra-narrow pulse radar to collect gesture data and then uses this method for gesture recognition and classification, achieving a high gesture recognition accuracy with a simpler algorithm and lower computational complexity.

[0006] A millimeter-wave radar gesture recognition method based on envelope data feature extraction includes the following steps:

[0007] Step S1: Use millimeter-wave radar to collect data of different gestures to form envelope data. The envelope data of each gesture is envelope data of equal frame length, and the data is packaged and saved in txt or mat format for subsequent processing.

[0008] Step S2: Read all the packaged data from Matlab, directly calculate the feature quantities of all the data, and draw a scatter plot of the feature quantities according to all permutations and combinations of the feature quantities. That is, use three optional feature quantities as coordinate axes. For each group of data, the values ​​of these three feature quantities are the x, y, and z values ​​in the coordinate system. Plot the point positions corresponding to each group of data in the spatial rectangular coordinate system. Points of different gesture data are represented by different shapes, and points of the same gesture have the same shape.

[0009] Step S3: Select the three features that have the largest separation between different gestures, that is, the most obvious separation between different shapes of scatter points, and use this set of features to distinguish gestures.

[0010] Step S4: Divide the intervals, and different intervals of each feature quantity correspond to different gesture judgments.

[0011] Preferably, the millimeter-wave radar of the present invention is connected to a PC via a USB interface, and the radar data receiving host computer program reads and saves the data.

[0012] Preferably, the millimeter-wave radar of the present invention is a continuous wave or pulse radar of any type, capable of generating envelope data of range cells.

[0013] Preferably, when collecting gesture data, the present invention places the hand directly in front of the millimeter-wave radar, with the radar placed flat on a table or held in the palm of the hand; the hand makes a corresponding gesture within a certain period of time, and the pulse radar signal is received by the receiving antenna after being reflected by the hand and pre-processed inside the radar chip. The resulting envelope data is then directly transmitted back to the PC and packaged and saved by the host computer program.

[0014] Preferably, the feature quantities of the present invention are the following eight types:

[0015] ① The cumulative difference value of each frame's distance unit;

[0016] ② The cumulative difference of the maximum amplitude in each frame;

[0017] ③ The variance of each distance cell in each frame;

[0018] ④ Variance of the maximum amplitude in each frame;

[0019] ⑤ Maximum amplitude;

[0020] ⑥ Amplitude mean;

[0021] ⑦ The maximum difference between the distance cells corresponding to the largest amplitude;

[0022] ⑧ The average distance cell difference between the first half and the second half of the frame sequence;

[0023] The scatter plot is a three-dimensional scatter plot, meaning it is plotted in total. A scatter plot is generated; from this, the three features that have the largest separation between different gestures are selected to form a group, that is, the separation between scatter plots of different shapes is the most obvious, and this group of features is used for gesture discrimination.

[0024] The recognition method of this invention is much simpler in terms of technical principle. First, this invention has lower requirements for the radar system, requiring only a simple single-transmitter, single-receiver system, resulting in lower cost. Second, this invention does not require any preprocessing of the obtained gesture envelope data; feature extraction can be performed directly on the raw echo signal data. Finally, this invention uses a method of defining feature ranges for gesture recognition and classification, which has low computational complexity, high accuracy, good robustness, and fast recognition speed, making it applicable to integrated systems of various levels. Attached Figure Description

[0025] Figure 1a This is a diagram illustrating a gesture approaching from directly in front.

[0026] Figure 1b A diagram illustrating a gesture moving away from directly in front;

[0027] Figure 1c This is a diagram illustrating a hand gesture of waving back and forth at a certain distance directly in front of you.

[0028] Figure 1d This is a diagram illustrating the gesture of making a fist at a certain distance directly in front of you.

[0029] Figure 2 This is an envelope diagram (two-dimensional, amplitude normalized) of one set of envelope data acquired by radar.

[0030] Figure 3 This is an envelope diagram (3D, amplitude unnormalized) of one set of envelope data acquired by radar.

[0031] Figure 4An embodiment of plotting a two-dimensional scatter plot for some feature combinations used in this invention, wherein: Figure 4 a is a two-dimensional scatter plot drawn using feature quantities ① and ⑧. Figure 4 b is a two-dimensional scatter plot drawn using feature quantities ① and ⑥. Figure 4 c is a two-dimensional scatter plot drawn using feature quantities ③ and ⑧. Figure 4 d is a two-dimensional scatter plot drawn using feature quantities ④ and ⑧;

[0032] Figure 5 This is a schematic diagram of a scatter plot (three-dimensional) of envelope data plotted using the optimal combination of feature quantities ①③⑧ adopted in this invention. Detailed Implementation

[0033] The invention will now be further described with reference to the accompanying drawings.

[0034] This invention is a gesture recognition algorithm based on a millimeter-wave radar system. It utilizes the envelope signal of the millimeter-wave radar. Data acquired by the millimeter-wave radar is transmitted back to the PC in real time and stored by the host computer program. The radar chip is only used for signal transmission and reception; all data processing is completed on the PC. All experiments can be conducted without a special microwave anechoic chamber. The millimeter-wave radar used can be any type of continuous wave or pulse radar, as long as it can generate envelope data for range units. The radar development board is not limited, but it should be equipped with a data transmission interface and a program burning interface. Relevant parameters should be adjusted according to specific circumstances during use.

[0035] Due to the limited data available from millimeter-wave radar, this invention uses only envelope data. Considering the performance and practical applications of millimeter-wave radar, this embodiment designs four simple and highly distinguishable hand gestures, encompassing movements of different directions and amplitudes. The basic hand gestures designed in this invention include: Figures 1a-1d As shown, these correspond to the following gestures: approaching from directly in front of the radar, moving away from directly in front of the radar, waving back and forth at a certain distance from directly in front of the radar, and clenching a fist at a certain distance from directly in front of the radar. Figures 1a-1d The images shown all represent the state when the millimeter-wave radar is not connected.

[0036] Since the millimeter-wave radar parameters are pre-configured and detailed parameters are provided in the user manual, they will not be described in detail here. The recognition method of this invention does not require the use of many millimeter-wave radar parameters; it only needs to adjust the effective measurement range of the millimeter-wave radar. Within this range, the number of distance units is valid. In the measurement of the recognition method of this invention, the millimeter-wave radar measurement range is approximately 10cm to 30cm. In preliminary testing, the distance unit of the millimeter-wave radar was measured to be 0.5mm. When collecting gesture data, the baud rate was set to 92160, the chirp of a single data packet was 30 (actually 29 frames were acquired), the total number of distance units was 414, and the data was saved in txt and mat formats, using envelope data. Using a fixed 30 frames of data ensures the integrity of gesture acquisition, facilitates subsequent data processing, and is more consistent with actual conditions.

[0037] The pulse radar signal, reflected by the hand, is received by the receiving antenna and pre-processed within the radar chip. The resulting envelope data is then directly transmitted back to the PC and packaged and saved by the host computer program, requiring no further data preprocessing. The collected gesture data is saved locally in txt and mat formats. The mat data is saved as a 29*414 matrix and can be directly opened with Matlab software; the txt file is a text document, with each frame containing a single range cell on a separate line. Each frame of 414 data points begins with '$' and ends with '#'. Radar data for a single gesture is packaged into the same file, meaning each file contains 29 frames of data for the same gesture. Through repeated acquisition, multiple sample data points for multiple gestures are obtained, and these samples are saved on the computer for subsequent feature extraction algorithms.

[0038] The recognition method of this invention is programmed using Matlab, therefore the data used is .mat data. This recognition method can be easily ported to algorithms in other languages, and the data read from .txt files is identical. On the computer, Matlab scripts are used to directly read all .mat files. The .mat data is organized in the format of each chirp sample number (distance unit number) × chirp number, resulting in a distance unit-frame order matrix S. ra The matrix is ​​a 29*414 matrix. In the range dimension, the nth column represents the envelope amplitude of the target at the range radar position n·ΔR+R0. A larger amplitude indicates a stronger reflected wave signal at that position, meaning the effective reflective surface of the object is near that position. In the time dimension, the mth row represents the envelope amplitude of the radar across all range dimensions in the slow time m·ΔT. Here, ΔR is the range cell, R0 is the range cell offset (i.e., the range at which the radar begins frequency sweeping), and ΔT is the frame duration of each frame sampled by the PCR radar, i.e., the slow time.

[0039] For Sra The raw envelope data is used to directly plot the two-dimensional and three-dimensional distance cell-frame sequence graphs. An image plotted using data from a single wave is shown below. Figure 2 Figure 3 As shown. The two-dimensional envelope image is the image after normalizing the envelope amplitude. A distance cell-normalized amplitude map is plotted for each frame, and a two-dimensional distance cell-frame sequence map is plotted to convert the amplitude into color. The three-dimensional image is not normalized; the x-axis represents the distance cell, the y-axis represents the frame sequence, and the z-axis represents the envelope data amplitude. From the two-dimensional image ( Figure 2 It can be seen that the distance unit of the waving data fluctuates slightly over time, as shown in the three-dimensional image ( Figure 3 It can be seen that the echo energy of the waving data changes significantly over time, which can be used as a basis for judging the waving action. The same applies to other gesture data.

[0040] Next, S for each gesture behavior ra Find its eigenvalues, which are the following 8 types:

[0041] ① The cumulative difference value of each frame's distance unit;

[0042] ② The cumulative difference of the maximum amplitude in each frame;

[0043] ③ The variance of each distance cell in each frame;

[0044] ④ Variance of the maximum amplitude in each frame;

[0045] ⑤ Maximum amplitude;

[0046] ⑥ Amplitude mean;

[0047] ⑦ The maximum difference between the distance cells corresponding to the largest amplitude;

[0048] ⑧ The average distance cell difference between the first half of the frame sequence and the second half of the frame sequence.

[0049] First, a two-dimensional scatter plot analysis was performed on the envelope data. Two statistical measures were arbitrarily selected as coordinate axes to create a two-dimensional scatter plot, with different gestures marked by dots of different shapes. The scatter plot showed the distribution of different statistical characteristics for each gesture data. Generally, the more concentrated the distribution of dots of the same shape (smaller intra-class distance), and the more dispersed the distribution of dots of different shapes (larger inter-class distance), the better the classification effect of that statistical measure. Figure 4 In the figure, closing, leaving, waving, and fisting represent the two-dimensional data point distribution of hand gestures indicating approach, departure, waving, and fist clenching, respectively, and are marked with □, ○, +, and · in the graph. Observing the two-dimensional statistical scatter plot, it is found that the classification effect is better when statistic ① or ⑧ is used as one of the parameters, such as... Figure 4 .in: Figure 4 a is a two-dimensional scatter plot drawn using feature quantities ① and ⑧. Figure 4b is a two-dimensional scatter plot drawn using feature quantities ① and ⑥. Figure 4 c is a two-dimensional scatter plot drawn using feature quantities ③ and ⑧. Figure 4 d is a two-dimensional scatter plot drawn using feature quantities ④ and ⑧.

[0050] Therefore, parameters ① and ⑧ are fixed, and a three-dimensional scatter analysis is further performed. Another feature quantity other than ① and ⑧ is selected. Using these three feature quantities as coordinate axes, the values ​​of these three feature quantities for each set of data are the x, y, and z values ​​in the coordinate system. The point positions corresponding to each set of data are plotted in a spatial rectangular coordinate system. Points for different gesture data are represented by different shapes, while points for the same gesture have the same shape.

[0051] The scatter plot is a three-dimensional scatter plot, meaning it is plotted in total. A scatter plot is generated. From this scatter plot, a set of three features that best separate the scatter points of different gestures is selected. That is, the scatter points of different shapes are most clearly separated, and this set of features can be used for gesture discrimination.

[0052] Then, intervals are divided, and different intervals of each feature quantity correspond to different gesture judgments.

[0053] The three optimal feature quantities selected are: ① the cumulative difference of distance units per frame, ③ the variance of distance units per frame, and ⑧ the average distance unit difference between the first and second half of the frame sequence. A 3D feature quantity scatter plot based on these three feature quantities is shown below. Figure 5 . Figure 5 In the figure, closing, leaving, waving, and fisting represent the two-dimensional data point distribution of the hand gestures of approaching, moving away, waving, and clenching fist, respectively, and are marked by □, ○, +, and ·. It can be seen that the separation between each gesture is relatively clear, meaning that this method can distinguish these four gestures well, and the statistical scatter plot achieves small intra-class spacing and large inter-class spacing. Dividing each feature value into intervals based on the scatter plot can serve as the basis for gesture discrimination. For example, if the value of feature value ① is [m...] a ,n a ), the value of characteristic quantity ③ is [m b ,n b The value of characteristic quantity ⑧ is [m] c ,n c When the envelope data is defined as gesture 1, it is determined to be gesture 1, and so on. Where m a ,n a ,m b .n b ,m c ,n c This represents the upper and lower boundaries of the interval.

[0054] Let the four gestures be numbered n as follows: approach (n=1); away (n=2); wave (n=3); clench fist (n=4).

[0055] For parameters ①, ③, and ⑧, define the statistical feature discrimination space for each gesture, that is, define an interval for each statistic. Let feature ① be inte d Characteristic ③ is var d The characteristic quantity ⑧ is d half When all three parameters of the gesture data fall within the specified interval, the gesture can be identified as such. Let ST denote the discrimination space of gesture n. n ,have

[0056] ST n ={inte d ,var d ,d half |inte d ∈(I d ,I u ),var d ∈(V d V u ),d half ∈(d d ,d u )}

[0057] Among them I d ,I u Let ST be the discrimination space for gesture n. n The upper and lower boundaries of parameter ①. V d V u Let ST be the discrimination space for gesture n. n The upper and lower boundaries of parameter ③. d ,d u Let ST be the discrimination space for gesture n. n The upper and lower boundaries of the intermediate parameter ⑧.

[0058] gesture data S ra Find S ra The parameter inte d var d d half If all three conditions are met, then... n If the gesture is valid, it is identified as gesture n; otherwise, it is identified as an invalid gesture.

[0059]

[0060] Supplement: The specific explanations for each characteristic quantity are as follows:

[0061] Let S be the data matrix for each gesture. ra =[s1,s2,…,s n] T ,(n=29), where Where n represents the total number of data frames, s i Let m represent the row vector of the i-th frame of data, and m represent the total number of distance units.

[0062] ① The cumulative difference value of each frame distance unit, inte d First, calculate the distance cell d corresponding to the maximum envelope magnitude of the i-th frame. i Then, for every two frames between d i Find the difference between each pair and add them together.

[0063]

[0064]

[0065] Where N is the total number of frames, which is 29 rows in the matrix; This represents the magnitude of the envelope amplitude at the j-th distance unit of the i-th frame of data.

[0066] ② The cumulative difference of the maximum amplitude in each frame, inte m First, calculate the maximum envelope amplitude mag of the i-th frame. i Then, for every two frames, the mag i Find the difference between each pair and add them together.

[0067]

[0068]

[0069] Where N is the number of frames, which is 29 rows in the matrix; This represents the magnitude of the envelope amplitude at the j-th distance unit of the i-th frame of data.

[0070] ③ Variance var of each frame distance unit d : The distance unit d of all frames obtained in ① i Find the variance

[0071]

[0072] Where N is the number of frames, i.e., the number of rows in the matrix, which is 29. μ d The mean distance unit value across all frames, i.e.

[0073]

[0074] ④ Variance of the maximum amplitude per frame (var) m The maximum amplitude mag of all frames obtained in ② i Find the variance

[0075]

[0076] Where N is the number of frames, i.e., the number of rows in the matrix, which is 29. μ m The average maximum amplitude across all frames, i.e.

[0077]

[0078] ⑤ Maximum amplitude (mag) max That is, to find the maximum value of all envelope amplitudes (29 * 414 values).

[0079]

[0080] in This represents the magnitude of the envelope amplitude at the j-th distance unit of the i-th frame of data.

[0081] ⑥ Amplitude mean mag ave That is, to calculate the mean of all envelope amplitudes (29 * 414 values).

[0082]

[0083] Where N is the number of frames, i.e., the number of rows in the matrix (29). M is the number of distance units per frame, i.e., the number of columns in the matrix (414). This represents the magnitude of the envelope amplitude at the j-th distance unit of the i-th frame of data.

[0084] ⑦ The maximum difference d between the distance cells corresponding to the maximum amplitude value dif That is, the distance unit d between any i-th frame and j-th frame obtained in ①. i ,d j Find the maximum value of each pairwise difference.

[0085]

[0086] Where N is the number of frames, which is the number of rows in the matrix, 29.

[0087] ⑧ The average distance cell difference d between the first half and the second half of the frame sequence half That is, the distance unit d of the i-th frame obtained in ①. i Find the first N / 2 d i The average value, calculate the last N / 2 d i The average of the two values ​​is then subtracted.

[0088]

[0089] in Let N be the largest even number not greater than N, where N is the number of frames, i.e., the number of rows in the matrix, which is 29. It should be 28.

Claims

1. A millimeter-wave radar gesture recognition method based on envelope data feature extraction, characterized in that, Includes the following steps: Step S1: Use millimeter-wave radar to collect data of different gestures to form envelope data. The envelope data of each gesture is envelope data of equal frame length, and the data is packaged and saved in txt or mat format for subsequent processing. Step S2: Read all the packaged data from Matlab, directly calculate the feature quantities of all the data, and draw a scatter plot of the feature quantities according to all permutations and combinations of the feature quantities. That is, use three optional feature quantities as coordinate axes. For each group of data, the values ​​of these three feature quantities are the x, y, and z values ​​in the coordinate system. Plot the point positions corresponding to each group of data in the spatial rectangular coordinate system. Points of different gesture data are represented by different shapes, and points of the same gesture have the same shape. Specifically: Matlab scripts are used to directly read all .mat files. The .mat data is organized in the format of sample count per chirp × chirp count, resulting in a distance cell-frame order matrix. In the range dimension, the nth column of data characterizes the range radar. The target's envelope amplitude indicates that a larger amplitude signifies a stronger reflected wave signal at that location, meaning the effective reflecting surface of the object is near that location. In the time dimension, the m-th row of data characterizes the radar's performance at that location. The envelope magnitude of the entire distance dimension in slow time; where For distance units, For distance cell offset, The frame duration for each frame sampled by the PCR radar, i.e., the slow time; right The raw envelope data is used to directly plot the two-dimensional and three-dimensional plots of the distance cell-frame sequence, and the image is plotted using data from a single wave. The two-dimensional envelope map is the image after the envelope amplitude is normalized. For each frame, a distance cell-normalized amplitude map is drawn, and a two-dimensional distance cell-frame sequence map is drawn to convert the amplitude into color. The three-dimensional map is not normalized. The x-axis is the distance cell, the y-axis is the frame sequence, and the z-axis is the envelope data amplitude. Step S3: Select the three features that have the largest separation between different gestures, that is, the most obvious separation between different shapes of scatter points, and use this set of features to distinguish gestures. Step S4: Divide the intervals, and different intervals of each feature quantity correspond to different gesture judgments.

2. The millimeter-wave radar gesture recognition method based on envelope data feature extraction according to claim 1, characterized in that, The millimeter-wave radar connects to a PC via a USB interface, and the host computer program reads and saves the radar data.

3. The millimeter-wave radar gesture recognition method based on envelope data feature extraction according to claim 1, characterized in that, Millimeter-wave radar is a continuous wave or pulse radar of any type, capable of generating envelope data for range cells.

4. The millimeter-wave radar gesture recognition method based on envelope data feature extraction according to claim 1, characterized in that, When collecting gesture data, place your hand directly in front of the millimeter-wave radar, with the radar placed flat on a table or held in your palm. Make the corresponding gesture within a certain time. The pulse radar signal is reflected by the hand and received by the receiving antenna, pre-processed inside the radar chip, and then the resulting envelope data is directly transmitted back to the PC and packaged and saved by the host computer program.

5. The millimeter-wave radar gesture recognition method based on envelope data feature extraction according to claim 1, characterized in that, The feature quantities are the following 8 types: ① The cumulative difference value of each frame's distance unit; ② The cumulative difference of the maximum amplitude in each frame; ③ The variance of each distance cell in each frame; ④ Variance of the maximum amplitude in each frame; ⑤ Maximum amplitude; ⑥ Amplitude mean; ⑦ The maximum difference between the distance cells corresponding to the largest amplitude; ⑧ The average distance cell difference between the first half and the second half of the frame sequence; The scatter plot is a three-dimensional scatter plot, meaning it is plotted in total. A scatter plot is generated; from this, the three features that have the largest separation between different gestures are selected to form a group, that is, the separation between scatter plots of different shapes is the most obvious, and this group of features is used for gesture discrimination.

Images