A gesture recognition method based on millimeter wave radar point cloud signal

Abstract

A gesture recognition method based on manual feature of millimeter wave radar point cloud signal, comprising the following steps: step 1, preparation stage: prepare a millimeter wave radar, place the millimeter wave radar in a fixed position indoors, and use it for subsequent gesture action collection; Step 2, offline construction stage: using the millimeter wave radar in step 1 to collect data, millimeter wave radar gesture data, and mark the corresponding classification label, after signal pretreatment of the millimeter wave radar gesture data, get the point cloud signal vector D s , the obtained point cloud signal vector is manually extracted to obtain a manual feature vector; Step 3, online recognition stage: the manual feature vector obtained in step 2 is classified and recognized by using a classification method. The present application avoids privacy leakage, avoids wearing complicated equipment at the same time, and further improves the gesture recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of gesture recognition systems, and particularly relates to a millimeter wave radar point cloud signal gesture recognition method based on handcrafted features. BACKGROUND

[0002] Gesture recognition is a non-contact human-computer interaction (HCI) method. When a gesture recognition system is introduced into a smart home system, visual-based gesture recognition has the problem of privacy leakage and is affected by light intensity; wearable device-based gesture recognition has the problem of not being easy to carry. SUMMARY

[0003] In order to overcome the deficiencies of the prior art, the present application provides a millimeter wave radar point cloud signal gesture recognition method based on handcrafted features, which deforms and extracts handcrafted features from the four-dimensional features (distance, Doppler velocity, pitch angle, and horizontal angle) of the point cloud signal, and classifies the obtained handcrafted features using a classification network, thereby achieving high-resolution gesture recognition while taking into account not leaking privacy and not wearing devices.

[0004] The technical solution adopted by the present application to solve its technical problems is:

[0005] A millimeter wave radar point cloud signal gesture recognition method based on handcrafted features, comprising the following steps:

[0006] Step 1, preparation phase: prepare a millimeter wave radar, place the millimeter wave radar at a fixed position indoors for subsequent gesture action collection;

[0007] Step 2, offline construction phase: collect data using the millimeter wave radar in step 1, millimeter wave radar gesture data, and mark the corresponding classification label, and obtain a point cloud signal vector D after signal preprocessing of the millimeter wave radar gesture data s , perform handcrafted feature extraction on the obtained point cloud signal vector to obtain a handcrafted feature vector;

[0008] Step 3, online recognition phase: classify and recognize the handcrafted feature vector obtained in step 2 using a classification method.

[0009] Further, in the step 2, the point cloud signal vector D s is established as follows:

[0010] Step 2-1, design a dynamic gesture action, then collect data at a fixed position at a distance from the millimeter wave radar described in step 1, millimeter wave radar gesture data X s , and mark the corresponding classification label F s ;

[0011] Step 2-2, process the millimeter-wave radar gesture data X obtained in step 2-1. s Filtering is performed to remove stationary targets and moving targets outside the range of hand gesture speed, resulting in the hand gesture dynamic data X. d ;

[0012] Step 2-3, process the gesture dynamic data X obtained in step 2-2. d A two-dimensional Fast Fourier Transform (2D-FFT) is performed, that is, a fast time-direction Fast Fourier Transform is performed on each pulse signal to obtain the distance information corresponding to each pulse. Then, the signals are stitched together along the slow time direction of the 2D-FFT to obtain the distance change vector V between the motion gesture and the millimeter-wave radar. d ;

[0013] Step 2-4, for the distance change vector V obtained in step 2-3 d Time-frequency analysis was performed using short-time Fourier transform to obtain the radial velocity variation vector V of the gesture relative to the millimeter-wave radar. v ;

[0014] Steps 2-5, for gesture dynamic data X d Perform eigenvalue decomposition to construct the signal subspace E of the signal. S and noise subspace E N By utilizing the orthogonality between the signal subspace and the noise subspace, a spatial spectral function can be constructed:

[0015]

[0016] In the formula, the denominator is the signal vector a(θ) and the noise matrix E. N The inner product, a H (θ) denotes the conjugate transpose of a(θ). E represents N The conjugate transpose of the signal and noise space, when the signal and noise spaces are orthogonal, gives the spatial spectrum P. MUSIC Since (θ) reaches its maximum value, the value of θ can be changed to find the spectral peak of the spatial spectrum. The θ value at this point is the direction of arrival (DOA) of the signal. The DOA values ​​θ form the angle change vector V. j ;

[0017] Steps 2-6: The signal vector of a point cloud in the point cloud signal. Obtain point cloud signal vector Where n is the total number of points in the point cloud signal;

[0018] Step 2-7, for the point cloud signal vector D obtained in step 2-6 s Manual feature extraction is performed to obtain the manual feature vector G. s .

[0019] Furthermore, in steps 2-7, the handcrafted feature vector G s The extraction steps are as follows:

[0020] Step 2-7-1, for the point cloud signal vector D obtained in step 2-6 s V d V v V j Deformation is performed, and the angle change vector V j Deformed pitch angle vector V f and horizontal angular vector V s Point cloud signal vector D s Deformation yields a three-dimensional coordinate vector V 3D =[V x V y V z ], where V x V is the X-axis vector. y V is the Y-axis vector. z The Z-axis vector represents the three-dimensional coordinates of a point, which are (x, y, z).

[0021] Step 2-7-2, combine the point cloud signal vector D from step 2-6. s And the three-dimensional coordinate vector V in step 2-7-1 3D And use the elevation vector V f and horizontal angular vector V s Replace the angle change vector V j The combined vector V is obtained. zu =[D s V 3D ] = [V d V v V f V s V x V y V z ], a combination vector of point clouds Where n is the total number of points in the point cloud signal;

[0022] Step 2-7-3: Within a certain three-dimensional coordinate range, set a suitable range F for the point cloud signal of the dynamic gesture. 3D = (x1~x2, y1~y2, z1~z2), for each of the steps in step 2-7-2 In With the set range F 3D In comparison, only and All in F 3D The point cloud within the range is retained, that is Other point clouds are deleted as interference points, and after deleting the interference points, a point deletion combined signal vector V is obtained szu ;

[0023] Step 2-7-4, the point deletion combined signal vector V obtained in step 2-7-3 is subjected to step 2-7-4 szu In the time t of the dynamic gesture action, that is, each frame of the point cloud signal, all point clouds in each frame are subjected to step 2-7-4 Manual feature extraction is performed, including maximum value, minimum value, arithmetic mean value, harmonic mean value, standard deviation, correlation coefficient, peak-peak amplitude, signal average power, skewness or kurtosis, to obtain a frame of manual feature vectors The manual feature vector G is obtained Where T is the total time of a dynamic gesture action.

[0024] Further, in step 3, the classification and recognition model classification step is as follows:

[0025] Step 3-1, select the classification and recognition model type, including support vector machine, decision tree, random forest, convolutional neural network or LSTM, and preliminarily design the model structure and each hyperparameter;

[0026] Step 3-2, input the manual feature vector G obtained in step 2-7-4 s and the corresponding classification label F s as a data set, randomly divided into a training set and a test set in proportion, the model is trained and iteratively optimized, the optimal model structure and hyperparameters are selected, and the final test model is obtained.

[0027] With the maturity of millimeter wave (30GHz-300GHz) radar technology, due to the high frequency of millimeter wave radar, less interference, high reliability, high precision, no privacy leakage problem and not affected by the lighting environment, etc. Advantages, millimeter wave radar gesture recognition as a non-contact human-computer interaction behavior can be widely used in smart furniture environment. The present application provides a millimeter wave radar point cloud signal gesture recognition method based on manual features, which uses a millimeter wave radar, solves the problems of privacy leakage and difficulty in carrying equipment, and uses manual features, point cloud signals and classification networks for the small amount of information obtained by the millimeter wave radar to achieve high resolution.

[0028] The beneficial effects of the present application mainly manifest in: the millimeter wave radar gesture recognition system can avoid the problem of privacy leakage compared with the gesture recognition system based on vision, and can avoid the cumbersome of wearing equipment compared with the gesture recognition system based on wearable devices. At the same time, in the small amount of signal information obtained and many interferences, point cloud signals and manual feature extraction methods are used to extract as many useful features as possible, and the optimal classification network is combined to achieve high resolution. Attached Figure Description

[0029] Fig. 1 This is a flowchart illustrating the overall process of the present invention.

[0030] Fig. 2 The design includes 12 hand gestures.

[0031] Fig. 3 This is the confusion matrix obtained using random forest classification. Detailed Implementation

[0032] The invention will now be further described with reference to the accompanying drawings.

[0033] Reference Figs. 1-3 A gesture recognition method for millimeter-wave radar point cloud signals based on handcrafted features includes the following steps:

[0034] Step 1, Preparation stage: Prepare a millimeter-wave radar and place it in a fixed indoor location for subsequent hand gesture data acquisition.

[0035] Step 2, Offline Construction Phase: Data acquisition is performed using the millimeter-wave radar from Step 1. Millimeter-wave radar gesture data is collected and labeled with corresponding classification tags. After signal preprocessing, the point cloud signal vector D is obtained. s Manual feature extraction is performed on the obtained point cloud signal vector to obtain a manual feature vector;

[0036] In step 2, the point cloud signal vector D s The steps to create it are as follows:

[0037] Step 2-1: Design dynamic hand gestures, and then collect data at a fixed position at a certain distance from the millimeter-wave radar described in Step 1. The millimeter-wave radar hand gesture data X... s And mark the corresponding category label F. s ;

[0038] Step 2-2, process the millimeter-wave radar gesture data X obtained in step 2-1. s Filtering is performed to remove stationary targets and moving targets outside the range of hand gesture speed, resulting in the hand gesture dynamic data X. d ;

[0039] Step 2-3, process the gesture dynamic data X obtained in step 2-2. d Performing a two-dimensional Fourier transform (2D-FFT) involves applying a fast time-direction Fast Fourier Transform to each pulse signal to obtain the distance information corresponding to each pulse. Then, the data is concatenated along the slow time direction of the 2D-FFT to obtain the distance change vector V between the motion gesture and the millimeter-wave radar. d ;

[0040] Step 2-4, the distance change vector V d is obtained by using short-time Fourier transform to analyze the time-frequency of the gesture dynamic data X v ;

[0041] Step 2-5, the gesture dynamic data X d is decomposed into signal subspace E S and noise subspace E N , and the spatial spectrum function P N (θ) is constructed by using the orthogonality between the signal subspace and the noise subspace.

[0042]

[0043] In the formula, the denominator is the inner product of the signal vector a(θ) and the noise matrix E H (θ) represents the conjugate transpose of a(θ), represents the conjugate transpose of E N , and when the signal and noise spaces are orthogonal, the spatial spectrum P MUSIC (θ) reaches the maximum value, so the spectral peak of the spatial spectrum can be obtained by changing the value of θ, and the value of θ at this time is the value of the signal direction of arrival, and the angle change vector V j is composed of the direction of arrival value θ.

[0044] Step 2-6, the signal vector of a point cloud in the point cloud signal is obtained. Wherein n is the total number of point clouds in the point cloud signal.

[0045] Step 2-7, the point cloud signal vector D s obtained in step 2-6 is manually extracted to obtain a manual feature vector G s .

[0046] In the step 2-7, the extraction step of the manual feature vector G s is as follows:

[0047] Step 2-7-1, the V s , V d , V v , V j of the point cloud signal vector D j obtained in step 2-6 is deformed, and the angle change vector V f is deformed into the pitch angle vector V s and the horizontal angle vector V s , and the three-dimensional coordinate vector V 3D is obtained by deforming the point cloud signal vector D [= [Vx , y , z , where V x is the X-axis vector, V y is the Y-axis vector, and V z is the Z-axis vector, and the three-dimensional coordinate value of a point is (x, y, z);

[0048] Step 2-7-2, combine the point cloud signal vector D s in step 2-6 with the three-dimensional coordinate vector V 3D in step 2-7-1, and replace the angle change vector V f with the elevation angle vector V s and the horizontal angle vector V j to obtain the combined vector V zu = [D s , V 3D ] = [V d , V v , V f , V s , V x , V y , V z ], and the combined vector of a point cloud is where n is the total number of point clouds in the point cloud signal;

[0049] Step 2-7-3, the point cloud signal of the dynamic gesture is within a certain three-dimensional coordinate range, and a suitable range F 3D = (x1~x2, y1~y2, z1~z2) is set. For each in step 2-7-2, compare with the set range F 3D , and only the point cloud whose and are within the range F 3D is retained, i.e. Other point clouds are deleted as interference points. After deleting the interference points, the deleted point combined signal vector V szu is obtained.

[0050] Step 2-7-4, for the deleted point combined signal vector V szu obtained in step 2-7-3, at the time t of the dynamic gesture action, i.e. each frame of the point cloud signal, the of all point clouds in each frame is manually extracted, including the maximum value, the minimum value, the arithmetic mean value, the harmonic mean value, the standard deviation, the correlation coefficient, the peak-peak amplitude, the signal average power, the skewness or the kurtosis, to obtain a frame of hand-crafted feature vector to obtain the hand-crafted feature vector where T is the total time of a dynamic gesture motion.

[0051] Step 3, online recognition stage: the manual feature vector obtained in step 2 is classified and recognized using a classification method.

[0052] In step 3, the classification and recognition model classification step is as follows:

[0053] Step 3-1, select the classification and recognition model type, including support vector machine, decision tree, random forest, convolutional neural network or LSTM, and preliminarily design the model structure and various hyperparameters;

[0054] Step 3-2, input the manual feature vector G obtained in step 2-7-4 s and the corresponding classification label F s as a data set, randomly divided into training set and test set in proportion, trained and iteratively optimized the model, selected the optimal model structure and hyperparameters, and obtained the final test model.

[0055] Compared with other methods, the gesture recognition method based on manual features of the millimeter wave radar point cloud signal avoids privacy leakage. Compared with the gesture recognition system based on wearable devices, the method avoids the cumbersome of wearing devices, and extracts as many useful features as possible through the point cloud signal and manual feature extraction method, and cooperates with the random forest classification network to realize high resolution. At the same time, the method proposed in the patent can also be combined with other gesture recognition technologies to further improve the performance. The present example is deployed in a real office scene. The experimental site is located in the conference room on the fifth floor of the old student building of Yuquan campus of Zhejiang University. When collecting data, the distance between the experimental personnel and the radar is 2.5 meters. A total of 12 gesture actions are designed as shown in Fig. 2 , which are S0: left hand spread; S1: left hand side up and down swing; S2: double hands in front of the chest and small arm rotation; S3: middle down swing in front (not limited to left and right hands); S4: middle up swing in front (not limited to left and right hands); S5: right hand spread; S6: right hand side up and down swing; S7: clockwise circle (not limited to left and right hands); S8: double hands spread; S9: double hands side up and down swing; S10: front up and down swing (not limited to left and right hands); S11: counterclockwise circle (not limited to left and right hands); the left and right hands and the direction are subject to the volunteer, the volunteer faces the radar, the direction of the x-axis is the direction of the radar, which is also the front of the volunteer, the direction of the y-axis is the right hand direction of the volunteer, and the direction of the z-axis is directly above, and the x-axis is represented by a black dot, the x-axis is perpendicular to the paper surface and outward, the y-axis is represented by a black circle, and the y-axis is perpendicular to the paper surface and inward. Here, random forest (RF) is selected as the classification model, after optimization, the forest hyperparameters are set, the number of trees N=500, and the depth of the tree depth=24.

[0056] From Fig. 3 It can be seen from the above table that the resolution of multiple gestures reaches 100% when using the method for gesture recognition classification, wherein the resolution of S7: clockwise circle drawing and S11: counterclockwise circle drawing is relatively low due to the too complex action, and the resolution of the remaining actions all reaches 94%.

[0057] The above embodiments are only used for describing the present application, but not for limiting the present application. Although the present application is explained in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications or equivalent replacements of the technical solutions of the present application do not deviate from the spirit and scope of the present application, and should be covered in the scope of claims of the present application.

Claims

1. A method for gesture recognition based on hand-crafted features of millimeter wave radar point cloud signals, characterized in that, The method comprises the following steps: Step 1, preparation phase: prepare a millimeter wave radar, and place the millimeter wave radar in a fixed position indoors for subsequent gesture action collection; Step 2, offline construction phase: data acquisition using the millimeter wave radar in step 1, millimeter wave radar gesture data, and mark the corresponding classification label, signal pretreatment of the millimeter wave radar gesture data to obtain a point cloud signal vector Manual feature extraction is performed on the obtained point cloud signal vector to obtain a manual feature vector; Step 3, online identification phase: the manual feature vector obtained in step 2 is classified and identified using a classification method; In said step 2, the point cloud signal vector is established in the following steps: Step 2-1, design dynamic gesture action, then collect data at a fixed position at a distance from the millimeter wave radar described in step 1, millimeter wave radar gesture data , and mark the corresponding classification label ; Step 2-2, filtering the millimeter wave radar gesture data obtained in step 2-1 Filtering is performed to filter out stationary targets and moving targets in the non-gesture motion speed range, to obtain gesture dynamic data ; Step 2-3: Process the gesture dynamic data obtained in Step 2-2. Perform two-dimensional Fourier transform 2D-FFT That is, perform a fast time to fast Fourier transform on each pulse signal to obtain the distance information corresponding to each pulse, and then follow the... 2D-FFT By stitching together the slow time-series data, we obtain the distance change vector between the motion gesture and the millimeter-wave radar. ; Step 2-4, to the distance change vector obtained in step 2-3 The time-frequency analysis is performed by using the short-time Fourier transform to obtain a radial velocity change vector of the motion gesture relative to the millimeter wave radar ; Step 2-5, gesture dynamic data Eigen-decomposition is performed to construct the signal subspace of the signal and the noise subspace Using the orthogonality between the signal subspace and the noise subspace, the spatial spectrum function can be constructed: = ； In the formula, the denominator is the signal vector. a ( θ ) and noise matrix The inner product, express a ( θ The conjugate transpose of ). express The conjugate transpose of the signal and noise space, when the signal and noise spaces are orthogonal, is the spatial spectrum. It has reached its maximum value, so it can be changed. θ The value of is used to calculate the spectral peaks of the spatial spectrum, and then the value of is obtained. θ The value is the direction of arrival (DOA) value of the signal, derived from the DOA value. θ Composition angle change vector ; Step 2-6, signal vector of one point cloud in the point cloud signal , to obtain the point cloud signal vector = [ , , ], where n is the total number of point clouds in the point cloud signal; Step 2-7, obtaining a hand-crafted feature vector by performing hand-crafted feature extraction on the point cloud signal vector obtained in step 2-6 Step 2-7, obtaining a hand-crafted feature vector by performing hand-crafted feature extraction on the point cloud signal vector obtained in step 2-6 Step 2-7, obtaining a hand-crafted feature vector by performing hand-crafted feature extraction on the point cloud 2.The hand-crafted feature based millimeter wave radar point cloud signal gesture recognition method of claim 1, wherein, In the step 2-7, the manual feature vector is extracted as follows: Step 2-7-1, deforming the point cloud signal obtained in Step 2-6 of , , , an angle change vector , a deformed pitch angle and a horizontal angle vector , a point cloud signal is deformed to obtain a three-dimensional coordinate vector [ , ] where is an X axis vector, is an Y axis vector, is an Z axis vector, and the three-dimensional coordinate value of a point is x, y, z ; Step 2-7-2: Combine the point cloud signal vectors from step 2-6. and the three-dimensional coordinate vector in step 2-7-1 and using elevation angle and horizontal angle vector Replace angle change vector , thus obtaining the combined vector = [ , ] = [ , , , , , , ], a combination vector of point clouds =[ , , ], = ; Step 2-7-3, the point cloud signal of dynamic gesture is in a certain three-dimensional coordinate range, and a suitable range is set ), compare each in step 2-7-2 with the set range , only are in the range , that is , other point clouds are deleted as interference points, and after deleting the interference points, the point deletion combination signal vector is obtained;​ Step 2-7-4, obtaining the punctured combined signal vector from the punctured combined signal vector obtained in step 2-7-3 at the time of the dynamic gesture action t , i.e. for each frame of the point cloud signal, obtaining a hand-crafted feature vector for all points in the frame by hand-crafted feature extraction including maximum, minimum, arithmetic mean, harmonic mean, standard deviation, correlation coefficient, peak-to-peak amplitude, signal average power, skewness or kurtosis , obtaining a hand-crafted feature vector = , wherein T is the total time of the dynamic gesture action.

3. The hand-crafted feature based millimeter wave radar point cloud signal gesture recognition method of claim 2, wherein, In the step 3, the classification and identification model classification step is as follows: Step 3-1, select the classification recognition model type, including support vector machine, decision tree, random forest, convolutional neural network or LSTM and preliminarily design the model structure and various hyperparameters; Step 3-2, input the hand-crafted feature vector obtained in step 2-7-4 and the corresponding classification label As a data set, randomly divided into training set and test set in proportion, the model is trained iteratively optimized, the optimal model structure and hyperparameters are selected, and the final test model is obtained.

Images