Millimeter wave radar gesture recognition method and recognition system based on multi-head self-attention mechanism

Abstract

The application relates to a millimeter wave radar gesture recognition method and a recognition system based on a millimeter wave radar. The application aims to solve the problem that existing radar-based gesture recognition technologies mostly utilize feature spectrum diagrams and convolutional neural networks to classify and recognize gestures, have long training time, occupy large storage space and do not consider attention mechanisms. The process is as follows: I. millimeter wave radar is adopted to collect gesture data to form a gesture data training set; II. a range-Doppler diagram is obtained; III. a range-time spectrum, a velocity-time spectrum, an azimuth spectrum and a pitch spectrum are simplified to obtain a 28*4-dimensional mixed feature vector; IV. a trained gesture recognition network is obtained; and V. gesture data to be measured collected by the millimeter wave radar is input into the trained gesture recognition network according to II and III to obtain a gesture data to be measured recognition result. The application is used in the field of gesture recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to a fast and light gesture recognition method and system based on millimeter wave radar. BACKGROUND

[0002] As a new type of human-computer interaction, non-contact gesture recognition conforms to people's body language habits and has broad application prospects. In the medical field, doctors can control medical equipment through gestures to achieve non-contact medical operations. In the automotive field, drivers and passengers can issue instructions to the car center console through gestures. In the smart home field, people can control air conditioners, televisions and other commonly used electrical appliances through gestures. In the AR / VR field, players can control objects in the game through gestures to enhance the sense of immersion. Because of its non-contact, strong perception of micro-motion targets, all-day, all-weather operation, no light influence, no privacy leakage and other advantages, gesture recognition based on millimeter wave radar has become an important human-computer interaction method. At present, most millimeter wave radars used for gesture recognition use frequency-modulated continuous wave technology and multiple transmitting and receiving antennas, which is the premise of the present application. The evaluation index of gesture recognition is mainly the type and recognition accuracy of gestures that can be recognized.

[0003] Generally, millimeter wave radar gesture recognition can be divided into three steps: first, use a millimeter radar sensor to detect and collect dynamic gesture information of a user; then, pre-process the echo signal to maximize the extraction of dynamic gesture features while filtering out interference clutter; finally, based on the results of gesture feature preprocessing, select the appropriate algorithm to classify and recognize gestures. Existing radar-based gesture recognition techniques mostly use feature spectrograms and convolutional neural networks for gesture classification and recognition, which takes a long time to train and occupies a large storage space, without considering the attention mechanism. SUMMARY

[0004] The purpose of the present application is to solve the problem of existing radar-based gesture recognition techniques that mostly use feature spectrograms and convolutional neural networks for gesture classification and recognition, which takes a long time to train and occupies a large storage space, without considering the attention mechanism, by proposing a millimeter wave radar gesture recognition method and system based on a multi-head self-attention mechanism.

[0005] The specific process of the millimeter wave radar gesture recognition method based on the multi-head self-attention mechanism is as follows:

[0006] Step 1: Use a millimeter wave radar with a signal form of frequency-modulated continuous wave to collect gesture data and form a gesture data training set.

[0007] Step 2: Pre-process the collected gesture data to obtain a range-Doppler (RD) map.

[0008] Step three, based on the range Doppler (RD) map obtained in step two, obtain a range-time map (RTM), a velocity-time map (DTM), an azimuth spectrum (ATM) and an elevation spectrum (ETM), simplify the range-time map (RTM), the velocity-time map (DTM), the azimuth spectrum (ATM) and the elevation spectrum (ETM), and finally obtain a mixed feature vector of 28x4 dimensions;

[0009] Step four, construct a gesture recognition network 8HBi-GRU, input the mixed feature vector into the gesture recognition network Bi-GRU to obtain a trained gesture recognition network Bi-GRU;

[0010] Step five, pre-process the gesture data collected by the millimeter wave radar to obtain a range Doppler (RD) map, based on the obtained range Doppler (RD) map, obtain a range-time map (RTM), a velocity-time map (DTM), an azimuth spectrum (ATM) and an elevation spectrum (ETM), simplify the range-time map (RTM), the velocity-time map (DTM), the azimuth spectrum (ATM) and the elevation spectrum (ETM), and finally obtain a mixed feature vector of 28x4 dimensions; input the obtained mixed feature vector of 28x4 dimensions into the trained gesture recognition network Bi-GRU to obtain a gesture data recognition result.

[0011] The millimeter wave radar gesture recognition system based on the multi-head self-attention mechanism is used to execute the millimeter wave radar gesture recognition method based on the multi-head self-attention mechanism.

[0012] The beneficial effects of the present application are:

[0013] The present application uses a millimeter wave radar with a signal form of a frequency-modulated continuous wave to collect gesture data, performs target detection and feature extraction on the data set, and finally performs gesture recognition of 12 gestures from the perspective of lightweight by means of a neural network.

[0014] To realize gesture recognition, the present application not only extracts common distance and velocity features, but also extracts azimuth angle and elevation angle. Currently, some radar-based gesture recognition methods directly put the original radar data into a neural network, and some extract distance, Doppler and angle of arrival information before putting them into a neural network. Most of them do not use elevation angle features. In comparison, the present application fully processes the data, and the feature extraction is more comprehensive and sufficient, which can represent a variety of micro-motion gestures.

[0015] The application uses a weighted average method to compress RTM, DTM, ATM and ETM data, extracts feature values as accurately as possible, and obtains a 28*4-dimensional mixed feature vector, which is 28 frames in total, each frame containing 4 feature values of distance, speed, azimuth angle and pitch angle, greatly reducing the data amount. The mixed feature vector and the proposed gesture recognition network 8HBi-GRU are used for classification, which can fully fuse the four features and extract the time correlation of gesture data. The experimental results show that for 12 kinds of micro-motion gestures, the recognition accuracy can reach 98.24%, and the model training and recognition speed is very fast, realizing fast and lightweight gesture recognition. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 Flowchart of the application

[0017] Figure 2a Example diagram of RTM data compression into feature vector;

[0018] Figure 2b Example diagram of DTM data compression into feature vector;

[0019] Figure 2c Example diagram of ATM data compression into feature vector;

[0020] Figure 2d Example diagram of ETM data compression into feature vector;

[0021] Figure 3 Structure diagram of the 8HBi-GRU network proposed by the application;

[0022] Figure 4 Schematic diagram of GRU model principle;

[0023] Figure 5 Schematic diagram of Bi-GRU model principle;

[0024] Figure 6 Schematic diagram of self-attention mechanism model;

[0025] Figure 7a Accuracy (Accuracy) change curve diagram in the training process of 8HBi-GRU network;

[0026] Figure 7b Loss (Loss) change curve diagram in the training process of 8HBi-GRU network;

[0027] Figure 8 Confusion matrix diagram obtained by testing the 8HBi-GRU network. DETAILED DESCRIPTION

[0028] Specific implementation method one: the specific process of the millimeter wave radar gesture recognition method based on the multi-head self-attention mechanism is as follows:

[0029] Step one: gesture data is collected by using a millimeter wave radar with a signal form of frequency modulation continuous wave (FMCW), and a gesture data training set is formed;

[0030] Step two: the collected gesture data is preprocessed to obtain a range-doppler (RD) map;

[0031] Step three: based on the range-doppler (RD) map obtained in step two, a range-time map (RTM), a velocity-time map (DTM), an azimuth spectrum (ATM), and an elevation spectrum (ETM) are obtained, the range-time map (RTM), the velocity-time map (DTM), the azimuth spectrum (ATM), and the elevation spectrum (ETM) are simplified, and finally a 28x4-dimensional mixed feature vector is obtained;

[0032] Step four: from the perspective of lightweight, a gesture recognition network 8HBi-GRU is constructed, the mixed feature vector is input into the gesture recognition network Bi-GRU, a trained gesture recognition network Bi-GRU is obtained, and the obtained lightweight model can finally achieve a recognition accuracy of 98.24%, having the advantages of short training time, fast training speed, and small data volume;

[0033] Step five: the gesture data to be measured collected by the millimeter wave radar is preprocessed to obtain a range-doppler (RD) map; based on the obtained range-doppler (RD) map, a range-time map (RTM), a velocity-time map (DTM), an azimuth spectrum (ATM), and an elevation spectrum (ETM) are obtained, the range-time map (RTM), the velocity-time map (DTM), the azimuth spectrum (ATM), and the elevation spectrum (ETM) are simplified, and finally a 28x4-dimensional mixed feature vector is obtained; the obtained 28x4-dimensional mixed feature vector is input into the trained gesture recognition network Bi-GRU, and a gesture data to be measured recognition result is obtained.

[0034] Specific implementation method two: the difference between this embodiment and specific implementation method one is that in step two, the collected gesture data is preprocessed to obtain a range-doppler (RD) map;

[0035] The specific process is as follows:

[0036] The static object component in the gesture data (echo signal) collected by the millimeter wave radar is filtered out by means of the MTI moving target indication technology, and gesture data after filtering out the static object component is obtained;

[0037] A 2D-FFT is performed on the gesture data after filtering out the static object component in the range dimension and the velocity dimension, and a range-doppler (RD) map is obtained;

[0038] The interference target in the obtained range Doppler (RD) graph is filtered out by using a constant false alarm rate (CFAR) detector (the interference target is detected and removed by the constant false alarm rate (CFAR) detector), and a range Doppler (RD) graph only containing the hand target after filtering out the interference target is obtained.

[0039] The other steps and parameters are the same as those in the first embodiment.

[0040] The third embodiment is different from the first or second embodiment in that the gesture data after filtering out the stationary object component is subjected to 2D-FFT in the range dimension and the velocity dimension to obtain a range Doppler (RD) graph.

[0041] The specific process is as follows:

[0042] The gesture data after filtering out the stationary object component is subjected to 2D-FFT in the range dimension and the velocity dimension to obtain a range Doppler (RD) graph (the horizontal axis is the velocity index, the vertical axis is the range index, and the value size is reflected on the RD graph as color), and the expression is as follows:

[0043]

[0044] wherein s IF (m, n) is the gesture data collected by a frequency-modulated continuous wave (FMCW) radar, N c is the number of chirps, N adc is the number of original gesture data collected by the frequency-modulated continuous wave (FMCW) radar; j is an imaginary unit, j 2 = -1; m is the original chirp index, and n is the original sampling point index.

[0045] The target will appear on the RD graph as a cluster of pixel points with relatively high energy on the RD graph.

[0046] The other steps and parameters are the same as those in the first or second embodiment.

[0047] The fourth embodiment is different from any one of the first to third embodiments in that, in step three, based on the range Doppler (RD) graph obtained in step two, a range-time spectrum (RTM), a velocity-time spectrum (DTM), an azimuth spectrum (ATM), and an elevation spectrum (ETM) are obtained, the range-time spectrum (RTM), the velocity-time spectrum (DTM), the azimuth spectrum (ATM), and the elevation spectrum (ETM) are simplified, and finally a 28x4-dimensional hybrid feature vector is obtained.

[0048] The specific process is as follows:

[0049] In step three, the range Doppler (RD) graph only containing the hand target is projected on the vertical axis, and frame-by-frame splicing is performed to obtain a range-time spectrum (RTM) graph.

[0050] Step three two, project the RD map containing only the human hand target on the horizontal axis and splice frame by frame to obtain a DTM map;

[0051] Step three three, perform DOA estimation on the human hand target points detected in the RD map containing only the human hand target, i.e., perform angle FFT in the horizontal channel dimension and splice frame by frame to obtain an ATM map; perform angle FFT in the vertical channel dimension and splice frame by frame to obtain an ETM map.

[0052] Step three four, simplify the RTM, DTM, ATM and ETM to obtain a 28x4-dimensional hybrid feature vector.

[0053] The other steps and parameters are the same as one of the first to third embodiments.

[0054] The fifth embodiment is different from one of the first to fourth embodiments in that the RTM, DTM, ATM and ETM are simplified in step three four to obtain a 28x4-dimensional hybrid feature vector;

[0055] The specific process is as follows:

[0056] The RTM, DTM, ATM and ETM are data compressed to obtain a 28x4-dimensional hybrid feature vector, where 28 represents 28 frames of data, and 4 represents four features of distance, velocity, azimuth and elevation, as shown in Figs. 2c and 2d. Figure 2a 、 2b

[0057] Step three four one, for the RTM map, the number of rows (the number of distance FFT points) and the number of columns of the RTM map are R=128 and L=28, respectively, i.e., the RTM map is spliced from the distance distribution of the target in 28 frames of data, the distance distribution of the target in each frame of data is divided into 128 distance units to represent, assuming that the energy value of the pixel point in the lth column and the rth row is represented as E_r(l,r), then the distance estimation f_r(l) of the target in the lth frame of data can be represented as:

[0058]

[0059] wherein, l=1, 2, …, L, r=1, 2, …, R;

[0060] The RTM map with the size of RxL is simplified to a feature vector with the size of 1xL by performing this operation on 28 frames of data, this dimension reduction method is efficient and direct, and one number can reflect the distance information of the target in a frame of data;​

[0061] Step three four two, for the velocity-time spectrum DTM, the number of rows (the number of points of distance FFT) and the number of columns of the velocity-time spectrum DTM are D=128 and L=28 respectively, assuming that the energy value of the pixel point of the lth column and the dth row is represented as E_d(l,d), then the distance estimation f_d(l) of the target in the lth frame data can be represented as:

[0062]

[0063] wherein l=1,2,…,L, d=1,2,…,D;

[0064] After the operation on the 28 frames of data, the velocity-time spectrum DTM with the size of D×L is simplified into a feature vector with the size of 1×L;

[0065] Step three four three, for the azimuth spectrum ATM, the number of rows (the number of points of distance FFT) and the number of columns of the azimuth spectrum ATM are A=160 and L=28 respectively, assuming that the energy value of the pixel point of the lth column and the ath row is represented as E_a(l,a), then the distance estimation f_a(l) of the target in the lth frame data can be represented as:

[0066]

[0067] wherein l=1,2,…,L, a=1,2,…,A;

[0068] After the operation on the 28 frames of data, the azimuth spectrum ATM with the size of A×L is simplified into a feature vector with the size of 1×L;

[0069] Step three four four, for the elevation spectrum ETM, the number of rows (the number of points of distance FFT) and the number of columns of the elevation spectrum ETM are E=50 and L=28 respectively, assuming that the energy value of the pixel point of the lth column and the e th row is represented as E_e(l,e), then the distance estimation f_e(l) of the target in the lth frame data can be represented as:

[0070]

[0071] wherein l=1,2,…,L, e=1,2,…,E;

[0072] After the operation on the 28 frames of data, the elevation spectrum ETM with the size of E×L is simplified into a feature vector with the size of 1×L;

[0073] Step three four five, time alignment and splicing are performed on the above four feature vectors, to obtain a mixed feature vector with the size of 28×4 (four 28-dimensional vectors are spliced into a 4×28 vector).

[0074] Other steps and parameters are the same as one of embodiments one to four.

[0075] Embodiment six: different from one of embodiments one to five, in step four, from the perspective of light weight, the gesture recognition network 8HBi-GRU is constructed, the mixed feature vector is input into the gesture recognition network Bi-GRU, the trained gesture recognition network Bi-GRU is obtained, and the light weight model finally achieves 98.24% of the recognition accuracy, has the advantages of short training time, fast training speed and small data volume;

[0076] The specific process is:

[0077] Step four one, the gesture recognition network 8HBi-GRU is constructed; the specific process is:

[0078] The gesture recognition network 8HBi-GRU sequentially includes a first bidirectional GRU layer, a second bidirectional GRU layer, a multi-head self-attention mechanism layer, a summation layer and a full connection layer;

[0079] Step four two, the 28x4-dimensional mixed feature vector obtained in step three is input into the gesture recognition network Bi-GRU until convergence, and the trained gesture recognition network Bi-GRU is obtained.

[0080] Other steps and parameters are the same as one of embodiments one to five.

[0081] Embodiment seven: different from one of embodiments one to six, in step four two, the 28x4-dimensional mixed feature vector obtained in step three is input into the gesture recognition network Bi-GRU until convergence, and the trained gesture recognition network Bi-GRU is obtained.

[0082] The specific process is:

[0083] The 28x4-dimensional mixed feature vector obtained in step three is input into the first bidirectional GRU layer, the first bidirectional GRU layer outputs the feature vector, the feature vector is input into the multi-head self-attention mechanism layer, the multi-head self-attention mechanism layer outputs the feature vector, the feature vector is input into the summation layer for summation operation (28x512 is changed to 512 after the summation layer), the summation layer outputs the feature vector, the feature vector is input into the full connection layer, and the full connection layer outputs a 12-dimensional vector, as shown in Figure 2a 、 2b , 2c, 2d, wherein the maximum value corresponds to the class, i.e. the recognition result, until convergence, and the trained gesture recognition network Bi-GRU is obtained.

[0084] The 28 represents the time, and the 4 represents the vector, respectively, the distance-time spectrum (RTM), the velocity-time spectrum (DTM), the azimuth spectrum (ATM) and the pitch spectrum (ETM);

[0085] The other steps and parameters are the same as one of embodiments one to six.

[0086] Embodiment eight: This embodiment is based on a multi-head self-attention mechanism-based millimeter wave radar gesture recognition system, which is used to perform a multi-head self-attention mechanism-based millimeter wave radar gesture recognition method.

[0087] The principle of the GRU network is as follows:

[0088] The recurrent unit of the GRU mainly consists of a reset gate and an update gate, which are represented by r t and z t respectively, and the om operator is defined as 1 minus the operation of the input data, and the structure of the GRU is shown in Figure 4 .

[0089] The reset gate fuses the new input information x t with the reserved information h t-1 at the last moment; the greater the value of the reset gate, the more useful information is reserved, and the calculation method is

[0090] r t =σ(W r ×[h t-1 ,x t ]) (6)

[0091] where W r is a parameter to be trained, r t is the value of the reset gate, and σ() is a sigmoid activation function.

[0092] The update gate determines the degree of influence of the information at the last moment on the current moment, and the greater the value of the update gate, the greater the influence of the information at the last moment on the current moment, and the update gate calculation formula is

[0093] z t =σ(W z ×[h t-1 ,x t ]) (7)

[0094] where W z is a parameter to be trained, and z t is the value of the update gate.

[0095] The calculation formula of the candidate hidden layer state is

[0096]

[0097] where Tanh() is a Tanh activation function, W xg , and W hgis the weight matrix (to be trained parameters), b g is the bias vector (to be trained parameters), It can be seen as the weighted fusion of hidden layer information at two moments, which needs to use the reset gate r t to calculate how much information is retained at the last moment, when r t = 0, the previous moment information is not memorized in the calculation process.

[0098] Then update the hidden layer state, the calculation method is

[0099]

[0100] GRU can obtain the inter-frame features of time series, but can only capture the forward and backward time series correlation, which is far from enough in complex classification problems.

[0101] The Bi-GRU network model (bidirectional GRU network model) is used to extract the time correlation of the sequence in both directions, and the context features of the time series are obtained, that is, the hidden layer state at t moment is determined by the hidden layer state at t+1 moment and t-1 moment (that is, h t is determined by h t-1 and h t+1 , so it can extract in both directions), and its structure is shown in Figure 5 ;

[0102] In the present application, n is the frame number, x t is a 4-dimensional vector, t = 1, 2,..., 28, the hidden layer size is 256 (that is, the dimension of h), and since the bidirectional GRU network is used, the output size is 28x512 (if the ordinary one-way GRU is used, the output size is 28x256, if the bidirectional is used, the output size is doubled, and the output size is 28x512).

[0103] The principle of self-attention mechanism is as follows:

[0104] The calculation of the self-attention mechanism depends on the query vector q, the key value vector k and the value vector v, which are obtained by multiplying the input sequence x (as shown in Figure 3 , the size of the sequence x is 4x28) and the weight matrix:

[0105] q = x x W Q (10)

[0106] k = x x W K (11)

[0107] v = x x W V (12)

[0108] Where, WQ , W K , W V indicates a corresponding weight matrix to realize the conversion of feature dimension, is a to-be-learned parameter in the self-attention mechanism, and is realized by using a linear layer to obtain q, k and v;

[0109] The self-attention mechanism refers to first obtaining a weight value allocated by q and k, then normalizing by using a Softmax function, finally weighting and summing the obtained weight value and a value vector v to obtain an Attention value, and the process is as shown in Figure 6 ;

[0110] For the feature vector x i at the i moment, it is assumed that the attention weight obtained by the feature vector x i at the j moment is a ij , and the final output is y i , and the related calculation formula is as follows:

[0111]

[0112]

[0113] , wherein d k is the depth of the query vector or the key value vector, in the application, d k = 512; q i is the corresponding query vector of the feature vector x i at the i moment, k j is the corresponding key value vector of the feature vector at the j moment, T is transposition, and v i is the value vector corresponding to the feature vector;

[0114] The multi-head self-attention mechanism is used for attention allocation, and the specific process is as follows:

[0115] The m-head self-attention mechanism is used, for a certain frame feature vector x i , the corresponding query vector q i , key value vector k i and value vector v i are respectively divided into m sub-vectors, and m groups of q, k and v combinations can be obtained;

[0116] The 28*512 size vector q becomes m groups of size vectors;

[0117] The 28*512 size vector k becomes m groups of size vectors;

[0118] The 28*512 size vector v becomes m groups of size vectors;

[0119] The group performs a self-attention mechanism operation, that is, a plurality of groups of outputs y are obtained, and then the plurality of groups of outputs y are merged to obtain a final attention output, as shown in Figure 2a 、 2b , 2c, 2d.

[0120] For a continuous input time sequence x, the time sequence length is n, and the final attention output u obtained after m head self-attention mechanism processing can be expressed by a formula as follows:

[0121] u = C (h1(x1, x2, ···, x n ), ···, h k (x1, x2, ···, x n ), ···, h m (x1, x2, ···, x n ))(15)

[0122] Wherein, h k (x1, x2, ···, x n ) represents the result of the kth self-attention operation, C(·) represents the merging processing of the output y in order, and the calculation process of each single head self-attention is completely same, but the weight matrix is different.

[0123] In the present application, m is selected as 8, and the output size of the multi-head self-attention mechanism is 28x512.

[0124] The beneficial effects of the present application are verified by the following embodiments:

[0125] Embodiment one: 8HBi-GRU is used to classify and identify 12 kinds of micro-motion gestures.

[0126] The 12 kinds of micro-motion gestures include: the operator facing the millimeter wave radar, 1) tick, 2) cross, 3) clockwise circle, 4) counterclockwise circle, 5) left and right waving hands, 6) left fan, 7) right fan, 8) waving hands, 9) waving hands, 10) single finger TAP, 11) palm clenched, 12) palm open. The present application invites 10 experimental personnel (6 men and 4 women) to participate in gesture data collection, the gesture action distance is between 20-60 cm away from the radar plane, the horizontal direction angle range is limited to ±80°, the vertical direction angle range is limited to ±25°, each person collects approximately the same number of gestures, and finally forms a gesture data set of 600 groups of samples for each kind of gesture, a total of 7200 groups of samples, wherein 70% is randomly extracted to process into a feature data set for training, and the remaining 30% is used for testing.

[0127] The experiment was conducted in the Python3.8, Pytorch1.12.0 environment, the CPU was i7-12700H, the GPU was RTX3060, and the system was Windows10. The training learning rate was set to 0.001, the Adam optimizer was used, and the iteration was 60 rounds.

[0128] The accuracy and loss curves during model training are shown in Figure 7a 7b As can be seen, with the progress of model training, the accuracy and loss of the training set and test set gradually converge after enough rounds of iteration, and the final overall classification accuracy is 98.24%.

[0129] The model was tested, and the confusion matrix is shown in Figure 8 The precision, recall, and F1-score of the 12 gestures are shown in Table 1.

[0130] Table 1: Performance of 12 gestures

[0131]

[0132] It can be seen that for left fan, right fan, waving hand and other large amplitude palm movement gestures, this classification scheme can achieve nearly 100% recognition rate, and for tick, cross, circle and other large amplitude finger movement gestures, the recognition effect of this classification scheme is also ideal. But for the pair of relatively micro-movement and easy-to-confuse gestures of clenched fist and open palm, the recognition effect is not very good, and the recognition rate can only reach about 90%.

[0133] Example 2: In order to verify the optimization effect of multi-head self-attention mechanism on Bi-GRU network and find the appropriate number of taps of multi-head self-attention mechanism, Bi-GRU network and Bi-GRU network with multi-head self-attention mechanism with attention head number of 0, 1, 2, 4, 8, 16 were experimented. The training learning rate was set to 0.001, the Adam optimizer was used, and the iteration was 60 rounds, and the test accuracy is shown in Table 2.

[0134] Table 2: Bi-GRU and multi-head self-attention mechanism experimental results

[0135]

[0136] ​1H Bi-GRU represents a Bi-GRU network combined with a single-head self-attention mechanism, 2H Bi-GRU represents a network combined with a multi-head self-attention mechanism with a tap number of 2 and Bi-GRU, and so on. It can be seen from the experimental results that the effect of the multi-head self-attention mechanism is greatly affected by the number of taps. When the tap number is appropriate, the multi-head self-attention mechanism has a certain improvement effect on the performance of the Bi-GRU network. When the tap number is not appropriate, it may even appear a negative optimization phenomenon. It can be found from the experimental data that the optimization effect of the multi-head self-attention mechanism on the Bi-GRU network is the largest when the tap number is set to 8, and the final recognition accuracy is the highest.

[0137] Example three: compare the 8HBi-GRU scheme of the application with other schemes to verify the advancement and superiority of the application. The experimental results are shown in Table 3.

[0138] Among them, VGG16, Resnet50, Resnet101, DenseNet121 and DenseNet161 take mixed feature spectrum graphs (RTM, DTM, ATM and ETM) as input, and the input data occupies a storage space of 1.15GB. The experiment shows that this kind of method has large data volume, many training parameters and too long training time.

[0139] In contrast, the method of taking mixed feature vectors as input, such as CNN, CNN-LSTM, CNN-Bi-GRU, LSTM, Bi-LSTM, Bi-GRU and 8HBi-GRU, only occupies a storage space of 0.82MB, has fewer training parameters and shorter training time, and can complete training within 3 minutes. Among them, CNN is composed of 4 one-dimensional convolution layers and 2 fully connected layers, LSTM is composed of two LSTM layers and one classifier, CNN-LSTM is composed of CNN and LSTM in cascade, and CNN-Bi-GRU is composed of CNN and Bi-GRU in cascade.

[0140] Experiments show that compared with CNN, CNN-LSTM, CNN-Bi-GRU, LSTM, Bi-LSTM and Bi-GRU, our 8HBi-GRU model has the highest recognition rate, strong superiority and application prospect.

[0141] Table 3 Comparison of the performance of our model and other models

[0142]

[0143] The present application can have other various embodiments, and those skilled in the art can make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application, and these corresponding changes and modifications shall all belong to the protection scope of the claims of the present application.

Claims

1. A millimeter-wave radar gesture recognition method based on a multi-head self-attention mechanism, characterized in that: The specific process of the method is as follows: Step 1: Use a millimeter-wave radar with frequency-modulated continuous wave signal to collect gesture data and form a gesture data training set; Step 2: Preprocess the collected gesture data to obtain the distance Doppler RD map; Step 3: Based on the range-Doppler RD map obtained in Step 2, obtain the range-time spectrum (RTM), velocity-time spectrum (DTM), azimuth spectrum (ATM), and elevation spectrum (ETM). Simplify the range-time spectrum (RTM), velocity-time spectrum (DTM), azimuth spectrum (ATM), and elevation spectrum (ETM) to finally obtain a 28×4-dimensional mixed feature vector. Step 4: Construct the gesture recognition network 8HBi-GRU, input the hybrid feature vector into the gesture recognition network Bi-GRU, and obtain the trained gesture recognition network Bi-GRU. Step 5: Preprocess the hand gesture data collected by the millimeter-wave radar to obtain the range-Doppler RD map; based on the obtained range-time spectrum (RTM), velocity-time spectrum (DTM), azimuth spectrum (ATM), and elevation spectrum (ETM), simplify the range-time spectrum (RTM), velocity-time spectrum (DTM), azimuth spectrum (ATM), and elevation spectrum (ETM) to obtain a 28×4-dimensional mixed feature vector; input the obtained 28×4-dimensional mixed feature vector into the trained hand gesture recognition network Bi-GRU to obtain the hand gesture data recognition result.

2. The millimeter-wave radar gesture recognition method based on multi-head self-attention mechanism according to claim 1, characterized in that: In step two, the collected gesture data is preprocessed to obtain a distance Doppler RD map; The specific process is as follows: By using MTI moving target display technology to filter out stationary object components in gesture data acquired by millimeter-wave radar, gesture data after filtering out stationary object components is obtained. Perform 2D-FFT on the distance and velocity dimensions of the gesture data after filtering out stationary object components to obtain the range-Doppler RD map; A constant false alarm rate (CFAR) detector was used to filter out interfering targets in the obtained range-Doppler RD map, resulting in a range-Doppler RD map containing only the human hand target after filtering out the interfering targets.

3. The millimeter-wave radar gesture recognition method based on multi-head self-attention mechanism according to claim 2, characterized in that: The gesture data after filtering out stationary object components is subjected to 2D-FFT in the distance and velocity dimensions to obtain the range Doppler RD map. The specific process is as follows: Performing a 2D-FFT on the gesture data after filtering out stationary objects in the distance and velocity dimensions yields the range-Doppler RD map, expressed as: Among them, s IF (m,n) represents the gesture data acquired by the frequency modulated continuous wave (FMCW) radar, N c N is the chirp number of the pulse signal. adc The number of hand gesture data points originally acquired by the Frequency Modulated Continuous Wave (FMCW) radar; j is the imaginary unit, j 2 =-1; m is the chirp index of the original pulse signal, and n is the index of the original sampling point.

4. The millimeter-wave radar gesture recognition method based on multi-head self-attention mechanism according to claim 3, characterized in that: In step three, based on the range-Doppler RD map obtained in step two, range-time spectrum (RTM), velocity-time spectrum (DTM), azimuth spectrum (ATM), and elevation spectrum (ETM) are obtained. The range-time spectrum (RTM), velocity-time spectrum (DTM), azimuth spectrum (ATM), and elevation spectrum (ETM) are simplified to finally obtain a 28×4-dimensional mixed feature vector. The specific process is as follows: Step 3: Project the range-Doppler (RD) map containing only the human hand target onto the vertical axis and stitch them together frame by frame to obtain the range-time spectrum (RTM) map. Step 32: Project the range Doppler RD map containing only the human hand target onto the horizontal axis and stitch them together frame by frame to obtain the velocity-time spectrum DTM map; Step 3: Perform Direction of Arrival (DOA) estimation on the detected hand target points in the range Doppler RD map containing only hand targets. That is, perform angle FFT in the horizontal channel dimension and stitch the images frame by frame to obtain the azimuth spectrum ATM. Angle FFT is performed in the vertical channel dimension, and the frames are stitched together to obtain the elevation spectrum ETM; Steps 3 and 4: Simplify the range-time spectrum (RTM), velocity-time spectrum (DTM), azimuth spectrum (ATM), and elevation spectrum (ETM) to obtain a 28×4 dimensional mixed feature vector.

5. The millimeter-wave radar gesture recognition method based on multi-head self-attention mechanism according to claim 4, characterized in that: In steps three and four, the range-time spectrum (RTM), velocity-time spectrum (DTM), azimuth spectrum (ATM), and elevation spectrum (ETM) are simplified to obtain a 28×4-dimensional mixed feature vector. The specific process is as follows: Step 341: For the range-time spectrum (RTM) map, the number of rows and columns of the RTM map are R = 128 and L = 28, respectively. That is, the range-time spectrum (RTM) map is composed of the distance distribution of the target in 28 frames of data. The distance distribution of the target in each frame of data is divided into 128 distance units. Assuming that the energy value of the pixel in the l-th column and r-th row is represented as E_r(l,r), then the distance estimate f_r(l) of the target in the l-th frame of data can be expressed as: Where l = 1, 2, ..., L, r = 1, 2, ..., R; Performing this operation on 28 frames of data simplifies the R×L distance-time spectrum RTM plot into a feature vector of size 1×L. Step 342: For the velocity-time spectrum DTM map, the number of rows and columns of the velocity-time spectrum DTM map are D=128 and L=28, respectively. Assuming that the energy value of the pixel in the l-th column and d-th row is represented as E_d(l,d), then the distance estimate f_d(l) of the target in the l-th frame of data can be expressed as: Where l = 1, 2, ..., L, d = 1, 2, ..., D; Performing this operation on 28 frames of data simplifies the D×L size velocity-time spectrum DTM plot into a feature vector of size 1×L. Step 3.

4. For the azimuth spectrum ATM, the number of rows and columns of the azimuth spectrum ATM are A = 160 and L = 28, respectively. Assuming the energy value of the pixel in the l-th column and a-th row is represented as E_a(l,a), then the distance estimate f_a(l) of the target in the l-th frame of data can be expressed as: Where, l = 1, 2, ..., L, a = 1, 2, ..., A; Performing this operation on 28 frames of data simplifies the A×L azimuth spectrum ATM into a feature vector of size 1×L. Step 344: For the elevation spectrum ETM, the number of rows and columns of the elevation spectrum ETM are E=50 and L=28, respectively. Assuming the energy value of the pixel in the l-th column and e-th row is represented as E_e(l,e), then the target distance estimate f_e(l) in the l-th frame of data can be expressed as: Where l = 1, 2, ..., L, e = 1, 2, ..., E; Performing this operation on 28 frames of data simplifies the E×L-sized elevation spectrum ETM into a feature vector of size 1×L. Steps 3, 4, and 5: Time-align and concatenate the above four feature vectors to obtain a 28×4 dimensional mixed feature vector.

6. The millimeter-wave radar gesture recognition method based on multi-head self-attention mechanism according to claim 5, characterized in that: In step four, a gesture recognition network 8HBi-GRU is constructed, and the hybrid feature vector is input into the gesture recognition network Bi-GRU to obtain the trained gesture recognition network Bi-GRU. The specific process is as follows: Step 41: Construct the gesture recognition network 8HBi-GRU; the specific process is as follows: The gesture recognition network 8HBi-GRU consists of a first bidirectional GRU layer, a second bidirectional GRU layer, a multi-head self-attention mechanism layer, a summation layer, and a fully connected layer. Step 4.2: Input the 28×4-dimensional mixed feature vector obtained in Step 3 into the gesture recognition network Bi-GRU until convergence, and obtain the trained gesture recognition network Bi-GRU.

7. The millimeter-wave radar gesture recognition method based on multi-head self-attention mechanism according to claim 6, characterized in that: In step four, the 28×4-dimensional mixed feature vector obtained in step three is input into the gesture recognition network Bi-GRU until convergence, thus obtaining the trained gesture recognition network Bi-GRU. The specific process is as follows: The 28×4 dimensional mixed feature vector obtained in step 3 is input into the first bidirectional GRU layer. The output feature vector of the first bidirectional GRU layer is input into the multi-head self-attention mechanism layer. The output feature vector of the multi-head self-attention mechanism layer is input into the summation layer for summation. The output feature vector of the summation layer is input into the fully connected layer. The fully connected layer outputs a 12-dimensional vector, where the class corresponding to the maximum value is the recognition result. This process continues until convergence, resulting in the trained gesture recognition network Bi-GRU.

8. A millimeter-wave radar gesture recognition system based on a multi-head self-attention mechanism, characterized in that: The system is used to execute the millimeter-wave radar gesture recognition method based on a multi-head self-attention mechanism as described in any one of claims 1 to 7.

Images