Gesture recognition method of FMCW millimeter wave radar based on multi-task learning

Abstract

The present invention provides a gesture recognition method for FMCW millimeter wave radar based on multi-task learning, including: obtaining a target feature vector time series sequence of a gesture to be recognized; inputting the target feature vector time series sequence into a multi-task learning gesture recognition model, and outputting a gesture category result; wherein the multi-task learning gesture recognition model is trained based on a sample target feature vector time series sequence, a corresponding gesture category label, and a corresponding gesture trajectory label, and the target feature vector time series sequence and the sample target feature vector time series sequence are both obtained by processing the original gesture recognition data obtained by the FMCW millimeter wave radar through a preset algorithm. The method realizes the effective and convenient acquisition of multi-dimensional features of user gestures, and improves the accuracy of gesture recognition classification through multi-task learning.

Description

Technical Field

[0001] The present invention relates to the technical field of gesture recognition, and in particular to a gesture recognition method of an FMCW (Frequency Modulated Continuous Wave) millimeter wave radar based on multi-task learning. Background Art

[0002] In recent years, human-computer interaction technology has been gradually applied to daily life. Gesture recognition, as one of the most intuitive means of human interaction, enables users to interact with machines in a more natural way through the movement of their palms or fingers, and has gradually become a research hotspot. Existing gesture recognition schemes can be divided into gesture recognition methods based on visual sensors, inertial sensors, and wireless radio frequency sensors according to the data source. The gesture recognition scheme based on optical sensors uses a camera to capture scene videos containing gestures, and then uses computer algorithms to identify, extract and classify gesture features in the image. Its hardware system is relatively simple, and the collected images and videos can provide information with high spatial resolution. However, because it needs to process two-dimensional or even three-dimensional image data, the data volume is large and the calculation cost is high. The high power consumption makes it difficult to apply in portable devices. In addition, gesture recognition based on optical sensors is sensitive to ambient light, prone to visual blind spots and light occlusion, which limits the user's scope of use and poses the risk of user privacy leakage. Inertial sensors can measure the acceleration and velocity of objects, but they require users to wear the device all the time when performing actions, which results in a poor user experience.

[0003] With the development of radio frequency technology and integrated circuits, gesture recognition methods based on wireless signals such as radar and WiFi have gradually attracted the attention of scholars. Compared with optical sensors and inertial sensors, wireless signals have low environmental requirements. Among them, millimeter-wave radar has many unique advantages. First, the radar transmits and receives signals and collects gesture information without being affected by weather conditions and ambient light, which greatly improves the application scenarios that the system can cover. Radar can also penetrate occlusions and propagate, making it possible to achieve gesture interaction under partial or complete occlusion. Secondly, the millimeter-wave radar receives echoes with lower energy consumption than optical solutions, can be better integrated into embedded devices, and is more convenient than wearable device solutions. At the same time, radar signals are more secure and confidential, and basically do not leak user privacy. In addition, compared with other low-frequency RF sensors, millimeter-wave radar sensors have narrower beams and higher spatial resolution. Millimeter-wave radar has attracted the attention of researchers in recent years due to its many advantages mentioned above. However, since gesture recognition methods based on millimeter-wave radar have inherent differences in the design of data acquisition, feature extraction and fusion, and gesture classification algorithm design, their algorithm development is still not very mature and there are still some limitations. For example, it fails to fully utilize the information that radar can provide, and the recognition model size and time complexity are large.

[0004] From the perspectives of feature extraction and gesture classification, radar-based gesture recognition has achieved certain results in the technical research and application of radar gesture recognition, but there are still some challenges:

[0005] (1) Construction of labeled datasets. When using deep learning for gesture recognition, sufficient gesture data is the basis for model training. However, based on existing hardware platforms, the acquisition of gesture data requires a lot of time and is very expensive.

[0006] (2) Multi-dimensional gesture feature extraction. Existing algorithms use micro-Doppler spectra and range Doppler spectra as network input data, but the data volume is large and there is no correspondence between the features of each target point. In addition, there is a lack of utilization and fusion of azimuth and elevation angle information, and the information provided by the radar cannot be effectively and fully utilized, which affects the recognition accuracy of gestures.

[0007] (3) Classifier design. When using deep learning to classify radar gestures, a single type of recognition model is often used. The recognition algorithm has a large amount of computation and high model complexity. The effective features of the input data cannot be fully extracted, resulting in low gesture classification accuracy.

[0008] In summary, how to effectively and conveniently extract multi-dimensional gesture features, and how to design a classifier to improve the accuracy of gesture classification, are still urgent problems to be solved by those skilled in the art. Summary of the invention

[0009] The present invention provides a gesture recognition method of FMCW millimeter wave radar based on multi-task learning, which is used to solve the problems that the existing multi-dimensional gesture feature extraction is not effective and convenient enough and the accuracy of gesture classification is not high.

[0010] The present invention provides a gesture recognition method of FMCW millimeter wave radar based on multi-task learning, comprising:

[0011] Obtain a time series sequence of target feature vectors of the gesture to be recognized;

[0012] Inputting the target feature vector time series sequence into a multi-task learning gesture recognition model, and outputting a gesture category result;

[0013] Among them, the multi-task learning gesture recognition model is trained based on the sample target feature vector time series sequence, the corresponding gesture category label and the corresponding gesture trajectory label. The target feature vector time series sequence and the sample target feature vector time series sequence are both obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm.

[0014] According to a gesture recognition method of an FMCW millimeter wave radar based on multi-task learning provided by the present invention, the multi-task learning gesture recognition model includes two learning tasks of gesture recognition classification and gesture trajectory reconstruction during the training process.

[0015] According to a gesture recognition method of an FMCW millimeter-wave radar based on multi-task learning provided by the present invention, the overall loss function of the multi-task learning gesture recognition model during the training process is the weighted loss function of two parts: a gesture recognition classification part and a gesture trajectory reconstruction part.

[0016] According to a gesture recognition method of an FMCW millimeter wave radar based on multi-task learning provided by the present invention, the training network results of the multi-task learning gesture recognition model during the training process include:

[0017] A one-dimensional CNN (Convolutional Neural Network) layer, an LSTM (Long Short-Term Memory) layer, and a one-dimensional transposed convolution layer connected in sequence;

[0018] As well as, the one-dimensional CNN layer, LSTM layer and fully connected layer connected sequentially.

[0019] According to a gesture recognition method for an FMCW millimeter-wave radar based on multi-task learning provided by the present invention, the one-dimensional CNN layer has 128 convolution kernels, the fully connected layer has a softmax activation function, and the one-dimensional transposed convolution uses 128 deconvolution kernels of size 7×1.

[0020] According to a gesture recognition method of an FMCW millimeter wave radar based on multi-task learning provided by the present invention, the target feature vector time series sequence and the sample target feature vector time series sequence are obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm, specifically including:

[0021] The spatial and temporal features of the gesture signal acquired by the FMCW millimeter-wave radar are extracted using a specific multi-dimensional fusion feature extraction algorithm. The distance, angle and speed information at each moment are fused one by one in the form of vectors to obtain the target feature vector time series sequence and the sample target feature vector time series sequence.

[0022] The present invention also provides a gesture recognition method for an FMCW millimeter wave radar based on multi-task learning, wherein the target feature vector time series sequence and the sample target feature vector time series sequence are obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm, specifically comprising:

[0023] For the distance value in the original features of gesture recognition obtained by the FMCW millimeter-wave radar in each frame, first perform fast-time FFT on the intermediate frequency signal, remove static clutter, and then use CFAR to detect the possible target distance;

[0024] Then, the Capon algorithm is used to calculate the azimuth angle spectrum at the corresponding distance and the CFAR (Constant False Alarm Rate) is used to detect the azimuth of the target that may exist at a specific distance;

[0025] The Capon algorithm is used to calculate the pitch angle spectrum at the corresponding distance and azimuth, and the pitch angle of the target at a specific distance and azimuth is detected by CFAR;

[0026] Finally, beamforming is used to calculate the target velocity of each target point;

[0027] The target distance, target azimuth, target pitch angle and target speed are constructed into target gesture recognition features, and Doppler FFT processing is performed. Then, based on density clustering, the cluster centers are extracted as the target feature vector time series sequence and the sample target feature vector time series sequence.

[0028] According to the present invention, a gesture recognition device of an FMCW millimeter wave radar based on multi-task learning is provided, comprising:

[0029] An acquisition unit, used for acquiring a time series sequence of a target feature vector of a gesture to be recognized;

[0030] A recognition unit, used for inputting the target feature vector time series sequence into a multi-task learning gesture recognition model and outputting a gesture classification result;

[0031] Among them, the multi-task learning gesture recognition model is trained based on the sample target feature vector time series sequence, the corresponding gesture category label and the corresponding gesture trajectory label. The target feature vector time series sequence and the sample target feature vector time series sequence are both obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm.

[0032] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning as described in any one of the above are implemented.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the gesture recognition method of FMCW millimeter-wave radar based on multi-task learning as described in any one of the above.

[0034] The FMCW millimeter wave radar gesture recognition method based on multi-task learning provided by the present invention obtains the target feature vector time series sequence of the gesture to be recognized; inputs the target feature vector time series sequence into the multi-task learning gesture recognition model, and outputs the gesture category result; wherein the multi-task learning gesture recognition model is trained based on the sample target feature vector time series sequence, the corresponding gesture category label and the corresponding gesture trajectory label, and the target feature vector time series sequence and the sample target feature vector time series sequence are obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm. The multi-dimensional features of the user gesture are effectively and conveniently obtained, and the accuracy of gesture recognition classification is improved through multi-task learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1A schematic diagram of a flow chart of a gesture recognition method of an FMCW millimeter wave radar based on multi-task learning provided by the present invention;

[0037] Figure 2 The physical structure diagram of the FMCW millimeter wave radar hardware platform provided by the present invention;

[0038] Figure 3 It is a schematic diagram of an interface for configuring radar transmission signal parameters according to the present invention;

[0039] Figure 4 A schematic diagram of a multi-task learning network provided by the present invention;

[0040] Figure 5 A schematic diagram of a multi-task learning network framework provided by the present invention;

[0041] Figure 6 A multi-dimensional feature extraction flow chart provided by the present invention;

[0042] Figure 7 The change of verification accuracy during the training process provided by the present invention;

[0043] Figure 8 The confusion matrix of the test set classification results provided by the present invention;

[0044] Fig. 9 A schematic diagram of the structure of a gesture recognition device of an FMCW millimeter-wave radar based on multi-task learning provided by the present invention;

[0045] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The existing gesture recognition and classification technologies generally have the problem that the multi-dimensional gesture feature extraction is not effective and convenient enough, and the accuracy of gesture classification is not high. Figure 1 The present invention describes a gesture recognition method of a FMCW millimeter wave radar based on multi-task learning. Figure 1 The flowchart of the gesture recognition method of FMCW millimeter wave radar based on multi-task learning provided by the present invention is as follows: Figure 1 As shown, the method includes:

[0048] Step 110: Obtain a time series sequence of target feature vectors of the gesture to be recognized.

[0049] Specifically, a target feature vector time series sequence of the gesture to be recognized is obtained, that is, a target feature vector is obtained after multi-dimensional feature fusion extraction of the feature vector formed by the distance, speed, azimuth and pitch angle of the gesture to be recognized, and then multiple frames of target feature vectors are spliced ​​together in time series to obtain a target feature vector time series sequence for identifying the corresponding gesture type.

[0050] Step 120, inputting the target feature vector time series sequence into a multi-task learning gesture recognition model, and outputting a gesture classification result;

[0051] Among them, the multi-task learning gesture recognition model is trained based on the sample target feature vector time series sequence, the corresponding gesture category label and the corresponding gesture trajectory label. The target feature vector time series sequence and the sample target feature vector time series sequence are both obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm.

[0052] Specifically, the target feature vector time series sequence is input into the multi-task learning gesture recognition model, and the predicted label is output, that is, the gesture category recognition result corresponding to the input target feature vector time series sequence. Among them, the multi-task learning gesture recognition model is obtained after training with a large number of target feature vector time series sequence samples and gesture category labels and gesture trajectory labels corresponding to the samples one by one. The more sample labels are used, the higher the accuracy of the model after training. The training model used in the embodiment of the present invention is a multi-task learning type, that is, in addition to outputting gesture classification results, the model can also output results similar to trajectory reconstruction. In this way, the two learning tasks adjust the parameters to be adjusted in the network under the supervision of their respective labels, so that the model obtained after the model training is completed is used to recognize gestures. The results are more accurate. It should be noted here that the target feature vector time series sequence and the sample target feature vector time series sequence are obtained by processing the original gesture recognition data obtained by the FMCW millimeter wave radar through a preset algorithm.

[0053] It should be further explained here that this method uses the hardware experimental platform and software of the FMCW millimeter-wave radar to collect data, and then designs a multi-dimensional fusion feature extraction method for the collected radar gesture signals. Based on the established radar gesture data set, gesture recognition algorithm research is carried out, and a multi-task based neural network structure is proposed to make full use of the parameter information extracted by the radar to achieve higher classification accuracy and faster classification speed.

[0054] Figure 2 The physical structure diagram of the FMCW millimeter wave radar hardware platform provided by the present invention is as follows: Figure 2 As shown, the FMCW millimeter wave radar platform used in the embodiment of the present invention is mainly composed of two parts, including the IWR 1443 radar module of Texas Instruments and the DCA1000 data acquisition board. Among them, the IWR1443 radar development board is mainly responsible for generating and transmitting linear frequency modulation waves, receiving and demodulating target echoes, and obtaining intermediate frequency signals carrying target information. The DCA1000 data acquisition board is responsible for transmitting the intermediate frequency signal to a personal computer for the next step of data processing. In terms of software, Figure 3 It is a schematic diagram of the interface for configuring the radar transmission signal parameters of the present invention, such as Figure 3 As shown, the embodiment of the present invention uses the mmWave studio host computer software developed by TI to configure the radar and collect data to store it on the PC.

[0055] Since the gain of the directional pattern of commercial radar drops quickly, the needs of gesture recognition applications are not taken into account. For the application scenarios of gestures and gesture recognition to be adopted by the present invention, an antenna optimization algorithm is designed to optimize the directional pattern of the array antenna to improve the directional pattern gain of the target area of ​​interest and reduce interference in other areas. It is also expected that by analyzing the configuration parameters of the radar signal acquisition platform, the optimal configuration of the FMCW millimeter-wave radar system for gesture recognition can be found, the radar performance can be maximized, and gesture data can be collected on this basis to construct a gesture data set.

[0056] The gesture recognition method of FMCW millimeter wave radar based on multi-task learning provided by the embodiment of the present invention obtains the target feature vector time series sequence of the gesture to be recognized; inputs the target feature vector time series sequence into the multi-task learning gesture recognition model, and outputs the gesture category result; wherein the multi-task learning gesture recognition model is trained based on the sample target feature vector time series sequence and the corresponding gesture category label, and the target feature vector time series sequence and the sample target feature vector time series sequence are obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm. It realizes the effective and convenient acquisition of the multi-dimensional features of the user gesture, and improves the accuracy of gesture recognition classification through multi-task learning.

[0057] Based on the above embodiment, in the method, the multi-task learning gesture recognition model includes two learning tasks during the training process: gesture recognition classification and reconstruction of gesture trajectory.

[0058] Specifically, during the training process, the multi-task learning gesture recognition model includes two learning tasks: gesture recognition classification and reconstruction of gesture trajectory. That is, the labels used in the training process include gesture category labels corresponding to the time series sequence of sample target gesture feature vectors, and gesture trajectory labels corresponding to the time series sequence of sample target gesture feature vectors. These two labels are respectively used to merge the parameters to be adjusted in the adjustment model network while "correcting" the predicted labels and reconstructing the trajectories.

[0059] Based on the above embodiment, in this method, the overall loss function of the multi-task learning gesture recognition model during the training process is the weighted loss function of the two parts of the gesture recognition classification part and the gesture trajectory reconstruction part.

[0060] Specifically, the compressed data is used as the input of the gesture trajectory reconstruction part and the gesture recognition classification part, and the weighted sum of the loss functions of the gesture trajectory reconstruction part and the gesture recognition classification part is used as the overall loss function. The purpose is to assist the feature extraction module in the network to better extract features from the input data through the reconstruction task, so as to improve the performance of the gesture recognition classification part.

[0061] Based on the above embodiment, in this method, the training network results of the multi-task learning gesture recognition model during the training process include:

[0062] The one-dimensional CNN layer, LSTM layer and one-dimensional transposed convolution layer are connected sequentially;

[0063] As well as, the one-dimensional CNN layer, LSTM layer and fully connected layer connected sequentially.

[0064] Specifically, Figure 4 The schematic diagram of the multi-task learning network provided by the present invention is as follows: Figure 4 As shown in the figure, the training network results of the multi-task learning gesture recognition model during the training process include: sequentially connected one-dimensional CNN layer, LSTM layer and one-dimensional transposed convolution layer (1D CNN Transpose); and sequentially connected one-dimensional CNN layer, LSTM layer and fully connected layer. The network framework diagram is shown in Figure 4As shown, the neural network structure used in the present invention can be mainly divided into four stages: feature extraction, encoding, classification and reconstruction. In the feature extraction stage, one-dimensional convolution operation is used to extract appropriate and sufficient features from the input time series trajectory such as distance and angle. The extracted features are mapped to the hidden layer through the LSTM layer in the encoding stage. The LSTM in the model converts the input data into a learned compressed representation, which is used as the input of the classification and reconstruction stages. The classification stage consists of a fully connected layer, which calculates the conditional probability of the output based on the learned input trajectory features by applying the softmax function. In the reconstruction or decoding stage, the compressed representation of the input trajectory is sent to several one-dimensional transposed convolution layers, which attempt to reconstruct the input feature trajectory.

[0065] Based on the above embodiment, in the method, the one-dimensional CNN layer has 128 convolution kernels, the fully connected layer has a softmax activation function, and the one-dimensional transposed convolution layer uses 128 deconvolution kernels of size 7×1.

[0066] Specifically, Figure 5 The schematic diagram of the multi-task learning network framework provided by the present invention is as follows: Figure 5 As shown in the figure, the input of the one-dimensional CNN is 4×100, and this layer has 128 convolution kernels. The one-dimensional CNN layer is followed by an LSTM layer, and the output of the LSTM layer is input to a fully connected layer with a softmax activation function. The output of the fully connected layer is then input to seven one-dimensional transposed convolution layers. The first two layers use 32 deconvolution kernels of size 7×1, the next two layers use 64 deconvolution kernels of size 7×1, the next two layers use 128 deconvolution kernels of size 7×1, and the last one-dimensional transposed convolution layer uses 128 deconvolution kernels of size 7×1. In addition, all convolution and transposed layers use rectified linear units (ReLU) as activation functions. In order to avoid overfitting of the model, dropout is used in the last transposed convolution layer of the model.

[0067] Based on the above embodiment, in the method, the target feature vector time series sequence and the sample target feature vector time series sequence are both obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm, specifically including:

[0068] The spatial and temporal features of the gesture signal acquired by the FMCW millimeter-wave radar are extracted using a specific multi-dimensional fusion feature extraction algorithm. The distance, angle and speed information at each moment are fused one by one in the form of vectors to obtain the target feature vector time series sequence and the sample target feature vector time series sequence.

[0069] Specifically, a multi-dimensional fusion feature extraction method was designed to extract the spatial and temporal features of gesture signals, and the distance, angle, and speed information at each moment were matched one by one to perform feature fusion in the form of vectors. This solved the problem of large data volume when using pictures as network input and no correspondence between the features of each target point, and more accurately reflected the changes of the hand over time during movement.

[0070] Based on the above embodiment, in the method, the target feature vector time series sequence and the sample target feature vector time series sequence are both obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm, specifically including:

[0071] For the distance value in the original features of gesture recognition obtained by the FMCW millimeter-wave radar in each frame, first perform fast-time FFT on the intermediate frequency signal, remove static clutter, and then use CFAR to detect the possible target distance;

[0072] Then, the Capon algorithm is used to calculate the azimuth angle spectrum at the corresponding distance and the CFAR is used to detect the possible target azimuth at a specific distance;

[0073] The Capon algorithm is used to calculate the pitch angle spectrum at the corresponding distance and azimuth, and the pitch angle of the target at a specific distance and azimuth is detected by CFAR;

[0074] Finally, beamforming is used to calculate the target velocity of each target point;

[0075] The target distance, target azimuth, target pitch angle and target speed are constructed into target gesture recognition features, and Doppler FFT processing is performed. Then, based on density clustering, the cluster centers are extracted as the target feature vector time series sequence and the sample target feature vector time series sequence.

[0076] Specifically, in order to make full use of the effective information provided by the radar and integrate various features, the embodiment of the present invention designs a gesture feature extraction method of multi-dimensional feature fusion, so that the distance, speed, azimuth and pitch angle features of each target at each moment correspond to each other, more accurately reflecting the change law of the hand over time during movement, reducing the input size of the classification model, and reducing the calculation amount of the forward propagation of the classification model. Figure 6 The multidimensional feature extraction flow chart provided by the present invention includes the following specific steps: Figure 6As shown. In each frame, first perform fast-time FFT on the intermediate frequency signal, remove static clutter, and then use CFAR to detect the distance of the possible target. Then use the Capon algorithm to calculate the azimuth angle spectrum at the corresponding distance and use CFAR to detect the azimuth of the target at a specific distance. Similarly, detect the pitch angle of the target at the corresponding distance and azimuth. After calculating the distance, azimuth and pitch angle of the target, beamform the 8-channel echo signals, and then perform Doppler FFT on the slow time dimension to calculate the speed of each target point. Cluster the target points in each frame according to their positions in space, and use the cluster center as the position of the target in the current frame to construct the feature vector.

[0077] After preprocessing the original radar echo signal, each gesture is represented by a time series trajectory of distance, angle, and speed, and a gesture dataset is established with 100 consecutive frames of distance, azimuth, pitch angle, and speed vector as the input of the classifier. In order to make full use of the extracted gesture feature vectors, a network framework based on multi-task learning is used for gesture recognition.

[0078] The key points of the embodiments of the present invention are: 1. First, a multi-dimensional fusion feature extraction method is designed to extract the spatial and temporal features of gesture signals, and the distance, angle, and speed information at each moment are matched one by one to perform feature fusion in the form of vectors, which solves the problem that the amount of data used as network input is large and there is no correspondence between the features of each target point, and more accurately reflects the change law of the hand over time during movement; while using Kalman filtering to smooth the gesture trajectory, the result is fine-tuned by changing the parameters, which increases the number of samples while ensuring the filtering effect, and realizes sample enhancement; 2. Secondly, a multi-task learning network framework is designed, and the overall network framework is divided into two task modules: trajectory reconstruction and gesture classification. The reconstruction task assists the shared part in the network to better extract features from the input data. First, the features of the input information are extracted through a one-dimensional convolutional layer, and then the extracted information is encoded using the LSTM layer, which is a compressed representation. In the next step, the compressed data is used as the input of the reconstruction part and the classification part, and the weighted sum of the loss functions of the reconstruction part and the classification part is used as the overall loss function. The purpose is to assist the feature extraction module in the network to better extract features from the input data through the reconstruction task, so as to improve the performance of the classification part.

[0079] Compared with the existing technology, the advantages of the embodiments of the present invention are: first, a multi-dimensional fusion feature extraction method is designed to extract the spatial and temporal features of gesture signals, and the distance, angle, and speed information at each moment are matched one by one to perform feature fusion in the form of vectors, which solves the problem that the amount of data used as network input is large and there is no correspondence between the features of each target point, and more accurately reflects the change pattern of the hand over time during movement; experiments show that there are obvious differences in the trajectories of different gesture actions, and they all show different characteristics.

[0080] Experimental verification process:

[0081] The designed multi-task learning model is used to train and classify the gesture dataset. 80% of the total samples are used to train the model, and the remaining 20% ​​of the samples are used for testing. According to the network structure and loss function used, the present invention uses the Adam optimizer, the learning rate is set to 0.0005, the epoch is set to 50, and the batch size is set to 32. Then the network is trained and tested, and the experimental results are statistically analyzed. Finally, the average recognition accuracy of the model for the six gestures is 99%, Figure 7 The change of verification accuracy during the training process provided by the present invention, Figure 8 This is the confusion matrix of the test set classification results provided by the present invention. Figure 8 As shown in the figure, the overall recognition accuracy of the eight types of gestures finally converged to 99%, among which the misclassified ones were mainly the third type of gesture sliding to the left and the fourth type of gesture sliding to the right. According to the analysis, the main reason may be that these two types of gestures only differ in the azimuth features, while the other feature trajectories are basically the same, so the probability of confusion is greater.

[0082] The multi-task network structure in the present invention is compared with the single-task network structure without the reconstruction module and the work of Infineon. In order to verify the recognition performance, the present invention uses the following parameters to compare the algorithms from two aspects: classification performance and time complexity:

[0083] (1) Accuracy: The ratio of the number of correctly classified samples in the test set to the total number of samples, which can reflect the recognition algorithm's ability to judge the data set.

[0084] (2) Floating Point Operations (FLOPs): The amount of computation that can be used to measure the complexity of the model.

[0085] (3) Number of parameters: The number of neural network parameters can measure the size of the model.

[0086] Table 1 is a performance comparison of different methods. The samples collected in the present invention are used to verify the recognition scheme. The comparison of classification performance and algorithm complexity is shown in Table 1.

[0087] Table 1 Performance comparison of different methods

[0088]

[0089] As shown in Table 1, in terms of classification performance, the multi-task network structure used in the present invention has an average recognition accuracy of 3% higher than that of the single-task module without the reconstruction module for the same data set, indicating that the multi-task structure can better extract the spatial features and temporal features in the gesture parameters with the assistance of the reconstruction task compared to the network structure without adding the multi-task module; while the method used by Infineon only achieved a recognition accuracy of 87.5%, mainly because it used the range Doppler image sequence as the network input, only used the distance information and speed information of the gesture but failed to use the lateral angle information, so it was difficult to recognize gestures with angle changes but no obvious changes in radial speed and radial distance, resulting in a decrease in the overall recognition accuracy. In terms of algorithm time complexity, the number of parameters is used to represent the size of the classification model, and FLOPs is used to evaluate the calculation amount of the model during forward propagation. It can be seen that the work of the present invention can have a higher recognition accuracy than the single-task network when the time complexity is the same; compared with 3D CNN, the number of parameters, model parameters and FLOPs are less and have a higher recognition accuracy, indicating that the algorithm proposed by the present invention is superior to the method used by Infineon. The algorithm is compared with other algorithms, and the factors affecting the classification performance are analyzed. The experimental results show that the average recognition accuracy of the invention can reach 99%, which is better than the compared algorithms.

[0090] The gesture recognition device of the FMCW millimeter wave radar based on multi-task learning provided by the present invention is described below. The gesture recognition device of the FMCW millimeter wave radar based on multi-task learning described below and the gesture recognition method of the FMCW millimeter wave radar based on multi-task learning described above can be referred to each other.

[0091] Fig. 9 A structural diagram of a gesture recognition device based on a FMCW millimeter wave radar for multi-task learning provided by the present invention is shown in FIG. Fig. 9 As shown, the device includes an acquisition unit 910 and an identification unit 920, wherein:

[0092] The acquisition unit 910 is used to acquire a time series sequence of a target feature vector of a gesture to be recognized;

[0093] The recognition unit 920 is used to input the target feature vector time series sequence into a multi-task learning gesture recognition model and output a gesture classification result;

[0094] Among them, the multi-task learning gesture recognition model is trained based on the sample target feature vector time series sequence, the corresponding gesture category label and the corresponding gesture trajectory label. The target feature vector time series sequence and the sample target feature vector time series sequence are both obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm.

[0095] The FMCW millimeter wave radar gesture recognition device based on multi-task learning provided by the present invention obtains the target feature vector time series sequence of the gesture to be recognized; inputs the target feature vector time series sequence into the multi-task learning gesture recognition model, and outputs the gesture category result; wherein the multi-task learning gesture recognition model is trained based on the sample target feature vector time series sequence, the corresponding gesture category label and the corresponding gesture trajectory label, and the target feature vector time series sequence and the sample target feature vector time series sequence are obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm. It realizes the effective and convenient acquisition of the multi-dimensional features of the user gesture, and improves the accuracy of gesture recognition classification through multi-task learning.

[0096] On the basis of the above-mentioned embodiment, in the device, the multi-task learning gesture recognition model includes two learning tasks of gesture recognition classification and reconstruction of gesture trajectory during the training process.

[0097] On the basis of the above embodiment, in the device, the overall loss function of the multi-task learning gesture recognition model during the training process is the weighted loss function of the two parts of the gesture recognition classification part and the gesture trajectory reconstruction part.

[0098] Based on the above embodiment, in the device, the training network results of the multi-task learning gesture recognition model during the training process include:

[0099] The one-dimensional CNN layer, LSTM layer and one-dimensional transposed convolution layer are connected sequentially;

[0100] As well as, the one-dimensional CNN layer, LSTM layer and fully connected layer connected sequentially.

[0101] Based on the above embodiment, in the device, the one-dimensional CNN layer has 128 convolution kernels, the fully connected layer has a softmax activation function, and the one-dimensional transposed convolution layer uses 128 deconvolution kernels of size 7×1.

[0102] On the basis of the above embodiment, in the device, the target feature vector time series sequence and the sample target feature vector time series sequence are obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm, specifically including:

[0103] The spatial and temporal features of the gesture signal acquired by the FMCW millimeter-wave radar are extracted using a specific multi-dimensional fusion feature extraction algorithm. The distance, angle and speed information at each moment are fused one by one in the form of vectors to obtain the target feature vector time series sequence and the sample target feature vector time series sequence.

[0104] On the basis of the above embodiment, in the device, the target feature vector time series sequence and the sample target feature vector time series sequence are obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm, specifically including:

[0105] For the distance value in the original features of gesture recognition obtained by the FMCW millimeter-wave radar in each frame, first perform fast-time FFT on the intermediate frequency signal, remove static clutter, and then use CFAR to detect the possible target distance;

[0106] Then, the Capon algorithm is used to calculate the azimuth angle spectrum at the corresponding distance and the CFAR is used to detect the possible target azimuth at a specific distance;

[0107] The Capon algorithm is used to calculate the pitch angle spectrum at the corresponding distance and azimuth, and the pitch angle of the target at a specific distance and azimuth is detected by CFAR;

[0108] Finally, beamforming is used to calculate the target velocity of each target point;

[0109] The target distance, target azimuth, target pitch angle and target speed are constructed into target gesture recognition features, and Doppler FFT processing is performed. Then, based on density clustering, the cluster centers are extracted as the target feature vector time series sequence and the sample target feature vector time series sequence.

[0110] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the gesture recognition method of the FMCW millimeter wave radar based on multi-task learning, the method comprising: obtaining a target feature vector time series sequence of a gesture to be recognized; inputting the target feature vector time series sequence into a multi-task learning gesture recognition model, and outputting a gesture category result; wherein the multi-task learning gesture recognition model is obtained by training based on a sample target feature vector time series sequence, a corresponding gesture category label and a corresponding gesture trajectory label, and the target feature vector time series sequence and the sample target feature vector time series sequence are obtained by processing the gesture recognition raw data obtained by the FMCW millimeter wave radar through a preset algorithm.

[0111] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0112] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the gesture recognition method of FMCW millimeter-wave radar based on multi-task learning provided by the above methods, and the method includes: obtaining a target feature vector time series sequence of a gesture to be recognized; inputting the target feature vector time series sequence into a multi-task learning gesture recognition model, and outputting a gesture category result; wherein the multi-task learning gesture recognition model is trained based on a sample target feature vector time series sequence, a corresponding gesture category label and a corresponding gesture trajectory label, and the target feature vector time series sequence and the sample target feature vector time series sequence are both obtained by processing the gesture recognition raw data obtained by the FMCW millimeter-wave radar through a preset algorithm.

[0113] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned gesture recognition method based on multi-task learning of FMCW millimeter-wave radar, the method comprising: obtaining a target feature vector time series sequence of a gesture to be recognized; inputting the target feature vector time series sequence into a multi-task learning gesture recognition model, and outputting a gesture category result; wherein the multi-task learning gesture recognition model is trained based on a sample target feature vector time series sequence, a corresponding gesture category label and a corresponding gesture trajectory label, and the target feature vector time series sequence and the sample target feature vector time series sequence are both obtained by processing the gesture recognition raw data obtained by the FMCW millimeter-wave radar through a preset algorithm.

[0114] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A gesture recognition method for FMCW millimeter wave radar based on multi-task learning, characterized in that: include: Obtain a time series sequence of target feature vectors of the gesture to be recognized; Inputting the target feature vector time series sequence into a multi-task learning gesture recognition model, and outputting a gesture category result; The multi-task learning gesture recognition model is obtained by training based on the sample target feature vector time series, the corresponding gesture category label and the corresponding gesture trajectory label; The target feature vector time series sequence and the sample target feature vector time series sequence are both obtained by processing the gesture recognition raw data acquired by the FMCW millimeter wave radar through a preset algorithm, specifically including: For the distance value in the original features of gesture recognition obtained by the FMCW millimeter-wave radar in each frame, first perform fast-time FFT on the intermediate frequency signal, remove static clutter, and then use CFAR to detect the possible target distance; Then, the Capon algorithm is used to calculate the azimuth angle spectrum at the corresponding distance and the possible target azimuth is detected through CFAR; The Capon algorithm is used to calculate the pitch angle spectrum at the corresponding distance and azimuth, and the pitch angle of the target is detected by CFAR; Finally, beamforming is used to calculate the target velocity of each target point; The target distance, target azimuth, target pitch angle and target speed are constructed into target gesture recognition features, and Doppler FFT processing is performed. Then, based on density clustering, the cluster centers are extracted as the target feature vector time series sequence and the sample target feature vector time series sequence.

2. The gesture recognition method of FMCW millimeter wave radar based on multi-task learning according to claim 1 is characterized in that: The multi-task learning gesture recognition model includes two learning tasks, gesture recognition classification and gesture trajectory reconstruction, during the training process.

3. The gesture recognition method of FMCW millimeter wave radar based on multi-task learning according to claim 2 is characterized in that: The overall loss function of the multi-task learning gesture recognition model during the training process is the weighted loss function of the two parts of the gesture recognition classification part and the gesture trajectory reconstruction part.

4. The gesture recognition method of FMCW millimeter wave radar based on multi-task learning according to claim 1, characterized in that: The training network results of the multi-task learning gesture recognition model during the training process include: The one-dimensional CNN layer, LSTM layer and one-dimensional transposed convolution layer are connected sequentially; As well as, the one-dimensional CNN layer, LSTM layer and fully connected layer connected sequentially.

5. The gesture recognition method of FMCW millimeter wave radar based on multi-task learning according to claim 4 is characterized in that: The one-dimensional CNN layer has 128 convolution kernels, the fully connected layer has a softmax activation function, and the one-dimensional transposed convolution layer uses 128 deconvolution kernels of size 7×1.

6. The gesture recognition method of FMCW millimeter wave radar based on multi-task learning according to claim 1, characterized in that: The target feature vector time series sequence and the sample target feature vector time series sequence are both obtained by processing the gesture recognition raw data acquired by the FMCW millimeter wave radar through a preset algorithm, specifically including: The spatial and temporal features of the gesture signal acquired by the FMCW millimeter-wave radar are extracted using a multi-dimensional fusion feature extraction algorithm. The distance, angle and speed information at each moment are fused one by one in the form of vectors to obtain the target feature vector time series sequence and the sample target feature vector time series sequence.

7. A gesture recognition device based on FMCW millimeter wave radar with multi-task learning, characterized in that: include: An acquisition unit, used for acquiring a time series sequence of a target feature vector of a gesture to be recognized; A recognition unit, used for inputting the target feature vector time series sequence into a multi-task learning gesture recognition model and outputting a gesture classification result; The multi-task learning gesture recognition model is obtained by training based on the sample target feature vector time series, the corresponding gesture category label and the corresponding gesture trajectory label; The target feature vector time series sequence and the sample target feature vector time series sequence are both obtained by processing the gesture recognition raw data acquired by the FMCW millimeter wave radar through a preset algorithm, specifically including: For the distance value in the original features of gesture recognition obtained by the FMCW millimeter-wave radar in each frame, first perform fast-time FFT on the intermediate frequency signal, remove static clutter, and then use CFAR to detect the possible target distance; Then, the Capon algorithm is used to calculate the azimuth angle spectrum at the corresponding distance and the possible target azimuth is detected through CFAR; The Capon algorithm is used to calculate the pitch angle spectrum at the corresponding distance and azimuth, and the pitch angle of the target is detected by CFAR; Finally, beamforming is used to calculate the target velocity of each target point; The target distance, target azimuth, target pitch angle and target speed are constructed into target gesture recognition features, and Doppler FFT processing is performed. Then, based on density clustering, the cluster centers are extracted as the target feature vector time series sequence and the sample target feature vector time series sequence.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning are implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the gesture recognition method of FMCW millimeter wave radar based on multi-task learning as described in any one of claims 1 to 6 are implemented.

Images