Gesture recognition method and device, equipment, storage medium and program product

Abstract

The invention discloses a gesture recognition method and device, equipment, a storage medium and a program product, and relates to the technical field of data processing, and the disclosed gesture recognition method comprises the steps: obtaining a gesture thermodynamic diagram collected by a millimeter wave radar; performing dimension reduction processing and space-time compression processing on the gesture thermodynamic diagram in sequence through a space-time feature extraction module of the gesture recognition model to obtain deep space-time features of the gesture; and performing category prediction on the deep spatial-temporal features of the gestures through a classification module of the gesture recognition model to obtain a gesture recognition result. The data size of the gesture deep-level spatial-temporal features after dimension reduction and compression is small, gesture recognition is performed based on the gesture deep-level spatial-temporal features, the requirements for hardware computing power and storage resources are low, a classification module of a gesture recognition model can quickly perform category prediction to obtain a gesture recognition result, and the gesture recognition efficiency is improved. The technical problem of how to process the distance-Doppler thermodynamic diagram and improve the gesture recognition efficiency is solved, and the gesture recognition efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a gesture recognition method, apparatus, device, storage medium, and program product. Background Art

[0002] In related technologies, gesture recognition methods based on millimeter-wave radar generate range-Doppler heat maps to reflect the location and shape of target objects. However, these range-Doppler heat maps are large in data volume, complex to process, and require high hardware computing power and storage resources, resulting in low efficiency. Therefore, the problem of how to process range-Doppler heat maps and improve gesture recognition efficiency remains urgent. Summary of the Invention

[0003] The main purpose of this application is to provide a gesture recognition method, device, equipment, storage medium and program product, aiming to solve the technical problem of how to process range-Doppler thermal images and improve the efficiency of gesture recognition.

[0004] To achieve the above objectives, this application proposes a gesture recognition method, which includes:

[0005] Obtain gesture heatmaps collected by millimeter-wave radar;

[0006] The gesture heat map is processed by the spatiotemporal feature extraction module of the gesture recognition model, and the deep spatiotemporal features of the gesture are obtained.

[0007] The classification module of the gesture recognition model is used to predict the category of the deep temporal and spatial features of the gesture to obtain the gesture recognition results.

[0008] In one embodiment, the step of performing dimensionality reduction processing on the gesture heat map by the spatiotemporal feature extraction module of the gesture recognition model includes:

[0009] The gesture heat map is extracted through the residual block in the encoding unit of the spatiotemporal feature extraction module to obtain the deep spatiotemporal feature matrix of the gesture, wherein the spatiotemporal feature extraction module is configured as an autoencoder;

[0010] The deep-level spatiotemporal feature matrix of gestures is reduced in dimension by the flattening layer in the encoding unit to obtain the one-dimensional deep-level spatiotemporal feature of gestures;

[0011] The one-dimensional gesture deep spatiotemporal features are converted into initial gesture deep spatiotemporal features of a first preset size through the first fully connected layer in the encoding unit.

[0012] In one embodiment, the step of performing spatiotemporal compression processing on the gesture heat map by the spatiotemporal feature extraction module of the gesture recognition model includes:

[0013] The spatial compression unit of the spatiotemporal feature extraction module is used to convert the initial gesture deep spatiotemporal features into intermediate gesture deep spatiotemporal features of a second preset size, wherein the number of channels of the intermediate gesture deep spatiotemporal features is smaller than the number of channels of the initial gesture deep spatiotemporal features;

[0014] The time compression unit of the spatiotemporal feature extraction module is used to perform time compression processing on the intermediate gesture deep spatiotemporal features to obtain the gesture deep spatiotemporal features, wherein the first time step of the gesture deep spatiotemporal features is smaller than the second time step of the intermediate gesture deep spatiotemporal features.

[0015] In one embodiment, before the step of performing category prediction on the deep spatiotemporal features of the gesture using the classification module of the gesture recognition model to obtain the gesture recognition result, the gesture recognition method further includes:

[0016] The gated recurrent module of the gesture recognition model performs gated recurrent encoding on the deep-level spatiotemporal feature points of the gesture at adjacent sampling moments within the first time step to obtain the temporal variation features of the gesture. The temporal variation features of the gesture are then superimposed with the deep-level spatiotemporal features of the gesture to obtain the deep-level spatiotemporal features of the target gesture. The gated recurrent module is used to capture the continuous change information of the arm posture performing the gesture at adjacent sampling moments.

[0017] The steps to obtain gesture recognition results by using the classification module of the gesture recognition model to predict the category of the deep temporal and spatial features of the gesture include:

[0018] The classification module of the gesture recognition model is used to predict the category of the deep spatiotemporal features of the target gesture and obtain the gesture recognition result.

[0019] In one embodiment, the gesture recognition method further includes:

[0020] Construct a gesture heatmap sample set;

[0021] The gesture heat map sample set and the connection temporal classification loss function are used to train the initial gesture recognition model to obtain the gesture recognition model.

[0022] In one embodiment, the step of obtaining a gesture heat map collected by a millimeter-wave radar includes:

[0023] Obtain multi-frame radar intermediate frequency signals collected by millimeter wave radar;

[0024] For each radar intermediate frequency signal, performing discrete Fourier transform on each first linear frequency modulation signal in the radar intermediate frequency signal to obtain a plurality of distance data;

[0025] For each range data, performing discrete Fourier transform on the second linear frequency modulation signal in each range unit in the range data to obtain a plurality of range-Doppler data;

[0026] Generate gesture heatmap based on multiple range-Doppler data.

[0027] In addition, to achieve the above-mentioned purpose, the present application also proposes a gesture recognition device, which includes:

[0028] The acquisition module is used to obtain the gesture heat map collected by the millimeter wave radar;

[0029] The feature extraction module is used to perform dimensionality reduction and spatiotemporal compression on the gesture heat map through the spatiotemporal feature extraction module of the gesture recognition model to obtain the deep spatiotemporal features of the gesture;

[0030] The recognition module is used to predict the category of the deep temporal and spatial features of gestures through the classification module of the gesture recognition model to obtain the gesture recognition results.

[0031] In addition, to achieve the above objectives, the present application also proposes a gesture recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the gesture recognition method described above.

[0032] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the gesture recognition method as described above are implemented.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the gesture recognition method as described above are implemented.

[0034] One or more technical solutions proposed in this application have at least the following technical effects:

[0035] The present application provides a gesture recognition method, apparatus, device, storage medium, and program product. The method utilizes a spatiotemporal feature extraction module of a gesture recognition model to perform dimensionality reduction and spatiotemporal compression on a gesture heat map to obtain deep spatiotemporal features of the gesture for gesture recognition. Compared with direct gesture recognition based on the gesture heat map, the data volume of the deep spatiotemporal features of the gesture after dimensionality reduction and compression is smaller. Gesture recognition based on the deep spatiotemporal features of the gesture has low requirements for hardware computing power and storage resources, allowing the classification module of the gesture recognition model to quickly perform category prediction to obtain gesture recognition results, thereby improving gesture recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 This is a flowchart of the first embodiment of the gesture recognition method of the present application;

[0039] Figure 2 A schematic diagram of a portion of the structure of the gesture recognition model provided by the gesture recognition method of this application;

[0040] Figure 3 A schematic diagram of the structure of the autoencoder in the gesture recognition model provided by the gesture recognition method of this application;

[0041] Figure 4 This is a module diagram of the first embodiment of the gesture recognition device of the present application;

[0042] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the gesture recognition method in the embodiment of the present application.

[0043] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0045] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0046] The main solution of the embodiment of the present application is: obtaining a gesture heat map collected by a millimeter-wave radar; performing dimensionality reduction and spatiotemporal compression processing on the gesture heat map in sequence through the spatiotemporal feature extraction module of the gesture recognition model to obtain deep spatiotemporal features of the gesture; and performing category prediction on the deep spatiotemporal features of the gesture through the classification module of the gesture recognition model to obtain the gesture recognition result.

[0047] Gesture recognition is gaining increasing attention in human-computer interaction (HCI) across industry and research, particularly in the development of diverse applications leveraging the Internet of Things (IoT). For example, in smart home systems, users can automatically play music by snapping their fingers or open curtains with a simple gesture.

[0048] In related technologies, gesture recognition methods can be solutions based on cameras or wearable sensors. Although these solutions also perform well by utilizing deep learning networks, the requirements of specific conditions limit their potential in practical applications. For example, in camera-based solutions, multiple cameras must be installed in the use space to obtain high-quality videos from different angles. However, the cost of cameras is very high, which makes these solutions very expensive. Camera-based solutions also have strict requirements on light intensity and are difficult to implement. In addition, camera-based solutions cannot guarantee user privacy. As for wearable sensor-based solutions, there is a risk that users may feel uncomfortable when wearing wearable sensors, resulting in a poor user experience.

[0049] Considering the shortcomings and deficiencies of solutions based on cameras or wearable sensors, researchers have proposed a gesture recognition solution based on wireless signals, which offers advantages such as contactless operation, privacy protection, and reduced dependence on light. However, Wi-Fi signals have relatively long wavelengths and low range resolution, making it difficult to capture detailed gesture information. Furthermore, wireless signals are easily affected by the surrounding environment, different users, and room layout.

[0050] In contrast, gesture recognition solutions based on millimeter wave (mmWave) have attracted greater attention due to their superior performance at higher frequency bands. On the one hand, mmWave radar, with its high range and angular resolution, can provide more detailed motion information, which facilitates gesture recognition. On the other hand, with the development of advanced manufacturing and processing technologies, individuals can purchase commercial, miniaturized chips at low cost, enabling the implementation of mmWave radar. To this end, researchers have explored the potential of range Doppler spectra, Doppler shifts, point clouds, and centralized location Doppler profiles. Furthermore, because gesture recognition requires more fine-grained analysis and perception than activity recognition, various feature extraction algorithms and classification methods have been proposed, including bidirectional long short-term memory networks (LSTMs), bifurcated convolutional neural networks (CNNs), and voting mechanisms based on hidden Markov models.

[0051] For example, the millimeter-wave radar gesture recognition system Soli uses a tiny chip that operates in the 60GHz frequency band and supports digital beamforming. Soli models the scattering center of the hand and combines slow and fast time processing to generate a range-Doppler spectrum. It then extracts multiple features and ultimately uses a machine learning classifier to recognize gestures. Experimental results show that using a Bayesian filter achieves a recognition rate of 92.1%.

[0052] For example, mHomeGes, a real-time arm gesture recognition system for smart home scenarios, first captures a series of point clouds with a fixed-length sliding window. It then extracts the concentrated position-Doppler profile (CPDP) and compresses the intensity of each point into the distance and Doppler dimensions. The resulting sum of the distance-Doppler spectrum in the time dimension is input into a customized CNN recognition model to identify fine-grained gestures. To eliminate the multipath effect caused by surrounding reflectors, mHomeGes proposes a novel ghost image separation algorithm that leverages the speed synchronization between the user and the corresponding ghost image and adopts a Hidden Markov Model with Voting Mechanism (HMM-VM) to achieve continuous gesture recognition. mHomeGes is implemented based on the commercial millimeter-wave radar TIIWR1443 development board. mHomeGes achieved a recognition accuracy of 97.96% for 25 volunteers in five home scenarios.

[0053] Finally, a novel hybrid model, Pantomime, is proposed for accurate gesture recognition using sparse point clouds. Pantomime first aggregates the point clouds of gesture fragments to obtain more points in a single frame, then resamples the aggregated point cloud using upsampling and downsampling algorithms to meet a fixed number of points. To further extract spatiotemporal features from the aggregated point cloud, Pantomime adopts a hybrid architecture that combines the point cloud deep learning model Pointnet++ and an LSTM module; Pointnet++ is designed to extract spatial features from 3D point clouds, while the LSTM module continues to extract temporal features from the obtained features; finally, the two feature vectors are concatenated to form the final feature vector, which is then classified through a fully connected layer. Pantomime achieved 95% accuracy and 99% area under the ROC (receiver operating characteristic curve) curve (AUC) for 21 gestures.

[0054] In summary, existing millimeter wave-based solutions still have the following shortcomings:

[0055] 1) The heat map-based solution requires a large amount of data: This method generates a heat map to reflect the position and shape of the target object, but the amount of image data generated is large, the processing complexity is high, and the hardware computing power and storage resources are high. This makes the gesture recognition method less efficient and is not conducive to real-time processing.

[0056] 2) Solutions based on statistical information lack information and have low accuracy: Relying solely on statistical information (such as radar signal strength and frequency) for gesture recognition ignores the dynamic characteristics and details of gestures, resulting in low recognition accuracy and difficulty meeting high-precision application requirements.

[0057] 3) Point cloud-based solutions suffer from low accuracy when there are few antennas: When antenna configurations are limited (e.g., 3 transmitters and 4 receivers or even 1 transmitter and 1 receiver), the generated point cloud data is sparse and cannot effectively capture the complete form and subtle changes of gestures, thus affecting recognition accuracy and reliability. This shortcoming is particularly evident in complex or rapidly changing gesture recognition scenarios.

[0058] Therefore, the present application provides a gesture recognition method, which uses the spatiotemporal feature extraction module of the gesture recognition model to perform dimensionality reduction and spatiotemporal compression on the gesture heat map to obtain deep spatiotemporal features of the gesture for gesture recognition. Compared with directly using the gesture heat map for gesture recognition, the data volume of the deep spatiotemporal features of the gesture after dimensionality reduction and compression is smaller. Gesture recognition based on the deep spatiotemporal features of the gesture has low requirements on hardware computing power and storage resources, so that the classification module of the gesture recognition model can quickly perform category prediction to obtain gesture recognition results, thereby improving the efficiency of gesture recognition.

[0059] Moreover, compared with solutions based on statistical information, the deep spatiotemporal features of gestures focus on the dynamic characteristics and details of gestures, improve the accuracy of gesture recognition, and are suitable for scenarios with high-precision application requirements.

[0060] In addition, compared with point cloud-based solutions, the deep spatiotemporal features of gestures are extracted from gesture heat maps, which are abundant in number and can effectively capture the complete form and subtle changes of gestures. The accuracy and reliability of gesture recognition are higher.

[0061] It should be noted that the execution subject of this embodiment can be a computing service device with gesture recognition, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device or gesture recognition device capable of implementing the above functions. The following uses a gesture recognition device as an example to illustrate this embodiment and the following embodiments.

[0062] Based on this, the embodiment of the present application provides a gesture recognition method, referring to Figures 1 to 3 , Figure 1 This is a flow chart of the first embodiment of the gesture recognition method of this application. Figure 2 A schematic diagram of a portion of the structure of the gesture recognition model provided by the gesture recognition method of this application, Figure 3 Schematic diagram of the structure of the autoencoder in the gesture recognition model provided by the gesture recognition method of this application.

[0063] In this embodiment, the gesture recognition method may include steps S100 to S300:

[0064] Step S100: Acquire a gesture heat map collected by a millimeter-wave radar.

[0065] It should be noted that the millimeter wave radar may include nTx transmitting antennas and nRx receiving antennas.

[0066] During gesture recognition, the millimeter-wave radar transmits an FMCW (Frequency Modulated Continuous Wave) signal s(t) through nTx transmitting antennas, as shown in the following formula 1:

[0067]

[0068] Where A is the signal amplitude, f0 is the initial frequency, μ is the frequency modulation slope, t is the time, and j is the imaginary unit.

[0069] After that, the FMCW signal propagates in space and hits the arm, which is then Rx Finally, the millimeter-wave radar can mix the reference signal generated inside the millimeter-wave radar with the reflected signal to generate an intermediate frequency (IF) signal. Based on the IF signal, it can generate a range-Doppler heat map as a gesture heat map.

[0070] In step S200 , the spatiotemporal feature extraction module of the gesture recognition model performs dimensionality reduction and spatiotemporal compression on the gesture heat map in sequence to obtain deep spatiotemporal features of the gesture.

[0071] It should be noted that the deep spatiotemporal features of gestures can reflect the inherent patterns and dynamic changes in gesture data, revealing the patterns and interactions of gesture data at different temporal and spatial scales. Dimensionality reduction can be performed by converting the image features of a gesture heatmap into lower-dimensional gesture features. Spatiotemporal compression can include temporal compression and spatial compression. Temporal compression can compress gesture features in the temporal dimension, while spatial compression can compress gesture features in the spatial dimension.

[0072] It is understandable that deep spatiotemporal features of gestures can be extracted from gesture heatmaps using a pre-trained gesture recognition model. The gesture recognition model can be implemented using deep learning methods, attention mechanisms, or multi-scale feature fusion.

[0073] Step S300 , performing category prediction on the deep temporal and spatial features of the gesture through the classification module of the gesture recognition model to obtain a gesture recognition result.

[0074] It should be noted that the gesture recognition model may also include a classification module, which can perform category prediction based on the deep temporal and spatial features of gestures and convert the extracted deep temporal and spatial features of gestures into gesture recognition results of gesture categories. Figure 2 As shown, the classification module 103 may include multiple fully connected layers (FCLs), and the features of each time step of the deep spatiotemporal features of the gesture will pass through multiple fully connected layers to generate the output of the category score.

[0075] Therefore, this embodiment provides a gesture recognition method, which uses the spatiotemporal feature extraction module of the gesture recognition model to perform dimensionality reduction and spatiotemporal compression on the gesture heat map to obtain deep spatiotemporal features of the gesture for gesture recognition. Compared with directly using the gesture heat map for gesture recognition, the data volume of the deep spatiotemporal features of the gesture after dimensionality reduction and compression is smaller. Gesture recognition based on the deep spatiotemporal features of the gesture has low requirements on hardware computing power and storage resources, so that the classification module of the gesture recognition model can quickly perform category prediction to obtain gesture recognition results, thereby improving the efficiency of gesture recognition.

[0076] Moreover, compared with solutions based on statistical information, the deep spatiotemporal features of gestures focus on the dynamic characteristics and details of gestures, improve the accuracy of gesture recognition, and are suitable for scenarios with high-precision application requirements.

[0077] In addition, compared with point cloud-based solutions, the deep spatiotemporal features of gestures are extracted from gesture heat maps, which are abundant in number and can effectively capture the complete form and subtle changes of gestures. The accuracy and reliability of gesture recognition are higher.

[0078] In a feasible implementation, step S200 may include steps A210 to A230:

[0079] In step A210 , features of the gesture heat map are extracted using the residual block in the encoding unit of the spatiotemporal feature extraction module to obtain a deep spatiotemporal feature matrix of the gesture.

[0080] Among them, the spatiotemporal feature extraction module is configured as an autoencoder.

[0081] In step A220 , the deep-level spatiotemporal feature matrix of the gesture is subjected to dimensionality reduction processing by the flattening layer in the encoding unit to obtain a one-dimensional deep-level spatiotemporal feature of the gesture.

[0082] Step A230: Convert the one-dimensional gesture deep spatiotemporal features into initial gesture deep spatiotemporal features of a first preset size through the first fully connected layer in the encoding unit.

[0083] It should be noted that the spatiotemporal feature extraction module may include an autoencoder, such as Figure 3 As shown, the encoding unit 1011 of the autoencoder may include a residual block, a flattening layer Flatten and a first fully connected layer connected in sequence. Among them, the residual block may include 5 residual blocks ResBlock connected in sequence. The residual connection can help solve the problem of gradient disappearance or gradient explosion in deep network training. Each residual block passes through a convolutional layer (conv3x3) to reduce the spatial dimension (ie, height and width) of the gesture deep spatiotemporal feature matrix and increase the number of feature channels. The flattening layer can convert the multi-dimensional gesture deep spatiotemporal feature matrix into a one-dimensional vector to obtain a one-dimensional gesture deep spatiotemporal feature for subsequent processing by the first fully connected layer. Through the first fully connected layer, the flattened one-dimensional gesture deep spatiotemporal feature can be further compressed and output as the initial gesture deep spatiotemporal feature of the first preset size. The first preset size can be determined according to the hyperparameters of the first fully connected layer, and the hyperparameters of the first fully connected layer can be set according to the actual usage scenario.

[0084] In addition, if Figure 2 As shown, the decoding unit 1012 of the autoencoder may include a second fully connected layer, an unflattening layer UnFlatten and a deconvolution block connected in sequence, wherein the deconvolution block may include 5 deconvolution layers DeConv connected in sequence, and the decoding unit 1012 may reconstruct a gesture heat map based on the deep spatiotemporal features of the gesture output by the encoding unit.

[0085] In one example, the dimension of the gesture heat map is (1, N, M), where N and M are the number of distance dimension points and Doppler dimension points of the distance-Doppler heat map, respectively. If N = 128 and M = 256, the initial gesture deep spatiotemporal features can be a latent vector of size (1, 1024).

[0086] Therefore, this embodiment provides a gesture recognition method. By configuring the spatiotemporal feature extraction module of the gesture recognition model as an autoencoder and configuring the encoding unit of the autoencoder as a sequentially connected residual block, flattening layer, and fully connected layer, this method extracts deep spatiotemporal features of arm gestures from gesture heatmaps. This method fully utilizes the distance and speed information in the gesture heatmaps to capture changes in the arm at different time points and spatial positions for gesture recognition, thereby improving gesture recognition accuracy. Furthermore, the gesture recognition model can adapt to heatmap processing in different scenarios through simple hyperparameter adjustments, resulting in high robustness.

[0087] In a feasible implementation, step S200 may further include steps A240 to A250:

[0088] Step A240 , using the spatial compression unit of the spatiotemporal feature extraction module, the initial gesture deep spatiotemporal feature is converted into an intermediate gesture deep spatiotemporal feature of a second preset size, the number of which is smaller than the channel number of the initial gesture deep spatiotemporal feature.

[0089] Step A250 : Using the time compression unit of the spatiotemporal feature extraction module to perform time compression processing on the intermediate gesture deep spatiotemporal features to obtain the gesture deep spatiotemporal features.

[0090] Among them, the first time step of the deep-level spatiotemporal feature of the gesture is smaller than the second time step of the deep-level spatiotemporal feature of the intermediate gesture.

[0091] It should be noted that the spatiotemporal feature extraction module can also include a spatial compression unit and a temporal compression unit. The spatial compression unit is used to perform spatial compression processing on the deep spatiotemporal features of the initial gesture, and the temporal compression unit is used to perform temporal compression processing on the deep spatiotemporal features of the intermediate gesture.

[0092] like Figure 2 As shown, the spatial compression unit 1013 may include two second fully connected layers, the second preset size is smaller than the first preset size, and correspondingly, the number of channels of the intermediate gesture deep spatiotemporal features is smaller than the number of channels of the initial gesture deep spatiotemporal features. The second preset size can be determined according to the hyperparameters of the second fully connected layer, and the hyperparameters of the second fully connected layer can be set according to the actual usage scenario. The spatial compression unit 1013 can convert the initial gesture deep spatiotemporal features into more discriminative spatial features, i.e., the intermediate gesture deep spatiotemporal features, through nonlinear transformation, thereby focusing more on the key shape and position changes of the arm to improve the accuracy of gesture recognition.

[0093] The time compression unit 1014 can be implemented by two residual blocks. The time compression unit 1014 reduces the time step of the intermediate gesture deep-level spatiotemporal features in the time dimension through the residual blocks, that is, reduces the sampling points of the intermediate gesture deep-level spatiotemporal features in time, thereby removing redundant time information and retaining the key time nodes of gesture changes. The first time step and the second time step can be determined according to the actual usage scenario and purpose of the gesture recognition model. In this embodiment, the first time step can be rounded to one-quarter of the second time step. The first residual block can convert the intermediate gesture deep-level spatiotemporal features into a feature vector in a specific format, and the second residual block can perform time compression on the feature vector of each output channel of the first residual block to obtain the gesture deep-level spatiotemporal features.

[0094] In one example, if the initial gesture deep spatiotemporal feature can be a feature vector of size (B, T, 1024), then the intermediate gesture deep spatiotemporal feature can be a feature vector of size (B, T, 64), and the first residual block of the time compression unit reshapes the feature vector of size (B, T, 64) to obtain (B*64, 1, T), and the second residual module of the time compression unit performs time compression on (B*64, 1, T) to obtain a gesture deep spatiotemporal feature of size (B*64, 1, T / / 4), where B is the batch of deep spatiotemporal features of gesture and T is the time step.

[0095] Therefore, this embodiment provides a gesture recognition method, which further compresses the deep spatiotemporal features of the initial gesture in the spatial dimension and the temporal dimension respectively through the spatial compression unit and the temporal compression unit of the spatiotemporal feature extraction module, converts the deep spatiotemporal features of the initial gesture into more discriminative spatial features, focuses more on the key shape and position changes of the arm, removes redundant time information in the deep spatiotemporal features of the initial gesture, and retains the key time nodes of the gesture changes, further increasing the amount of data required to be processed by the classification module and improving the efficiency and accuracy of gesture recognition.

[0096] In a feasible embodiment, before step S300, the gesture recognition method may further include: performing gated loop encoding on the deep-level spatiotemporal feature points of the gesture at adjacent sampling moments within the first time step through a gated loop module of the gesture recognition model to obtain a time-varying feature of the gesture, and superimposing the time-varying feature of the gesture with the deep-level spatiotemporal feature of the gesture to obtain a deep-level spatiotemporal feature of the target gesture. The gated loop module is used to capture the continuous change information of the posture of the arm performing the gesture at adjacent sampling moments.

[0097] Correspondingly, step S300 may include: performing category prediction on the deep spatiotemporal features of the target gesture by a classification module of the gesture recognition model to obtain a gesture recognition result.

[0098] It should be noted that if Figure 2 As shown, the gesture recognition model may further include a gated recurrent unit (GRU) 102, which may iteratively process the deep-level spatiotemporal feature points of the gesture at adjacent sampling moments in the deep-level spatiotemporal feature of the gesture, i.e., gated recurrent encoding, and retain the historical information of the useful deep-level spatiotemporal feature points of the gesture, convert the deep-level spatiotemporal feature of the gesture into a time feature representation useful for subsequent classification tasks, and after obtaining the temporal change feature of the gesture, superimpose the temporal change feature of the gesture with the deep-level spatiotemporal feature of the gesture to obtain the deep-level spatiotemporal feature of the target gesture, so that the deep-level spatiotemporal feature of the target gesture can represent the temporal evolution law of the gesture, thereby further improving the accuracy of gesture recognition.

[0099] It can be understood that when GRU performs gated recurrent encoding on the deep spatiotemporal features of gestures, it first reshapes the deep spatiotemporal features of a gesture to obtain the deep spatiotemporal features of the gesture to be encoded, and then performs gated recurrent encoding on the deep spatiotemporal features of the gesture to be encoded to obtain the temporal change features of the gesture.

[0100] In one example, the GRU input is a gesture deep spatiotemporal feature of size (B*64, 1, T / / 4). The gesture deep spatiotemporal feature is first reshaped into a gesture deep spatiotemporal feature to be encoded of size (B, T / / 4, 64). Then, the gesture deep spatiotemporal feature to be encoded is gated and cyclically encoded to obtain a gesture deep spatiotemporal feature of size (B, T / / 4, d gru ) gesture time variation characteristics. Among them, d gru It is the hyperparameter of the gated loop module and is set according to the actual usage scenario of the model.

[0101] Therefore, this embodiment provides a gesture recognition method. After obtaining the deep-level spatiotemporal features of the gesture, the deep-level spatiotemporal features of the gesture are also encoded by GRU, and the temporal change features of the gesture are obtained and superimposed with the deep-level spatiotemporal features of the gesture to obtain the deep-level spatiotemporal features of the target gesture for gesture recognition, so that the deep-level spatiotemporal features of the target gesture can represent the temporal evolution law of the gesture, thereby further improving the accuracy of gesture recognition.

[0102] In a feasible implementation, the gesture recognition method further includes: constructing a gesture heat map sample set; and using the gesture heat map sample set and a connection temporal classification loss function to train an initial gesture recognition model to obtain a gesture recognition model.

[0103] It should be noted that before running the gesture recognition method of this embodiment, a gesture heatmap sample set can also be constructed to train the initial gesture recognition module and pre-train a gesture recognition model. The gesture heatmap sample set can include multiple gesture heatmap samples and corresponding gesture category labels.

[0104] It is understandable that during model training, CTCLoss (Connectionist Temporal Classification Loss) can be used to address the temporal misalignment between the gesture heatmap samples input to the encoding unit and the output labels of the classification module. This loss also considers the contextual information of the entire gesture heatmap sample set, rather than individual gesture heatmap samples, to optimize the model's prediction performance. Additionally, during model training, the ADAM (Adaptive Moment Estimation) optimizer can be used with a learning rate of 0.001.

[0105] Therefore, this embodiment provides a gesture recognition method. During model training, CTCLoss is used to process the temporal misalignment problem between the gesture heat map samples input by the encoding unit and the output labels of the classification module, and the contextual information of the entire gesture heat map sample set is considered instead of a single gesture heat map sample to optimize the prediction performance of the model. This allows the model to more accurately predict the category of the deep spatiotemporal features of the gesture, further improving the accuracy of gesture recognition.

[0106] In a feasible implementation, step S100 may include: obtaining multiple frames of radar intermediate frequency signals collected by a millimeter-wave radar; performing a discrete Fourier transform on each first linear frequency modulation signal in the radar intermediate frequency signal for each radar intermediate frequency signal to obtain multiple distance data; performing a discrete Fourier transform on each second linear frequency modulation signal in each distance unit in the distance data for each distance data to obtain multiple range-Doppler data; and generating a gesture heat map based on the multiple range-Doppler data.

[0107] It should be noted that the multi-frame radar intermediate frequency signal can be obtained by mixing the reflected signals received by multiple transmitting antennas with a reference signal.

[0108] Using formula 2, for each radar intermediate frequency signal, a discrete Fourier transform is performed on each first linear frequency modulation signal in the radar intermediate frequency signal to obtain multiple distance data; formula 2 is:

[0109]

[0110] Among them, s range (k,m) is the distance data of the mth Chirp (linear frequency modulation signal), s dig (n,m) is the digital signal of the nth sampling point of the mth chirp, N is the number of sampling points, and k is the frequency index after DFT (Discrete Fourier Transform) (corresponding to the arm distance).

[0111] Using Formula 3, for each range data, a discrete Fourier transform is performed on the second linear frequency modulation signal in each range unit in the range data to obtain multiple range-Doppler data; Formula 3 is:

[0112]

[0113] Among them, s range (k,m) is the range-Doppler data of the mth chirp, M is the total number of chirps in the radar intermediate frequency signal, and l is the frequency index after Doppler DFT (corresponding to the arm speed).

[0114] In addition, it should be noted that before the step of "performing a discrete Fourier transform on the second linear frequency modulation signal within each range unit in the range data to obtain multiple range-Doppler data," the multiple range data may also be subjected to static clutter elimination to remove fixed interference (such as ground reflections, buildings, etc.) that does not change over time in the range data. Preferably, static clutter elimination can be achieved by calculating the average range spectrum of the range data of the radar intermediate frequency signal for multiple consecutive frames and subtracting the average range spectrum from the consecutive frames.

[0115] It can be understood that by using Formula 4 to perform incoherent accumulation on multiple range-Doppler data, the power spectrum after incoherent accumulation can be obtained. The power spectrum after incoherent accumulation P(k,l) can be converted into a heat map to obtain a gesture heat map. Wherein, Formula 4 is:

[0116]

[0117] The basic principle behind generating a gesture heatmap is to map the power values ​​of the incoherently accumulated power spectrum to a visually recognizable color or brightness range. In a gesture heatmap, the color or brightness of each pixel corresponds to the power value at the corresponding position in the incoherently accumulated power spectrum, visually demonstrating the distribution of the target object in both distance and speed dimensions.

[0118] In one example, a TII WR1843BOOST millimeter-wave radar was used to collect 2,470 gesture heatmaps from 10 volunteers. The gesture recognition method of this embodiment was used for gesture recognition. The 2,470 gesture heatmaps included nine gestures: one hand up, one hand down, both hands up, both hands down, both hands inward, both hands outward, right swipe, left swipe, and circle. As shown in Table 1 below, the gesture recognition method of this embodiment achieved high accuracy.

[0119] Table 1

[0120] gesture Accuracy Recall F1 score One-hand lift 100.00％ 97.22％ 98.59％ One-handed push-down 98.53％ 100.00％ 99.26％ Raise your hands 100％ 100.00％ 100.00％ Press down with both hands 100.00％ 100.00％ 100.00％ Hands adducted 97.30％ 100.00％ 98.63％ Hands outstretched 100.00％ 100.00％ 100.00％ Swipe right 100.00％ 100.00％ 100.00％ Swipe left 100.00％ 100.00％ 100.00％ Draw a circle 98.31％ 98.31％ 98.31％

[0121] Therefore, this embodiment provides a gesture recognition method that generates a gesture heat map by performing two consecutive discrete Fourier transforms on multi-frame radar intermediate frequency signals. The gesture heat map contains the changes in gesture features in both time and space dimensions, allowing the gesture recognition model to extract deep temporal and spatial features of gestures from both time and space dimensions. Compared with gesture recognition that relies on statistical information, this method takes into account the dynamic characteristics and details of gestures, thereby improving recognition accuracy.

[0122] This application provides a gesture recognition device, such as Figure 4 As shown, the gesture recognition device may include:

[0123] An acquisition module 10 is used to acquire a gesture heat map collected by a millimeter-wave radar;

[0124] The feature extraction module 20 is used to perform dimensionality reduction and spatiotemporal compression processing on the gesture heat map in sequence through the spatiotemporal feature extraction module of the gesture recognition model to obtain deep spatiotemporal features of the gesture;

[0125] The recognition module 30 is used to perform category prediction on the deep temporal and spatial features of the gesture through the classification module of the gesture recognition model to obtain a gesture recognition result.

[0126] For more implementation details of the above-mentioned gesture recognition device, please refer to the description of the specific implementation of the gesture recognition method in the above embodiment. For the sake of brevity, they will not be repeated here.

[0127] The present application provides a gesture recognition device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the gesture recognition method of the above-mentioned embodiment 1.

[0128] Reference below Figure 5 , which shows a schematic diagram of the structure of a gesture recognition device suitable for implementing the embodiments of the present application. The gesture recognition device in the embodiments of the present application may include, but is not limited to, mobile terminals such as laptop computers, PDAs (Personal Digital Assistants), and PADs (Portable Application Descriptions), as well as fixed terminals such as desktop computers. Figure 5 The gesture recognition device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0129] like Figure 5As shown, the gesture recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the gesture recognition device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input devices 1007, such as a touch screen, touchpad, keyboard, mouse, image sensor, microphone, etc.; output devices 1008, such as a liquid crystal display (LCD) and speakers; storage devices 1003, such as a magnetic tape and hard disk; and communication devices 1009. Communication device 1009 can allow the gesture recognition device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a gesture recognition device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or provided instead.

[0130] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0131] The gesture recognition device provided in this application, utilizing the gesture recognition method described in the aforementioned embodiment, solves the technical problem of processing range-Doppler heat maps and improving gesture recognition efficiency. Compared to related technologies, the beneficial effects of the gesture recognition device provided in this application are the same as those of the gesture recognition method described in the aforementioned embodiment. Other technical features of this gesture recognition device are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.

[0132] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0133] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0134] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, and the computer-readable program instructions are used to execute the gesture recognition method in the above embodiment.

[0135] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0136] The computer-readable storage medium may be included in the gesture recognition device, or may exist independently without being incorporated into the gesture recognition device.

[0137] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a gesture recognition device, the gesture recognition device: obtains a gesture heat map collected by a millimeter-wave radar; performs dimensionality reduction and spatiotemporal compression on the gesture heat map in sequence through a spatiotemporal feature extraction module of a gesture recognition model to obtain deep spatiotemporal features of the gesture; and performs category prediction on the deep spatiotemporal features of the gesture through a classification module of the gesture recognition model to obtain a gesture recognition result.

[0138] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0140] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0141] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned gesture recognition method. This computer-readable storage medium addresses the technical problem of processing range-Doppler heat maps and improving gesture recognition efficiency. Compared to related technologies, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the gesture recognition method provided in the aforementioned embodiments and are not further elaborated here.

[0142] The present application also provides a computer program product, including a computer program, which implements the steps of the gesture recognition method as described above when executed by a processor.

[0143] The computer program product provided in this application can solve the technical problem of how to process range-Doppler heat maps and improve the efficiency of gesture recognition. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the gesture recognition method provided in the above embodiments, and will not be elaborated here.

[0144] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A gesture recognition method, characterized in that: The gesture recognition method comprises: Obtain gesture heatmaps collected by millimeter-wave radar; The gesture heat map is processed by the spatiotemporal feature extraction module of the gesture recognition model in sequence with dimensionality reduction and spatiotemporal compression to obtain the deep spatiotemporal features of the gesture; The classification module of the gesture recognition model is used to perform category prediction on the deep temporal and spatial features of the gesture to obtain a gesture recognition result.

2. The gesture recognition method according to claim 1, wherein: The step of performing dimensionality reduction processing on the gesture heat map by using the spatiotemporal feature extraction module of the gesture recognition model includes: Extracting features from the gesture heat map using a residual block in an encoding unit of the spatiotemporal feature extraction module to obtain a deep spatiotemporal feature matrix of the gesture, wherein the spatiotemporal feature extraction module is configured as an autoencoder; Performing dimensionality reduction processing on the gesture deep-level spatiotemporal feature matrix through the flattening layer in the encoding unit to obtain a one-dimensional gesture deep-level spatiotemporal feature; The one-dimensional gesture deep spatiotemporal features are converted into initial gesture deep spatiotemporal features of a first preset size through a first fully connected layer in the encoding unit.

3. The gesture recognition method according to claim 2, wherein: The step of performing spatiotemporal compression processing on the gesture heat map by using the spatiotemporal feature extraction module of the gesture recognition model includes: Using the spatial compression unit of the spatiotemporal feature extraction module, the initial gesture deep spatiotemporal feature is converted into an intermediate gesture deep spatiotemporal feature of a second preset size, wherein the number of channels of the intermediate gesture deep spatiotemporal feature is smaller than the number of channels of the initial gesture deep spatiotemporal feature; The time compression unit of the spatiotemporal feature extraction module is used to perform time compression processing on the intermediate gesture deep spatiotemporal feature to obtain the gesture deep spatiotemporal feature, wherein the first time step of the gesture deep spatiotemporal feature is smaller than the second time step of the intermediate gesture deep spatiotemporal feature.

4. The gesture recognition method according to claim 3, wherein: Before the step of performing category prediction on the deep spatiotemporal features of the gesture by the classification module of the gesture recognition model to obtain the gesture recognition result, the gesture recognition method further includes: Performing gated cyclic encoding on the deep-level spatiotemporal feature points of the gesture at adjacent sampling moments within the first time step through the gated cyclic module of the gesture recognition model to obtain a temporal variation feature of the gesture, and superimposing the temporal variation feature of the gesture with the deep-level spatiotemporal feature of the gesture to obtain a deep-level spatiotemporal feature of the target gesture. The gated cyclic module is used to capture continuous variation information of the posture of the arm performing the gesture at adjacent sampling moments; The step of performing category prediction on the deep temporal and spatial features of the gesture by using a classification module of the gesture recognition model to obtain a gesture recognition result includes: The classification module of the gesture recognition model is used to perform category prediction on the deep spatiotemporal features of the target gesture to obtain a gesture recognition result.

5. The gesture recognition method according to any one of claims 1 to 4, characterized in that: The gesture recognition method further includes: Construct a gesture heatmap sample set; The gesture heat map sample set and the connection temporal classification loss function are used to train the initial gesture recognition model to obtain the gesture recognition model.

6. The gesture recognition method according to any one of claims 1 to 4, characterized in that: The step of obtaining the gesture heat map collected by the millimeter wave radar includes: Obtain multi-frame radar intermediate frequency signals collected by millimeter wave radar; For each of the radar intermediate frequency signals, performing discrete Fourier transform on each of the first linear frequency modulation signals in the radar intermediate frequency signal to obtain a plurality of distance data; For each of the distance data, performing discrete Fourier transform on the second linear frequency modulation signal in each distance unit in the distance data to obtain a plurality of range-Doppler data; A gesture heat map is generated based on the plurality of range-Doppler data.

7. A gesture recognition device, characterized in that: The gesture recognition device comprises: The acquisition module is used to obtain the gesture heat map collected by the millimeter wave radar; A feature extraction module is used to perform dimensionality reduction and spatiotemporal compression processing on the gesture heat map in sequence through the spatiotemporal feature extraction module of the gesture recognition model to obtain deep spatiotemporal features of the gesture; The recognition module is used to perform category prediction on the deep temporal and spatial features of the gesture through the classification module of the gesture recognition model to obtain a gesture recognition result.

8. A gesture recognition device, characterized in that: The gesture recognition device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and is configured to implement the steps of the gesture recognition method according to any one of claims 1 to 6 through the computer program.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the gesture recognition method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the gesture recognition method according to any one of claims 1 to 6 are implemented.

Images