A millimeter wave radar point cloud gesture recognition method and system based on double-flow space-time feature coding

Abstract

This application provides a method and system for gesture recognition based on millimeter-wave radar point clouds using dual-stream spatiotemporal feature coding, relating to the field of radar target recognition technology. The method includes: acquiring gesture echo point cloud data collected by millimeter-wave radar; preprocessing the gesture echo point cloud data to generate a basic spatiotemporal feature matrix and a higher-order feature matrix; inputting the basic spatiotemporal feature matrix into a first-stream spatial feature branch to extract a basic semantic feature vector; inputting the higher-order feature matrix into a second-stream temporal feature branch to extract a higher-order dynamic feature vector; fusing the basic semantic feature vector and the higher-order dynamic feature vector to obtain a dual-stream joint feature vector; and inputting the dual-stream joint feature vector into a classifier to output the gesture recognition result. This application solves the problem of low detection rate of key action frames in millimeter-wave radar point clouds and improves the efficiency of spatiotemporal fusion.

Description

Technical Field

[0001] This application relates to the fields of radar target recognition and human-computer interaction technology, and in particular to a millimeter-wave radar point cloud gesture recognition method and system based on dual-stream spatiotemporal feature coding. Background Technology

[0002] With the implementation of human-computer interaction and smart IoT applications, millimeter-wave radar signals, compared to optical sensors, have the advantages of stronger resistance to ambient light interference and no privacy leaks. The demand for gesture recognition using millimeter-wave radar as a new interaction method continues to rise. In recent years, its application scenarios, such as in-vehicle cockpits, smart homes, and sterile medical devices, have been continuously expanding.

[0003] Millimeter-wave radar point clouds are the core carrier for completing spatiotemporal feature mapping in motion perception. They can capture minute hand movements and simultaneously acquire high-precision features in multiple dimensions such as distance, speed, and spatial angle. However, when performing gesture recognition feature extraction, this type of data inherently suffers from a series of structural defects. For example, its spatial distribution is sparse, and different users' operating habits can cause significant fluctuations in the length of the action sequence. These inherent properties lead to a high degree of spatiotemporal coupling of continuous gestures, hindering the accurate discrimination of micro-movements. Currently, a typical challenge in the field of gesture recognition using radar point cloud data is the submergence of keyframe features, specifically manifested in the following three categories: 1) high-order statistical distributions are difficult to capture; 2) spatiotemporal information fusion is prone to folding problems; and 3) compared to conventional optical images, fine-grained distinguishability is not high. These defects reduce the effective perceptual feature surface of the model, decrease feature decoupling ability, and ultimately lead to misjudgment of similar gestures. The occurrence of misjudgment is also directly related to the common problems in practical application scenarios, such as inconsistent gesture amplitude and the simultaneous presence of redundant background noise. Traditional machine recognition and evaluation work relies heavily on manual feature engineering and single-stream network architecture, which has problems such as subjectively preset frame truncation and insufficient generalization ability. When dealing with complex high-dimensional sparse point clouds, the analysis effect is not good and cannot meet the ever-increasing requirements of actual deployment in the era of intelligent interconnection.

[0004] In recent years, with the continuous development of deep learning technology, data-driven neural network architectures have provided a new technical direction for radar gesture recognition. Compared with traditional methods that rely mainly on physical formula derivation, deep learning technology has advantages in both extracting the consistency of high-dimensional implicit features and modeling long program sequences. However, existing deep learning-based radar gesture recognition methods mostly adopt single-branch network structures, coupling the low-level physical features and high-order statistical features in gesture point cloud data. This makes it difficult to fully capture the spatiotemporal information of key action frames in dynamic gestures, resulting in insufficient feature decoupling capabilities and persistent misjudgment of similar gestures. Furthermore, while stacked 3D convolutional or large-scale Transformer models have achieved breakthroughs in recognition accuracy, their high computational complexity and large number of parameters make them difficult to deploy on resource-constrained edge devices, limiting the practical application of gesture recognition technology in IoT terminals and embedded scenarios.

[0005] Therefore, how to effectively decouple the low-level physical features and high-order statistical features that are coupled in millimeter-wave radar point clouds under a lightweight network architecture in order to improve the accuracy of gesture recognition and meet the needs of edge deployment is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] This application provides a method and system for hand gesture recognition based on millimeter-wave radar point clouds using dual-stream spatiotemporal feature coding, aiming to solve the following technical problems: 1) In view of the inherent defects of dynamic gesture sparse point cloud data captured by MIMO millimeter-wave radar, such as inconsistent frame lengths, noise interference, and high coupling of spatiotemporal features, how to obtain characteristic sequences that adapt to the input requirements of different branch networks by performing targeted preprocessing and feature reconstruction on the original point cloud data, so as to achieve effective decoupling between low-level physical features and high-order statistical features.

[0007] 2) How to design a lightweight dual-stream spatiotemporal fusion network to encode the underlying physical micro-motion features and the high-order macro-motion features respectively, and enhance the network's ability to perceive key action frames through an attention mechanism, thereby improving the accuracy of gesture recognition with low computational overhead.

[0008] 3) How to improve the generalization performance and convergence accuracy of the model by optimizing the training strategy to alleviate the gradient instability problem in long sequence training.

[0009] In a first aspect, this application provides a millimeter-wave radar point cloud gesture recognition method based on dual-stream spatiotemporal feature coding, the method comprising: Acquire gesture echo point cloud data collected by millimeter-wave radar, wherein the gesture echo point cloud data includes a variable-length frame sequence; The gesture echo point cloud data is preprocessed to generate a basic spatiotemporal feature matrix and a higher-order feature matrix; The basic spatiotemporal feature matrix is ​​input into the spatial feature branch of the first-flow LSTM to extract the basic semantic feature vector; The higher-order feature matrix is ​​input into the second-stream Attention-BiGRU temporal feature branch to extract the higher-order dynamic feature vector. By fusing the basic semantic feature vector and the higher-order dynamic feature vector, a dual-stream joint feature vector is obtained; The dual-stream joint feature vector is input into the classifier, and the gesture recognition result is output.

[0010] As a preferred technical solution, the preprocessing of the gesture echo point cloud data to generate a basic spatiotemporal feature matrix and a higher-order feature matrix specifically includes: Based on each frame in the variable-length frame sequence, physical features including spatial location coordinates, Doppler components, and planar radial distance are extracted, and the number of points in each frame is normalized to a fixed length to reconstruct the basic spatiotemporal feature matrix. For each frame in the variable-length frame sequence, multiple echo features are extracted and intra-frame spatial statistical vectors are calculated. Combined with the first-order temporal difference features between frames, the higher-order feature matrix is ​​reconstructed.

[0011] As a preferred technical solution, the step of extracting physical features including spatial location coordinates, Doppler components, and planar radial distance from each frame in the variable-length frame sequence, and performing fixed-length normalization on the number of points in each frame to reconstruct the basic spatiotemporal feature matrix, specifically includes: For the gesture echo point cloud data of frame t, extract its spatial position x-coordinate. y-axis Doppler components and planar radial distance , wherein the radial distance Calculate according to the following formula: In the formula, This indicates the radial distance of the reflection point on the horizontal plane; The horizontal coordinates of the spatial positions of all reflection points in the current frame y-axis Doppler components and planar radial distance The vectors are unfolded and concatenated in sequence to form a one-dimensional vector, which serves as the basic feature vector for the t-th frame. The dimension of the vector is determined based on the number of extracted physical feature channels and the maximum number of effective reflection points after alignment of a single frame. The basic feature vectors of each frame are arranged in chronological order and reconstructed into the basic spatiotemporal feature matrix.

[0012] As a preferred technical solution, the step of extracting multiple echo features and calculating intra-frame spatial statistical vectors for each frame in the variable-length frame sequence, and reconstructing the higher-order feature matrix by combining the first-order temporal difference features between frames, specifically includes: Based on multiple echo features in the gesture echo point cloud data of frame t, calculate the statistics corresponding to each echo attribute, and combine them with the total effective dynamic points of a single frame to form an intra-frame spatial statistical vector. Calculate the first-order temporal difference feature based on the spatial statistical vector of the current frame and the spatial statistical vector of the previous frame; The spatial statistical vector and the temporal difference feature are concatenated on the feature channel to form the high-order feature vector of the t-th frame; The higher-order feature vectors of each frame are arranged in chronological order and reconstructed into the higher-order feature matrix.

[0013] As a preferred technical solution, the step of inputting the basic spatiotemporal feature matrix into the first-stream LSTM spatial feature branch to extract the basic semantic feature vector specifically includes: The basic spatiotemporal feature matrix is ​​input into the LSTM network, and the hidden state at each time step is output. After flattening the hidden states at each time step, feature dimensionality reduction is performed using projection blocks consisting of fully connected layers, batch normalization layers, and ReLU activation functions to obtain the basic semantic feature vector.

[0014] As a preferred technical solution, the step of inputting the high-order feature matrix into the second-stream Attention-BiGRU temporal feature branch to extract the high-order dynamic feature vector specifically includes: The higher-order feature matrix is ​​input into the projection layer for feature dimension transformation to generate the input feature sequence; The input feature sequence is processed through a bidirectional gated recurrent unit (BiGRU) network to extract the hidden state at each time step. The hidden state is input into the temporal attention layer, the unnormalized attention score at each time step is calculated, and the attention weight is obtained by normalization using the Softmax function. The attention weights are weighted and summed with the hidden states at each time step to obtain the global context vector; The global context vector is passed through a fully connected layer and a batch normalization layer to output the high-order dynamic feature vector.

[0015] As a preferred technical solution, the fusion of the basic semantic feature vector and the higher-order dynamic feature vector to obtain the dual-stream joint feature vector specifically includes: The basic semantic feature vector and the higher-order dynamic feature vector are concatenated and spliced ​​along the channel dimension to obtain a dual-stream joint feature vector.

[0016] As a preferred technical solution, the step of inputting the dual-stream joint feature vector into the classifier and outputting the gesture recognition result specifically includes: The classifier employs a multilayer perceptron model, and the dual-stream joint feature vector... The gesture is processed through multiple fully connected hidden layers for dimensionality reduction, and then mapped to the category space of the corresponding gesture category by the output layer. The probability of each category is calculated by the Softmax function, and the gesture recognition result is output.

[0017] As a preferred technical solution, the method further includes: A dual-stream gesture recognition model is constructed, which includes a first-stream LSTM spatial feature branch, a second-stream Attention-BiGRU temporal feature branch, a feature fusion module, and a classifier module. The basic spatiotemporal feature matrix and high-order feature matrix obtained after preprocessing the gesture echo point cloud data constitute the dataset, and the dataset is divided into training set and validation set. Samples are taken out in batches from the training set, and the samples are processed using the dual-stream gesture recognition model to obtain the predicted probability distribution of each category. The loss value is calculated using the cross-entropy loss function. Backpropagation is performed based on the loss value to calculate the gradient vector of the model parameters; Before the optimizer performs parameter updates, the gradient vector is clipped according to the following formula to obtain the clipped gradient vector. : in, This represents the gradient vector of the model parameters calculated during backpropagation. Represents the gradient vector Norm, This represents the preset maximum norm threshold for the gradient. This is an operation to retrieve the minimum value; The optimizer is used with the clipped gradient vector The gradient is used as a parameter to update the model parameters; After each round of training, the Macro-F1 score of the current model is evaluated on the validation set, and the optimal model parameters are saved based on the Macro-F1 score. After the iteration is completed, the dual-stream gesture recognition model is configured using the optimal model parameters to obtain the trained dual-stream gesture recognition model.

[0018] Secondly, this application provides a millimeter-wave radar point cloud gesture recognition system based on dual-stream spatiotemporal feature coding, used to implement the method described above, the system comprising: The data acquisition module is configured to acquire gesture echo point cloud data collected by millimeter-wave radar, wherein the gesture echo point cloud data includes a variable-length frame sequence. The point cloud preprocessing module is configured to preprocess the gesture echo point cloud data to generate a basic spatiotemporal feature matrix and a higher-order feature matrix. The first feature extraction module is configured to input the basic spatiotemporal feature matrix into the first-stream LSTM spatial feature branch to extract the basic semantic feature vector. The second feature extraction module is configured to input the high-order feature matrix into the second-stream Attention-BiGRU temporal feature branch to extract the high-order dynamic feature vector. The feature fusion module is configured to fuse the basic semantic feature vector and the higher-order dynamic feature vector to obtain a dual-stream joint feature vector. The gesture recognition module is configured to input the dual-stream joint feature vector into the classifier and output the gesture recognition result.

[0019] The millimeter-wave radar point cloud gesture recognition method and system based on dual-stream spatiotemporal feature coding provided in this application have at least the following beneficial effects: This application innovatively proposes a model architecture that decouples feature splitting from physical meaning by constructing a dual-stream spatiotemporal fusion network consisting of an LSTM spatial feature branch for underlying physical micro-motion tracking and an Attention-BiGRU temporal feature branch for macroscopic dynamic weight allocation. Specifically, this application performs targeted processing on the raw point cloud data during the preprocessing stage to generate a basic spatiotemporal feature matrix and a high-order feature matrix, effectively decoupling the underlying physical features from the high-order statistical features. The first-stream LSTM branch models the spatial coordinates and Doppler velocity in the basic spatiotemporal feature matrix to accurately capture subtle changes in key action frames such as palm translation and finger micro-movements. The second-stream Attention-BiGRU branch extracts the high-order temporal evolution laws of the macroscopic kinematic features in the high-order feature matrix and uses an attention mechanism to adaptively enhance the weight allocation of key action frames. The dual-stream features are fused through feature concatenation to avoid information folding. A gradient pruning mechanism is introduced during the training stage to alleviate gradient instability in long sequence training and improve the model's adaptive alignment ability and generalization performance on unbalanced variable-length sequence datasets. Furthermore, this application deploys the above method in an embedded system to verify its real-time processing effectiveness from data acquisition to recognition result output on edge devices, meeting the actual needs of lightweight deployment in IoT terminals and embedded scenarios. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] Figure 1 A flowchart of a millimeter-wave radar point cloud gesture recognition method based on dual-stream spatiotemporal feature coding provided for embodiments of this application; Figure 2 A flowchart of point cloud preprocessing provided in the embodiments of this application; Figure 3 This is a schematic diagram of the DSTG-Net model provided in an embodiment of this application; Figure 4 This is a schematic diagram of the LSTM spatial feature branch structure provided in the embodiments of this application; Figure 5 This is a schematic diagram of the Attention-BiGRU temporal branch structure provided in an embodiment of this application; Figure 6 The graphs showing the relationship between the training accuracy and the number of training epochs of different methods provided in the embodiments of this application on a public dataset; Figure 7 A comparison chart of the best accuracy of different methods provided in the embodiments of this application on a public dataset; Figure 8The confusion matrix comparison results of the temporal method provided in the embodiments of this application on a public dataset are shown in the figure; where (a) is STGNet; and (b) is the proposed dual-stream fusion gesture recognition method (DSTG-Net). Figure 9 The graphs showing the relationship between the training accuracy and the number of training rounds for different methods provided in the embodiments of this application on a self-built dataset; Figure 10 A comparison chart of the best accuracy of different methods provided in the embodiments of this application on a self-built dataset; Figure 11 The confusion matrix comparison results of the temporal method provided in the embodiments of this application on a self-built dataset are shown in the figure; where (a) is STGNet; and (b) is the proposed dual-stream fusion gesture recognition method (DSTG-Net). Figure 12 A graph showing the relationship between training accuracy and training rounds in an ablation experiment provided in this application embodiment; Figure 13 A comparison chart of the best accuracy in ablation experiments provided in the embodiments of this application; Figure 14 A graph showing the relationship between the validation accuracy and the number of training rounds in an ablation experiment provided in this embodiment of the application; Figure 15 A comparison chart of the best accuracy on a self-built dataset in ablation experiments provided in this application embodiment; Figure 16 This is a diagram illustrating the architecture of a gesture recognition system provided in an embodiment of this application. Figure 17 This is a structural diagram of a millimeter-wave radar point cloud gesture recognition system based on dual-stream spatiotemporal feature coding, provided as an embodiment of this application.

[0022] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0024] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0025] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0026] Currently, a pressing technical problem to be solved is how to obtain feature sequences for different branch input requirements in lightweight gesture recognition methods and systems, especially when the dynamic gesture sparse point cloud data captured by multiple input multiple output (MIMO) millimeter-wave radar has varying frame lengths and high noise levels. This requires designing a dual-stream spatiotemporal fusion network to encode the gesture point cloud data captured by millimeter-wave radar and achieving better gesture recognition accuracy through optimized training strategies. To address this technical problem, this application provides a millimeter-wave radar point cloud gesture recognition method based on dual-stream spatiotemporal feature encoding. This method combines the underlying physical micro-motion tracking branch of the Long Short-Term Memory (LSTM) network with macroscopic dynamic weight allocation using an attention-based bidirectional gated recurrent unit (Attention-BiGRU) network. It proposes the DSTG-Net model, which decouples feature splitting from physical meaning, solving the problem of low detection rate of key action frames in millimeter-wave radar point clouds and improving the efficiency of spatiotemporal fusion. The improved DSTG-Net model described above can accurately capture key motion changes such as palm translation and finger micro-movements in radar sequences, and also exhibits good adaptive alignment capabilities on unbalanced variable-length sequence datasets. Furthermore, this embodiment deploys the DSTG-Net dual-stream fusion gesture recognition method onto a Raspberry Pi embedded system, verifying its effectiveness in gesture recognition.

[0027] like Figure 1As shown, this millimeter-wave radar point cloud gesture recognition method based on dual-stream spatiotemporal feature coding first acquires gesture data input, which is millimeter-wave radar point cloud data. Specifically, it is three-dimensional gesture point cloud data carrying frame number, velocity, peak value, and spatial position information. The acquired data undergoes point cloud preprocessing, and the preprocessed data is input into the dual-stream gesture recognition model DSTG-Net to obtain gesture recognition results. The gesture recognition results specifically include arm to the left (arm waving to the left), arm to the right (arm waving to the right), hand away (hand away from radar), hand close (hand close to radar), horizontal fist, vertical fist, hand to the right (hand translating to the right), hand to the left (hand translating to the left), palm down rotation (palm turning down), palm up rotation (palm turning up), hand up (hand translating up), and hand down (hand translating down). Figure 1 The example identified is a vertically clenched fist. The gesture recognition result also includes a curve showing the change in classification probability over time. The vertical axis of the curve represents the classification probability of the current predicted gesture category output by the model, that is, the probability value of the model determining the target category corresponding to the current input gesture. The horizontal axis represents the time dimension of the recognition process, which is used to intuitively present the process of the probability of the target gesture category gradually increasing, maintaining a peak, and then gradually decreasing throughout the entire process of the complete gesture action from start to finish.

[0028] based on Figure 1 The process shown includes the following steps S10-S60 in the specific implementation of this millimeter-wave radar point cloud gesture recognition method based on dual-stream spatiotemporal feature coding.

[0029] S10: Acquire gesture echo point cloud data collected by millimeter-wave radar. The gesture echo point cloud data includes a variable-length frame sequence.

[0030] Specifically, this embodiment uses millimeter-wave radar to collect user hand gestures in real time. The radar emits a linear frequency modulated continuous wave signal, which is reflected by the hand and received by a receiving antenna array. A two-dimensional fast Fourier transform is performed on the intermediate frequency signal to extract the range and Doppler dimensions. Then, a constant false alarm rate (CFAR) detection method is used to filter out valid reflection points from the background noise, generating raw point cloud data containing the spatial coordinates, Doppler components, and reflection intensity of each reflection point. Each frame of point cloud data corresponds to a snapshot of the hand's spatial distribution at the time of acquisition. The number of point cloud frames contained in a complete hand gesture varies with the user's operation speed and the duration of the gesture, thus forming a variable-length frame sequence. This variable-length frame sequence completely records the spatial evolution and velocity change trajectory of each reflection point of the hand during the hand gesture.

[0031] S20: Preprocess the gesture echo point cloud data to generate the basic spatiotemporal feature matrix and the higher-order feature matrix.

[0032] This embodiment preprocesses the millimeter-wave radar point cloud data (i.e., gesture echo point cloud data) obtained in step S10 to obtain the data input for DSTG-Net.

[0033] In some embodiments, such as Figure 2 The diagram shown is a flowchart of point cloud preprocessing provided in an embodiment of this application. Point cloud preprocessing includes two processing branches, namely branch 1 and branch 2.

[0034] 1) Processing flow of branch 1 First, the original point cloud data of the input variable-length frame sequence is normalized and reconstructed to obtain the fixed-length input tensor required for the LSTM branch. The maximum number of observation frames for the gesture recognition system is set to... The maximum number of pixels per frame is For the first Extract the spatial location x-coordinate from the gesture echo point cloud data of the frame. y-axis Doppler components and planar radial distance Planar radial distance The calculation formula is: In the formula, This indicates the radial distance of the reflection point on the horizontal plane; and These correspond to the x and y coordinates of the reflection point in the radar Cartesian coordinate system, respectively.

[0035] Then, when the number of points in a single frame is insufficient Use zero to fill values ​​when the value exceeds a certain threshold. Then truncation is performed. Each frame of data is flattened and stitched together to form the first... The basic feature vector of a frame Its dimensions The calculation is as follows: In the formula, This represents the total dimension of the basic feature vector of a single frame. Represents the number of extracted physical feature channels, including , , , There are 4 features in total. This represents the maximum number of effective reflection points in a single frame after alignment.

[0036] The final variable-length frame sequence is reconstructed into a dimension of Basic spatiotemporal feature matrix , which serves as the input to the spatial feature branch of the first-order LSTM.

[0037] 2) Processing flow of branch 2 First, based on the first point in the original point cloud dataset... The five echo features within the frame, namely , , , ,in To calculate the reflection intensity, seven statistics corresponding to the reflection point attributes are calculated: mean, standard deviation, median, maximum, minimum, and... quantiles and quantiles, and thus obtain 3D statistical features, combined with the total number of effective dynamic points in a single frame, constitute a total of A dimensional intra-frame spatial statistical vector.

[0038] To enhance the network's perception of action trends, first-order temporal difference features are introduced. : In the formula, Indicates the first The first-order temporal difference feature vector of a frame; This is the 36-dimensional spatial statistical vector of the current frame; This is the spatial statistics vector of the previous frame.

[0039] Then, the spatial statistical vector and the temporal difference vector are concatenated on the feature channels to form the first... The higher-order features of a frame are represented as follows: Where, in the formula, Indicates the first The 72-dimensional high-order feature vector after frame fusion; This indicates that the statistical features of the current frame will be used. Compared with first-order temporal difference features Perform arithmetic splicing operations.

[0040] Finally, the reconstructed sequence is transformed into a fixed-length sequence with dimension 1. higher-order feature matrix This helps reduce the learning difficulty for the subsequent Attention-BiGRU network when processing gestures with similar trajectories.

[0041] S30: Input the basic spatiotemporal feature matrix into the first-flow LSTM spatial feature branch to extract the basic semantic feature vector.

[0042] Based on the fundamental spatiotemporal matrix obtained in step S20 and higher-order characteristic matrix The schematic diagram of the DSTG-Net structure proposed in this embodiment is as follows: Figure 3 As shown, it comprises three modules: a first-stream LSTM spatial feature branch, a second-stream Attention-BiGRU temporal feature branch, and a dual-stream feature fusion and classifier. The first-stream LSTM spatial feature branch and the second-stream Attention-BiGRU temporal feature branch together serve as the feature extraction module within the three modules.

[0043] In some embodiments, the first-stream LSTM spatial feature branch is used to extract the low-level temporal features of gesture spatial evolution. This first-stream LSTM spatial feature branch employs a basic spatiotemporal matrix. As input, its network structure is as follows Figure 4 As shown. The core task of the first-order LSTM spatial feature branch is to analyze the fundamental spatiotemporal matrix. The spatial coordinates and Doppler velocities contained therein are modeled using a low-level temporal dependency model, which responds to the underlying spatiotemporal matrix of the input. The data sequence is first processed through two LSTM layers, outputting a hidden state tensor of (80, 32) for all time steps. Then, the hidden state of this sequence is flattened to obtain a tensor of dimension . The extracted one-dimensional features are then processed. Finally, these features are input into a non-linear projection block, which consists of a fully connected layer, a batch normalized (BN) layer, and a ReLU activation function. This reduces the features to 128 dimensions, yielding the basic semantic feature vector. .

[0044] S40: Input the high-order feature matrix into the second-stream Attention-BiGRU temporal feature branch to extract the high-order dynamic feature vector.

[0045] In some embodiments, the second-stream Attention-BiGRU temporal feature branch is used to capture the macroscopic kinematic features and high-order temporal evolution of gestures. By statistically abstracting the point cloud from a single frame, the impact of feature discretization caused by the sparsity of the original data can be effectively reduced. This second-stream Attention-BiGRU temporal feature branch employs a high-order feature matrix. As input, its network structure is as follows Figure 5 As shown. First, the input high-order feature matrix... First, the model enters the projection layer, then the first fully connected layer, increasing the feature dimension from 72 to 128. After dimensionality increase, batch normalization and ReLU activation functions are applied sequentially to enhance non-linear expressiveness. A Dropout layer with a dropout rate of 0.4 is then added, randomly deactivating some neurons to improve the model's generalization performance. Next, the second fully connected layer performs dimensionality reduction, lowering the dimension to 64, ultimately generating a more compact input feature sequence. .

[0046] Then, By using a two-layer bidirectional gated recurrent unit (BiGRU) network, including both forward and reverse temporal directions, the extracted sequence feature dimension can be increased to 128, resulting in a dimension... The timing characteristics of BiGRU output, i.e., BiGRU in Hidden state output at all times .

[0047] Next, BiGRU in Hidden state output at all times Enter the temporal attention layer. This temporal attention layer first considers the currently acquired hidden state. Using fully connected layers with Tanh activation functions, importance scores for each time step are calculated. The scalar mapping is then completed through a linear output layer, and normalization is performed using the Softmax function to finally obtain the attention weights. The above calculation process is expressed as follows: in, For BiGRU in The hidden state that is constantly being output. and This refers to the learnable weight matrix in the fully connected layer of the attention mechanism. and For the corresponding bias vector, Indicates the current time step Unnormalized attention score For the summation index variable of time steps, It is an exponential function. This represents the first number obtained after normalization using the Softmax function. Attention weights at each time step This represents the total number of time steps of the input sequence, in this embodiment. .

[0048] Finally, attention weights The weighted sum of the hidden states at all time steps is used to generate an adaptive global context vector. The global context vector Further, through fully connected layers and batch normalization layers, the output is the final 128-dimensional high-order dynamic feature vector. .

[0049] S50: The basic semantic feature vector and the high-order dynamic feature vector are fused to obtain the dual-stream joint feature vector.

[0050] In some embodiments, the dual-stream feature integration module will fuse features using a feature concatenation approach. and Two 128-dimensional feature vectors are concatenated along the channel dimension: in, This represents the joint feature vector of the two streams after fusion. This represents the basic semantic feature vector extracted by the LSTM spatial branch. This represents the high-order dynamic feature vector extracted by the Attention-BiGRU branch. This indicates that concatenation is performed along the feature channel dimension. The fused features are represented as a 256-dimensional real vector. This direct concatenation preserves the orthogonal features extracted by each of the two-stream networks, avoiding problems such as high-dimensional feature cancellation or information folding that may occur with element-wise addition.

[0051] S60: Input the joint feature vector of the two streams into the classifier and output the gesture recognition result.

[0052] In some embodiments, the dual-stream joint feature vector The classifier, which employs a deep multilayer perceptron model, is designed with a funnel-shaped dimensionality reduction structure as follows: (1) First hidden layer: Since the input is a 256-dimensional feature, it is first mapped to a 128-dimensional feature using a fully connected method, and then the BN and ReLU functions are used to perform the first reasonable cross-feature cross learning. (2) Second hidden layer: The result after Dropout is set to 0.4 is mapped to 64 dimensions.

[0053] (3) Output layer: Since Dropout processing has been performed beforehand, a linear layer is used to project the extracted features into a category space with 12 neurons, namely the 12 gesture categories under study.

[0054] (4) Calculate the unnormalized log probability of each category using the Softmax layer, and directly use the cross-entropy loss function for global iterative optimization.

[0055] In some embodiments, a specific training process for the dual-stream gesture recognition model DSTG-Net is also provided. The dual-stream gesture recognition model includes a first-stream LSTM spatial feature branch, a second-stream Attention-BiGRU temporal feature branch, a feature fusion module, and a classifier module. The first-stream LSTM spatial feature branch, the second-stream Attention-BiGRU temporal feature branch, the feature fusion module, and the classifier module are used to perform the above steps S30-S60 respectively. The specific data processing flow is as described above and will not be repeated here.

[0056] When training the dual-stream gesture recognition model DSTG-Net, the dataset is constructed by preprocessing the gesture echo point cloud data to obtain the basic spatiotemporal feature matrix and high-order feature matrix. The dataset is then divided into a training set and a validation set. Samples are extracted from the training set in batches, and the dual-stream gesture recognition model is used to process the samples to obtain the predicted probability distribution for each category. The cross-entropy loss function is used to calculate the loss value. Backpropagation is performed based on the loss value to calculate the gradient vector of the model parameters. Before the optimizer updates the parameters, the gradient vector is pruned according to the following formula to obtain the pruned gradient vector. An optimizer is used to select the trimmed gradient vector. The gradient is used as a parameter to update the model parameters. After each training round, the Macro-F1 score of the current model is evaluated on the validation set, and the optimal model parameters are saved based on the Macro-F1 score. After the iteration ends, the dual-stream gesture recognition model is configured using the optimal model parameters to obtain the trained dual-stream gesture recognition model.

[0057] In practical implementation, a stratified sampling method was used to divide the collected radar gesture dataset. The training set comprised 80% of the dataset, and the validation set comprised 20%. The sample proportions for each of the 12 gesture categories strictly corresponded to the overall distribution, thus resolving the issue of feature loss in smaller class samples during random partitioning. Correspondingly, an iterator was constructed using PyTorch's DataLoader during data loading, with a selected batch size of [missing information]. =32. Using the standard cross-entropy loss function. To measure the probability distribution made by the network With real and unique hot tags The difference was analyzed using the AdamW optimizer, which has weight decay decoupling properties. The initial learning rate used in the experiment was... =0.001, and the weight decay coefficient is 0.0001. A cosine annealing learning rate scheduler is introduced in the experiment, allowing the learning rate to smoothly decrease from the initial value to near zero within one and a half cosine cycles. The cycle parameter is set to 50 to improve the final convergence accuracy of the model on the validation set. Furthermore, during backpropagation, long sequences are prone to gradient explosion, leading to abnormal model parameter updates and disrupting the learned feature distribution. Therefore, this embodiment incorporates the following gradient pruning mechanism before the optimizer executes step() to update: (8) in, This represents the gradient vector of the model parameters calculated during backpropagation. Represents the gradient vector Norm, This represents the preset maximum norm threshold for the gradient, which is set to 1.0 here; This is an operation to find the minimum value. During training, the maximum gradient of all model parameters is reached. The norm design constraint threshold is 1.0.

[0058] In addition, to reasonably reflect the model's ability to distinguish various gestures in a balanced way, the model calculates the Macro F1 score as a reference standard, tracks the performance of the validation set in real time, and uses this to determine the optimal model. The training process of the above dual-stream fusion gesture model is shown in Table 1.

[0059] Table 1 Training Flow of the Dual-Stream Fusion Gesture Model (DSTG-Net)

[0060] The feasibility and advancement of the method proposed in this application will be further illustrated below with specific experiments. This embodiment of the experiment selected two millimeter-wave radar gesture datasets: one is a publicly available standard dataset [P. Grobelny and A. Narbudowicz, “Mm-wave radar-based recognition of multiple hand gestures using long short-term memory (LSTM) neural network,” Electronics, vol. 11, no. 5, pp. 787–798, 2022.], and the other is a self-built dataset. Both datasets used the TI IWR1642 millimeter-wave radar with a center frequency of 76-81 GHz, a frame period of 33 ms, a range resolution of 0.039, a maximum unambiguous range of 2.41 m, and a radial velocity resolution of 0.13 m / s. The publicly available dataset contained 4600 valid gesture samples collected by four subjects of different heights and ages. The self-built dataset contained 2100 valid gesture samples collected by three subjects. The 12 gesture categories are specifically defined as follows: arm waving to the left (AL), arm waving to the right (AR), horizontal fist clench (CFH), vertical fist clench (CFP), hand away from radar (HA), hand near radar (HC), hand translating downwards (HD), palm flipping upwards (HRPD), palm flipping downwards (HRPU), hand translating to the left (HL), hand translating to the right (HR), and hand translating upwards (HU). This dataset encompasses the spatial coordinate features (x, y) of point clouds, fully records frame sequence and target number, and strictly preserves the underlying physical parameters of radar measurements such as radial distance, Doppler velocity, and peak reflectance intensity.

[0061] In the first experiment, Figures 6-7This paper presents a comparison of the training accuracy and category accuracy of the proposed DSTG-Net dual-stream fusion gesture recognition system with four mainstream deep learning networks on public datasets. The training epochs were 200. The results show that traditional VGGNet, ResNet, and CNN single-stream networks have significant limitations in actual performance on public datasets; after 200 training epochs, their performance bottleneck essentially stagnates at around 79%. The STGNet (Spatial-Temporal Gesture Network) architecture, which possesses temporal modeling capabilities [Wu H, Wang X, DongY, et al. STGNet: Adaptive Millimeter-Wave Gesture Recognition via Spatial-Temporal Gesture Network[C] / / 2025 International Joint Conference on NeuralNetworks (IJCNN). 2025: 1-8], maintains a stable accuracy between 93% and 94% across all testing phases. The proposed DSTGNet achieved an accuracy of 97.40%, which is because the dual-branch fusion architecture still has excellent feature extraction capabilities and generalization stability when processing data with different distributions. Figure 8 The results show the confusion matrix of gesture recognition using two methods, STGNet and DSTG-Net, based on temporal modeling. This verifies that the proposed DSTG-Net dual-branch decoupling structure combined with the attention mechanism can better uncover action details, suppress redundant noise, and improve detection performance.

[0062] Table 2 Performance comparison of various models at different iteration cycles on the public dataset.

[0063] In the second experiment, as shown in Table 2 and Figures 9-11 As shown, Figures 9-11 The performance of the proposed DSTG-Net dual-stream fusion gesture recognition system on a self-built dataset is demonstrated. Figure 9 The curves showing the relationship between training accuracy and number of epochs for the proposed DSTG-Net dual-stream fusion gesture recognition system and four mainstream deep learning networks on a self-built dataset are shown. Figure 10 It is a comparison of the accuracy rates of each category using bar charts. Figure 11The results are from the confusion matrix. It can be seen that, compared to conventional deep learning models such as CNN, ResNet, VGGNet, and STGNet, the method in this application achieves an accuracy of 98.25% after 80 training cycles, surpassing STGNet. Furthermore, compared to STGNet, the proposed DSTG-Net method can dynamically adjust attention weights, focusing on the true physical distribution of key action frames. This architecture has advantages in feature extraction, and its efficient fusion mechanism lays a solid lightweight foundation for subsequent porting to embedded hardware.

[0064] Table 3 Performance comparison of each model on the self-built dataset at different iteration cycles

[0065] In the third experiment, combining Table 3 and Figure 12 and Figure 13 As shown, Figure 12 and Figure 13 Together, they presented the ablation experiment results of the proposed DSTG-Net dual-stream fusion gesture recognition system on a public dataset. Figure 12 The validation accuracy of Attention-BiGRU, LSTM, and the proposed method on a public dataset with 200 iterations is shown. Figure 13 This comparison shows the best accuracy of Attention-BiGRU, LSTM, and the proposed method for classifying 12 gestures on a public dataset with 200 iterations. It can be seen that the LSTM and Attention-BiGRU models, which rely solely on a single time-series feature extraction branch, have an upper limit to their feature representation capabilities. Our proposed model effectively improves the accuracy of gesture recognition by fusing the spatiotemporal features of a two-stream network.

[0066] In the fourth experiment, Figures 14-15 The performance of the proposed DSTG-Net dual-stream fusion gesture recognition system on a self-built dataset is demonstrated. Figure 14 The validation accuracy of Attention-BiGRU, LSTM, and the proposed method on a public dataset with 200 iterations is shown. Figure 15 This is a comparison of the best accuracy of Attention-BiGRU, LSTM and the proposed method for classifying 12 gestures on a public dataset with 200 iterations.

[0067] In the fifth experiment, the DSTG-Net gesture recognition method proposed in this application was ported to Raspberry Pi for real-time recognition testing. Figure 15 This is a diagram of the gesture recognition system architecture. Figure 16This is a real-time recognition result image. The experiment deployed the proposed DSTG-Net dual-stream fusion network gesture recognition algorithm from an offline PC to a Raspberry Pi embedded system. It achieves a real-time response of "data perception - feature extraction - model inference - hardware linkage." At the physical level, the system consists of three parts: a perception layer, an edge computing layer, and an interactive feedback layer. First, the system uses a TI IWR1642 to capture and unpack the TLV format radar radio frequency data stream in real time, writing the parsed spatial coordinates, Doppler velocity, and peak intensity into a globally shared variable in memory, achieving physical-level decoupling between data acquisition and system control. Then, the raw point cloud stored in the buffer queue is first sent to the preprocess_of_fusion() function for processing, obtaining a basic spatiotemporal tensor of size (80, 320) and a higher-order statistical tensor of size (80, 72). These are then passed to the dual-stream fusion model running on the CPU for forward computation, directly outputting the unnormalized classification probability. The interactive feedback module simultaneously extracts the "real labels" at the file directory level and compares these labels with the "prediction results" output by the system, intuitively presenting the model's actual performance on specific edge samples. Experimental results verify that the constructed gesture recognition system achieves a low-latency link from radar-level data acquisition to GPIO hardware-linked feedback, validating the feasibility and practical engineering value of the proposed gesture recognition system design.

[0068] In summary, this application's method utilizes dual-branch feature fusion to achieve gesture recognition. First, addressing the issues of inconsistent frame lengths and high noise levels in the raw point cloud data output by millimeter-wave radar, a data preprocessing and feature reconstruction module is designed. This generates a basic spatiotemporal matrix focusing on microscopic evolution and a higher-order feature matrix focusing on macroscopic motion trends, effectively reducing the impact of feature discretization, high noise levels, and inconsistent frame lengths caused by the sparsity of the original data. Then, two branches, LSTM and Attention-BiGRU, are constructed to encode the aforementioned feature matrices, and a feature fusion and classifier module is designed to achieve gesture recognition. Finally, the method is experimentally tested on public and self-built datasets, achieving an average accuracy of 98.75%. Furthermore, real-time testing on a Raspberry Pi 4B (RAM 1G) verifies the effectiveness of the proposed method.

[0069] The method described in this application can be applied to the field of human-computer interaction using millimeter-wave radar in computer vision. By installing millimeter-wave radar acquisition equipment, it can be used for common gesture recognition, such as in smart homes, vehicle cockpits, and sterile medical settings. It is suitable for intelligent edge computing applications that comprehensively capture various non-contact interaction processes by learning dual-stream spatiotemporal features to determine gesture feature maps.

[0070] This application also provides a millimeter-wave radar point cloud gesture recognition system based on dual-stream spatiotemporal feature coding, such as... Figure 17 As shown, the millimeter-wave radar point cloud gesture recognition system based on dual-stream spatiotemporal feature coding includes: The data acquisition module 1701 is configured to acquire gesture echo point cloud data collected by millimeter-wave radar, wherein the gesture echo point cloud data includes a variable-length frame sequence. The point cloud preprocessing module 1702 is configured to preprocess the gesture echo point cloud data to generate a basic spatiotemporal feature matrix and a higher-order feature matrix. The first feature extraction module 1703 is configured to input the basic spatiotemporal feature matrix into the first-stream LSTM spatial feature branch to extract the basic semantic feature vector. The second feature extraction module 1704 is configured to input the high-order feature matrix into the second-stream Attention-BiGRU temporal feature branch to extract the high-order dynamic feature vector. The feature fusion module 1705 is configured to fuse the basic semantic feature vector and the higher-order dynamic feature vector to obtain a dual-stream joint feature vector. The gesture recognition module 1706 is configured to input the dual-stream joint feature vector into a classifier and output the gesture recognition result.

[0071] It should be noted that the above-mentioned millimeter-wave radar point cloud gesture recognition system based on dual-stream spatiotemporal feature coding and the prior method belong to the same technical concept and can achieve the same technical effect, so it will not be elaborated here.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A millimeter wave radar point cloud gesture recognition method based on double-flow space-time feature coding, characterized in that, The method includes: Acquire gesture echo point cloud data collected by millimeter-wave radar, wherein the gesture echo point cloud data includes a variable-length frame sequence; The gesture echo point cloud data is preprocessed to generate a basic spatiotemporal feature matrix and a higher-order feature matrix; The basic spatiotemporal feature matrix is ​​input into the spatial feature branch of the first-flow LSTM to extract the basic semantic feature vector; The higher-order feature matrix is ​​input into the second-stream Attention-BiGRU temporal feature branch to extract the higher-order dynamic feature vector. By fusing the basic semantic feature vector and the higher-order dynamic feature vector, a dual-stream joint feature vector is obtained; The dual-stream joint feature vector is input into the classifier, and the gesture recognition result is output.

2. The method of claim 1, wherein, The preprocessing of the gesture echo point cloud data to generate a basic spatiotemporal feature matrix and a higher-order feature matrix specifically includes: Based on each frame in the variable-length frame sequence, physical features including spatial location coordinates, Doppler components, and planar radial distance are extracted, and the number of points in each frame is normalized to a fixed length to reconstruct the basic spatiotemporal feature matrix. For each frame in the variable-length frame sequence, multiple echo features are extracted and intra-frame spatial statistical vectors are calculated. Combined with the first-order temporal difference features between frames, the higher-order feature matrix is ​​reconstructed.

3. The method according to claim 2, characterized in that, The step involves extracting physical features, including spatial location coordinates, Doppler components, and planar radial distance, from each frame in the variable-length frame sequence, and performing fixed-length normalization on the number of points in each frame to reconstruct the basic spatiotemporal feature matrix. Specifically, this includes: For the gesture echo point cloud data of frame t, extract its spatial position x-coordinate. y-axis Doppler components and planar radial distance , wherein the radial distance Calculate according to the following formula: In the formula, This indicates the radial distance of the reflection point on the horizontal plane; The horizontal coordinates of the spatial positions of all reflection points in the current frame y-axis Doppler components and planar radial distance The vectors are unfolded and concatenated in sequence to form a one-dimensional vector, which serves as the basic feature vector for the t-th frame. The dimension of the vector is determined based on the number of extracted physical feature channels and the maximum number of effective reflection points after alignment of a single frame. The basic feature vectors of each frame are arranged in chronological order and reconstructed into the basic spatiotemporal feature matrix.

4. The method according to claim 2, characterized in that, The step of extracting multiple echo features and calculating intra-frame spatial statistical vectors for each frame in the variable-length frame sequence, and then reconstructing the high-order feature matrix by combining the first-order temporal difference features between frames, specifically includes: Based on multiple echo features in the gesture echo point cloud data of frame t, calculate the statistics corresponding to each echo attribute, and combine them with the total effective dynamic points of a single frame to form an intra-frame spatial statistical vector. Calculate the first-order temporal difference feature based on the spatial statistical vector of the current frame and the spatial statistical vector of the previous frame; The spatial statistical vector and the temporal difference feature are concatenated on the feature channel to form the high-order feature vector of the t-th frame; The higher-order feature vectors of each frame are arranged in chronological order and reconstructed into the higher-order feature matrix.

5. The method according to claim 1, characterized in that, The step of inputting the basic spatiotemporal feature matrix into the first-stream LSTM spatial feature branch to extract the basic semantic feature vector specifically includes: The basic spatiotemporal feature matrix is ​​input into the LSTM network, and the hidden state at each time step is output. After flattening the hidden states at each time step, feature dimensionality reduction is performed using projection blocks consisting of fully connected layers, batch normalization layers, and ReLU activation functions to obtain the basic semantic feature vector.

6. The method according to claim 1, characterized in that, The step of inputting the high-order feature matrix into the second-stream Attention-BiGRU temporal feature branch to extract the high-order dynamic feature vector specifically includes: The higher-order feature matrix is ​​input into the projection layer for feature dimension transformation to generate the input feature sequence; The input feature sequence is processed through a bidirectional gated recurrent unit (BiGRU) network to extract the hidden state at each time step. The hidden state is input into the temporal attention layer, the unnormalized attention score at each time step is calculated, and the attention weight is obtained by normalization using the Softmax function. The attention weights are weighted and summed with the hidden states at each time step to obtain the global context vector; The global context vector is passed through a fully connected layer and a batch normalization layer to output the high-order dynamic feature vector.

7. The method according to claim 1, characterized in that, The process of fusing the basic semantic feature vector and the higher-order dynamic feature vector to obtain the dual-stream joint feature vector specifically includes: The basic semantic feature vector and the higher-order dynamic feature vector are concatenated and spliced ​​along the channel dimension to obtain a dual-stream joint feature vector.

8. The method according to claim 1, characterized in that, The step of inputting the dual-stream joint feature vector into the classifier and outputting the gesture recognition result specifically includes: The classifier employs a multilayer perceptron model, and the dual-stream joint feature vector... The gesture is processed through multiple fully connected hidden layers for dimensionality reduction, and then mapped to the category space of the corresponding gesture category by the output layer. The probability of each category is calculated by the Softmax function, and the gesture recognition result is output.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: A dual-stream gesture recognition model is constructed, which includes a first-stream LSTM spatial feature branch, a second-stream Attention-BiGRU temporal feature branch, a feature fusion module, and a classifier module. The basic spatiotemporal feature matrix and high-order feature matrix obtained after preprocessing the gesture echo point cloud data constitute the dataset, and the dataset is divided into training set and validation set. Samples are taken out in batches from the training set, and the samples are processed using the dual-stream gesture recognition model to obtain the predicted probability distribution of each category. The loss value is calculated using the cross-entropy loss function. Backpropagation is performed based on the loss value to calculate the gradient vector of the model parameters; Before the optimizer performs parameter updates, the gradient vector is clipped according to the following formula to obtain the clipped gradient vector. : in, This represents the gradient vector of the model parameters calculated during backpropagation. Represents the gradient vector Norm, This represents the preset maximum norm threshold for the gradient. This is an operation to retrieve the minimum value; The optimizer is used with the clipped gradient vector The gradient is used as a parameter to update the model parameters; After each round of training, the Macro-F1 score of the current model is evaluated on the validation set, and the optimal model parameters are saved based on the Macro-F1 score. After the iteration is completed, the dual-stream gesture recognition model is configured using the optimal model parameters to obtain the trained dual-stream gesture recognition model.

10. A millimeter-wave radar point cloud gesture recognition system based on dual-stream spatiotemporal feature coding, used to implement the method as described in any one of claims 1 to 9, characterized in that, The system includes: The data acquisition module is configured to acquire gesture echo point cloud data collected by millimeter-wave radar, wherein the gesture echo point cloud data includes a variable-length frame sequence. The point cloud preprocessing module is configured to preprocess the gesture echo point cloud data to generate a basic spatiotemporal feature matrix and a higher-order feature matrix. The first feature extraction module is configured to input the basic spatiotemporal feature matrix into the first-stream LSTM spatial feature branch to extract the basic semantic feature vector. The second feature extraction module is configured to input the high-order feature matrix into the second-stream Attention-BiGRU temporal feature branch to extract the high-order dynamic feature vector. The feature fusion module is configured to fuse the basic semantic feature vector and the higher-order dynamic feature vector to obtain a dual-stream joint feature vector. The gesture recognition module is configured to input the dual-stream joint feature vector into the classifier and output the gesture recognition result.

Images