A dynamic gesture recognition method under complex scene interference by a multi-feature fusion network

Abstract

The application relates to a dynamic gesture recognition method of a multi-feature fusion network under complex scene interference, which comprises the following steps: acquiring a baseband intermediate frequency signal, processing the baseband intermediate frequency signal, acquiring a range-time graph and a Doppler-time graph, inputting the range-time graph and the Doppler-time graph into a gesture recognition model, acquiring a gesture recognition result, training a light neural network model by using a training set in the gesture recognition model, extracting spatial displacement features and gesture speed change information of the range-time graph and the Doppler-time graph based on a DSCC module in the gesture recognition model, acquiring a feature graph, inputting the feature graph combined with position coding into a Transform module for global feature extraction, and acquiring the gesture recognition result. Through extraction and fusion of multiple features, distance, speed and time information of a target gesture are more comprehensively expressed, and the richness and expression capability of gesture features in a complex scene are improved.

Description

Technical Field

[0001] This invention relates to the fields of wireless communication and artificial intelligence, and in particular to a dynamic gesture recognition method using a multi-feature fusion network under complex scene interference. Background Technology

[0002] In recent years, driven by both wireless communication and artificial intelligence technologies, Human-Computer Interaction (HCI) technology has become deeply integrated into daily life, becoming one of the key technologies supporting modern social development. As a core research direction in the field of HCI, Hand Gesture Recognition (HGR) technology has gained widespread attention due to its intuitive and natural interactive characteristics. Currently, this technology has achieved breakthroughs in multiple application areas such as intelligent driving, smart homes, assistive devices for the disabled, and sign language translation, demonstrating enormous application value and development potential.

[0003] Currently, gesture recognition systems are mainly divided into four categories: systems based on wearable sensors, computer vision, WiFi, and FMCW radar. Systems based on wearable sensors rely on data gloves, capturing hand movement information through sensors such as accelerometers and gyroscopes. For example, in 2016, Zheng Y and his research team innovatively combined bending sensors with resistance sensors to successfully develop a functional sensor glove, providing a method to visualize and quantify coordination anomalies between joints. Kanokoda et al. used artificial neural networks for real-time gesture prediction. In 2019, Glauser et al. proposed an innovative design based on a stretch array sensor to address the spatial resolution limitations of traditional sensors. By using a dense array of stretch sensors, they achieved comprehensive capture of hand surface deformation. However, these systems suffer from problems such as easy device damage, limited functionality, high cost, and inconvenience in wearing. Systems based on computer vision collect images through imaging devices such as cameras and extract hand features from them. Wu J et al. proposed an innovative dual-branch convolutional neural network architecture, which improves the accuracy and robustness of gesture recognition by fusing features from depth data and optical flow data. In 2018, MR Islam et al. used deep convolutional neural networks (DCNN) and multi-class support vector machines (SVM) to recognize gestures of 26 alphabetic symbols. In 2019, Nguyen's team designed a neural network model based on positive definite matrix manifold learning and applied it to gesture recognition using human skeletal data collected by depth-sensing cameras. However, such systems are susceptible to lighting and blind spots and are not conducive to user privacy protection. Wi-Fi-based gesture recognition systems use Wi-Fi signals to recognize human gestures in other applications. In 2013, Pu et al. designed the WiSee system, which senses gestures in the home environment through Doppler frequency shift. In 2021, Wang et al. proposed the WiDG system based on CSI and deep learning, capable of recognizing digits 0-9 with recognition accuracies of 97.2% and 95.3% in wall-penetrating and wall-free scenarios, respectively. However, the limited detection range of Wi-Fi systems restricts their widespread application. Systems based on FMCW radar utilize radar technology to collect gesture signals and classify and recognize them using machine learning or deep learning methods. In 2019, Choi et al. successfully recognized 10 gestures with an accuracy of 99.10% using Google's 60GHz FMCW radar (Soli) combined with a Long Short-Term Memory (LSTM) network, and the accuracy for recognizing gestures from new participants was 98.48%. In 2021, a study introduced a reusable LSTM (RLSTM) network based on range-Doppler angular trajectories, using a 77GHz FMCW MIMO radar, achieving an average precision of 99%.FMCW radar boasts advantages such as small size, light weight, strong anti-interference capability, and high resolution. Furthermore, operating in non-contact mode, it is unaffected by line-of-sight, lighting conditions, or inclement weather, significantly enhancing user experience and privacy protection. In summary, gesture recognition systems based on FMCW radar exhibit superior performance, applicability, user experience, and privacy protection. They can simultaneously acquire multi-dimensional motion parameters such as distance, speed, and angle of a target using frequency-modulated signals with a fixed slope, providing richer information dimensions for gesture recognition. Compared to wearable sensors, computer vision, and Wi-Fi systems, it possesses significant advantages and holds substantial development potential in the fields of intelligent technology and human-computer interaction.

[0004] Current research on existing gesture recognition methods based on FMCW radar reveals some achievements in the field. However, several challenges remain in feature extraction and gesture classification. First, when selecting gesture features as model input, methods often employ only micro-Doppler features, or extract three-dimensional features of range-Doppler-angle, or extract range, Doppler, and azimuth features individually. These single features are susceptible to subjective influences and interference, and lack sufficient signal representation capabilities, making them unsuitable for more complex scenarios. Second, while existing methods have achieved significant progress in classification accuracy in millimeter-wave radar-based gesture recognition research, they still face several key challenges. Most recognition systems rely on traditional convolutional modules to build network models. While effective, this architecture leads to an exponential increase in model complexity and parameter count as convolutional neural networks deepen. Furthermore, frequent downsampling operations with increasing network depth often result in the loss of substantial detail, potentially negatively impacting final classification accuracy. Finally, if the dataset is collected only in a single environment, it will lack environmental diversity, resulting in weak generalization ability of the trained model and failing to fully reflect multipath effects and complex background interference in real-world scenarios, thus limiting the model's robustness. Such datasets cannot comprehensively verify the model's adaptability in practical applications, potentially leading to a significant gap between performance in experimental environments and real-world scenarios. Summary of the Invention

[0005] To address the problems of existing technologies, this invention proposes a dynamic gesture recognition method using a multi-feature fusion network under complex scene interference. By processing radar signals, it extracts and fuses the RTM and DTM of gestures, improving feature richness and signal expressiveness. Multi-scene data acquisition summarizes the specific characteristics of each scene. To overcome the problem of low model efficiency, a lightweight neural network model, Multi-DSCC-Transformer, is designed.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A dynamic gesture recognition method using a multi-feature fusion network under complex scene interference includes:

[0008] Acquire the baseband intermediate frequency signal, process the baseband intermediate frequency signal, and obtain the distance-time map and the Doppler-time map;

[0009] The distance-time map and the Doppler-time map are input into the gesture recognition model to obtain the gesture recognition result. The gesture recognition model is obtained by training a lightweight neural network model using a training set, which includes: a distance-time spectrum map, a Doppler-time spectrum map, and gesture labels. Based on the DSCC module in the gesture recognition model, the spatial displacement features and gesture velocity change information of the distance-time map and the Doppler-time map are extracted to obtain a feature map. The feature map is combined with the position encoding and input into the Transformer module for global feature extraction to obtain the gesture recognition result.

[0010] Optionally, acquiring the baseband intermediate frequency signal includes:

[0011] The baseband intermediate frequency signal is generated using an FMCW radar device:

[0012] The detection signal whose frequency changes linearly with time is acquired, and the detection signal is split into two paths. One path of the detection signal is amplified and radiated into space to detect the target, while the other path of the detection signal is used as a reference signal.

[0013] When the radiated signal detects the target, the signal returns and is mixed with the reference signal to generate an intermediate frequency (IF) signal containing target information. The IF signal is then filtered to obtain the baseband IF signal.

[0014] Optionally, obtaining the distance-time map and the Doppler-time map includes:

[0015] The baseband intermediate frequency signal is rearranged into a two-dimensional data matrix; wherein the horizontal axis of the two-dimensional data matrix represents the slow time dimension and the vertical axis represents the fast time dimension.

[0016] Perform a fast Fourier transform on the fast time dimension to obtain a distance-time matrix, and then perform a slight scaling transformation on the distance-time matrix to obtain the distance-time map;

[0017] A short-time Fourier transform is performed on the slow time dimension, and the short-time Fourier transform results of each interval are coherently superimposed to obtain the Doppler-time map.

[0018] Optionally, the gesture recognition model includes:

[0019] The DSCC module is used to extract the spatial displacement features and gesture velocity change information of the distance-time map and the Doppler-time map to generate the feature map; the feature map includes: a first feature map and a second feature map;

[0020] The position encoding module is used to perform weighted feature fusion on the first feature map and the second feature map, and then introduce position encoding;

[0021] The Transformer module is used to extract global features of dynamic gestures using positional encoding results;

[0022] A fully connected module is used to process the global features, perform gesture classification, and obtain the gesture recognition results.

[0023] Optionally, the DSCC module includes two parallel convolutional submodules, used to characterize the spatial displacement features of the gesture motion and capture the gesture velocity change information, respectively.

[0024] The convolutional submodule includes: depthwise separable convolutional layers, and pooling layers are alternately arranged. The pooling layer at the end is connected to multiple standard convolutional layers in sequence, and the standard convolutional layer at the end is connected to a CAM unit. Each convolutional layer is followed by a batch normalization layer and a nonlinear layer.

[0025] The depthwise separable convolutional layer is used for feature extraction from the distance-time map and the Doppler-time map;

[0026] The pooling layer is used to reduce the spatial dimension of the output of the depthwise separable convolutional layer;

[0027] The standard convolutional layer is used to perform secondary feature extraction on the output of the pooling layer;

[0028] The CAM unit is used to perform channel attention enhancement on the output of the standard convolutional layer.

[0029] Optionally, the Transformer module includes: multiple encoding sub-modules for extracting global features of dynamic gestures using position encoding results;

[0030] The encoding submodule includes: a Dropout layer, a layer normalization layer, a multi-head self-attention layer, a feedforward neural network layer, and a normalization layer connected in sequence; wherein, residual connections are added after the multi-head self-attention layer and the feedforward neural network layer;

[0031] The multi-head self-attention layer is used to perform a linear transformation on the normalization result to generate a query vector, a key vector, and a value vector. The query vector and the key vector are then multiplied by a dot product to calculate the attention weights.

[0032] Optionally, generating the query vector, key vector, and value vector includes:

[0033]

[0034] Where Q is the query vector, K is the key vector, V is the value vector, T is the transpose of the vector, and d k The dimensions of the key vector and query vector.

[0035] Optionally, calculating the attention weights includes:

[0036]

[0037] Where Weight ts is the weight.

[0038] Optionally, the fully connected module includes:

[0039] An adaptive average pooling layer is used to reduce the dimensionality of the global features;

[0040] A fully connected layer is used to map the reduced-dimensional global features to the label space to obtain the gesture recognition result.

[0041] The beneficial effects of this invention are as follows:

[0042] This invention designs a lightweight gesture recognition model, Multi-DSCC-Transformer, which employs a depthwise separable convolutional structure, significantly reducing model complexity. Two identical DSCC modules extract gesture features from distance-time and Doppler-time feature maps, respectively, and fuse these features to increase the diversity of gesture information. Based on this, a channel attention model (CAM) is used to effectively learn the important parts of the features, and a Transformer network is further introduced to extract global features of dynamic gestures. By extracting and fusing multiple features, this invention more comprehensively expresses the distance, speed, and time information of the target gesture, improving the richness and expressiveness of gesture features in complex scenes, enhancing the model's robustness, making it more stable in complex environments, and thus improving recognition performance.

[0043] This invention can significantly improve the performance and practicality of the model. It collects experimental data in four non-ideal environments containing various interferences, noises, and clutter, effectively simulating real-world environments. This not only enriches the diversity of data samples but also increases the learning difficulty of the model, thereby enhancing its adaptability and generalization ability in complex environments. On the other hand, it also increases the learning difficulty of the model for gesture recognition, thereby improving the model's adaptability and generalization ability in complex scenarios, laying a solid foundation for the deployment and promotion of gesture recognition systems. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a block diagram of the FMCW radar system according to an embodiment of the present invention;

[0046] Figure 2 This is a flowchart of the raw echo data preprocessing process according to an embodiment of the present invention;

[0047] Figure 3 This is a diagram of the DSCC-transformer network structure according to an embodiment of the present invention;

[0048] Figure 4 This is a structural diagram of the Channel Attention Mechanism (CAM) according to an embodiment of the present invention;

[0049] Figure 5 This is a structural diagram of the DSCC block according to an embodiment of the present invention;

[0050] Figure 6 This is a structural diagram of the Transformer according to an embodiment of the present invention;

[0051] Figure 7 This is a schematic diagram of the experimental gesture design for an embodiment of the present invention; wherein, (a) is left and right swinging, (b) right and left swinging, (c) push, (d) pull, (e) clockwise rotation, (f) counterclockwise rotation, (g) double-tap, and (h) clapping;

[0052] Figure 8 This is a schematic diagram of the data acquisition environment according to an embodiment of the present invention; wherein, (a) represents different acquisition environments and (b) represents different acquisition locations;

[0053] Figure 9 This is a schematic diagram of the confusion matrix of various features in the embodiments of the present invention; wherein, (a) is the confusion matrix of RTM, (b) is the confusion matrix of DTM, and (c) is the confusion matrix of RTM and DTM fusion;

[0054] Figure 10 These are the model loss and validation curves of an embodiment of the present invention; where (a) is the validation accuracy curve and (b) is the training loss curve.

[0055] Figure 11 This is a schematic diagram illustrating the gesture accuracy in different scenarios according to embodiments of the present invention;

[0056] Figure 12 This is a schematic diagram illustrating the accuracy of different distances and angles in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] In the context of rapid technological advancements, millimeter-wave radar is increasingly used in complex scenarios. However, the numerous interference factors in these complex scenarios, along with the inherent high complexity of the models themselves, present unprecedented challenges to the practical implementation of gesture recognition methods. To effectively address this challenge, this invention proposes a dynamic gesture recognition method using a multi-feature fusion network under complex scene interference. This algorithm obtains a self-built dataset by collecting data in four interference-prone environments, significantly enhancing the complexity and richness of the data. Simultaneously, the advanced clutter filtering technique MTI significantly improves the signal-to-noise ratio. Based on this, multiple key modes, such as the range-time map (RTM) and Doppler-time map (DTM), are successfully acquired, and the richness and expressive power of the features are further enhanced through mode fusion. In terms of gesture recognition model construction, this invention proposes a lightweight neural network model, Multi-DSCC-Transformer. The DSCC module (Depthwise Separable Convolution with CAM Channel Attention Mechanism) utilizes depthwise separable convolution to reduce the size of model parameters and uses CAM (Channel Attention Module) to enhance the feature representation of each channel, thereby improving the feature representation capability of the convolutional neural network. The Transformer module captures the global features of dynamic gestures through stacking and uses a multi-head attention mechanism to focus on basic features. Extensive experimental results show that the gesture recognition algorithm based on multimodal fusion combined with the Multi-DSCC-Transformer model exhibits excellent generalization and adaptability, and possesses strong robustness even in complex scenarios. Specifically, this method achieves a recognition accuracy of up to 97.40% for eight common gestures, providing a highly promising solution for gesture recognition applications of millimeter-wave radar in complex scenarios.

[0060] This embodiment discloses a dynamic gesture recognition method using a multi-feature fusion network under complex scene interference, including: acquiring a baseband intermediate frequency signal; processing the baseband intermediate frequency signal to acquire a distance-time map and a Doppler-time map; inputting the distance-time map and the Doppler-time map into a gesture recognition model to obtain gesture recognition results; the gesture recognition model is obtained by training a lightweight neural network model using a training set, which includes a distance-time spectrum map, a Doppler-time spectrum map, and gesture labels; extracting spatial displacement features and gesture velocity change information from the distance-time map and the Doppler-time map based on the DSCC module in the gesture recognition model to obtain feature maps; and inputting the feature maps, combined with position encoding, into a Transformer module for global feature extraction to obtain gesture recognition results.

[0061] Furthermore, acquiring the baseband intermediate frequency signal includes: generating the baseband intermediate frequency signal using an FMCW radar device; acquiring a detection signal whose frequency changes linearly with time; splitting the detection signal into two paths; amplifying one detection signal and radiating it into space to detect the target; using the other detection signal as a reference signal; when the radiated signal detects the target, the signal returns and is mixed with the reference signal to generate an intermediate frequency signal containing target information; filtering the intermediate frequency signal to acquire the baseband intermediate frequency signal.

[0062] Specifically, a simplified block diagram of the FMCW radar used in this invention is shown below. Figure 1 As shown, it mainly consists of a transmitter, receiver, analog-to-digital converter, and antenna. Its basic working principle is that a voltage-controlled oscillator generates a signal whose frequency varies linearly with time. This signal is split into two paths by a power divider: one path is amplified by a power amplifier and radiated into space by the transmitting antenna for target detection; the other path is retained as a reference signal. When the transmitted signal encounters a target, some energy is reflected back and captured by the receiving antenna. The reflected echo signal is initially amplified by a low-noise amplifier to improve signal quality. Subsequently, the amplified echo signal is mixed with the reference signal to generate an intermediate frequency (IF) signal containing target information. The mixed signal is then filtered to extract the baseband IF signal, which contains the target's range and velocity information. By analyzing and processing the IF signal, the target's range and relative velocity can be accurately calculated, thereby achieving target detection and tracking.

[0063] For FMCW radar, the transmitted signal can be represented as:

[0064]

[0065] In the formula: A tx f is the amplitude of the transmitted signal. c Let α be the carrier frequency (starting frequency), α be the slope of frequency modulation (rate of change of frequency), and t be time. This is the initial phase.

[0066] The received signal is:

[0067]

[0068] In the formula: A rx The amplitude of the received signal is typically smaller than the amplitude of the transmitted signal, and τ is the round-trip time delay of the signal to the target. Where R is the target distance and c is the speed of light.

[0069] The intermediate frequency signal formed by combining the transmitted and received signals can be represented as:

[0070]

[0071] In the formula: A if f is the amplitude of the intermediate frequency signal. if This is a mid-frequency range, typically determined by the target's distance and velocity. This represents the phase of the intermediate frequency signal.

[0072] Furthermore, obtaining the distance-time map and the Doppler-time map includes: rearranging the baseband intermediate frequency signal into a two-dimensional data matrix; wherein the horizontal axis of the two-dimensional data matrix represents the slow time dimension and the vertical axis represents the fast time dimension; performing a fast Fourier transform on the fast time dimension to obtain the distance-time matrix; performing a slight scaling transformation on the distance-time matrix to obtain the distance-time map; performing a short-time Fourier transform on the slow time dimension; and coherently superimposing the short-time Fourier transform results of each interval to obtain the Doppler-time map.

[0073] Specifically, by processing the intermediate frequency signals acquired by the FMCW radar, range-time maps (RTM) and Doppler-time maps (DTM) can be generated, which provide key feature information for gesture recognition. Figure 2 The preprocessing flow of the raw echo data is detailed. First, the raw echo data is rearranged into a two-dimensional data matrix, where the horizontal axis represents the slow time dimension and the vertical axis represents the fast time dimension. Next, a Fast Fourier Transform (FFT) is performed on the fast time dimension to obtain the range-time matrix. This matrix is ​​then slightly scaled to obtain the range-time spectrum (RTM) to capture changes in the target's range dimension. To eliminate static background interference, a fourth-order Butterworth high-pass filter with a cutoff frequency of 0.0075Hz is used as a Moving Target Indicator (MTI) to effectively filter out irrelevant static signals. For changes in the distance between the arm and radar caused by different hand gestures, the system selects a range of 0.2 meters to 1.2 meters for further processing. Within each range, a Short Time Fourier Transform (STFT) is performed along the slow time dimension with a window length of 0.2 seconds and an overlap rate of 95% to capture the dynamic characteristics of the hand gestures. By coherently superimposing the STFT results from each range, a Doppler-time map is finally generated, reflecting changes in the velocity dimension of the hand gesture. Combining range-time maps with Doppler-time maps enables high-precision gesture classification. Since each dataset contains unique and important feature information, efficiently fusing multi-dimensional features is crucial for improving gesture classification accuracy. This multimodal fusion method not only enhances the model's discriminative ability but also provides reliable technical support for gesture recognition in complex scenarios.

[0074] Furthermore, the gesture recognition model includes: a DSCC module, used to extract spatial displacement features and gesture velocity change information from distance-time maps and Doppler-time maps to generate feature maps; the feature maps include: a first feature map and a second feature map; a position encoding module, used to perform weighted feature fusion of the first feature map and the second feature map and introduce position encoding; a Transformer module, used to extract global features of dynamic gestures using the position encoding results; and a fully connected module, used to process global features and perform gesture classification to obtain gesture recognition results.

[0075] Specifically, to address the needs of millimeter-wave radar gesture recognition in complex interference scenarios, this invention proposes a Multi-DSCC-Transformer hybrid network, the core architecture of which is as follows: Figure 3 As shown, the model's two inputs, the Distance-Time Map (RTM) and the Doppler-Time Map (DTM), are first fed into two depthwise separable convolutional modules to extract local features along two paths for dynamic gestures. Then, a Channel Attention (CAM) module is introduced to enhance the feature representation of each channel. Secondly, to address the lack of positional information in the Transformer's parallel input mechanism, positional encoding is added during the dual-branch weighted feature fusion stage to ensure consistent input-output feature order in the MHA mechanism. Next, the Transformer architecture is used for global feature extraction and spatiotemporal feature enhancement. Finally, a fully connected (FC) module is used to process the output features and classify the gestures.

[0076] Furthermore, the DSCC module includes: two parallel convolutional sub-modules, used to represent the spatial displacement features of gesture motion and capture gesture velocity change information, respectively; the convolutional sub-modules include: depthwise separable convolutional layers, depthwise separable convolutional layers, and pooling layers alternately, with the pooling layer at the end connected to multiple standard convolutional layers in sequence, and the standard convolutional layer at the end connected to the CAM unit; the depthwise separable convolutional layer is used to extract features from the distance-time map and the Doppler-time map; the pooling layer is used to reduce the spatial dimension of the output of the depthwise separable convolutional layer; the standard convolutional layer is used to perform secondary feature extraction on the output of the pooling layer; and the CAM unit is used to perform channel attention enhancement on the output of the standard convolutional layer.

[0077] Specifically, this module is a lightweight deep feature extractor that uses a symmetrical dual-branch structure to process the input image, as shown in the following diagram. Figure 3 As shown.

[0078] like Figure 5As shown, the inputs to the DSCC module are RTM and DTM, representing the spatial displacement features of the gesture motion and capturing the velocity change information of the gesture, respectively. The input dimensions are both 64×64×3. Each convolutional module consists of two depthwise separable convolutional layers, three standard convolutional layers, and two pooling layers. The convolutional kernel sizes are 3×3 and 1×1, and the pooling kernel size is 2×2. Following the convolutional layers are batch normalization layers and nonlinear layers. After feature extraction, the outputs are input into the Channel Attention Module (CAM), as shown... Figure 4 As shown, the features of important channels are enhanced. Then, the feature maps enhanced by channel attention are weighted and fused. The self-attention mechanism of the Transformer module is global and lacks a concept of order. To address the shortcomings of the Transformer network's positional information processing mechanism and capture the positional relationships of elements in the sequence, positional information must be explicitly added before entering the Transformer network. The positional encoding used in this invention is absolute positional encoding, which can adapt to input sequences of different lengths. During training, the model can learn the relationship between positional encoding and sequence elements. When encountering new sequences of different lengths, it can still use these learned relationships for reasoning, thereby recovering the structural information of the sequence to a certain extent and improving the model's performance and stability.

[0079] Furthermore, the Transformer module includes: multiple encoding sub-modules for extracting global features of dynamic gestures using positional encoding results; the encoding sub-modules include: a Dropout layer, a layer normalization layer, a multi-head self-attention layer, a feedforward neural network layer, and a normalization layer connected in sequence; among them, residual connections are added after the multi-head self-attention layer and the feedforward neural network layer; the multi-head self-attention layer is used to perform a linear transformation on the normalization result to generate a query vector, a key vector, and a value vector, and to perform a dot product on the query vector and the key vector to calculate the attention weights.

[0080] This invention uses a Transformer encoder to extract global features of dynamic gestures, such as... Figure 6 As shown, the Transformer encoder consists of multiple identical layers stacked together. Each layer mainly includes layer normalization, a multi-head self-attention layer, and a feedforward neural network layer. The most crucial layer is the multi-head attention layer, which operates as follows: First, the input vector is linearly transformed to generate a query vector (Q), a key vector (K), and a value vector (V). The formulas for calculating these three vectors are:

[0081]

[0082] Secondly, the attention weights are calculated by performing a dot product on the query vector and all key vectors. Then, the weights are normalized using the Softmax function. The calculation formula is as follows:

[0083]

[0084] Finally, the weight coefficients are multiplied by the value vector to obtain the attention values ​​for different samples, thus yielding the output of the self-attention mechanism. The multi-head self-attention mechanism is a concatenation of multiple self-attention mechanisms. First, the input sequence is added to the positional encoding and passed through a Dropout layer. Then, it is processed through multiple block modules, each containing a multi-head attention network and a feedforward network, connected using residual connections. Finally, the output is obtained through a final normalization layer.

[0085] Furthermore, the fully connected module includes: an adaptive average pooling layer, used to reduce the dimensionality of global features; and a fully connected layer, used to map the dimensionality-reduced global features to the label space to obtain gesture recognition results.

[0086] Specifically, in the final stage, an adaptive average pooling layer and a fully connected layer are used to output gesture labels. The adaptive average pooling layer plays a crucial role in reducing the dimensionality of the feature vectors, which helps reduce the number of model parameters. The fully connected layer maps the feature vectors to the label space, thereby achieving dynamic gesture classification.

[0087] Experimental Analysis and Discussion

[0088] Hardware Platform: This embodiment uses TI's AWR 1843BOOST millimeter-wave radar, operating at a frequency of 77-81 GHz with a maximum bandwidth of 4 GHz, in conjunction with a DCA 1000 high-speed data acquisition card to collect radar echoes from gestures. Table 1 shows the radar parameter configuration.

[0089] Table 1 Radar parameter configuration

[0090]

[0091]

[0092] Data Acquisition: This embodiment designs eight dynamic gestures for data collection, such as... Figure 7 As shown in (a)-(h). To improve the robustness of the model, data was collected in four non-ideal environments, including an empty laboratory, an office, a conference room, and a crowded laboratory. Five volunteers participated in the data collection process. The distances and angles of the collection locations 1-6 were (0.6m, 0°), (0.6m, 45°), (0.8m, 0°), (0.8m, -30°), (1.0m, 0°), and (1.0m, 30°), respectively. Figure 8As shown in (a)-(b), each volunteer performed each action 48 times, resulting in a total of 1920 dynamic gesture radar echo data files. The collected data underwent signal processing to obtain the RTM and DTM for each gesture, and the resulting dataset was divided into training, testing, and validation sets in a 6:2:2 ratio.

[0093] Experimental Results Analysis: In order to comprehensively evaluate the overall performance of the identification method proposed in this invention, this invention conducts systematic experimental verification from three aspects: feature verification, model performance comparison, and method robustness.

[0094] (1) Feature verification:

[0095] This invention, through experimental analysis and comparison of distance-time maps (RTM), Doppler-time maps (DTM), and fused features, deeply analyzes the limitations of single features and verifies the reliability and advantages of fused features. Figure 9 The confusion matrix generated for all features of the model reveals significant misidentification and confusion when using a single feature for gesture classification. For example, the RTM confusion matrix shows that gesture 4 has a 100% accuracy rate, but gesture 2 only has a 40% accuracy rate. The DTM confusion matrix shows that gestures 3 and 4 achieve high accuracy rates, while gestures 2 and 5 have less than ideal accuracy rates. This highlights the limitations and inadequacies of single features in representing gestures under complex scene interference. Specifically, while RTM and DTM demonstrate high accuracy rates for specific gestures, they each have obvious shortcomings. For instance, RTM primarily represents gestures based on changes in distance between the target object and the radar, thus performing poorly when handling gestures involving rapid speed changes. DTM, on the other hand, has limited ability to capture distance information for static or slowly moving gestures, failing to provide sufficient discriminative information and leading to a decrease in recognition rate. These shortcomings indicate that a single feature cannot fully cover the dynamic characteristics of gestures, resulting in limited classification performance.

[0096] To overcome the limitations of single features, this invention effectively integrates features from RTM and DTM. This feature fusion fully leverages the complementary advantages of both, thereby significantly improving the accuracy and robustness of gesture recognition. Figure 9 (a) Figure 9 (b) and Figure 9 (c) The comparison shows that the fused features perform well in reducing misidentification and confusion. The confusion matrix clearly shows that the fused features significantly reduce errors in gesture recognition, further verifying the effectiveness of the feature fusion strategy.

[0097] Model performance analysis:

[0098] To verify the overall performance of the recognition method proposed in this invention, a comparative experiment with different learning rates was first conducted. Then, the method was compared with four neural network models, namely the classic neural network model VGG19, ResNet50, MobileNetV3, and the self-designed neural network model CNN, to verify its performance.

[0099] Comparison of different learning rates:

[0100] In the training process of deep learning models, the learning rate and decay strategy are key parameters affecting network performance, directly determining the model's convergence speed and final performance. To examine the impact of different learning rates on the Multi-DSCC-Transformer model, parameter optimization was performed, testing different learning rates: 1e-3, 1e-4, 1e-5, 2e-4, and 2e-5. The training loss curve and validation accuracy curve of the model were then observed. Figure 10 As shown in (a)-(b), a learning rate of 1e-5 results in the highest accuracy, the fastest convergence speed of the loss-accuracy curves, and the greatest stability. Therefore, the learning rate used in the experiments of this invention is 1e-5.

[0101] Model accuracy analysis: Model accuracy is an important indicator for evaluating the performance of classification models. This invention compares different neural network models with the designed Multi-DSCC-transformer, comparing not only the accuracy of each model but also analyzing their time complexity (Flops) and number of parameters (Params). As shown in Table 2...

[0102] The model in this invention achieves a recognition accuracy of 97.40%, significantly outperforming other comparative models. Compared to the CNN model with the highest accuracy (92.97%) among the comparative models, our model shows a 4.43% improvement, demonstrating stronger feature extraction and classification capabilities. In terms of time complexity, the model in this invention has 42.79 GFLOPs, second only to MobileNet V3-small's 4.04 GFLOPs, and significantly lower than VGG19 (206.61 GFLOPs), ResNet50 (19.22 GFLOPs), and MobileNet V3-large (15.24 GFLOPs). Although the model's FLOPs are higher than MobileNet V3-small, its recognition accuracy is 18.23% higher, significantly compensating for the slight increase in computational complexity. The model in this paper has 2.24M parameters, which is significantly lower than VGG19 (73.67M), ResNet50 (21.29M), and MobileNet V3-large (8.42M), and only slightly higher than MobileNet V3-small (3.05M) and CNN (0.96M). Nevertheless, the Multi-DSCC-transformer model achieves a better balance between parameter count and computational complexity while maintaining the highest recognition accuracy.

[0103] Table 2. Gesture recognition results of different models

[0104]

[0105] Model robustness verification (different environments, different locations):

[0106] Different Environments: Environmental factors in different scenarios significantly affect the propagation characteristics of millimeter-wave signals, including the reflection, absorption, and scattering of signals by obstacles. This difference leads to significant variations in the intensity and clutter interference of gesture signals acquired in different scenarios, thus increasing the difficulty of gesture recognition. Therefore, verifying the generalization ability and robustness of the proposed method in different scenarios is crucial. To this end, we conducted experiments in four scenarios: a crowded laboratory, an office, a conference room, and an empty laboratory, to comprehensively evaluate the scenario adaptability of the method. Experimental results show that the recognition accuracy varies significantly in different scenarios. Figure 11As shown, the highest recognition accuracy was achieved in the conference room setting (98.95%), while the lowest was achieved in the crowded laboratory setting (90.63%). Comparing the recognition results between crowded and empty laboratories revealed that factors such as scene size, interference, and clutter significantly impacted the recognition results. Specifically, the conference room, with its relatively large space and less interference, exhibited more stable signal propagation, resulting in the highest recognition accuracy. In contrast, the crowded laboratory, with its confined space, numerous obstacles, and personnel activity, led to complex signal reflection and scattering, increasing clutter interference and thus reducing recognition accuracy. The recognition accuracy in the office and empty laboratory settings was 96.85% and 92.71%, respectively. Although the office setting was smaller, the relatively orderly arrangement of items resulted in less interference, leading to higher recognition accuracy. While the empty laboratory setting had a larger space, the lack of sufficient reflective objects may have resulted in insufficient signal strength, affecting recognition accuracy. These experimental results validate the adaptability and robustness of the proposed method in different scenarios and provide important references for subsequent optimization of gesture recognition technology.

[0107] Different positions: The different positions of the experimental gestures relative to the radar cause differences in electromagnetic signal attenuation and multipath reflection, resulting in variations in the amplitude and phase of the received signal, i.e., differences in distance and angle, ultimately leading to differences in gesture features. These feature differences have a significant impact on the accuracy of the gesture recognition method and therefore require further verification. To evaluate the performance of gesture recognition at different distances and angles, we designed nine experimental positions: (0.6m, 45°), (0.6m, 30°), (0.6m, 0°), (0.8m, 45°), (0.8m, 30°), (0.8m, 0°), (1.0m, 45°), (1.0m, 30°), and (1.0m, 0°). For each experimental position, we collected 20 sets of data to comprehensively evaluate the recognition performance. Figure 12 As shown, the recognition accuracy varies significantly across different locations. Specifically, the highest accuracy of 97.60% is achieved at (0.6m, 0°), while the lowest accuracy of only 88.75% is achieved at (1.0m, 45°). Further analysis reveals that at the same angle, recognition accuracy gradually decreases with increasing distance; and at the same distance, recognition accuracy significantly decreases as the angle increases from 0° to 45°. The main reasons for these differences include: the radar's field of view, signal transmission power, and receiver sensitivity limit its ability to capture gestures at long distances and specific angles; and obstacles, clutter, and other interference sources in complex environments further weaken signal quality, especially at long distances and non-optimal angles. These analyses further validate the adaptability and robustness of the method presented in this invention at different locations, providing an important reference for gesture recognition in complex scenarios.

[0108] To address the interference problem faced by dynamic gesture recognition using millimeter-wave radar in complex scenarios, this invention proposes a multi-feature fusion network method for dynamic gesture recognition under complex scene interference. First, radar data is collected in four different scenarios to enhance data diversity and complexity, thereby better simulating the challenges of real-world environments. Second, gesture data is filtered and denoised, significantly improving the signal-to-noise ratio. Based on this, feature maps such as range-time maps (RTM) and Doppler-time maps (DTM) are extracted. These feature maps comprehensively reflect the motion state of the gesture in the distance, velocity, and time dimensions, and are analyzed in depth. To further reduce model complexity, this invention designs a lightweight Multi-DSCC–Transformer model. This model combines depthwise separable convolution (DSCC) and Transformer modules, achieving high recognition accuracy while maintaining low memory requirements. By comparing the recognition results of different features, this invention verifies the correctness of feature extraction and the necessity of multi-modal feature fusion. Furthermore, this embodiment was compared with five different neural network models. The results show that the proposed Multi-DSCC–Transformer method outperforms other models in recognition accuracy, while not having the highest memory requirements. The experiments further demonstrate that the model proposed in this invention exhibits strong robustness to changes in position, angle, and scene, showcasing its adaptability in complex environments. This lays the foundation for subsequent continuous gesture recognition based on millimeter-wave radar.

[0109] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A dynamic gesture recognition method using a multi-feature fusion network under complex scene interference, characterized in that, include: Acquire the baseband intermediate frequency signal, process the baseband intermediate frequency signal, and obtain the distance-time map and the Doppler-time map; Input the distance-time map and the Doppler-time map into the gesture recognition model to obtain the gesture recognition result; The gesture recognition model is obtained by training a lightweight neural network model using a training set, which includes a distance-time spectrum, a Doppler-time spectrum, and gesture labels. Based on the DSCC module in the gesture recognition model, spatial displacement features and gesture velocity change information of the distance-time spectrum and the Doppler-time spectrum are extracted to obtain a feature map. The feature map is then combined with the position encoding and input into the Transformer module for global feature extraction to obtain the gesture recognition result. The gesture recognition model includes: The DSCC module is used to extract the spatial displacement features and gesture velocity change information of the distance-time map and the Doppler-time map to generate the feature map; the feature map includes: a first feature map and a second feature map; The position encoding module is used to perform weighted feature fusion on the first feature map and the second feature map, and then introduce position encoding; The Transformer module is used to extract global features of dynamic gestures using positional encoding results; A fully connected module is used to process the global features, perform gesture classification, and obtain the gesture recognition result; The DSCC module includes two parallel convolutional submodules, which are used to characterize the spatial displacement features of the gesture motion and capture the gesture velocity change information, respectively. The convolutional submodule includes: depthwise separable convolutional layers, and pooling layers are alternately arranged. The pooling layer at the end is connected to multiple standard convolutional layers in sequence, and the standard convolutional layer at the end is connected to a CAM unit. Each convolutional layer is followed by a batch normalization layer and a nonlinear layer. The depthwise separable convolutional layer is used for feature extraction from the distance-time map and the Doppler-time map; The pooling layer is used to reduce the spatial dimension of the output of the depthwise separable convolutional layer; The standard convolutional layer is used to perform secondary feature extraction on the output of the pooling layer; The CAM unit is used to perform channel attention enhancement on the output of the standard convolutional layer.

2. The dynamic gesture recognition method for complex scene interference using a multi-feature fusion network according to claim 1, characterized in that, Acquiring the baseband intermediate frequency signal includes: The baseband intermediate frequency signal is generated using an FMCW radar device: The detection signal whose frequency changes linearly with time is acquired, and the detection signal is split into two paths. One detection signal is amplified and radiated into space to detect the target, while the other detection signal is used as a reference signal. When the radiated signal detects the target, the signal returns and is mixed with the reference signal to generate an intermediate frequency (IF) signal containing target information. The IF signal is then filtered to obtain the baseband IF signal.

3. The dynamic gesture recognition method for complex scene interference using a multi-feature fusion network according to claim 1, characterized in that, Obtaining the distance-time map and the Doppler-time map includes: The baseband intermediate frequency signal is rearranged into a two-dimensional data matrix; wherein the horizontal axis of the two-dimensional data matrix represents the slow time dimension and the vertical axis represents the fast time dimension. Perform a fast Fourier transform on the fast time dimension to obtain a distance-time matrix, and then perform a scaling transformation on the distance-time matrix to obtain the distance-time map; A short-time Fourier transform is performed on the slow time dimension, and the short-time Fourier transform results of each interval are coherently superimposed to obtain the Doppler-time map.

4. The dynamic gesture recognition method for complex scene interference using a multi-feature fusion network according to claim 1, characterized in that, The Transformer module includes: multiple encoding sub-modules, used to extract global features of dynamic gestures using position encoding results; The encoding submodule includes: a Dropout layer, a first normalization layer, a multi-head self-attention layer, a feedforward neural network layer, and a second normalization layer connected in sequence; wherein, residual connections are added after the multi-head self-attention layer and the feedforward neural network layer; The multi-head self-attention layer is used to perform a linear transformation on the normalization result to generate a query vector, a key vector, and a value vector. The query vector and the key vector are then multiplied by a dot product to calculate the attention weights.

5. The dynamic gesture recognition method for complex scene interference using a multi-feature fusion network according to claim 4, characterized in that, Generating the query vector, key vector, and value vector includes: in, For query vector, For key vectors, For value vectors, This is the transpose of the vector. The dimensions of the key vector and query vector.

6. The dynamic gesture recognition method for complex scene interference using a multi-feature fusion network according to claim 4, characterized in that, The calculation of the attention weights includes: in, As weight.

7. The dynamic gesture recognition method for complex scene interference using a multi-feature fusion network according to claim 1, characterized in that, The fully connected module includes: An adaptive average pooling layer is used to reduce the dimensionality of the global features; A fully connected layer is used to map the reduced-dimensional global features to the label space to obtain the gesture recognition result.

Images