FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet

Abstract

The application discloses a FMCW radar gesture recognition method based on double-channel feature fusion and ResNet, relates to the technical field of radar signal processing and intelligent recognition, and comprises the following steps: a frequency-modulated continuous wave radar gesture data acquisition system is constructed, echo signal collection of preset category gesture actions is carried out by using a millimeter wave radar, and original echo data collected is stored; the FMCW radar gesture recognition method based on double-channel feature fusion and ResNet has the advantages of high recognition accuracy, strong environmental adaptability, good privacy protection and high system stability, and can effectively improve the application performance of the FMCW radar in a non-contact human-computer interaction scene.

Description

Technical Field

[0001] This invention relates to the field of radar signal processing and intelligent recognition technology, specifically to an FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet. Background Technology

[0002] With the continuous development of information technology and intelligent sensing technology, human-computer interaction is evolving from traditional physical contact interaction to natural, contactless interaction. Gesture recognition, as one of the important technical means to achieve natural human-computer interaction, has been widely used in fields such as intelligent driving assistance systems, smart healthcare, smart home control, and virtual reality. In existing technologies, based on the different methods of acquiring gesture data, gesture recognition technology is mainly divided into recognition methods based on wearable devices and recognition methods based on contactless sensing devices.

[0003] Gesture recognition methods based on wearable devices typically acquire hand movement data through devices such as smart gloves, rings, or inertial sensors. These methods can obtain relatively accurate movement information, but require users to wear specialized devices, leading to inconvenience, poor comfort, and device wear and tear, thus limiting their widespread application. Vision-based non-contact gesture recognition methods acquire image or video data through cameras and combine it with computer vision algorithms for recognition. While achieving non-contact interaction to some extent, they are susceptible to lighting conditions, occlusion, and background complexity, and also pose a certain risk of privacy breaches. Recognition performance degrades significantly in low light or complex environments. In recent years, radar-based gesture recognition technology has gradually gained attention. Frequency-modulated continuous wave millimeter-wave radar acquires target distance and velocity information by transmitting linear frequency-modulated signals and receiving reflected echoes from targets. It offers advantages such as being unaffected by lighting conditions, strong penetration capabilities, good privacy protection, and adaptability to complex environments. By processing the echo signal using Fast Fourier Transform (FFT) and Moving Target Indication (MTI) filtering, feature representations such as range-time maps and Doppler-time maps can be obtained, reflecting the spatial displacement and velocity change characteristics of the gesture. Existing gesture recognition methods based on FMCW radar often use a single feature map as the neural network input, such as relying solely on RTM or DTM for classification. While these methods can distinguish gestures to some extent, different gestures may exhibit similar local features in the range or velocity dimensions. Relying on a single feature is insufficient to fully represent the complete dynamic information of the gesture, easily leading to misclassification. For example, some push, pull, and push-pull gestures show similar trends in the range-time map, while overlapping areas exist in the Doppler dimension, making it difficult for a single feature input model to achieve high-precision differentiation. Furthermore, some existing methods use neural network structures with shallow or relatively simple structures, limiting their ability to mine potential multi-level features in radar echo data and failing to fully utilize the complementary information contained in RTM and DTM. On the other hand, traditional convolutional neural networks are prone to gradient vanishing or feature degradation problems as the network deepens, affecting the model's ability to extract complex dynamic gesture features. Although some studies have attempted to use multi-stream networks or feature-level fusion strategies, there is still room for improvement in fusion structure design, feature level utilization, and network depth optimization. Summary of the Invention

[0004] The purpose of this invention is to provide an FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet, thereby solving the problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet, comprising: S1. Construct a frequency modulated continuous wave radar gesture data acquisition system, use millimeter wave radar to collect echo signals of preset categories of gestures, and store the collected raw echo data. S2. Preprocess the raw echo data, including reconstructing the echo data into a data matrix arranged according to the transmission period, performing windowing processing and spectrum transformation along the fast time dimension to obtain distance information, performing moving target filtering to suppress static target interference, performing windowing processing and spectrum transformation along the slow time dimension to obtain speed information, and generating distance-time feature maps and Doppler-time feature maps corresponding to the gestures by accumulating them frame by frame. S3. The distance-time feature map and the Doppler-time feature map are cropped and normalized to serve as input data for the dual-channel neural network. S4. Construct a dual-channel feature extraction model based on residual networks. Set up two convolutional network channels with the same structure, and perform multi-layer convolution operations, pooling operations and residual block feature extraction on the distance-time feature map and the Doppler-time feature map respectively to obtain the corresponding high-dimensional feature representation. S5. Concatenate and fuse the feature vectors output from the two channels, and input the fused features into a fully connected layer for classification training. Obtain the recognition results of each gesture category through the probability output layer. S6. Optimize the parameters of the dual-channel feature fusion residual network using the training set, and verify the model's recognition accuracy and generalization ability using the test set to achieve automatic recognition of different gestures.

[0006] Preferably, the linear frequency modulated signal transmitted by the frequency modulated continuous wave radar is represented as: ; Where s(t) represents the transmitted signal; A represents the amplitude of the transmitted signal; f0 represents the starting frequency; B represents the frequency modulation bandwidth; T represents the duration of a single frequency modulation cycle; t represents the time variable; φ(t) represents the phase noise; and j represents the imaginary unit.

[0007] Preferably, the output of the residual block is represented as: H(x) = F(x) + x; Where x represents the input feature of the residual block; F(x) represents the residual mapping after two layers of convolution, batch normalization and activation function processing; H(x) represents the output feature of the residual block, which is obtained by adding the input feature and the residual mapping element by element.

[0008] Preferably, during the preprocessing process, windowing is performed before spectral transformation of the fast time dimension data to reduce the impact of sidelobe leakage on range resolution and improve the main lobe concentration of the range spectrum.

[0009] Preferably, the moving target filtering process includes performing differential operations on the echo data between consecutive frames to eliminate static background reflection signals, thereby enhancing the energy characteristics of the dynamic gesture target.

[0010] Preferably, the distance-time feature map and Doppler-time feature map are normalized before being input into the neural network to ensure that the amplitude distribution among different samples remains consistent, thereby improving the stability of model training.

[0011] Preferably, in the dual-channel feature extraction model, each channel includes a convolutional layer, a batch normalization layer, an activation function layer, a max pooling layer, and multiple residual blocks connected in sequence, used to extract low-level features and high-level semantic features of gestures layer by layer.

[0012] Preferably, when the number of channels or spatial size of the input feature map and the output feature map of the residual block are inconsistent, a convolutional layer is set to perform dimension matching processing on the input features in order to achieve element-wise addition of the features.

[0013] Preferably, the dual-channel network adopts a feature concatenation method in the feature fusion stage, which concatenates the high-dimensional features extracted from the two channels along the channel dimension to retain the complementary information of distance and velocity features.

[0014] Preferably, random deactivation layers are set between the fully connected layers to randomly shield some neuron connections during training, thereby reducing the risk of model overfitting and improving generalization ability.

[0015] As can be seen from the above technical solution, the present invention has the following beneficial effects: This FMCW radar gesture recognition method, based on dual-channel feature fusion and ResNet, systematically preprocesses the echo signal from frequency-modulated continuous wave millimeter-wave radar, extracting range-time and Doppler-time feature maps respectively. A dual-channel residual network is constructed to perform parallel deep feature extraction and cascaded fusion of the two types of features, fully leveraging the complementary characteristics of range and velocity dimensions. This effectively overcomes the inter-class misclassification problem caused by relying on only a single feature in existing technologies. Simultaneously, the residual network structure enhances the deep feature representation capability of the network, mitigating gradient vanishing and feature degradation phenomena, and improving the modeling ability for complex dynamic gestures. Through feature-level fusion and fully connected classification strategies, high-precision recognition of multiple types of gestures is achieved, maintaining good generalization performance under different test conditions. The overall solution has advantages such as high recognition accuracy, strong environmental adaptability, good privacy protection, and high system stability, effectively improving the application performance of FMCW radar in non-contact human-computer interaction scenarios. Attached Figure Description

[0016] Figure 1This is a flowchart of the FMCW radar gesture recognition method of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, this invention provides a technical solution: an FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet, comprising: S1. Construct a frequency modulated continuous wave radar gesture data acquisition system, use millimeter wave radar to collect echo signals of preset categories of gestures, and store the collected raw echo data. S2. Preprocess the raw echo data, including reconstructing the echo data into a data matrix arranged according to the transmission period, performing windowing processing and spectrum transformation along the fast time dimension to obtain distance information, performing moving target filtering to suppress static target interference, performing windowing processing and spectrum transformation along the slow time dimension to obtain speed information, and generating distance-time feature maps and Doppler-time feature maps corresponding to the gestures by accumulating them frame by frame. S3. The distance-time feature map and the Doppler-time feature map are cropped and normalized to serve as input data for the dual-channel neural network. S4. Construct a dual-channel feature extraction model based on residual networks. Set up two convolutional network channels with the same structure, and perform multi-layer convolution operations, pooling operations and residual block feature extraction on the distance-time feature map and the Doppler-time feature map respectively to obtain the corresponding high-dimensional feature representation. S5. Concatenate and fuse the feature vectors output from the two channels, and input the fused features into a fully connected layer for classification training. Obtain the recognition results of each gesture category through the probability output layer. S6. Optimize the parameters of the dual-channel feature fusion residual network using the training set, and verify the model's recognition accuracy and generalization ability using the test set to achieve automatic recognition of different gestures.

[0019] This implementation method leverages the joint range and velocity sensing capabilities of frequency-modulated continuous wave millimeter-wave radar. It achieves automatic recognition of dynamic gestures through dual-channel feature construction and residual network fusion modeling. The overall implementation process includes data acquisition, signal preprocessing, feature map construction, dual-channel deep feature extraction, feature fusion classification, and model training and optimization. These steps unfold sequentially according to signal processing logic and neural network training procedures.

[0020] First, in the data acquisition phase, a frequency-modulated continuous wave (FMCH) millimeter-wave radar gesture data acquisition system is constructed. The millimeter-wave radar continuously transmits FMCH signals. When a hand performs a preset gesture in front of the radar, the moving hand generates a reflected echo of the transmitted signal. The receiving antenna receives the echo signal, performs mixing processing, and outputs intermediate frequency (IF) signal data. The acquisition system records and stores each frame of data according to the transmission cycle and sampling point sequence, ensuring that each gesture has multiple samples under different personnel, different amplitudes of movement, and different speeds of motion, thus forming a representative dataset.

[0021] In the signal preprocessing stage, the raw echo data is first reconstructed according to frames, transmission periods, and sampling points to form a regularly arranged data matrix. For the sampled data within each transmission period, windowing is applied along the fast time direction to reduce sidelobe interference caused by spectral leakage, followed by spectral transformation to obtain the corresponding distance distribution information. To suppress stable echoes generated by stationary objects in the environment, differential processing or background mean elimination processing is performed on the data between consecutive transmission periods to retain dynamic target information that changes over time. Subsequently, windowing and spectral transformation are performed again along the slow time direction to extract velocity change information caused by hand movements. The distance information in consecutive time frames is stacked in chronological order to generate a distance-time feature map, and the velocity information is stacked in chronological order to generate a Doppler-time feature map, thus forming a complete temporal feature representation reflecting the dynamic changes of the gesture.

[0022] During the feature normalization stage, the generated distance-time and Doppler-time feature maps are effectively cropped to retain the main energy concentration regions and uniformly adjusted to a fixed size to meet the input requirements of the neural network. Subsequently, the feature map data is normalized to ensure that the data distribution among different samples is within a uniform scale, avoiding significant numerical differences that could affect the stability of network training. The two types of feature maps, after processing, are then used as input data for the dual-channel network.

[0023] In the feature extraction stage, a dual-channel convolutional neural network model based on a residual structure is constructed. The two channels have identical structures but independent parameters. The first channel inputs a distance-time feature map, and the second channel inputs a Doppler-time feature map. Each channel sequentially passes through convolutional layers to extract local spatial features, and then undergoes downsampling through pooling layers to enhance feature stability. High-level semantic features are extracted layer by layer through multiple residual blocks. The residual structure preserves shallow information through cross-layer connections, mitigating gradient decay during deep network training and improving model convergence efficiency and expressive power. After multi-layer feature extraction, global average pooling is used to convert the two-dimensional feature map into a one-dimensional high-dimensional feature vector.

[0024] In the feature fusion and classification stage, the feature vectors output from the two channels are concatenated and concatenated to form a fused feature representation, enabling the distance dynamic information and velocity dynamic information to complement each other. The fused features are input to a fully connected layer for feature remapping, and the predicted probability of each gesture category is obtained through a probability output layer. The category with the highest probability is selected as the final recognition result.

[0025] During the model training and optimization phase, the sample data is divided into training and test sets. During training, prediction results are obtained through forward propagation, and the loss value is calculated based on the difference between the predicted value and the true label. The network parameters are then updated layer by layer using the backpropagation algorithm. After training, the model's recognition accuracy and generalization ability are evaluated using the test set. Based on the evaluation results, the network structure or training parameters are adjusted until the model achieves stable recognition performance. Through these steps, a complete processing flow from the raw millimeter-wave echo signal to the gesture category output is achieved, forming a repeatable automatic recognition system.

[0026] The linear frequency modulated signal transmitted by a frequency modulated continuous wave radar is represented as: ; Where s(t) represents the transmitted signal; A represents the amplitude of the transmitted signal; f0 represents the starting frequency; B represents the frequency modulation bandwidth; T represents the duration of a single frequency modulation cycle; t represents the time variable; φ(t) represents the phase noise; and j represents the imaginary unit.

[0027] This embodiment provides a detailed explanation of the formation mechanism of frequency modulated continuous wave radar transmitted signals, so as to clarify its fundamental role in distance and velocity measurement.

[0028] Frequency-modulated continuous wave (FM-CW) radar generates continuous electromagnetic signals using a voltage-controlled oscillator (VCO) or a phase-locked loop (PLL) frequency synthesizer. Within one frequency modulation cycle, the signal frequency increases or decreases uniformly over time according to a preset slope. The amplitude of the transmitted signal remains constant to ensure stable transmission power, thereby improving the signal-to-noise ratio of the echo signal. The starting frequency of the signal determines the radar's center frequency band, typically located in the millimeter-wave band to achieve higher spatial resolution.

[0029] Within a single frequency modulation (FM) cycle, the frequency gradually changes from the starting frequency to the ending frequency within a fixed bandwidth range. The bandwidth directly determines the range resolution capability; the larger the bandwidth, the smaller the minimum range difference the system can distinguish. The duration of the FM cycle determines the rate of frequency change, i.e., the FM slope. A larger FM slope results in a more significant difference in echo frequency under the same target distance conditions, which is beneficial for improving ranging sensitivity.

[0030] When the transmitted signal encounters a target, it is reflected, and the echo signal has a time delay relative to the transmitted signal. The radar internally mixes the echo signal with the current transmitted signal. Since the transmission frequency changes over time, the echo signal corresponds to the transmission frequency at a previous moment, thus creating a frequency difference between the two. This frequency difference is proportional to the target distance. The target distance information can be obtained by performing spectral analysis on the mixed signal.

[0031] When a target exhibits radial motion, a Doppler frequency shift is superimposed on the echo signal. This shift alters the phase variation of the echo signal over time. By analyzing the phase change pattern across multiple frequency modulation cycles, target velocity information can be further extracted.

[0032] The phase term in the signal describes potential phase noise or artificially added modulation information within the system. In practical systems, phase noise originates from oscillator stability and environmental disturbances. By designing a reasonable phase stabilization mechanism, ranging errors can be reduced and system accuracy improved. Expressing the signal in complex exponential form allows for a unified description of the signal's amplitude and phase characteristics, facilitating subsequent mixing, filtering, and spectral processing. Therefore, this transmitted signal model provides the physical basis for subsequent distance and velocity analysis, and is a core prerequisite for realizing dynamic gesture perception.

[0033] The output of the residual block is represented as: H(x) = F(x) + x; Where x represents the input feature of the residual block; F(x) represents the residual mapping after two layers of convolution, batch normalization and activation function processing; H(x) represents the output feature of the residual block, which is obtained by adding the input feature and the residual mapping element by element.

[0034] This implementation method relates to the implementation mechanism of residual block structure in deep residual networks. The focus is on solving the problems of gradient decay and feature degradation during deep network training by introducing cross-layer connection.

[0035] In the specific implementation, the residual block first receives the input feature map from the previous layer. This input feature map contains spatial structure information and semantic information at a certain level. The input features first enter the first convolutional layer, where the convolution kernel slides across the feature map to extract local region features, generating a new feature map. After the convolution operation, batch normalization is performed to unify the feature distribution, reduce internal covariate bias, and improve training stability. Subsequently, nonlinear transformation capability is introduced through an activation function, enabling the network to express complex feature relationships.

[0036] The features processed by the first convolutional layer continue into the second convolutional layer for further feature extraction and mapping, followed by batch normalization. This results in a residual mapping. This mapping represents the incremental features of the input features after two layers of nonlinear transformation.

[0037] In the output stage, the original input features and the residual mapping result are added element-wise. Element-wise addition assumes that the input features and the residual mapping result are consistent in spatial size and number of channels. When their sizes are inconsistent, the convolution parameters can be adjusted by setting the stride, or a linear projection method can be used to match the dimensions of the input features. After the addition operation, the feature is processed by an activation function to form the final output features of the residual block.

[0038] With this structural design, the network does not need to directly learn the complete feature mapping relationship, but instead learns the difference between the input features and the target features. This difference is usually easier to optimize, thus reducing the training difficulty.

[0039] In the dual-channel gesture recognition network, residual blocks are applied to the distance-time feature channel and the Doppler-time feature channel, respectively, so that the original information of the two types of features is not lost during the process of deepening layer by layer, while obtaining a higher level of abstract representation.

[0040] During preprocessing, windowing is applied before spectral transformation of the fast time dimension data to reduce the impact of sidelobe leakage on range resolution and improve the main lobe concentration of the range spectrum.

[0041] In this embodiment, the spectral leakage problem in the range information extraction stage is optimized. The intermediate frequency signal obtained by the frequency-modulated continuous wave radar after mixing is essentially a discrete signal sampled within a finite time length. In actual processing, when performing spectral transformation on the fast time-dimensional data, the limited sampling length is equivalent to truncating the original continuous signal. Directly performing spectral transformation on the truncated signal will cause energy diffusion in the frequency domain, resulting in obvious sidelobes around the main frequency component, thereby reducing the range spectrum resolution.

[0042] To address the aforementioned issues, windowing is first applied to the fast time-dimensional data before spectral transformation. Specifically, the sampled data within each transmission cycle is multiplied by a preset window function, causing the signal to gradually and smoothly decay at time boundaries, rather than being abruptly truncated. This method reduces the distribution of sidelobe energy in the spectrum, concentrating the main energy more in the main lobe region.

[0043] In practice, the windowing step precedes the range spectrum calculation. For each transmission cycle, the sampled sequence is multiplied point-by-point by the window function coefficients to generate the windowed data sequence. Subsequently, a spectral transformation is performed on the windowed data to obtain the range spectrum result. Because windowing effectively suppresses spectral leakage, the main peak in the range spectrum is clearer, and the frequency position corresponding to the target distance is more stable.

[0044] In gesture recognition scenarios, hand targets are usually close to the radar and exhibit slight displacement changes. If the sidelobe energy is too strong, it can easily mask weak target information in adjacent range cells. Windowing processing can enhance the distinguishability between range cells, providing a more accurate range data foundation for subsequent moving target filtering and feature map construction.

[0045] Moving target filtering involves differential processing of echo data between consecutive frames to eliminate static background reflection signals, thereby enhancing the energy characteristics of dynamic gesture targets.

[0046] This implementation addresses the static background interference problem encountered by millimeter-wave radar in practical applications. In indoor or vehicle-mounted applications, fixed objects such as walls, desktops, and equipment casings continuously reflect radar signals, forming stable and high-intensity echo components. These static echoes remain essentially unchanged across multiple time frames, creating a stable background energy distribution in the range spectrum or range-time feature map, thereby masking the subtle changes in dynamic gesture targets.

[0047] To enhance the characteristics of dynamic targets, this implementation introduces moving target filtering in the preprocessing stage. Specifically, the continuously acquired radar echo data is first arranged frame by frame. For data from corresponding range cells or sampling points in two adjacent frames, a point-by-point difference operation is performed, i.e., the data from the previous frame is subtracted from the data from the current frame. Since the static background echo changes very little between adjacent frames, the difference result is close to zero, while the echo changes caused by dynamic hand movements are more significant, and the difference result retains a larger amplitude, thus highlighting the dynamic target component.

[0048] In the actual processing flow, this differential operation can be performed after the distance spectrum calculation is completed, or before the distance-time feature map is generated. For each distance unit, the amplitude difference between adjacent frames is calculated in the time dimension, forming a new dynamic feature sequence. After differential processing, the originally stable background reflection is significantly weakened, while the energy region that changes with gesture movement is enhanced.

[0049] After differential processing, the results can be processed by taking the absolute value of the amplitude or normalizing the energy to avoid the influence of sign caused by positive and negative differences and to enhance the trainability of the feature map. Subsequently, Doppler information extraction or feature map construction is performed to provide data input that highlights dynamic information for the subsequent dual-channel neural network. Through the above differential mechanism, the system can effectively suppress static interference and improve the separability of dynamic gesture targets in the feature space.

[0050] Distance-time feature maps and Doppler-time feature maps are normalized before being input into the neural network to ensure that the amplitude distribution among different samples is consistent, thereby improving the stability of model training.

[0051] This implementation addresses the issue of differences in amplitude scale among different gesture samples. In millimeter-wave radar gesture recognition, factors such as varying gesture execution speed, distance between the hand and the radar, reflection angle, and individual differences lead to significant differences in the amplitude distribution of the generated distance-time feature map and Doppler-time feature map. Directly inputting the original feature map into the neural network may result in large gradient fluctuations during model training, reduced convergence speed, and even affect the final recognition accuracy.

[0052] To address the aforementioned issues, normalization is performed before the feature maps are input into the neural network. Specifically, for each distance-time feature map or Doppler-time feature map, its amplitude range or overall distribution is first statistically analyzed. Then, amplitude mapping is performed according to a preset normalization rule. Normalization compresses the original amplitudes to a uniform numerical range, ensuring all samples are distributed within a similar data scale.

[0053] In practice, individual samples can be normalized to map their maximum and minimum values ​​to a uniform range; alternatively, standardization can be performed based on the statistical characteristics of the entire training dataset to distribute the sample data around a unified center. After normalization, the numerical ranges of the distance-time feature map and the Doppler-time feature map tend to be consistent, thus ensuring the balance of the dual-channel input data on the numerical scale.

[0054] After normalization, the two types of feature maps are then input into a dual-channel residual network for convolutional feature extraction. Because the input data distribution is more stable, the gradient changes more smoothly during forward and backward propagation, which is beneficial for the model to converge quickly and obtain stable training results.

[0055] In the dual-channel feature extraction model, each channel includes a convolutional layer, a batch normalization layer, an activation function layer, a max pooling layer, and multiple residual blocks connected in sequence, used to extract low-level features and high-level semantic features of gestures layer by layer.

[0056] This embodiment provides a detailed description of the network structure and hierarchical functions of the dual-channel feature extraction model. This dual-channel model extracts features independently from the distance-time feature map and the Doppler-time feature map, respectively. The two channels have identical structures but independent parameters to ensure that both types of physical features can be fully modeled.

[0057] In each channel, a convolutional layer is first set up. The convolutional layer performs local region scanning operations on the input feature map through the convolution kernel to extract edge contours, energy distribution areas, and local structural information in the gesture image. Since the distance-time feature map and the Doppler-time feature map are inherently image structural, the convolutional layer can effectively extract their spatial distribution features.

[0058] A batch normalization layer follows the convolutional layer. This layer adjusts the distribution of the convolutional output, keeping the numerical range of features in each channel stable and reducing the distribution offset between different batches of data, thereby improving the stability and convergence speed of the training process.

[0059] Subsequently, an activation function layer is set up. The activation function performs a non-linear transformation on the linear convolution results, enabling the network to express complex feature relationships. By introducing a non-linear mechanism, the model can learn the complex pattern changes of different gestures over time.

[0060] After initial feature extraction, a max pooling layer is applied. The max pooling layer reduces the feature map size by downsampling the maximum response value within a local region, while retaining the most significant feature responses, enhancing the translation invariance of features, and reducing computational complexity.

[0061] Following the aforementioned base layer, multiple residual blocks are stacked sequentially. By introducing a cross-layer connection mechanism, the residual blocks extract higher-level semantic features layer by layer while preserving the original feature information. Shallow residual blocks mainly extract local energy distribution features and basic motion trajectory information, while deep residual blocks can integrate time-series change trends and spatial distribution patterns to form a high-level semantic expression of the overall dynamic structure of the gesture.

[0062] In the dual-channel structure, the distance channel focuses on extracting trajectory features of the target's position changing over time, while the Doppler channel focuses on extracting information about velocity changes and motion direction. By constructing complete convolutional and residual extraction structures separately, it is ensured that the two types of features have formed sufficiently abstract high-dimensional representations before fusion.

[0063] When the number of channels or spatial dimensions of the input feature map and the output feature map of the residual block are inconsistent, a convolutional layer is set to perform dimension matching processing on the input features in order to achieve element-wise addition of the features.

[0064] This implementation describes the handling mechanism for situations where the input features and the residual mapping results in the residual structure have inconsistent dimensions. In deep convolutional neural networks, as the number of network layers increases, the spatial size of the feature map is often changed through stride convolution or pooling operations, and the number of feature channels may also be increased by increasing the number of convolutional kernels. When the input feature map of the residual block differs from the output feature map after convolution in terms of spatial size or number of channels, directly adding elements one by one will not satisfy the condition for adding at corresponding positions.

[0065] To ensure the proper execution of the residual connections, this implementation adds a dimension-matching path outside the residual branch. When a discrepancy is detected between the spatial dimensions or channel count of the input and output feature maps, a convolutional layer is applied to the input feature path for linear mapping. This convolutional layer typically uses the same number of channels as the output features, and its spatial dimensions are adjusted using appropriate stride parameters to ensure that the mapped input features maintain consistency with the residual mapping result in height, width, and channel count.

[0066] In the specific implementation, when the residual block needs to be downsampled, the convolutional layer can be set with a larger stride to simultaneously reduce the spatial resolution of the input feature map; when only the number of channels is inconsistent, a convolutional operation with a stride of one can be used to adjust the channels. After completing the dimension matching, the mapped input features and the residual mapping results are added element-wise, and then the final features are output from the activation function layer.

[0067] Through this structural design, residual connections can still be stably implemented even if the feature scale or channel depth changes at different stages of the network, thereby maintaining the continuity and trainability of deep networks.

[0068] In the dual-channel gesture recognition model, as the network deepens layer by layer, both the distance channel and the Doppler channel may experience changes in feature dimensions. Through the aforementioned dimension matching mechanism, it can be ensured that the residual block structure in the two channels always satisfies the condition of element-wise addition, thus guaranteeing the consistency of the network structure.

[0069] The dual-channel network employs a feature concatenation approach in the feature fusion stage, splicing the high-dimensional features extracted from the two channels along the channel dimension to preserve the complementary information of distance and velocity features.

[0070] This embodiment describes the fusion mechanism after dual-channel feature extraction. In this invention, the distance-time feature map mainly reflects the trajectory information of the gesture's spatial position changing over time, while the Doppler-time feature map mainly reflects the speed and direction changes of the gesture. These two types of features are physically distinct but complementary in distinguishing different gesture categories. Therefore, after feature extraction, a reasonable fusion strategy needs to be designed to fully preserve both types of feature information.

[0071] In the specific implementation, the dual-channel network passes through convolutional layers, normalization layers, activation layers, pooling layers, and multiple residual blocks, each outputting a high-dimensional feature representation. This high-dimensional feature is typically a one-dimensional feature vector, obtained by compressing the two-dimensional feature map using global average pooling. At this point, the distance channel outputs a feature vector representing the spatial distribution pattern of the gesture, while the Doppler channel outputs a feature vector representing the dynamic change pattern of the gesture.

[0072] In the fusion stage, a feature concatenation approach is used. Specifically, the two feature vectors are concatenated along the channel dimension, combining the distance and velocity feature vectors sequentially into a higher-dimensional fused feature vector. This concatenation process does not compress or weight the original features; instead, it fully preserves all feature information from both channels, ensuring that the fused feature includes both spatial position and velocity change information.

[0073] The fused feature vector is input into subsequent fully connected layers for discriminative learning. These fully connected layers automatically learn the correlation between distance and velocity features by performing linear mapping and nonlinear transformations on the concatenated high-dimensional features, thus forming a more discriminative comprehensive feature representation. Finally, a probability output layer determines the gesture category.

[0074] By employing a feature concatenation approach, we can avoid prematurely losing important information from any channel during the fusion stage, enabling the network to autonomously learn the optimal combination of the two types of features during training.

[0075] Random deactivation layers are set between fully connected layers to randomly mask some neuron connections during training, thereby reducing the risk of model overfitting and improving generalization ability.

[0076] This implementation addresses the overfitting problem that may occur during the training of deep neural networks through optimized design. After the dual-channel feature fusion is completed, the fused feature vector is input to a fully connected layer for high-dimensional mapping and classification. Because the fully connected layer has a large number of parameters, especially when the feature dimension is high, the model is prone to over-memorizing the training data, thereby reducing its ability to identify unknown samples.

[0077] To address the aforementioned issues, a random deactivation layer is introduced between adjacent fully connected layers. During the training phase, the random deactivation layer randomly selects some neurons and sets their outputs to zero according to a preset probability, temporarily blocking the connections between some neurons and preventing them from participating in forward and backward propagation calculations in the current training batch. The neurons that are not blocked continue to participate in feature mapping and parameter updates.

[0078] In the specific implementation, after the fused features are input into the first fully connected layer, they are processed by an activation function and then enter a random deactivation layer. The random deactivation layer generates a random mask based on a set deactivation ratio, filters the neuron outputs, sets the outputs of selected neurons to zero, and leaves the rest unchanged. The data then enters the next fully connected layer for further feature combination and classification learning.

[0079] During the model inference phase, i.e. the testing or actual deployment phase, the random deactivation mechanism no longer blocks neurons but enables all connections. At the same time, the neuron output is scaled proportionally to the deactivation ratio during the training phase to ensure that the output statistical characteristics are consistent between the training and inference phases.

[0080] In this way, the network cannot rely on a fixed neuron or local feature path to complete the classification task during training. Instead, it must learn more robust and distributed feature representations, thereby improving the overall generalization ability.

[0081] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet, characterized in that, include: S1. Construct a frequency modulated continuous wave radar gesture data acquisition system, use millimeter wave radar to collect echo signals of preset categories of gestures, and store the collected raw echo data. S2. Preprocess the raw echo data, including reconstructing the echo data into a data matrix arranged according to the transmission period, performing windowing processing and spectrum transformation along the fast time dimension to obtain distance information, performing moving target filtering to suppress static target interference, performing windowing processing and spectrum transformation along the slow time dimension to obtain speed information, and generating distance-time feature maps and Doppler-time feature maps corresponding to the gestures by accumulating them frame by frame. S3. The distance-time feature map and the Doppler-time feature map are cropped and normalized to serve as input data for the dual-channel neural network. S4. Construct a dual-channel feature extraction model based on residual networks. Set up two convolutional network channels with the same structure, and perform multi-layer convolution operations, pooling operations and residual block feature extraction on the distance-time feature map and the Doppler-time feature map respectively to obtain the corresponding high-dimensional feature representation. S5. Concatenate and fuse the feature vectors output from the two channels, and input the fused features into a fully connected layer for classification training. Obtain the recognition results of each gesture category through the probability output layer. S6. Optimize the parameters of the dual-channel feature fusion residual network using the training set, and verify the model's recognition accuracy and generalization ability using the test set to achieve automatic recognition of different gestures.

2. The FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet according to claim 1, characterized in that: The linear frequency modulated signal transmitted by the frequency modulated continuous wave radar is represented as: ； Where s(t) represents the transmitted signal; A represents the amplitude of the transmitted signal; f0 represents the starting frequency; B represents the frequency modulation bandwidth; T represents the duration of a single frequency modulation cycle; t represents the time variable; φ(t) represents the phase noise; and j represents the imaginary unit.

3. The FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet according to claim 1, characterized in that: The output of the residual block is represented as follows: H(x) = F(x) + x; Where x represents the input feature of the residual block; F(x) represents the residual mapping after two layers of convolution, batch normalization and activation function processing; H(x) represents the output feature of the residual block, which is obtained by adding the input feature and the residual mapping element by element.

4. The FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet according to claim 1, characterized in that: In the preprocessing process, windowing is performed before spectral transformation of the fast time dimension data to reduce the impact of sidelobe leakage on range resolution and improve the main lobe concentration of the range spectrum.

5. The FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet according to claim 1, characterized in that: The moving target filtering process includes differential operations on the echo data between consecutive frames to eliminate static background reflection signals, thereby enhancing the energy characteristics of dynamic gesture targets.

6. The FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet according to claim 1, characterized in that: The distance-time feature map and Doppler-time feature map are normalized before being input into the neural network to ensure that the amplitude distribution among different samples remains consistent, thereby improving the stability of model training.

7. The FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet according to claim 1, characterized in that: In the dual-channel feature extraction model, each channel includes a convolutional layer, a batch normalization layer, an activation function layer, a max pooling layer, and multiple residual blocks connected in sequence, used to extract low-level features and high-level semantic features of gestures layer by layer.

8. The FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet according to claim 7, characterized in that, When the number of channels or spatial dimensions of the input feature map and the output feature map of the residual block are inconsistent, a convolutional layer is set to perform dimension matching processing on the input features in order to achieve element-wise addition of the features.

9. The FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet according to claim 1, characterized in that: The dual-channel network employs a feature concatenation method in the feature fusion stage, splicing the high-dimensional features extracted from the two channels along the channel dimension to preserve the complementary information of distance and velocity features.

10. The FMCW radar gesture recognition method based on dual-channel feature fusion and ResNet according to claim 1, characterized in that: Random deactivation layers are set between the fully connected layers to randomly shield some neuron connections during training, thereby reducing the risk of model overfitting and improving generalization ability.

Images