Gesture control method and system based on UWB radar and multi-modal deep learning

Abstract

The application discloses a gesture control method and system based on UWB radar and multi-modal deep learning, and the method comprises the following steps: synchronously collecting UWB radar signals, infrared video streams and IMU signals of gestures through a UWB radar, an infrared camera and an inertial measurement unit, and performing timestamp alignment and labeling on the collected signals to construct a three-modal vehicle-mounted gesture dataset; performing pretreatment and feature extraction to obtain a UWB radar comprehensive feature set, an infrared vision comprehensive feature set and an IMU comprehensive feature set; inputting the feature set into a feature fusion and gesture recognition model based on cross-modal attention, obtaining a gesture category probability distribution through multi-modal joint feature representation, and converting the gesture category probability distribution into a function instruction based on a gesture-instruction mapping table; and training the model by using an adaptive reweighting and adversarial training strategy. The application solves the problems of low recognition accuracy and poor robustness of the prior art in a vehicle-mounted complex environment, and significantly improves the naturalness, safety and intelligent level of gesture interaction.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and in particular to a gesture control method and system based on UWB radar and multimodal deep learning. Background Technology

[0002] With the continuous development of intelligent vehicles and in-vehicle information systems, the interaction methods between drivers and these systems are also gradually upgrading. Traditional interaction methods mainly rely on physical buttons or touchscreens, which can easily distract drivers and increase operational risks. In recent years, although voice interaction has become more widespread, it still suffers from low recognition rates and frequent false triggers in noisy environments or multilingual scenarios. Therefore, developing a more natural, intuitive, and safe interaction method has become an important direction for the development of in-vehicle information systems.

[0003] Gesture interaction, as a contactless human-computer interaction method, allows drivers to operate in-vehicle information systems without taking their hands off the steering wheel or shifting their gaze. For example, common functions such as adjusting volume, switching navigation, and answering calls can all be completed with simple gestures, significantly reducing operational complexity and safety hazards. However, current gesture recognition methods based on single vision or inertial sensors are prone to instability or failure in complex lighting environments, occluded scenarios, and dynamic driving environments, making it difficult to meet the high real-time and reliability requirements of in-vehicle applications.

[0004] The invention with application number CN202410270481.1 provides a sensitivity-adjustable intelligent gesture control system, which realizes gesture interaction in a model display scene through a model display module, a camera module, a gesture recognition module, a frame rate adjustment module, a distance judgment module, a zoom module, and a precision control module. This system can adjust the sensitivity of gesture control through distance judgment and zoom control, thereby improving the interactive effect of model display. However, this method is mainly applicable to specific scenarios such as model display, and its application in complex in-vehicle environments is limited.

[0005] The invention with application number CN201810677591.4 provides a vehicle window control device and method based on Time-of-Flight (TOF) gesture recognition. This technology achieves gesture control of the vehicle window through an image recognition module, an image processing module, a TOF control module, and an execution module. This method can prevent the driver from looking down or operating blindly while driving, thus improving driving safety to a certain extent. However, this method mainly relies on a single TOF sensor for recognition, and its adaptability to lighting conditions, occlusion, and complex driving environments is insufficient.

[0006] Most existing technologies rely on single-modal sensors, which can easily lead to decreased recognition accuracy under varying lighting conditions, gesture occlusion, or dynamic driving environments. Their adaptability to complex gesture scenarios is limited. They cannot effectively support the complex gesture interaction requirements of multiple categories and actions. Most employ traditional image processing or single-modal recognition, making it difficult to capture the temporal and multi-dimensional features of gestures. The lack of adaptive learning to user habits, real-time optimization of recognition performance, and system-level optimization in real-world in-vehicle scenarios limits their application value in smart cockpits and autonomous driving scenarios.

[0007] Therefore, it is necessary to propose a solution to improve one or more problems existing in the above-mentioned related technical solutions.

[0008] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] The purpose of this disclosure is to provide a gesture control method and system based on UWB radar and multimodal deep learning, thereby overcoming, to at least some extent, one or more problems caused by the limitations and defects of related technologies.

[0010] According to one aspect of the embodiments of this disclosure, a gesture control method based on UWB radar and multimodal deep learning is provided, the method comprising the following steps: By simultaneously collecting UWB radar signals, infrared video streams, and IMU signals of gestures using UWB radar, infrared cameras, and inertial measurement units deployed in the driver's seat, and by performing timestamp alignment and gesture category labeling on the collected signals, a three-modal vehicle gesture dataset is constructed. The trimodal vehicle gesture dataset is preprocessed and features are extracted to obtain a trimodal integrated feature set containing UWB radar integrated feature set, infrared vision integrated feature set and IMU integrated feature set; A feature fusion and gesture recognition model based on cross-modal attention is constructed. The UWB radar integrated feature set, infrared vision integrated feature set and IMU integrated feature set are used as training samples, and the labeled gesture categories are used as training targets. The model is trained using a strategy based on adaptive optimization and adversarial training. The trained model is packaged into a lightweight inference engine and deployed in the in-vehicle infotainment system. The three-modal comprehensive feature set is acquired in real time. The trained model is used to perform dynamic weight allocation and feature interaction through intra-modal attention and cross-modal attention. Then, the in-vehicle gesture semantic enhancement multimodal Transformer recognition network outputs the probability distribution of gesture categories. Based on the gesture-command mapping table, it is converted into function commands. The function commands are used to realize real-time recognition of gestures and control of in-vehicle functions.

[0011] In an exemplary embodiment of this application, the step of simultaneously acquiring UWB radar signals, infrared video streams, and IMU signals of gestures by deploying a UWB radar, an infrared camera, and an inertial measurement unit in the driver's seat, and performing timestamp alignment and gesture category labeling on the acquired signals to construct a three-modal vehicle gesture dataset includes: The UWB radar sensor is deployed above the center console area in front of the driver's seat, the infrared camera is deployed on the left pillar of the steering wheel, and the inertial measurement unit is integrated into the smart bracelet worn by the driver to synchronously collect the micro-motion characteristics, spatial trajectory, contour movement, acceleration and angular velocity data of the gestures. The collected UWB radar signals, infrared video streams, and IMU signals are time-stamped and precisely aligned, and the gesture categories and gesture execution scenarios are labeled. The gesture categories include swiping, clicking, rotating, and grabbing, and the gesture execution scenarios include driving and parking.

[0012] In an exemplary embodiment of this application, the step of preprocessing and extracting features from the trimodal vehicle gesture dataset to obtain a trimodal integrated feature set including a UWB radar integrated feature set, an infrared vision integrated feature set, and an IMU integrated feature set includes: The UWB radar signal is preprocessed and reconstructed based on sparse representation and compressed sensing, and time-domain and frequency-domain features are extracted to form a comprehensive UWB radar feature set. The infrared video stream is denoised, gesture segmented, and key point tracked. Spatiotemporal features are extracted through a spatiotemporal neural network to form an infrared visual comprehensive feature set. The IMU signal is filtered and normalized, and motion features are extracted through a feature enhancement recurrent network to form a comprehensive IMU feature set.

[0013] In an exemplary embodiment of this application, the step of constructing a feature fusion and gesture recognition model based on cross-modal attention, using UWB radar integrated feature set, infrared visual integrated feature set, and IMU integrated feature set as training samples, and labeled gesture categories as training targets, and training the model using a strategy based on adaptive optimization and adversarial training, includes: An adaptive reweighting mechanism is adopted, which calculates the prediction confidence and class boundary distance of each training sample through sample difficulty assessment, assigns higher loss weights to high-difficulty samples based on the class boundary distance, and uses the cross-entropy loss function to calculate the classification loss. An adversarial training mechanism is introduced. The gradient of the classification loss on the multimodal joint feature representation is calculated by the fast gradient sign method. Adversarial perturbations are generated based on the gradient and the preset perturbation intensity coefficient. The adversarial perturbations are added to the multimodal joint feature representation to obtain adversarial training samples. The feature fusion and gesture recognition model based on cross-modal attention is trained by combining dynamic learning rate and batch size adjustment strategies.

[0014] In an exemplary embodiment of this application, the steps of inputting the real-time acquired trimodal comprehensive feature set into the trained model, dynamically allocating weights and interacting with features through intra-modal attention and cross-modal attention, then using an in-vehicle gesture semantic enhancement multimodal Transformer recognition network to output the gesture category probability distribution, and converting it into a function command based on a gesture-command mapping table include: The UWB radar integrated feature set, the infrared visual integrated feature set, and the IMU integrated feature set are combined using cross-modal attention-based feature fusion to obtain a multimodal joint feature representation; The multimodal joint feature representation is input into the multimodal Transformer recognition network, which outputs the probability distribution of gesture categories and converts them into function commands based on the gesture-command mapping table.

[0015] In an exemplary embodiment of this application, the step of obtaining a multimodal joint feature representation by fusing the UWB radar integrated feature set, the infrared visual integrated feature set, and the IMU integrated feature set using cross-modal attention-based feature fusion includes: Linear mapping is performed on the UWB radar integrated feature set, the infrared visual integrated feature set, and the IMU integrated feature set, respectively; The mapped features are sequentially subjected to intra-modal attention processing and cross-modal attention processing. The cross-modal attention processing includes multiple parallel attention heads, which are used to capture the correlation between features of different modalities. Each attention head generates independent cross-modal interaction weights. The intramodal attention output and crossmodal attention output are concatenated, and after layer normalization and residual connection, the multimodal joint feature representation is obtained.

[0016] In an exemplary embodiment of this application, the steps of inputting the multimodal joint feature representation into a multimodal Transformer recognition network, outputting a gesture category probability distribution, and converting it into a function instruction based on a gesture-instruction mapping table include: Construct a gesture semantic enhancement multimodal Transformer recognition network consisting of one-dimensional convolutional layers and attention distillation layers; The multimodal joint feature representation captures fine-grained semantic information output features of gesture actions through multi-layer encoding using multiple encoding blocks; wherein, each encoding block includes a multi-head self-attention layer, a gesture semantic enhancement multimodal Transformer recognition network layer, and a feedforward neural network layer; The output features are aggregated by a global average pooling layer to obtain a compact feature vector. The compact feature vector is then input into a fully connected layer and processed by a Softmax activation function to output the gesture category probability distribution. Based on the probability distribution of the gesture categories and the preset gesture-command mapping table, the identified gesture categories are converted into corresponding function commands.

[0017] In an exemplary embodiment of this application, the adjustment formula for the loss weight is: in, This represents the weight of the s-th sample. Indicates the basic weight. This represents the weighting adjustment coefficient. This represents the distance between the s-th sample and the class decision boundary.

[0018] In an exemplary embodiment of this application, the expression for calculating the classification loss using the cross-entropy loss function is as follows: in, This represents the classification loss, where N represents the batch size. This indicates that the s-th sample corresponds to the true label of the c-th class. Indicates the first Each sample corresponds to a predicted probability of class c. This represents the weight of the s-th sample.

[0019] According to another aspect of the embodiments of this disclosure, a gesture control system based on UWB radar and multimodal deep learning is provided, the system comprising: The data acquisition module includes a UWB radar sensor, an infrared camera, and an inertial measurement unit deployed in the driver's seat. It is used to simultaneously acquire UWB radar signals, infrared video streams, and IMU signals of gestures, and to perform time stamp alignment and annotation on the acquired signals to construct a three-modal vehicle gesture dataset. The data processing module is used to preprocess and extract features from the trimodal vehicle gesture dataset to obtain a UWB radar integrated feature set, an infrared vision integrated feature set, and an IMU integrated feature set. The feature fusion and gesture recognition module is used to input the UWB radar integrated feature set, the infrared vision integrated feature set, and the IMU integrated feature set into the feature fusion and gesture recognition model based on cross-modal attention, obtain the gesture category probability distribution through multimodal joint feature representation, and convert it into a function command based on the gesture-command mapping table; The training unit is used to train the feature fusion and gesture recognition model based on cross-modal attention using an adaptive reweighting and adversarial training strategy. The interactive control module is used to receive function commands output by the feature fusion and gesture recognition module, and control the in-vehicle infotainment system to realize real-time recognition of gestures and control in-vehicle functions through the function commands.

[0020] This application proposes a gesture control method and system based on UWB radar and multimodal deep learning. On the one hand, by fusing and recognizing feature fusion and gesture recognition models based on cross-modal attention of multi-sensor data from UWB radar, infrared vision, and IMU, it effectively overcomes the limitations of single-modal recognition technology in complex in-vehicle environments, achieving robust and accurate spatiotemporal feature capture and semantic understanding of gesture actions, significantly improving the accuracy and environmental adaptability of gesture recognition. On the other hand, by introducing adaptive reweighting and adversarial training strategies, it enhances the model's ability to focus on difficult samples, its defense against adversarial interference, and the convergence stability of the training process, thereby greatly improving the generalization performance and robustness of the recognition model. This ensures that the system can provide a highly reliable, low-latency real-time gesture interaction experience in real in-vehicle scenarios, while also taking into account driving safety and natural interaction. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 This diagram illustrates the steps of a gesture control method based on UWB radar and multimodal deep learning in an exemplary embodiment of this application. Figure 2 This diagram illustrates a gesture control method based on UWB radar and multimodal deep learning in an exemplary embodiment of this application. Figure 3 This diagram illustrates the actual deployment process of the gesture control method based on UWB radar and multimodal deep learning in an exemplary embodiment of this application. Figure 4This diagram illustrates a gesture control system based on UWB radar and multimodal deep learning in an exemplary embodiment of this application. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0024] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0025] This example implementation first provides a gesture control method based on UWB radar and multimodal deep learning. This method can be applied to a terminal device, such as a mobile terminal like a mobile phone, desktop computer, personal digital assistant, laptop, tablet, or smartwatch. (Reference) Figures 1-2 As shown, the method may include the following steps: Step S101: Simultaneously collect UWB radar signals, infrared video streams, and IMU signals of gestures by deploying UWB radar, infrared camera, and inertial measurement unit in the driver's seat, and perform time stamp alignment and gesture category labeling on the collected signals to construct a three-modal vehicle gesture dataset.

[0026] Step S102: Preprocess and extract features from the trimodal vehicle gesture dataset to obtain a trimodal integrated feature set containing UWB radar integrated feature set, infrared vision integrated feature set and IMU integrated feature set.

[0027] Step S103: Construct a feature fusion and gesture recognition model based on cross-modal attention. Use the UWB radar integrated feature set, infrared vision integrated feature set, and IMU integrated feature set as training samples, and the labeled gesture category as the training target. Use an adaptive optimization and adversarial training strategy to train the model.

[0028] Step S104: The trained model is packaged into a lightweight inference engine and deployed in the in-vehicle infotainment system. The three-modal comprehensive feature set is acquired in real time. The trained model is used to perform dynamic weight allocation and feature interaction through intra-modal attention and cross-modal attention. Then, the in-vehicle gesture semantic enhancement multimodal Transformer recognition network outputs the probability distribution of gesture categories. Based on the gesture-command mapping table, it is converted into function commands. The real-time recognition of gestures and control of in-vehicle functions are realized through the function commands.

[0029] This application proposes a gesture control method based on UWB radar and multimodal deep learning. On the one hand, by fusing and recognizing multi-sensor data from UWB radar, infrared vision, and IMU using cross-modal attention-based feature fusion and gesture recognition models, it effectively overcomes the limitations of single-modal recognition technology in complex in-vehicle environments. This achieves robust and accurate spatiotemporal feature capture and semantic understanding of gesture actions, significantly improving the accuracy and environmental adaptability of gesture recognition. On the other hand, by introducing adaptive reweighting and adversarial training strategies, it enhances the model's ability to focus on difficult samples, its defense against adversarial interference, and the convergence stability of the training process. This significantly improves the generalization performance and robustness of the recognition model, ensuring that the system can provide a highly reliable, low-latency real-time gesture interaction experience in real in-vehicle scenarios, while also taking into account driving safety and natural interaction.

[0030] Below, as Figures 1-4 As shown, a gesture control method based on UWB radar and multimodal deep learning proposed in this example embodiment will be described in more detail.

[0031] Step S101: Simultaneously collect UWB radar signals, infrared video streams, and IMU signals of gestures by deploying UWB radar, infrared camera, and inertial measurement unit in the driver's seat, and perform time stamp alignment and gesture category labeling on the collected signals to construct a three-modal vehicle gesture dataset.

[0032] Specifically, addressing the issues of susceptibility to interference and insufficient stability in single-sensor gesture recognition in vehicle environments, this study deploys three types of sensors—UWB radar, infrared camera, and inertial measurement unit (IMU)—at the driver's seat to collect multimodal raw data, including micro-motion features, spatial trajectories, contour movements, and acceleration and angular velocities of gestures. The collected UWB radar signals, infrared video streams, and IMU signals are precisely time-stamped and aligned. Experts then annotate the gesture categories and execution scenarios, including four types of gestures (swiping, clicking, rotating, and grasping) and two types of execution scenarios (driving and parking). Finally, the aligned multimodal data and their annotations are integrated to construct a three-modal vehicle gesture dataset: UWB radar, infrared vision, and IMU.

[0033] It's important to understand that the multi-source sensor deployment and data acquisition in the vehicle environment involves deploying three types of sensors: ultra-wideband radar (UWB radar), infrared cameras, and inertial measurement units (IMUs). The UWB radar sensor is mounted above the center console in front of the driver's seat, capturing the micro-motion characteristics and three-dimensional spatial trajectory data of the driver's gestures by emitting and receiving ultra-wideband electromagnetic waves. The infrared camera is deployed on the left pillar of the steering wheel, using infrared imaging technology to capture the contours and motion features of gestures in low-light / nighttime environments. The IMU sensor is integrated into the driver's smart bracelet, recording the acceleration and angular velocity signals of the hand in real time. All three sensors start acquiring data simultaneously: the UWB radar sensor and the IMU sensor acquire continuous time-series data at the same sampling frequency, while the infrared camera acquires video stream data at a fixed frame rate, together forming the multi-modal raw data.

[0034] It's also important to understand that data annotation and dataset integration involve precisely aligning the timestamps of the collected UWB radar signal data, infrared video stream data, and IMU signal data. A unified clock synchronization mechanism ensures a one-to-one correspondence between the three modalities of data at the same moment. Data samples are annotated according to the driver's gesture type and vehicle operating status. The annotations include gesture category and gesture execution scenario. Gesture categories are divided into four types: swipe, click, rotate, and grab. Gesture execution scenarios are divided into two types: while driving and while parked. The aligned UWB radar signal data, infrared video stream data, and IMU signal data, along with their corresponding annotation information, are integrated to form a "UWB radar-infrared vision-IMU" three-modal vehicle gesture dataset.

[0035] Step S102: Preprocess and extract features from the trimodal vehicle gesture dataset to obtain a trimodal integrated feature set containing UWB radar integrated feature set, infrared vision integrated feature set and IMU integrated feature set.

[0036] Specifically, the data obtained in step S101 is subjected to feature extraction based on multimodal signal analysis. First, adaptive Kalman filtering, time-domain synchronization, and background clutter suppression are applied to the UWB radar signal. Short-time Fourier transform and wavelet transform are used to extract time-frequency domain features. Sparse representation and compressed sensing methods are introduced to construct a low-sampling-rate signal reconstruction model. Time-domain and frequency-domain features are extracted based on the reconstructed signal to form a comprehensive UWB radar feature set. Second, nonlocal mean denoising, Gaussian mixture background modeling, and adaptive threshold segmentation are applied to the infrared video stream to obtain a gesture foreground image sequence. This sequence is input into the vehicle-mounted gesture recognition spatiotemporal neural network (VGRSNN) to extract spatiotemporal features. Dynamic features are calculated by combining the hand key point tracking results to form a comprehensive infrared visual feature set. Finally, moving average filtering and z-score normalization are applied to the IMU signal. The processed acceleration and angular velocity signals are concatenated into a joint sequence and input into the vehicle-mounted gesture feature enhancement recurrent network (VGFERN) to extract motion features, forming a comprehensive IMU feature set.

[0037] Step S102 includes sub-steps S2001-S2003: Step S2001: Preprocess the UWB radar signal and reconstruct the signal based on sparse representation and compressed sensing, and extract time-domain and frequency-domain features to form a comprehensive UWB radar feature set.

[0038] Specifically, the raw echo signals acquired by the UWB radar sensor are preprocessed. First, an adaptive Kalman filter algorithm is used for noise suppression to filter out electromagnetic interference signals in the vehicle environment. Then, time-domain synchronization of multiple frames of echo data is achieved based on the sensor trigger signal to ensure that the time base of the signal is consistent at different times. Finally, background clutter is suppressed using the background difference method to eliminate fixed clutter interference generated by the static structure of the vehicle body, resulting in the denoised UWB radar signal. For the denoised signal The time-frequency domain features were extracted using both Short-Time Fourier Transform (STFT) and Wavelet Transform to form an initial time-frequency feature matrix. A low-sampling-rate signal reconstruction model is constructed by introducing sparse representation and compressed sensing methods. This model uses the compressed sampled signal... Using the input as input, a high-quality gesture motion feature signal is reconstructed at a low sampling rate. The reconstruction formula is as follows: in This represents the reconstructed UWB radar gesture motion characteristic signal. The sparse basis representing the micro-motion features of hand gestures is generated through training with a large number of standard hand gesture motion signals. This indicates that UWB radar signals are in sparse base The sparsity coefficients are based on the reconstructed signal. Further, three time-domain features—peak intensity, rising edge slope, and duration—and three frequency-domain features—center frequency, bandwidth, and spectral entropy—are extracted from the signal, and compared with the initial time-frequency feature matrix. Fusion to form a comprehensive UWB radar feature set .

[0039] Step S2002: Denoise the infrared video stream, segment the gestures, and track key points. Extract spatiotemporal features through a spatiotemporal neural network to form an infrared visual comprehensive feature set.

[0040] Specifically, the video stream data acquired by the infrared camera is preprocessed. First, a non-local mean denoising algorithm is used to remove Gaussian noise from the video frames, resulting in denoised infrared video frames. ( (This represents the time index of the video frame); subsequently, a Gaussian mixture model is used for background modeling to generate a dynamic background template. Calculated using the frame difference method and The difference is used to perform gesture region segmentation by combining adaptive threshold segmentation, and the gesture foreground mask is obtained. Based on mask from Extracting gesture foreground images . Continuous gesture foreground image The Vehicle Gesture Recognition Spatiotemporal Neural Network (VGRSNN) is a sequence-input vehicle gesture recognition network that employs gesture keypoint perception on top of the traditional I3D network. This gesture keypoint perception outputs the coordinates of 21 hand keypoints through pre-trained hand pose estimation, and combines this with Kalman filtering to achieve keypoint trajectory tracking. The trajectory information is then encoded into feature vectors and fused with spatiotemporal convolutional features.

[0041] The network consists of four spatiotemporal convolutional blocks, each composed of a 3D convolutional layer, a batch normalization layer, and a GELU activation function, progressively extracting local spatial texture features, mesoscale motion features, and global spatiotemporal evolution features of gestures. The output feature map of the l-th spatiotemporal convolutional block... The calculation process is as follows: Here, Conv3D represents 3D convolution operation, BN represents batch normalization operation, and GELU is the Gaussian error linear unit activation function. Finally, the initial infrared visual features are output through a global average pooling layer. .

[0042] Combining attitude estimation and keypoint tracking results, two dynamic features are calculated: the rate of change of Euclidean distance and the amount of angle change between keypoints. Infrared visual integrated feature set .

[0043] Step S2003: Filter and normalize the IMU signal, and extract motion features through a feature enhancement recurrent network to form an IMU comprehensive feature set.

[0044] Specifically, the acceleration and angular velocity signals acquired by the IMU sensor are preprocessed. First, a moving average filtering algorithm is used to filter high-frequency noise. The filtering formula is as follows: in This represents the i-th acceleration signal value after filtering. This represents the filtered angular velocity signal value, and K represents the sliding window size. This represents the j-th original acceleration signal value. Let represent the j-th original angular velocity signal value. Then, the z-score normalization method is used to map the filtered signal to the standard normal distribution interval. The normalization formula is as follows: in This represents the normalized value of the i-th acceleration signal. This represents the normalized value of the i-th angular velocity signal. This represents the mean of the acceleration signal. This represents the standard deviation of the acceleration signal. This represents the mean of the angular velocity signal. The standard deviation of the angular velocity signal. The normalized acceleration signal sequence. and angular velocity signal sequence spliced ​​into an IMU joint signal sequence The input is a vehicle-mounted gesture feature enhancement recurrent network. This network adds a motion feature enhancement gate to the traditional GRU, dynamically adjusting the attention to acceleration and angular velocity signals through learnable parameters to enhance the extraction of hand dynamic motion features. The network contains a three-layer recurrent structure, with 32 hidden units in each layer, capturing the short-term dependence and long-term evolution features of gesture movements layer by layer, and finally outputting the IMU comprehensive feature set through a fully connected layer. .

[0045] Step S103: Construct a feature fusion and gesture recognition model based on cross-modal attention. Use the UWB radar integrated feature set, infrared vision integrated feature set, and IMU integrated feature set as training samples, and the labeled gesture category as the training target. Use an adaptive optimization and adversarial training strategy to train the model.

[0046] Specifically, an adaptive reweighting mechanism is adopted, which calculates the prediction confidence and class boundary distance of each training sample through sample difficulty assessment, assigns higher loss weights to high-difficulty samples based on the class boundary distance, and uses the cross-entropy loss function to calculate the classification loss; Specifically, during model training, a sample difficulty assessment module is introduced. This module determines the sample difficulty level by calculating the prediction confidence and class boundary distance for each training sample. Misclassified samples and boundary samples are assigned higher loss weights, and the weight adjustment formula is as follows: in This represents the weight of the s-th sample. Indicates the basic weight. This represents the weighting adjustment coefficient. This represents the distance between the s-th sample and the class decision boundary. The smaller the sample size, the more difficult the sample selection process. The larger the value, the greater the classification loss. The cross-entropy loss function is used to calculate the classification loss, and the formula for the loss function is as follows: in This represents the classification loss, where N represents the batch size. This represents the true label of the c-th class corresponding to the s-th sample (represented using one-hot encoding). This represents the predicted probability of the s-th sample corresponding to the c-th class, derived from the output of the improved multimodal Transformer recognition network.

[0047] An adversarial training mechanism is introduced. The gradient of the classification loss on the multimodal joint feature representation is calculated by the fast gradient sign method. Adversarial perturbations are generated based on the gradient and the preset perturbation intensity coefficient. The adversarial perturbations are added to the multimodal joint feature representation to obtain adversarial training samples. The feature fusion and gesture recognition model based on cross-modal attention is trained by combining dynamic learning rate and batch size adjustment strategies.

[0048] Specifically, an adversarial training mechanism is introduced to enhance model robustness. Adversarial examples are generated using the Fast Signed Gradient Method (FGSM). By calculating the gradient direction of the loss function relative to the input, small perturbations are added along that direction, forcing the model to learn more robust feature representations. In this vehicle-mounted gesture control system, FGSM is used to enhance the robustness of the original multimodal features. Add perturbation Generate adversarial features The calculation formula is as follows: in This indicates resistance to disturbances. Indicates the disturbance intensity coefficient. The classification loss represents the joint feature representation of multimodal modes. gradient, This represents the generated adversarial features. The in-vehicle gesture control recognition model is trained by inputting data to learn robust features on adversarial examples. A dynamic learning rate and batch size adjustment strategy is adopted, and the learning rate update formula is as follows: in This represents the learning rate in round t+1. This represents the learning rate in round t. This represents the learning rate adjustment factor. Indicates the first Average classification loss of the wheel, This represents the average classification loss in round t-1. The batch size is dynamically adjusted based on the rate of change of loss in the current training round. When the loss decreases slowly, the batch size is reduced to increase the parameter update frequency; when the loss fluctuates greatly, the batch size is increased to improve training stability.

[0049] Through the aforementioned adaptive reweighting and adversarial training strategies, a highly robust and accurate vehicle gesture control recognition model based on UWB radar and multimodal deep learning is finally generated. This model can accurately recognize various gesture commands and provide reliable control support for vehicle interaction systems.

[0050] Step S104: The trained model is packaged into a lightweight inference engine and deployed in the in-vehicle infotainment system. The three-modal comprehensive feature set is acquired in real time. The trained model is used to perform dynamic weight allocation and feature interaction through intra-modal attention and cross-modal attention. Then, the in-vehicle gesture semantic enhancement multimodal Transformer recognition network outputs the probability distribution of gesture categories. Based on the gesture-command mapping table, it is converted into function commands. The real-time recognition of gestures and control of in-vehicle functions are realized through the function commands.

[0051] Specifically, the feature fusion and gesture recognition model based on cross-modal attention includes: a feature fusion and multimodal Transformer recognition network based on cross-modal attention; The UWB radar integrated feature set, the infrared visual integrated feature set, and the IMU integrated feature set are used to obtain a multimodal joint feature representation by feature fusion based on cross-modal attention; Specifically, linear mapping is performed on the UWB radar integrated feature set, the infrared visual integrated feature set, and the IMU integrated feature set, respectively, to transform the UWB radar integrated feature set... Infrared visual integrated feature set Synthesis Feature Set with IMU The input cross-modal hierarchical attention fusion first performs linear mapping on the features of the three modalities, transforming them into a unified high-dimensional feature space. The mapping formula is as follows: in, This represents the mapped UWB radar feature vector. The weight matrix representing the characteristics of UWB radar. The bias vector representing the characteristics of UWB radar; This represents the mapped infrared visual feature vector. The weight matrix representing infrared visual features. The bias vector representing the infrared visual features; This represents the mapped IMU feature vector. The weight matrix representing the features of the IMU. The bias vector represents the IMU feature.

[0052] The mapped features are sequentially processed with intra-modal attention and cross-modal attention to construct a two-layer attention mechanism for feature fusion: the first layer is intra-modal attention, which strengthens the response of key feature components by calculating the self-attention weights within each modality feature; the second layer is cross-modal attention, which sets up 6 parallel attention heads, of which 2 focus on the spatial association between UWB radar and infrared vision features, 2 focus on the dynamic association between UWB radar and IMU features, and 2 focus on the temporal association between infrared vision and IMU features. Each attention head generates independent cross-modal interaction weights, calculated as follows: in This represents the cross-modal attention output between modal X and modal Y. This represents the query matrix generated from the modality X feature vector. This represents the key matrix generated from the eigenvectors of mode Y. This indicates the dimensions of the query matrix and the key matrix.

[0053] By concatenating the intra-modal attention outputs and cross-modal attention outputs, and then applying layer normalization and residual connections, a multimodal joint feature representation is obtained. .

[0054] The multimodal joint feature representation is input into the multimodal Transformer recognition network, which outputs the probability distribution of gesture categories and converts them into function commands based on the gesture-command mapping table.

[0055] Specifically, a gesture semantic enhancement multimodal Transformer recognition network is constructed, consisting of one-dimensional convolutional layers and attention distillation layers, to represent the joint features of the multimodal networks. The input is a multimodal Transformer recognition network for in-vehicle gesture semantic enhancement. This network introduces a gesture semantic enhancement module on the basis of a traditional Transformer encoder. This module consists of a one-dimensional convolutional layer and an attention distillation layer. The one-dimensional convolutional layer captures global gesture action features by expanding the feature receptive field; the attention distillation layer improves feature discriminativeness by transferring the attention weights of the pre-trained gesture model.

[0056] The multimodal joint feature representation captures fine-grained semantic information of gesture actions through multi-layer encoding using multiple encoding blocks. Each encoding block includes a multi-head self-attention layer, a gesture semantic enhancement multimodal Transformer recognition network layer, and a feedforward neural network layer; the network contains four encoding blocks. Through multi-layer encoding, the network can progressively deepen its understanding of cross-modal features and accurately capture fine-grained semantic information of gesture actions.

[0057] After encoding, the output features are aggregated by a global average pooling layer to obtain a compact feature vector. The input is then fed into a fully connected layer and processed by the Softmax activation function, outputting the probability distribution of gesture categories. (c represents the gesture category index, corresponding to four categories: swipe, click, rotate, and grab), and the category with the highest probability value is the recognition result.

[0058] Construct a gesture-command mapping table to establish a fixed association between the recognized gesture categories and the function commands of the vehicle information system, and realize the command mapping function of sliding to adjust the volume, clicking to answer the phone, rotating and zooming the navigation, and grabbing to switch menus.

[0059] The UWB radar integrated feature set, infrared vision integrated feature set, and IMU integrated feature set are input into a feature fusion and gesture recognition model based on cross-modal attention. First, the three-modal integrated feature sets are linearly mapped through cross-modal hierarchical attention fusion, and then dynamically weighted and interacted with each other through intra-modal attention and cross-modal attention to obtain a multimodal joint feature representation. Second, the joint features are input into an in-vehicle gesture semantic enhancement multimodal Transformer recognition network. The feature understanding is deepened step by step through four coding blocks to capture fine-grained semantic information of gestures. Finally, the gesture category probability distribution is output through global average pooling and Softmax classification, and then converted into functional commands based on a gesture-command mapping table.

[0060] The system integrates UWB radar sensors, infrared cameras, and IMU sensors into the in-vehicle infotainment system (IVI), achieving data communication via a CAN bus. The trained multimodal signal preprocessing module, feature fusion module, and gesture recognition model are packaged into a lightweight inference engine and deployed on the IVI main control chip to process multimodal data in real time and output gesture commands. The system adaptively optimizes the interaction strategy based on the driving scenario and ensures the continuity of interaction through the sensor status monitoring module, realizing reliable control functions such as sliding to adjust the volume, clicking to answer calls, rotating and zooming the map, and grabbing to switch menus.

[0061] Specifically, the multimodal sensor integration and system connection integrates UWB radar sensors, infrared cameras, and IMU sensors into the vehicle environment. The UWB radar sensor is fixed in the dashboard above the central control screen, the infrared camera is embedded in the multi-function button area on the left side of the steering wheel, and the IMU sensor establishes a wireless connection with the in-vehicle infotainment system (IVI) via Bluetooth. All three sensors communicate with the IVI system via a CAN bus. The sampling frequencies of the UWB radar sensor and IMU sensor are set to match the processing frequency of the IVI system, and the frame rate of the infrared camera adaptively adjusts according to ambient lighting conditions to ensure real-time and stable data transmission.

[0062] The model deployment and data processing flow encapsulates the trained multimodal signal preprocessing module, cross-modal feature fusion module, and gesture recognition model into a lightweight inference engine, deployed in the main control chip of the IVI system. The inference engine is optimized based on TensorRT to improve real-time processing speed. The IVI system receives raw data from the three sensors in real time. First, the preprocessing module filters, suppresses clutter, and performs sparse reconstruction on the UWB radar signal; denoises, models the background, and segments the gesture on the infrared video stream; and filters and normalizes the IMU signal, outputting comprehensive feature sets for UWB radar, infrared vision, and IMU, respectively. Then, the feature fusion module generates a multimodal joint feature representation. Finally, the gesture recognition model is input to obtain the gesture category, and then converted into the corresponding function instruction according to the mapping table.

[0063] In terms of interactive response and scenario application, after receiving the function command corresponding to the gesture, the IVI system immediately executes the corresponding operation. For example, it adjusts the audio volume when a swipe gesture is recognized, answers or hangs up a call when a tap gesture is recognized, zooms the navigation map when a rotation gesture is recognized, and switches the application menu when a grab gesture is recognized. The system then displays the operation feedback interface on the central control screen. The system optimizes response strategies for two scenarios: driving and parking. In the driving scenario, it prioritizes response speed and simplifies interactive feedback to avoid distracting the driver. In the parking scenario, it enhances the feedback of interactive details and supports more complex gesture combinations. The system has a built-in sensor status monitoring module that checks the data transmission status of each sensor in real time. When a sensor malfunction occurs, it immediately switches to a backup interaction mode to ensure the continuity and reliability of the in-vehicle human-machine interaction.

[0064] Furthermore, this example embodiment also provides a gesture control system based on UWB radar and multimodal deep learning, referencing... Figure 4 As shown, the system includes: The data acquisition module includes a UWB radar sensor, an infrared camera, and an inertial measurement unit deployed in the driver's seat. It is used to simultaneously acquire UWB radar signals, infrared video streams, and IMU signals of gestures, and to perform time stamp alignment and annotation on the acquired signals to construct a three-modal vehicle gesture dataset. The data processing module is used to preprocess and extract features from the trimodal vehicle gesture dataset to obtain a UWB radar integrated feature set, an infrared vision integrated feature set, and an IMU integrated feature set. The feature fusion and gesture recognition module is used to input the UWB radar integrated feature set, the infrared vision integrated feature set, and the IMU integrated feature set into the feature fusion and gesture recognition model based on cross-modal attention, obtain the gesture category probability distribution through multimodal joint feature representation, and convert it into a function command based on the gesture-command mapping table; The training unit is used to train the feature fusion and gesture recognition model based on cross-modal attention using an adaptive reweighting and adversarial training strategy. The interactive control module is used to receive function commands output by the feature fusion and gesture recognition module, and control the in-vehicle infotainment system to realize real-time recognition of gestures and control in-vehicle functions through the function commands.

[0065] To verify the effectiveness of the technical solution of this invention, a 20-day continuous test was conducted on a constructed in-vehicle gesture interaction test platform under simulated in-vehicle environment and real vehicle scenarios, covering various typical working conditions such as normal lighting, low-light nighttime, dynamic interference, and sensor malfunction. Data acquisition was carried out simultaneously using a UWB radar sensor, an infrared camera, and an IMU sensor, and gesture action baseline values ​​were obtained through a high-precision hand motion capture system. The dataset contains 24 scene variations, divided into training, validation, and test sets in a 7:2:1 ratio, covering samples of four types of gestures: swiping, clicking, rotating, and grasping.

[0066] 1. Overall performance of multimodal gesture recognition The performance of this method was validated in three dimensions: recognition accuracy, environmental adaptability, and operational robustness. The differences between this method and single-modal methods (UWB radar only, infrared vision only, IMU only) and baseline fusion methods were compared. Evaluation indicators: Gesture recognition accuracy: The percentage of samples that are correctly recognized; False recognition rate (FER): The percentage of samples that are incorrectly identified; Scene Adaptability Index (SAI): The coefficient of variation in accuracy under different scenarios; Table 1. Experimental Results of Comprehensive Performance of Multimodal Gesture Recognition 2. Performance of Multimodal Feature Fusion Module We conducted a specific validation of the cross-modal hierarchical attention fusion module, comparing its fusion performance with that of direct feature concatenation and single-modal feature input: Evaluation indicators: Feature alignment error (FAE): Distance deviation of cross-modal features in a uniform space; Modal Co-gain Gain (MCG): The performance improvement ratio of the fused mode relative to the single mode; Fusion computation latency: the time consumed by fusing features from a single sample.

[0067] Table 2. Experimental results of the multimodal feature fusion module performance. 3. Validation of the effectiveness of the model training strategy The adaptive reweighting and adversarial training strategies were validated, and the performance differences between the basic training strategy and the proposed training strategy were compared: Evaluation indicators: Hard sample identification accuracy (HSA): The rate at which misclassified / boundary samples are correctly identified; Combat Robustness Index (ARI): Accuracy retention rate against perturbations; Training convergence rate (CR): The number of iterations required to achieve stable accuracy.

[0068] Table 3. Experimental results verifying the effectiveness of the model training strategy. Experimental results show that the proposed method's multimodal gesture recognition performance is significantly better than that of single-modal schemes and baseline fusion methods, with particularly outstanding advantages in complex in-vehicle scenarios such as low light and dynamic interference, maintaining a recognition accuracy of over 93%. In real-vehicle testing, the system can achieve a real-time response within 15.6ms, meeting the real-time and reliability requirements of in-vehicle human-machine interaction.

[0069] Furthermore, the specific manner in which each module performs its operation in the apparatus described above has been described in detail in the embodiments of the method, and will not be elaborated upon here.

[0070] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units. Components shown as modules or units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0071] In an exemplary embodiment of this disclosure, an electronic device is also provided, which may include a processor and a memory for storing executable instructions of the processor. The processor is configured to perform the steps of the neural network-based real-time evaluation method for electronic countermeasures jamming effectiveness described in any of the above embodiments by executing the executable instructions.

[0072] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as “circuit,” “module,” or “system.”

[0073] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the above-described neural network-based real-time evaluation method for electronic countermeasures jamming effectiveness according to the embodiments of this disclosure.

[0074] In an exemplary embodiment of this disclosure, a computer storage medium is also provided, on which a computer program is stored, which, when executed by, for example, a processor, can implement the steps of the real-time evaluation method for electronic countermeasures jamming effectiveness based on neural networks described in any of the above embodiments.

[0075] In some possible implementations, various aspects of the present invention can also be implemented as a computer program product comprising a computer program or instructions which, when run on a terminal device, cause the terminal device to perform the steps described in the section on the real-time evaluation method for electronic countermeasures jamming effectiveness based on neural networks described in this specification, according to various exemplary embodiments of the present invention.

[0076] The aforementioned program products can be written in any combination of one or more programming languages ​​to perform the operations of this invention. These programming languages ​​include object-oriented programming languages—such as Java and C++—and conventional procedural programming languages—such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0077] The computer software product may be stored in a computer storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0078] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A gesture control method based on UWB radar and multimodal deep learning, characterized in that, Includes the following steps: By simultaneously collecting UWB radar signals, infrared video streams, and IMU signals of gestures using UWB radar, infrared cameras, and inertial measurement units deployed in the driver's seat, and by performing timestamp alignment and gesture category labeling on the collected signals, a three-modal vehicle gesture dataset is constructed. The trimodal vehicle gesture dataset is preprocessed and features are extracted to obtain a trimodal integrated feature set containing UWB radar integrated feature set, infrared vision integrated feature set and IMU integrated feature set; A feature fusion and gesture recognition model based on cross-modal attention is constructed. The UWB radar integrated feature set, infrared vision integrated feature set and IMU integrated feature set are used as training samples, and the labeled gesture categories are used as training targets. The model is trained using a strategy based on adaptive optimization and adversarial training. The trained model is packaged into a lightweight inference engine and deployed in the in-vehicle infotainment system. The three-modal comprehensive feature set is acquired in real time. The trained model is used to perform dynamic weight allocation and feature interaction through intra-modal attention and cross-modal attention. Then, the in-vehicle gesture semantic enhancement multimodal Transformer recognition network outputs the probability distribution of gesture categories. Based on the gesture-command mapping table, it is converted into function commands. The function commands are used to realize real-time recognition of gestures and control of in-vehicle functions.

2. The gesture control method based on UWB radar and multimodal deep learning according to claim 1, characterized in that, The steps of simultaneously acquiring UWB radar signals, infrared video streams, and IMU signals of gestures by deploying UWB radar, infrared cameras, and an inertial measurement unit in the driver's seat, and then performing timestamp alignment and gesture category labeling on the acquired signals to construct a three-modal vehicle gesture dataset include: The UWB radar sensor is deployed above the center console area in front of the driver's seat, the infrared camera is deployed on the left pillar of the steering wheel, and the inertial measurement unit is integrated into the smart bracelet worn by the driver to synchronously collect the micro-motion characteristics, spatial trajectory, contour movement, acceleration and angular velocity data of the gestures. The collected UWB radar signals, infrared video streams, and IMU signals are time-stamped and precisely aligned, and the gesture categories and gesture execution scenarios are labeled. The gesture categories include swiping, clicking, rotating, and grabbing, and the gesture execution scenarios include driving and parking.

3. The gesture control method based on UWB radar and multimodal deep learning according to claim 1, characterized in that, The steps for preprocessing and feature extraction of the trimodal vehicle gesture dataset to obtain a trimodal integrated feature set including UWB radar integrated feature set, infrared vision integrated feature set, and IMU integrated feature set include: The UWB radar signal is preprocessed and reconstructed based on sparse representation and compressed sensing, and time-domain and frequency-domain features are extracted to form a comprehensive UWB radar feature set. The infrared video stream is denoised, gesture segmented, and key point tracked. Spatiotemporal features are extracted through a spatiotemporal neural network to form an infrared visual comprehensive feature set. The IMU signal is filtered and normalized, and motion features are extracted through a feature enhancement recurrent network to form a comprehensive IMU feature set.

4. The gesture control method based on UWB radar and multimodal deep learning according to claim 1, characterized in that, The steps for constructing a feature fusion and gesture recognition model based on cross-modal attention, using UWB radar integrated feature sets, infrared visual integrated feature sets, and IMU integrated feature sets as training samples, and labeled gesture categories as training targets, and employing a strategy based on adaptive optimization and adversarial training to train the model include: An adaptive reweighting mechanism is adopted, which calculates the prediction confidence and class boundary distance of each training sample through sample difficulty assessment, assigns higher loss weights to high-difficulty samples based on the class boundary distance, and uses the cross-entropy loss function to calculate the classification loss. An adversarial training mechanism is introduced. The gradient of the classification loss on the multimodal joint feature representation is calculated by the fast gradient sign method. Adversarial perturbations are generated based on the gradient and the preset perturbation intensity coefficient. The adversarial perturbations are added to the multimodal joint feature representation to obtain adversarial training samples. The feature fusion and gesture recognition model based on cross-modal attention is trained by combining dynamic learning rate and batch size adjustment strategies.

5. The gesture control method based on UWB radar and multimodal deep learning according to claim 1, characterized in that, The steps of inputting the real-time acquired trimodal comprehensive feature set into the trained model, dynamically allocating weights and interacting with features through intra-modal attention and cross-modal attention, and then using an in-vehicle gesture semantic enhancement multimodal Transformer recognition network to output the gesture category probability distribution, and converting it into function commands based on the gesture-command mapping table, include: The UWB radar integrated feature set, the infrared visual integrated feature set, and the IMU integrated feature set are combined using cross-modal attention-based feature fusion to obtain a multimodal joint feature representation; The multimodal joint feature representation is input into the multimodal Transformer recognition network, which outputs the probability distribution of gesture categories and converts them into function commands based on the gesture-command mapping table.

6. The gesture control method based on UWB radar and multimodal deep learning according to claim 5, characterized in that, The steps of obtaining a multimodal joint feature representation by fusing the UWB radar integrated feature set, the infrared visual integrated feature set, and the IMU integrated feature set using cross-modal attention-based feature fusion include: Linear mapping is performed on the UWB radar integrated feature set, the infrared visual integrated feature set, and the IMU integrated feature set, respectively; The mapped features are sequentially subjected to intra-modal attention processing and cross-modal attention processing. The cross-modal attention processing includes multiple parallel attention heads, which are used to capture the correlation between features of different modalities. Each attention head generates independent cross-modal interaction weights. The intramodal attention output and crossmodal attention output are concatenated, and after layer normalization and residual connection, the multimodal joint feature representation is obtained.

7. The gesture control method based on UWB radar and multimodal deep learning according to claim 6, characterized in that, The steps of inputting the multimodal joint feature representation into a multimodal Transformer recognition network, outputting a gesture category probability distribution, and converting it into a function instruction based on a gesture-instruction mapping table include: Construct a gesture semantic enhancement multimodal Transformer recognition network consisting of one-dimensional convolutional layers and attention distillation layers; The multimodal joint feature representation captures fine-grained semantic information output features of gesture actions through multi-layer encoding using multiple encoding blocks; wherein, each encoding block includes a multi-head self-attention layer, a gesture semantic enhancement multimodal Transformer recognition network layer, and a feedforward neural network layer; The output features are aggregated by a global average pooling layer to obtain a compact feature vector. The compact feature vector is then input into a fully connected layer and processed by a Softmax activation function to output the gesture category probability distribution. Based on the probability distribution of the gesture categories and the preset gesture-command mapping table, the identified gesture categories are converted into corresponding function commands.

8. The gesture control method based on UWB radar and multimodal deep learning according to claim 4, characterized in that, The adjustment formula for the loss weight is: in, This represents the weight of the s-th sample. Indicates the basic weight. This represents the weighting adjustment coefficient. This represents the distance between the s-th sample and the class decision boundary.

9. The gesture control method based on UWB radar and multimodal deep learning according to claim 8, characterized in that, The expression for calculating the classification loss using the cross-entropy loss function is as follows: in, This represents the classification loss, where N represents the batch size. This indicates that the s-th sample corresponds to the true label of the c-th class. This represents the predicted probability of the s-th sample corresponding to the c-th class. This represents the weight of the s-th sample.

10. A gesture control system based on UWB radar and multimodal deep learning, used to implement the method described in any one of claims 1-9, characterized in that, The system includes: The data acquisition module includes a UWB radar sensor, an infrared camera, and an inertial measurement unit deployed in the driver's seat. It is used to simultaneously acquire UWB radar signals, infrared video streams, and IMU signals of gestures, and to perform time stamp alignment and annotation on the acquired signals to construct a three-modal vehicle gesture dataset. The data processing module is used to preprocess and extract features from the trimodal vehicle gesture dataset to obtain a UWB radar integrated feature set, an infrared vision integrated feature set, and an IMU integrated feature set. The feature fusion and gesture recognition module is used to input the UWB radar integrated feature set, the infrared vision integrated feature set, and the IMU integrated feature set into the feature fusion and gesture recognition model based on cross-modal attention, obtain the gesture category probability distribution through multimodal joint feature representation, and convert it into a function command based on the gesture-command mapping table; The training unit is used to train the feature fusion and gesture recognition model based on cross-modal attention using an adaptive reweighting and adversarial training strategy. The interactive control module is used to receive function commands output by the feature fusion and gesture recognition module, and control the in-vehicle infotainment system to realize real-time recognition of gestures and control in-vehicle functions through the function commands.

Images