A non-radar modality to radar modality based cross-modality knowledge distillation method

Abstract

The present application relates to the technical field of knowledge distillation, and particularly relates to a cross-modal knowledge distillation method based on non-radar modal to radar modal. The method comprises the following steps: acquiring synchronous historical visual modal data and historical radar modal data, and obtaining time-aligned visual input data and radar input data after preprocessing; constructing a teacher-student knowledge distillation network, taking the visual input data as the input of the teacher network and the radar input data as the input of the student network for training; the teacher network and the student network both adopt a space-time feature decoupling architecture, the teacher network extracts long-term time features of the visual modal, the student network extracts long-term time features of the radar modal, and the student network learns the long-term time features extracted by the teacher network through knowledge distillation; after the training is completed, only the student network is deployed for inference and recognition of the radar modal data. The present application realizes stable knowledge migration from visual to radar, significantly improves the radar recognition accuracy, and meets the edge lightweight deployment requirements.

Description

Technical Field

[0001] This invention relates to the field of knowledge distillation technology, and specifically to a cross-modal knowledge distillation method based on non-radar mode to radar mode. Background Technology

[0002] Human Activity Recognition (HAR) is a key technology in the field of artificial intelligence, with broad application prospects in contactless human-computer interaction, health monitoring, and smart homes. Currently, HAR technology mainly relies on visual sensors or emerging radar sensors for data collection and recognition.

[0003] Vision-based HAR technology is relatively mature, capable of capturing rich texture, color, and appearance features to achieve high-precision behavior recognition. However, visual modalities are sensitive to environmental changes; their recognition performance drops sharply in low-light, backlit, or physically obstructed environments. Furthermore, the deployment of visual sensors in privacy-sensitive areas such as homes and hospitals is severely limited. In contrast, radar-based HAR technology (such as millimeter-wave radar) offers advantages such as strong environmental robustness, immunity to lighting conditions, support for non-line-of-sight detection, and effective protection of user privacy. However, due to the sparseness and low signal-to-noise ratio of radar signals, its extracted feature representation capabilities are weak, often resulting in single-modal recognition accuracy failing to meet practical application requirements. Therefore, single-modal HAR technology consistently struggles to simultaneously achieve high accuracy, strong robustness, and privacy security.

[0004] To integrate the advantages of different modalities, multimodal fusion technology has been proposed, such as joint inference using visual and radar data. While multimodal fusion can improve recognition accuracy, it faces three major challenges in practical deployment: First, high hardware synchronization costs. Visual and radar sensors typically require high-precision hardware triggering or GPS for time synchronization, increasing device complexity and cost. Second, difficulty in aligning heterogeneous features. The pixel features of vision and the distance-Doppler features of radar exhibit significant physical heterogeneity, making effective alignment in the feature space difficult. Third, excessive edge computing load. The computational demands of processing multiple high-dimensional data streams simultaneously far exceed the capacity of edge devices such as smart bracelets and home gateways, resulting in excessive inference latency and failing to meet real-time requirements.

[0005] To address the aforementioned issues, this field proposes the concept of Cross-Modal Knowledge Distillation (CMKD), which utilizes multimodal data during the training phase and uses only a single lightweight modality during the deployment phase. However, most existing CMKD methods are designed for scenarios where the feature differences between modalities are small, such as distillation from RGB (Red, Green, Blue) images to optical flow maps or depth maps. When applied to highly heterogeneous modal pairs like vision and radar (the former represented as a 2D pixel matrix, and the latter as a range-Doppler map or point cloud), traditional distillation methods often struggle to achieve effective knowledge transfer due to the inability to directly align spatial features, and may even lead to a performance degradation of the student network due to feature confusion.

[0006] Therefore, overcoming the significant feature differences between visual and radar modalities, achieving stable knowledge transfer from vision to radar, and enabling radar single-modal models to inherit the recognition accuracy of visual modalities while taking into account the environmental adaptability and privacy protection advantages of radar, and meeting the lightweight requirements of edge deployment, has become an urgent technical problem to be solved. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a cross-modal knowledge distillation method based on non-radar modality to radar modality. During the training phase, this method utilizes the non-radar modality as the teacher network and the radar modality as the student network for knowledge transfer. During the deployment phase, only the radar modality is used for inference. By constructing a spatiotemporally decoupled teacher-student network architecture, and extracting long-term temporal features shared between the visual and radar modalities as a knowledge transfer bridge, this method effectively overcomes the feature differences between strongly heterogeneous modalities, achieving stable knowledge transfer from non-radar modality to radar modality. While preserving the robustness and privacy protection advantages of the radar environment, it significantly improves recognition accuracy and meets the requirements for lightweight edge deployment.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A cross-modal knowledge distillation method based on non-radar mode to radar mode includes the following steps:

[0010] S1. Acquire synchronized historical visual modal data and historical radar modal data;

[0011] S2. Preprocess the acquired historical visual modal data and historical radar modal data to obtain time-aligned visual input data and radar input data;

[0012] S3. Construct a teacher-student knowledge distillation network, which includes a teacher network and a student network; use visual input data as input to the teacher network and radar input data as input to the student network for training.

[0013] Both the teacher network and the student network adopt a decoupled architecture that separates the spatial feature extraction module and the temporal feature extraction module. The teacher network is used to extract long-term temporal features of the visual modality, and the student network is used to extract long-term temporal features of the radar modality. Knowledge distillation enables the student network to learn the long-term temporal features extracted by the teacher network.

[0014] After training is complete, the student network is deployed only for inference and recognition of radar modal data.

[0015] Further, in S2, the preprocessing includes:

[0016] Historical radar mode data is coherently demodulated, filtered and amplified, and subjected to range-dimensional fast Fourier transform and Doppler-dimensional fast Fourier transform to generate range-Doppler maps; multiple consecutive frames of range-Doppler maps are stacked in time series to generate range-time-Doppler maps;

[0017] The distance-time-Doppler image and historical visual modal data are processed by frame rate alignment, size unification and channel normalization to obtain time-aligned radar input data and visual input data.

[0018] Furthermore, the long-term time characteristics include at least one of motion speed, acceleration change, motion continuity, pause rhythm, and motion start and end times.

[0019] Furthermore, the decoupled architecture separating the spatial feature extraction module and the temporal feature extraction module specifically includes: the spatial feature extraction module is used to extract the spatial features of each modality, and the temporal feature extraction module is used to extract long-term temporal features; during the knowledge distillation process, the student network uses the output of the temporal feature extraction module as the classification basis.

[0020] Furthermore, the spatial feature extraction module of the teacher network adopts a ResNet-50 backbone network, and the spatial feature extraction module of the student network adopts a ResNet-18 backbone network; the temporal feature extraction modules of the teacher network and the student network adopt the same network structure.

[0021] Furthermore, in S3, the training employs a combined loss function to optimize the student network. This combined loss function includes a cross-entropy loss based on the true labels of historical radar modal data and a KL divergence loss based on the soft labels output by the teacher network. The KL divergence loss is used to convey the dark knowledge extracted by the teacher network that is related to long-term temporal features.

[0022] Furthermore, the combined loss function Represented as:

[0023]

[0024] In the formula, For cross-entropy loss, For KL divergence loss, The weighting factor is the cross-entropy loss factor. This is the weighting factor for the KL divergence loss.

[0025] A cross-modal knowledge distillation system based on non-radar mode to radar mode for implementing the above method includes:

[0026] The data acquisition module is used to acquire synchronized historical visual modal data and historical radar modal data;

[0027] The data preprocessing module is used to preprocess the acquired historical visual modal data and historical radar modal data to obtain time-aligned visual input data and radar input data.

[0028] The neural network training module is used to build and train the teacher-student knowledge distillation network;

[0029] Deploy the inference module to perform inference and recognition of radar modal data by deploying only the student network after training is complete.

[0030] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the above-described method.

[0031] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0032] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0033] (1) This invention analyzes the physical characteristics of visual and radar modalities and finds that although they cannot be aligned in terms of spatial features, they have semantic consistency in terms of long-term temporal features. Based on this finding, this invention designs a decoupled architecture that separates the spatial feature extraction module and the temporal feature extraction module. During knowledge distillation, only shared long-term temporal features are transferred, avoiding interference from non-alignable heterogeneous spatial features on knowledge transfer. Experimental results show that after adopting the method of this invention, the radar single-modal recognition accuracy is improved from 74.33% to 83.83%, an improvement of 9.50%, which is significantly better than existing cross-modal distillation methods, realizing stable cross-modal knowledge transfer from vision to radar.

[0034] (2) In the training phase, this invention utilizes dual-modal data from both vision and radar, transferring the dark knowledge of the visual modality to the radar modality through knowledge distillation. In the deployment phase, only a lightweight single-modal radar model is used for inference and recognition, and the teacher network is discarded. This design retains the accuracy advantages of multimodal fusion while avoiding the problems faced by multimodal fusion in actual deployment, such as high hardware synchronization costs, difficulty in aligning heterogeneous features, and insufficient edge computing capabilities. At the same time, the deployment phase uses only the radar modality, fully inheriting the inherent advantages of radar modality, such as strong environmental robustness (unaffected by lighting or occlusion), good privacy protection (no imaging function), low hardware cost, and support for non-line-of-sight detection. It is suitable for deployment in privacy-sensitive scenarios such as homes and hospitals, as well as for edge devices.

[0035] (3) Radar single-mode data has feature sparsity, and the model performance is prone to saturation. Simply increasing the model complexity or performing intra-modal distillation cannot effectively improve the recognition accuracy. This invention compensates for the information deficiency of radar single-mode by introducing rich dark knowledge of visual modality, enabling the radar model to learn the fine-grained temporal feature patterns contained in visual modality, thereby breaking through the performance bottleneck of radar single-mode and achieving a leap in accuracy. Attached Figure Description

[0036] Figure 1 A flowchart of a cross-modal knowledge distillation method provided in an embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example

[0039] refer to Figure 1 This embodiment provides a cross-modal knowledge distillation method based on non-radar mode to radar mode, which is performed according to the following steps:

[0040] S1. Historical Modal Data Acquisition

[0041] The system acquires synchronized historical visual modal data and historical radar modal data of the human body. In this embodiment, the radar modal data is a millimeter-wave radar modal data.

[0042] Historical visual modal data was acquired using the Microsoft Azure Kinect DK depth camera; the camera is a 12-megapixel RGB camera that provides a color data stream that matches the depth data stream, and an accelerometer and gyroscope (IMU) are used for sensor orientation and spatial tracking.

[0043] Historical radar modal data (raw intermediate frequency signals from radar representing human activity) is acquired using a single antenna on the AWR6843 radar RF board. The main parameter settings for the AWR6843 radar RF board are as follows:

[0044] The linear frequency modulation (FM) pulse number is 128, the starting frequency is 60 GHz, the FM slope is 149.909 MHz / µs, the FM time is 26.68 µs, the FM cycle is 1526.68 µs, and the frame period is 200 ms.

[0045] To improve the robustness of the dataset and simulate environmental interference of visual modalities in real-world scenarios, this embodiment designed three lighting conditions for the data collection environment: normal lighting (indoor natural light, illuminance 550-560 lux), backlighting (light source located behind the volunteer, illuminance 795-805 lux), and low light (only the lowest brightness indoor light is turned on, illuminance 100-105 lux).

[0046] Twenty volunteers were selected (10 males and 10 females, aged 18-26, with no physical disabilities). They were tasked with writing six types of letters (AF), repeating each action 10 times. Each action lasted 1.8-8.0 seconds, corresponding to 9-40 frames of radar data (matched to a 200ms frame period). 1200 pairs of raw (video-radar) data were collected under each lighting condition, for a total of 3600 pairs of raw data.

[0047] S2, Data Preprocessing

[0048] The historical visual modal data and historical radar modal data acquired by S1 are preprocessed to obtain time-aligned visual input data and radar input data. The specific preprocessing steps are as follows:

[0049] S201. Historical radar mode data consists of the raw intermediate frequency (IF) signal from the radar. Its initial state contains a significant amount of environmental noise, clutter interference, and unwanted carrier components. First, coherent demodulation is performed, using a reference signal in phase and frequency with the transmitted signal to mix the IF signal, separating the target echo signal from the carrier and extracting the baseband signal containing target characteristics. Subsequently, a low-frequency filtering and amplification stage is used, employing a low-pass filter to remove residual high-frequency noise, electromagnetic interference, and clutter components after demodulation. Simultaneously, the useful baseband signal is amplified to obtain a clean baseband signal, ensuring that the amplitude of the clean baseband signal is within the effective dynamic range for subsequent processing.

[0050] The obtained clean baseband signal needs to be processed by Fast Fourier Transform (FFT) sequentially along the range dimension and Doppler dimension to achieve accurate analysis of the target's spatial and motion features:

[0051] Range-dimensional FFT processing: Based on the "range-time-frequency" correlation characteristics of electromagnetic wave propagation, the frequency changes of the intermediate frequency signal directly map the range difference between the target and the radar. By performing range-dimensional FFT processing on the clean baseband signal, the time-domain signal is converted into a frequency-domain signal, and the amplitude and phase information of each frequency component are quickly extracted. Then, combined with the radar system parameters, frequency-range mapping calculation is performed to generate an accurate range feature matrix.

[0052] Doppler FFT processing: Based on the linear frequency modulated pulse number set in S1 (Chirp Number = 128 in this embodiment), a stable phase difference exists between the continuous linear frequency modulated pulse signal emitted by the radar and the target echo signal. This phase difference is directly caused by the Doppler effect generated by the radial motion of the target. By performing Doppler FFT processing on the phase difference sequence of the continuous linear frequency modulated pulse signal, the phase domain information is converted into frequency domain Doppler features, and then based on the Doppler frequency shift... With radial velocity Physical conversion relationship (in (Using the radar wavelength), the radial velocity of the target is accurately retrieved to form a velocity characteristic matrix.

[0053] After two-dimensional FFT processing using the range and Doppler dimensions, a single-frame range-Doppler (RD) map is constructed by using the range feature as the horizontal axis and the Doppler feature (reflecting the target's radial velocity) as the vertical axis. To capture the temporal dynamic features of the target's motion, all continuously acquired single-frame RD maps are stacked in time sequence, and temporal integration is performed on the multi-frame RD maps to generate a range-time-Doppler (RTD) map. This map completely records the dynamic evolution of the target's range change trend and radial velocity throughout the entire motion cycle in a three-dimensional feature space.

[0054] The historical visual modal data acquired by S202 and S1 is RGB video. RGB video images are extracted from the historical visual modal data frame by frame to obtain the RGB video frame sequence corresponding to the RTD graph time.

[0055] S203. Perform frame rate alignment, size unification, and channel normalization on the RTD image obtained in S201 and the RGB video frame sequence obtained in S202 to obtain time-aligned radar input data and visual input data.

[0056] Specifically, action segments (in this embodiment, the start and end times of each letter in a single handwriting stroke constitute one action segment) are segmented by associating device trigger timestamps with file names. Due to individual differences in human movement speed, the number of frames in each segmented action segment fluctuates within a certain range (in this embodiment, the number of frames is between 9 and 40). To balance action integrity and computational overhead, all action segments are first unified to a preset number of frames (36 frames in this embodiment) through interpolation. The preset number of frames must cover the complete cycle of the action segment. Then, the frames are downsampled to a target number of frames (23 frames in this embodiment). This target number of frames ensures that temporal features are not lost while reducing computational costs, thus achieving frame rate standardization for different action segments.

[0057] To ensure the consistency of input size between visual data and radar data and eliminate format conflicts caused by modal differences, the resolution of RGB video frames is adjusted to a preset size (224×224 in this embodiment), and the RTD image is scaled to a preset size (224×224) using linear stretching technology, ensuring that the input specifications of the two modal data are consistent in the spatial dimension.

[0058] To address individual differences and inherent modal characteristics, visual and radar data are normalized. Specifically, this involves employing... Normalize each channel separately, where, For normalized data, The original data, As the preset mean, This is the preset standard deviation.

[0059] For visual data, the three-channel mean is set to a preset mean vector ([0.485, 0.456, 0.406] in this embodiment), and the standard deviation is set to a preset standard deviation vector ([0.229, 0.224, 0.225] in this embodiment). For RTD graphs, the single-channel mean is set to a preset mean (0.5 in this embodiment), and the standard deviation is set to a preset standard deviation (0.5 in this embodiment). This processing aims to offset the data distribution shift caused by individual and modal differences, ensure the consistency of multimodal data in the feature space, and improve the model's generalization ability.

[0060] S204, Dataset Partitioning

[0061] After the above preprocessing, time-aligned visual input data and radar input data are obtained. The preprocessed data is divided into training and test sets proportionally. In this embodiment, the preprocessed data is divided into training and test sets at a ratio of 5:1, with 3000 pairs in the training set and 600 pairs in the test set.

[0062] S3, Neural Network Training and Deployment

[0063] S301. Construct a teacher-student knowledge distillation network, which includes a teacher network and a student network.

[0064] Both the teacher and student networks employ a decoupled architecture that separates spatial feature extraction modules from temporal feature extraction modules. Spatial features of the visual modality and the radar modality (such as texture and color in vision, and distance and intensity in radar) cannot be directly aligned. However, the long-term temporal features of the two modalities (such as motion speed, acceleration changes, action continuity, pause rhythm, and start and end times of actions) have semantic consistency and can serve as a shared carrier for cross-modal knowledge transfer. Therefore, by separating and extracting spatial features from long-term temporal features, only shared long-term temporal features are transferred during knowledge distillation, avoiding interference from misaligned heterogeneous spatial features on knowledge transfer.

[0065] The spatial feature extraction module of the teacher network uses a ResNet-50 backbone network to extract spatial features of the visual modality; the temporal feature extraction module uses fully connected layers (in this embodiment, the input dimension is 2048 and the output dimension is 1024) to extract long-term temporal features of the visual modality. The specific structural parameters of the teacher network are shown in Table 1.

[0066] Table 1 Teacher Network Structure Parameters

[0067]

[0068] The spatial feature extraction module of the student network uses a ResNet-18 backbone network to extract spatial features of the radar modes; the temporal feature extraction module uses a fully connected layer (in this embodiment, the input dimension is 512 and the output dimension is 512) to extract long-term temporal features of the radar modes. The number of parameters in the student network is approximately 54.3% of that in the teacher network, meeting the requirements for lightweight deployment on edge devices. The specific structural parameters of the student network are shown in Table 2.

[0069] Table 2 Student Network Structure Parameters

[0070]

[0071] It should be noted that the temporal feature extraction modules of both the teacher and student networks employ the same network structure (both are fully connected layers) to ensure the spatial alignment of long-term temporal features during knowledge distillation. The spatial feature extraction module of the teacher network has higher complexity than that of the student network, used to fully extract dense features of the visual modality; the spatial feature extraction module of the student network has lower complexity, used to prevent overfitting of sparse features of the radar modality. In both the teacher and student networks, the output of the spatial feature extraction module serves as the input of the temporal feature extraction module, and the output of the temporal feature extraction module serves as the input of the classifier.

[0072] S302. Use the time-aligned visual input data from the training set obtained in S2 as the input to the teacher network and the radar input data as the input to the student network for training.

[0073] In this application, hard labels refer to the true category labels of the radar data, using one-hot encoding, meaning the probability of the correct category is 1, and the probability of other categories is 0. Soft labels refer to the probability distribution output by the teacher network, containing information on the similarity relationships between categories (i.e., hidden knowledge), and are in the form of a probability distribution vector for all categories, with the sum of all probability values ​​being 1.

[0074] During training, a combined loss function is used. The optimization objective is to minimize the input loss function, optimizing the student network so that it simultaneously learns the real labels from the radar data and the soft labels from the teacher network. The parameters of the teacher network are frozen during training, while only the parameters of the student network are updated. A combined loss function is used. Including cross-entropy loss and KL divergence loss, it is expressed as:

[0075]

[0076] In the formula, For cross-entropy loss, For KL divergence loss, The weighting factor is the cross-entropy loss factor. The weighting factor for KL divergence loss is... =1, in this embodiment =0.4, =0.6.

[0077] Cross-entropy loss: Based on the true hard labels (one-hot encoding) of radar data, it ensures the accurate classification boundary of students' online learning activities, and is expressed as:

[0078]

[0079] in, This represents the true label probability distribution of radar activity (the correct category probability is 1, and the others are 0). For student network The predicted probability distribution of action types, In this embodiment, the human behavior recognition task has a total of 6 types of actions (such as handwritten letters AF), which is the total number of classification categories. =6.

[0080] KL divergence loss: Based on the soft labels (temperature smoothed) of the teacher network output, it is used to convey the dark knowledge extracted by the teacher network that is related to long-term time features, so that the output distribution of the student network approximates the output distribution of the teacher network, and is expressed as:

[0081]

[0082] in, For temperature parameters, Provide teachers with online experience and temperature control. Smoothed Probability of soft tags for class actions For students, network output at the same temperature Smoothed Probability of class action, The smaller the value, the higher the consistency of knowledge distribution between the student network and the teacher network.

[0083] S303. After training, the teacher network is discarded, and only the student network is deployed to the target edge device (such as a smart bracelet, home gateway, etc.). During deployment, the radar sensor collects radar modal data in real time. The collected radar modal data is preprocessed and then input into the trained student network for inference, outputting behavior recognition results. This deployment method retains the environmental robustness and privacy protection advantages of radar modal data, while meeting the lightweight requirements of edge devices.

[0084] To verify the effectiveness of the method of this invention, a comparative experiment was designed in this embodiment. The experiment used a training set (3000 pairs) and a test set (600 pairs) partitioned by S204. All experiments were conducted under the same hardware environment and software framework to ensure the fairness of the comparison results.

[0085] (1) Verification of the effectiveness of spatiotemporal feature decoupling

[0086] The following control group was set up in this experiment:

[0087] (a) Traditional 3DCNN model group: 3DCNN models that employ undecoupled spatiotemporal features, including:

[0088] 3DCNN_2L: A two-layer spatiotemporal hybrid extraction network where spatial and temporal features are coupled within the network;

[0089] 3DCNN_LTF: Loss-based temporal feature extraction network, with weak temporal feature extraction capabilities;

[0090] 3DCNN_GTFE: Global Temporal Augmentation Network, which has long-term modeling capabilities, but spatiotemporal features are still coupled.

[0091] (b) Model group of the present invention: The spatiotemporal feature decoupling architecture proposed in this invention is used to extract time features based on two basic units:

[0092] BiLSTM_Net: Temporal Feature Extraction Module Based on Bidirectional Long Short-Term Memory Network (BiLSTM)

[0093] Transformer_Net: A temporal feature extraction module based on the Transformer network.

[0094] The comparative variable in the experiment was whether spatiotemporal feature decoupling was achieved. All models were trained and tested on the same dataset, and the recognition accuracy was recorded with and without distillation. The experimental results are shown in Table 3.

[0095] Table 3. Experimental results verifying the effectiveness of spatiotemporal feature decoupling.

[0096]

[0097] As shown in Table 3, the recognition accuracy of traditional 3DCNN models did not improve after using cross-modal knowledge distillation. Specifically, 3DCNN_2L decreased by 0.34%, 3DCNN_LTF by 4.67%, and 3DCNN_GTFE by 2.33%. This indicates that because the spatial features of the visual modality and the radar modality cannot be directly aligned, the traditional spatiotemporally coupled 3DCNN models are affected by heterogeneous spatial features during knowledge distillation, leading to knowledge transfer failure.

[0098] In comparison, the spatiotemporal feature decoupling models (BiLSTM_Net and Transformer_Net) proposed in this invention achieve significant improvements in recognition accuracy after using cross-modal knowledge distillation. Specifically, BiLSTM_Net improves from 74.33% to 83.83%, an increase of 9.50%; and Transformer_Net improves from 65.83% to 75.17%, an increase of 9.34%.

[0099] The above results fully verify the effectiveness of the spatiotemporal feature decoupling architecture proposed in this invention. By using an independent temporal feature extraction module, the inherent defects of spatiotemporal coupling are avoided, enabling knowledge distillation to focus on long-term temporal features shared by the two modalities, thereby achieving stable knowledge transfer.

[0100] (2) Comparison with existing cross-modal distillation methods

[0101] This experiment compares the method of this invention with existing cross-modal distillation methods. Existing cross-modal distillation methods include Kendall Rank Correlation Distillation (KRC), Decoupled Knowledge Distillation (DKD), Bidirectional Distillation (BD), and Comparative Representation Distillation (CRD). All comparison methods use the same teacher network (ResNet-50) and student network (ResNet-18) and are tested on the same dataset. The experimental results are shown in Table 4.

[0102] Table 4. Experimental results compared with existing cross-modal distillation methods

[0103]

[0104] As shown in Table 4, existing cross-modal distillation methods have low adaptability to knowledge transfer in the highly heterogeneous vision-to-radar mode, with distillation generally resulting in a decrease in recognition accuracy or only a slight improvement. This indicates that most existing methods are designed for similar modalities and lack specific processing for the highly heterogeneous characteristics of vision and radar modes.

[0105] In comparison, the method of this invention achieves an accuracy of 83.83% after distillation, significantly outperforming existing methods. This result fully demonstrates the superiority of the method of this invention. Through systematic analysis of prior knowledge from two modalities, long-term temporal features are identified as the only shareable knowledge carrier, and a spatiotemporal feature decoupling architecture is designed specifically to significantly improve the knowledge transfer capability from vision to radar.

[0106] (3) Verification of radar single-mode performance saturation

[0107] This experiment involved intramodal knowledge extraction. Only radar single-mode data was used to train the student network and perform intramodal distillation. The experimental results are shown in Table 5.

[0108] Table 5. Experimental Results of Radar Single-Mode Performance Saturation Verification

[0109]

[0110] As shown in Table 5, intramodal knowledge distillation can only achieve limited performance improvement (7.67% improvement under the Transformer architecture), and its effect is still lower than the cross-modal distillation method proposed in this invention. More importantly, intramodal distillation is essentially model compression, which cannot introduce new information to compensate for the information deficiencies of single-mode radar.

[0111] This result verifies the necessity of the cross-modal distillation proposed in this invention. By introducing rich dark knowledge from the visual modality (such as the speed change pattern of actions, acceleration patterns, pause rhythms, etc.), it compensates for the information defects of the single-modality radar, thereby achieving a leap in accuracy.

[0112] The specific embodiments of the present invention are provided to enable those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention.

[0113] It should be understood that the present invention is not limited to the content already described above, and various modifications and changes can be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A cross-modal knowledge distillation method based on non-radar mode to radar mode, characterized in that, Includes the following steps: S1. Acquire synchronized historical visual modal data and historical radar modal data; S2. Preprocess the acquired historical visual modal data and historical radar modal data to obtain time-aligned visual input data and radar input data; S3. Construct a teacher-student knowledge distillation network, which includes a teacher network and a student network; use visual input data as input to the teacher network and radar input data as input to the student network for training. Both the teacher network and the student network adopt a decoupled architecture that separates the spatial feature extraction module and the temporal feature extraction module. The teacher network is used to extract long-term temporal features of the visual modality, and the student network is used to extract long-term temporal features of the radar modality. Knowledge distillation enables the student network to learn the long-term temporal features extracted by the teacher network. After training is complete, the student network is deployed only for inference and recognition of radar modal data.

2. The cross-modal knowledge distillation method based on non-radar mode to radar mode as described in claim 1, characterized in that, In S2, the preprocessing includes: Historical radar mode data is coherently demodulated, filtered and amplified, and subjected to range-dimensional fast Fourier transform and Doppler-dimensional fast Fourier transform to generate range-Doppler maps; multiple consecutive frames of range-Doppler maps are stacked in time series to generate range-time-Doppler maps; The distance-time-Doppler image and historical visual modal data are processed by frame rate alignment, size unification and channel normalization to obtain time-aligned radar input data and visual input data.

3. The cross-modal knowledge distillation method based on non-radar mode to radar mode as described in claim 1, characterized in that, The long-term time characteristics include at least one of the following: motion speed, acceleration change, motion continuity, pause rhythm, and the start and end times of the motion.

4. The cross-modal knowledge distillation method based on non-radar mode to radar mode according to claim 1, characterized in that, The decoupled architecture separating the spatial feature extraction module and the temporal feature extraction module specifically includes: the spatial feature extraction module is used to extract the spatial features of each modality, and the temporal feature extraction module is used to extract long-term temporal features; during the knowledge distillation process, the student network uses the output of the temporal feature extraction module as the classification basis.

5. The cross-modal knowledge distillation method based on non-radar mode to radar mode according to claim 1, characterized in that, The spatial feature extraction module of the teacher network adopts a ResNet-50 backbone network, and the spatial feature extraction module of the student network adopts a ResNet-18 backbone network; the temporal feature extraction modules of the teacher network and the student network adopt the same network structure.

6. The cross-modal knowledge distillation method based on non-radar mode to radar mode according to claim 1, characterized in that, In S3, the training employs a combined loss function to optimize the student network. This combined loss function includes cross-entropy loss based on the true labels of historical radar modal data and KL divergence loss based on the soft labels output by the teacher network. The KL divergence loss is used to convey the dark knowledge extracted by the teacher network that is related to long-term temporal features.

7. The cross-modal knowledge distillation method based on non-radar mode to radar mode as described in claim 6, characterized in that, The combined loss function Represented as: In the formula, For cross-entropy loss, For KL divergence loss, The weighting factor is the cross-entropy loss factor. This is the weighting factor for the KL divergence loss.

8. A cross-modal knowledge distillation system based on non-radar mode to radar mode for implementing the method of any one of claims 1 to 7, characterized in that, include: The data acquisition module is used to acquire synchronized historical visual modal data and historical radar modal data; The data preprocessing module is used to preprocess the acquired historical visual modal data and historical radar modal data to obtain time-aligned visual input data and radar input data. The neural network training module is used to build and train the teacher-student knowledge distillation network; Deploy the inference module to perform inference and recognition of radar modal data by deploying only the student network after training is complete.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method of any one of claims 1 to 7.

Images