Radar target detection method and device based on cross-view cross-modal contrast learning and medium
By employing a self-supervised learning framework based on cross-view and cross-modal contrastive learning, the problems of dependence on labeled data and viewpoint limitations in radar target detection are solved, achieving a high-efficiency improvement in detection performance under limited labeled data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2024-11-29
- Publication Date
- 2026-05-29
AI Technical Summary
Existing radar target detection technologies rely on expensive human annotation and complex equipment, and self-supervised learning methods are limited to a single perspective, failing to effectively utilize the multi-view information of radar data, resulting in poor generalization ability.
A self-supervised learning framework of cross-view and cross-modal contrastive learning is adopted. By constructing a radar target detection model, self-supervised pre-training is performed using different views of radar data and visible light images. Combined with cross-view and cross-modal contrastive learning, the relationship between different views and modalities is captured, reducing the dependence on labeled data.
It significantly improves the performance of radar target detection, especially in the case of limited labeled data, enhancing feature representation capabilities and detection accuracy.
Smart Images
Figure CN122115820A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology and relates to radar target detection, specifically a radar target detection method, device and medium based on cross-view and cross-modal comparative learning. Background Technology
[0002] Frequency-modulated continuous wave radar is crucial for advanced driver assistance systems and has attracted significant attention compared to lidar and cameras due to its low cost and robustness to adverse weather conditions. The raw radar signal data (ADC) can be converted into a range-azimuth-Doppler (RAD) tensor using a fast Fourier transform. The RAD tensor contains information such as the distance, direction, and velocity of nearby objects and is widely used in radar perception tasks such as target detection, target tracking, and semantic segmentation.
[0003] However, radar signals are difficult for humans to understand, making human annotation extremely time-consuming and expensive. The lack of large-scale labeled radar data limits the performance of radar perception tasks. Some automated or semi-automated annotation methods utilize supervision from other modalities to generate labels for radar data. However, these methods also suffer from several problems. First, they require complex equipment, such as stereo cameras and LiDAR, leading to high data collection costs. Second, projecting masks from other modalities onto radar data requires a complex calibration and computation process, which can result in potential errors such as missing objects or inaccurate bounding boxes. Manual verification is often required to correct these errors. Finally, their generalization ability is generally poor. Due to variations in hardware and setup, proposed annotation methods are often only applicable to specific datasets. On the other hand, some studies have explored self-supervised learning for radar perception to address the challenge of lacking large-scale labeled data. However, these approaches are limited to a single viewpoint, such as bird's-eye view (BEV), range-azimuth (RA) view, or range-Doppler (RD) view, failing to provide comprehensive predictions and ignoring the correspondence between different viewpoints in the radar data. Summary of the Invention
[0004] This invention aims to address some problems in radar target detection technology by proposing a self-supervised approach to learn radar visually aligned RAD representations, thereby reducing the dependence on annotations and improving model performance without increasing computational overhead.
[0005] The technical solution of this invention is: a radar target detection method based on cross-view and cross-modal contrastive learning, comprising the following steps:
[0006] A self-supervised learning radar target detection model is constructed for radar target detection based on the range-azimuth-Doppler RAD tensor, including the following steps:
[0007] Step S1: Construct a radar dataset, perform data preprocessing, take the modulus and logarithm of the complex data of the radar RAD data, and then normalize it. The preprocessed data is divided into a training set and a fine-tuning dataset.
[0008] Step S2: Construct a radar target detection model and perform self-supervised pre-training, including positive and negative sample pair construction, feature extraction, cross-view contrastive learning, and cross-modal contrastive learning;
[0009] Step S2.1: Use transpose transformation on the preprocessed RAD data to obtain data from different viewpoints, including RA, RD, and AD views, which, together with the corresponding visible light images, are used as input to the radar target detection model;
[0010] Step S2.2: Use RadarResNet as the backbone network of the radar target detection model, extract the features of the RA, RD and AD three views respectively, use the encoder of YOLOv9 to extract the visible light image features, and project the four features onto a unified representation space through the projection head;
[0011] Step S2.3: Take three views from the same RAD cube and the corresponding RGB images as positive sample pairs, and data from different samples as negative sample pairs. Perform cross-view comparison learning on the features of different views encoded in step S2.2, and perform cross-modal comparison learning on the features of the RA view and the image features.
[0012] Step S2.4: Self-supervised learning is performed using the contrastive learning loss function, and the model is trained using the Adam optimizer until the loss converges, and the pre-trained model weights are obtained.
[0013] Step S3: Fine-tune the downstream task of the radar target detection model, load the pre-trained model weights from step S2, and use different task head prediction results for different downstream tasks;
[0014] Step S3.1: Use the RAD data from the RA view as input to the model;
[0015] Step S3.2: Extract RAD features using the backbone network of the pre-trained RA view from step S2;
[0016] Step S3.3: Use different task heads to predict results for different target detection tasks;
[0017] Step S3.4: Supervise network learning using the loss function corresponding to the task, including target box regression loss, target existence loss and classification loss, train the network and complete fine-tuning.
[0018] As a further improvement of the present invention, in step S2.2, the radar data... Features are extracted separately using multiple RadarResNet backbone networks, and then passed through projection layers. Project onto a unified representation space:
[0019]
[0020] In the above formula, V represents a view of the input RAD tensor, including RA, RD, and AD. This represents the vector after feature extraction and projection. It is a projection layer composed of two linear layers;
[0021] For image data Backbone network using YOLOv9-c Extract image features during training. Pre-trained weights from COCO object detection are used, and the parameters are kept frozen before being passed through a projection layer. Projecting onto a unified representation space:
[0022]
[0023] As a further improvement of the present invention, in step S2.3, positive sample pairs are formed for different views of the same RAD cube in a small batch, and negative sample pairs are formed for views from different RAD cubes. For two views... and The contrast loss is calculated using the NT-Xent loss proposed in SimCLR:
[0024]
[0025] In the above formula, 1 i≠k ∈{0,1} is the indicator function, which takes the value 1 if and only if i≠k, N is the mini-batch size, ω is the temperature coefficient, s(·) is the cosine similarity function, V1 and V2 represent views of the input RAD tensor, V1 and V2 are views of different kinds, and the contrastive learning loss function between views V1 and V2 is expressed as:
[0026]
[0027] For two pairs of views, RA and RD, RD and AD, and RA and AD, calculate their contrast loss. The total cross-view loss is obtained by linear combination. and To calculate:
[0028]
[0029] Similarly, for and Calculate the contrast loss using NT-Xent loss:
[0030]
[0031] The cross-modal contrast loss is expressed as:
[0032]
[0033] As a further improvement of the present invention, step 2.4 involves cross-view comparison loss. Combined with cross-modal contrastive loss Together, as the total loss function during the self-supervised pre-training phase, it is expressed as follows:
[0034]
[0035] The present invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded and executed by the processor to implement the radar target detection model of the radar target detection method based on cross-view and cross-modal contrastive learning. The model is pre-trained by self-supervised training using unlabeled registered radar data and visible light images, outputs the pre-trained weights of the model, and is used to output radar target detection results after fine-tuning in downstream tasks.
[0036] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores at least one instruction or at least one program, wherein when the at least one instruction or at least one program is executed, it implements the radar target detection model of the radar target detection method based on cross-view cross-modal contrastive learning.
[0037] The beneficial effects of this invention are as follows: This invention provides a self-supervised learning method for radar target detection, which has the following advantages compared with the prior art.
[0038] (1) This invention proposes a novel RAD tensor self-supervised learning framework, which allows the model to learn transferable representations from unlabeled data, reducing the dependence on labeled data and significantly improving the performance of radar target detection on limited labeled data.
[0039] (2) The cross-view contrast learning proposed in this invention captures the correspondence between different views of the RAD cube, thereby enabling the RAD model to obtain spatial distribution knowledge from multiple views.
[0040] (3) The cross-modal contrastive learning proposed in this invention captures the correspondence between RAD and visible light images, thereby transferring prior semantic knowledge from vision to radar.
[0041] (4) This invention combines cross-view and cross-modal comparative learning to enhance the feature representation capability of radar, allowing the RAD model to learn transferable feature representations from unlabeled data, and significantly improving the accuracy of downstream radar target detection tasks when labeled data is limited. Attached Figure Description
[0042] Figure 1 This is the overall flowchart of the present invention.
[0043] Figure 2 This is a diagram of the radar backbone network structure of the present invention.
[0044] Figure 3 The following are the detection results of the present invention in downstream tasks: (a) is the visible light image of the corresponding radar monitoring, (b) is the detection result of the existing radar detection model RADDet, and (c) is the detection result of the present invention. Detailed Implementation
[0045] This invention constructs a self-supervised learning framework for radar target detection based on range-azimuth-Doppler (RAD) tensors; it captures the relationships between different views of the RAD cube through cross-view contrastive learning, enabling the model to acquire spatial distribution knowledge from multiple views; and it captures the relationships between RAD and visual modalities through cross-modal contrastive learning, enabling the model to acquire semantic knowledge from the visual modalities.
[0046] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.
[0047] This invention proposes a radar target detection method based on cross-view and cross-modal contrastive learning. A self-supervised learning radar target detection model is constructed for radar target detection based on the range-azimuth-Doppler RAD tensor. The overall process is as follows: Figure 1 As shown, it includes the following steps.
[0048] Step S1: Construct a radar dataset, preprocess the publicly available millimeter-wave radar dataset, and divide the dataset for model pre-training and fine-tuning.
[0049] Step S1.1: Radar data preprocessing, taking the modulus and logarithm of the complex radar data, and then normalizing it.
[0050] Step S1.2: Dataset partitioning. The RADDet dataset is used. The model is pre-trained using all the unlabeled data in the training set, and fine-tuned for downstream tasks using a small portion of labeled data. Specifically, fine-tuning is performed on 5%, 10%, 20%, 50%, and 100% of the training set.
[0051] Step S2: Self-supervised pre-training of the model, including positive and negative sample pair construction, feature extraction, cross-view contrastive learning, and cross-modal contrastive learning.
[0052] Step S2.1: Use transpose transformation on the preprocessed RAD data to obtain data from different perspectives, including RA, RD, and AD views, and input them along with the corresponding visible light images into the network.
[0053] Step S2.2: Use RadarResNet as the backbone network to extract features from the three views respectively. Figure 2 The detailed network architecture is described. The residual blocks are based on ResNet, and the downsampling blocks are max pooling layers. In the RA view, the RadResNet backbone remains unchanged from the original setup. For the RD and AD views, since the channel dimension is much larger than in the RA view, fewer residual blocks and smaller kernel-sized convolutional blocks are used to simplify the model structure in order to reduce computational overhead.
[0054] Radar data under V-view Features are extracted separately using multiple RadarResNet backbone networks, and then passed through projection layers. Projecting onto a unified representation space:
[0055]
[0056] In the above formula, V represents a view of the input RAD tensor, including RA, RD, and AD. This represents the vector after feature extraction and projection. It is a projection layer composed of two linear layers.
[0057] For the visual branch, any pre-trained image backbone can be used to extract image features. This invention uses a YOLOv9 encoder to extract visible light image features. For image data... Backbone network using YOLOv9-c Extract image features during training. Pre-trained weights from COCO object detection are used, and the parameters are kept frozen before being passed through a projection layer. Projecting onto a unified representation space:
[0058]
[0059] Step S2.3: Treat the three views from the same RAD cube and the corresponding RGB images in a small batch as positive sample pairs, and the data from different samples as negative sample pairs. Perform cross-view comparison learning on the features of different views encoded in step S2.2, and perform cross-modal comparison learning on the features of the RA view and the image features.
[0060] Positive samples are formed for different views of the same cube within a mini-batch, while views of different cubes are formed. For two views... and The contrast loss is calculated using the NT-Xent loss proposed in SimCLR:
[0061]
[0062] In the above formula, 1 i≠l ∈{0,1} is the indicator function, which takes the value 1 if and only if i≠k, N is the mini-batch size, τ is the temperature coefficient, s(·) is the cosine similarity function, V1 and V2 represent views of the input RAD tensor, and V1 and V2 are different types of views, including RA, RD and AD. The contrastive learning loss function between views V1 and V2 can be expressed as:
[0063]
[0064] For the pairs of views RA and RD, RD and AD, and RA and AD, we calculate their contrast loss. The total cross-view loss is obtained through linear combination. and To calculate:
[0065]
[0066] Similarly, for and Calculate the contrast loss using NT-Xent loss:
[0067]
[0068] Cross-modal contrast loss can be expressed as:
[0069]
[0070] Step S2.4: Perform self-supervised learning using a contrastive learning loss function. The cross-view contrastive loss and cross-modal contrastive loss are combined as the total loss function for the self-supervised pre-training stage, as shown below:
[0071]
[0072] Train the model using the Adam optimizer until the loss converges, and finally obtain the pre-trained model parameters.
[0073] Step S3: Fine-tune the model for downstream tasks. Load the pre-trained model weights from step S2 and use different task head prediction results for different downstream tasks.
[0074] Step S3.1: Use the RAD data from the RA view as input to the model.
[0075] Step S3.2: Use the backbone network of the pre-trained RA view from step S2 to extract RAD features.
[0076] Step S3.3: Use different task head prediction results for different downstream target detection tasks, such as fine-tuning for 3DRAD target detection task and 2D BEV target detection task respectively.
[0077] Step S3.4: Supervise network learning using the loss function corresponding to the task, including target box regression loss, target existence loss and classification loss, and train the network to complete the fine-tuning for the downstream task.
[0078] This invention also provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the radar target detection model in the above method, performing self-supervised pre-training using unlabeled registered radar data and visible light images, outputting the pre-trained weights of the model, and using them for fine-tuning in downstream tasks to output radar target detection results. The electronic device of this invention is intended to represent various forms of digital electronic computer devices, such as laptops, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, smartphones, wearable devices, and other similar data computing devices. The processor can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processors include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc.
[0079] This invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer processor, implements the radar target detection model of the radar target detection method based on cross-view, cross-modal contrastive learning. The program code for implementing the method of this invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0080] This invention combines cross-view and cross-modal contrastive learning methods to construct a radar self-supervised learning framework. This framework allows the model to learn transferable representations from unlabeled data, reducing reliance on labeled data and improving radar target detection performance on limited labeled data. To verify the effectiveness of the salient model, experiments were conducted on the widely used RADDet dataset. The visualization results are shown below. Figure 3 As shown, Figure 3 (a) is the visible light image corresponding to radar monitoring. Figure 3 (b) shows the detection results of the existing radar detection model RADDet. Figure 3 (c) represents the detection results of this invention. The upper and lower lines represent two detection scenarios. It can be seen that there are missed detections and false detections in the results of RADDet. However, the model of this invention, which has been pre-trained under self-supervision, achieves accurate detection and improves performance compared to the RADDet model.
Claims
1. A radar target detection method based on cross-view, cross-modal contrastive learning, characterized in that, A self-supervised learning radar target detection model is constructed for radar target detection based on the range-azimuth-Doppler RAD tensor, including the following steps: Step S1: Construct a radar dataset, perform data preprocessing, take the modulus and logarithm of the complex data of the radar RAD data, and then normalize it. The preprocessed data is divided into a training set and a fine-tuning dataset. Step S2: Construct a radar target detection model and perform self-supervised pre-training, including positive and negative sample pair construction, feature extraction, cross-view contrastive learning, and cross-modal contrastive learning; Step S2.1: Use transpose transformation on the preprocessed RAD data to obtain data from different viewpoints, including RA, RD, and AD views, which, together with the corresponding visible light images, are used as input to the radar target detection model; Step S2.2: Use RadarResNet as the backbone network of the radar target detection model, extract the features of the RA, RD and AD three views respectively, use the encoder of YOLOv9 to extract the visible light image features, and project the four features onto a unified representation space through the projection head; Step S2.3: Take three views from the same RAD cube and the corresponding RGB images as positive sample pairs, and data from different samples as negative sample pairs. Perform cross-view comparison learning on the features of different views encoded in step S2.2, and perform cross-modal comparison learning on the features of the RA view and the image features. Step S2.4: Self-supervised learning is performed using the contrastive learning loss function, and the model is trained using the Adam optimizer until the loss converges, and the pre-trained model weights are obtained. Step S3: Fine-tune the downstream task of the radar target detection model, load the pre-trained model weights from step S2, and use different task head prediction results for different downstream tasks; Step S3.1: Use the RAD data from the RA view as input to the model; Step S3.2: Extract RAD features using the backbone network of the pre-trained RA view from step S2; Step S3.3: Use different task heads to predict results for different target detection tasks; Step S3.4: Supervise network learning using the loss function corresponding to the task, including target box regression loss, target existence loss and classification loss, train the network and complete fine-tuning.
2. The radar target detection method based on cross-view, cross-modal contrastive learning according to claim 1, characterized in that, In step S1, the radar dataset comes from a publicly available millimeter-wave radar dataset. The divided training set contains unlabeled data for pre-training the radar target detection model, while the fine-tuning dataset contains labeled data for fine-tuning in downstream tasks.
3. The radar target detection method based on cross-view, cross-modal contrastive learning according to claim 1, characterized in that, In step S2.2, the radar data Features are extracted separately using multiple RadarResNet backbone networks, and then passed through projection layers. Project onto a unified representation space: In the above formula, V represents the view of the input RAD tensor, including three views: RA, RD, and AD. This represents the vector after feature extraction and projection. It is a projection layer composed of two linear layers; For image data Backbone network using YOLOv9-c Extract image features during training. Pre-trained weights from COCO object detection are used, and the parameters are kept frozen before being passed through a projection layer. Project onto a unified representation space:
4. The radar target detection method based on cross-view, cross-modal contrastive learning according to claim 1, characterized in that, In step S2.3, positive sample pairs are formed for different views of the same RAD cube in the mini-batch, and negative sample pairs are formed for views from different RAD cubes. For two views... and The contrast loss is calculated using the NT-Xent loss proposed in SimCLR: In the above formula, 1 i≠k ∈{0,1} is the indicator function, which takes the value 1 if and only if i≠k, N is the mini-batch size, τ is the temperature coefficient, s(·) is the cosine similarity function, V1 and V2 represent views of the input RAD tensor, V1 and V2 are different types of views, and the contrastive learning loss function between views V1 and V2 is expressed as: For two pairs of views, RA and RD, RD and AD, and RA and AD, calculate their contrast loss. The total cross-view loss is obtained by linear combination. and To calculate: Similarly, for and Calculate the contrast loss using NT-Xent loss: The cross-modal contrast loss is expressed as:
5. The radar target detection method based on cross-view, cross-modal contrastive learning according to claim 1, characterized in that, Step 2.4 will compare the loss across views. Cross-modal contrast loss Combined, these can be expressed as the total loss function for the self-supervised pre-training stage, as follows:
6. An electronic device, characterized in that, The electronic device includes a processor and a memory. The memory stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded and executed by the processor to implement the radar target detection model of the radar target detection method based on cross-view and cross-modal contrastive learning as described in any one of claims 1-5. The model is pre-trained by self-supervised training using unlabeled registered radar data and visible light images, outputs the pre-training weights of the model, and is used to output the radar target detection results after fine-tuning in downstream tasks.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which, when executed, implements the radar target detection model of the radar target detection method based on cross-view cross-modal contrastive learning as described in any one of claims 1-5.