A millimeter wave radar three-dimensional target detection method, system, medium and equipment based on cross-modal visual semantic injection

Abstract

This invention discloses a method, system, medium, and device for 3D target detection using millimeter-wave radar based on cross-modal visual semantic injection, relating to the field of autonomous driving environmental perception technology. The method includes: simultaneously acquiring and extracting visual object query features and radar physical features; defining positive and negative sample pairs of radar and visual features within a bird's-eye view plane based on dynamic matching boundary constraints, and calculating spatial contrast loss; aligning historical visual features with current radar features based on the vehicle's motion compensation mechanism, and calculating temporal contrast loss; optimizing the radar network by combining the two contrast losses, decoupling the visual branch during the inference stage, and outputting 3D detection results solely from the millimeter-wave radar point cloud. This invention efficiently injects visual semantics into the radar network through spatial and temporal contrast learning, achieving high-precision and robust pure radar 3D target detection during the inference stage without relying on visual sensors, and is particularly suitable for optically degraded environments such as rain, fog, and dust.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving environment perception and autonomous mobile robot (AMR) navigation technology, specifically to a method, system, medium, and device for three-dimensional target detection using millimeter-wave radar based on cross-modal visual semantic injection. Background Technology

[0002] In environmental perception tasks of autonomous driving systems and autonomous mobile robots (AMRs), optical sensors (such as high-definition cameras and LiDAR) are currently the most mainstream source of information for 3D target detection. However, as the application boundaries of embodied intelligence and special-purpose robots continue to expand, robots often need to operate in extremely harsh, unstructured environments. For example, in construction operations in the field of intelligent buildings, the air is filled with high concentrations of cement dust and high-pressure paint mist; in autonomous passenger vehicle scenarios, extreme weather such as heavy rain, dense fog, or blizzards are frequently encountered. In such scenarios where optics are absolutely degraded, the lenses of optical sensors are easily contaminated and blinded, and high-density suspended particles can cause severe optical backscattering, leading to the complete failure of the perception system.

[0003] In contrast, millimeter-wave radar (mmWave Radar), due to its longer wavelength, has excellent physical penetration characteristics that are all-weather, dust-resistant, rain- and fog-resistant, and can directly measure the Doppler velocity of targets. Therefore, it has become the only reliable base for ensuring the spatial perception and safe obstacle avoidance of robots in complex environments.

[0004] However, existing millimeter-wave radar-based sensing methods suffer from the following serious drawbacks: Semantic blind spots and classification difficulties: Millimeter-wave radar point clouds are extremely sparse and lack visual texture information. Traditional pure radar algorithms (such as PointPillars and PillarNet) have difficulty accurately reconstructing and classifying complex targets (such as trucks, pedestrians, and metal road signs), resulting in serious semantic ambiguity.

[0005] The "weakest link" in multimodal fusion: Existing multimodal fusion algorithms (such as BEVFusion and RCBEVDet) heavily rely on simultaneous input from vision and radar during the inference phase. Once the vision sensor is blinded by severe weather or dust, the fusion network often suffers catastrophic performance crashes due to the lack of image features. In addition, the inference computing power of multimodal architectures is huge, making it difficult to deploy on resource-constrained edge computing devices.

[0006] Traditional distillation features overfitting: Existing cross-modal distillation methods (such as RadarDistill) attempt to force radar networks to mimic the dense geometry of lidar, which violates the sparse physical nature of radar point clouds. In actual training, the network is forced to memorize multipath noise to piece together non-existent geometric contours, which leads to an increase in localization and classification errors.

[0007] To address the aforementioned issues, this paper proposes a pure millimeter-wave radar 3D perception method that can leverage visual knowledge to overcome radar semantic bottlenecks during the training phase and completely decouple visual dependence during the inference phase. This method aims to solve the problems existing in the prior art and is a problem that urgently needs to be addressed by those skilled in the art. Summary of the Invention

[0008] In view of this, the present invention provides a method, system, medium and device for three-dimensional target detection of millimeter-wave radar based on cross-modal visual semantic injection, so as to achieve high-precision, high-robustness and resistance to extreme environmental interference low-cost pure radar three-dimensional perception.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A method for 3D target detection using millimeter-wave radar based on cross-modal visual semantic injection includes the following steps: S1 Data Acquisition Steps: Acquire synchronously collected visual image data and millimeter-wave radar point cloud data; S2 Feature extraction steps: Extract visual semantic features from visual image data, and extract radar physical features from millimeter-wave radar point cloud data; S3 Spatial Contrast Learning Steps: Based on dynamic matching boundary constraints, define positive and negative sample pairs of radar physical features and visual object query features in the bird's-eye view plane, and calculate spatial contrast loss in the latent space. S4 Spatiotemporal Consistency Modeling Steps: Based on the vehicle motion compensation mechanism, cross-frame spatial alignment of historical visual object query features is performed and matched with the radar physical features of the current frame to calculate the temporal comparison loss; S5 Single-Modal Inference Deployment Steps: The radar network parameters are optimized by combining spatial contrast loss and temporal contrast loss. During the inference deployment phase, the vision branch is decoupled, and the 3D target detection results are output through the optimized radar network, relying solely on millimeter-wave radar point cloud data.

[0010] The above method, optionally, includes the following specific content for S2: Visual object query feature extraction steps: Input visual image data into a pre-trained visual teacher network with frozen parameters to extract and generate a visual object query set containing high-level category semantics and target spatial location information; Radar physical feature extraction steps: Input millimeter-wave radar point cloud data into the radar student network. The radar point cloud data includes three-dimensional spatial coordinates, Doppler velocity and radar cross section. The radar point cloud data is subjected to self-attention feature extraction through radar dense encoder to obtain radar physical feature map.

[0011] The above method, optionally, includes the following specific content for S3: The classification confidence score is greater than a set threshold. Visual object query; Target length predicted based on visual object query and width Adaptive calculation of dynamic matching radius: ; in, This is the preset scaling factor; The radar physical features are projected onto the two-dimensional bird's-eye view XY plane to calculate the radar points. Visual Inquiry Center The Euclidean distance; when the Euclidean distance satisfies If a match is successful, the corresponding radar features and the visual query are combined to form a set of positive sample pairs. Radar points that exceed the current dynamic matching radius are classified as negative samples. Using a nonlinear projection head, radar physical features and visual object queries are mapped to a shared latent space, yielding the mapped features respectively. and ; Calculate the InfoNCE spatial contrast loss function To maximize the mutual information of positive sample pairs, the formula is as follows:

[0012] in, This is the temperature scaling factor.

[0013] The above method, optionally, includes the following specific content for S4: Maintain a first-in-first-out historical memory queue for caching high-confidence visual object queries from previous frames; Obtain the relative pose matrix of the vehicle's odometer and locate the physical center point of the historical query in the historical memory queue. Perform a rigid body transformation to align it to the current frame coordinate system. The rigid body transformation formula is: ; in, Let be a rotation matrix. It is a translation vector; Calculate the cosine similarity between the radar physical features of the current frame and the aligned historical queries, and select those with similarity greater than a threshold. The matching pairs are used to construct a set of time-series positive sample pairs, and the time-series contrast loss is calculated accordingly. .

[0014] The above method, optionally, includes the following specific content for S5: After completing multimodal joint training, the visual teacher network, historical memory queue, and contrastive learning projector were removed from the network architecture. The radar student network that incorporates cross-modal semantic priors is retained and then fine-tuned on a defined subset of radar data with bounding box labels. It only receives single millimeter-wave radar point cloud inputs and directly outputs 3D bounding box prediction results containing position, size, heading angle and velocity through a finely tuned radar student network.

[0015] Optionally, the above method may optimize radar network parameters by combining spatial contrast loss and temporal contrast loss, including: Construct the overall objective function for end-to-end joint optimization The calculation formula is as follows:

[0016] in, For object detection, classification, and regression loss; The weight hyperparameter for spatial contrast loss is... The weighting hyperparameters for time-series comparison loss.

[0017] A millimeter-wave radar 3D target detection system based on cross-modal visual semantic injection, used to execute any of the millimeter-wave radar 3D target detection methods based on cross-modal visual semantic injection described above, includes a data acquisition module, a feature extraction module, a spatial contrast learning module, a spatiotemporal consistency modeling module, and a single-modal inference deployment module connected in sequence; wherein, The data acquisition module is used to acquire synchronously collected visual image data and millimeter-wave radar point cloud data; The feature extraction module is used to extract visual semantic features from visual image data and radar physical features from millimeter-wave radar point cloud data. The spatial contrast learning module is used to define positive and negative sample pairs of radar physical features and visual object query features in the bird's-eye view plane based on dynamic matching boundary constraints, and to calculate spatial contrast loss in the latent space. The spatiotemporal consistency modeling module is used to perform cross-frame spatial alignment of historical visual object query features based on the vehicle motion compensation mechanism, and match them with the radar physical features of the current frame to calculate the temporal comparison loss. The single-modal inference deployment module is used to optimize radar network parameters by combining spatial contrast loss and temporal contrast loss. During the inference deployment phase, the vision branch is decoupled, and the three-dimensional target detection results are output through the optimized radar network based solely on millimeter-wave radar point cloud data.

[0018] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a millimeter-wave radar three-dimensional target detection method based on cross-modal visual semantic injection as described in any of the preceding claims.

[0019] An electronic device includes: a memory for storing computer programs; A processor for executing a computer program stored in memory to implement a millimeter-wave radar three-dimensional target detection method based on cross-modal visual semantic injection as described in any of the preceding claims; Electronic devices are used in autonomous vehicles or autonomous mobile robots.

[0020] As can be seen from the above technical solution, compared with the prior art, the present invention provides a method, system, medium, and device for three-dimensional target detection of millimeter-wave radar based on cross-modal visual semantic injection, which has the following beneficial effects: This invention employs an architecture of fusion distillation during the training phase and single-modal decoupling during the inference phase. Through dynamic matching of boundary constraints and contrastive learning, it achieves soft binding between sparse radar signals and high-level visual semantics in the latent space, breaking through the semantic blind spot of pure radar perception. Through temporal memory queues and motion compensation, it overcomes the problems of transient frame loss and multipath flicker in radar. Finally, under the premise of completely eliminating dependence on visual hardware, it achieves high-precision, low-latency pure radar 3D target detection, exhibiting extremely high robustness in extreme degradation environments such as dust, rain, and fog. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0022] Figure 1 This is a flowchart of a millimeter-wave radar three-dimensional target detection method based on cross-modal visual semantic injection disclosed in this invention; Figure 2 This is a schematic diagram of the overall framework of a millimeter-wave radar three-dimensional target detection system based on cross-modal visual semantic injection disclosed in this invention; Figure 3This is a schematic diagram illustrating the principle of query-point space comparison learning (positive and negative sample construction) for dynamic boundary constraints disclosed in this invention. Figure 4 This is a comparison chart of the robust detection performance of the algorithm disclosed in this invention and existing baseline algorithms under extreme weather conditions; Figure 5a This is a bar chart comparing the detection accuracy of a millimeter-wave radar three-dimensional target detection method under different motion states, as disclosed in an embodiment of the present invention. Figure 5b This is a line graph comparing the speed estimation error of a millimeter-wave radar three-dimensional target detection method disclosed in an embodiment of the present invention under different vehicle speed conditions. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0025] See Figure 1 As shown, this invention discloses a millimeter-wave radar three-dimensional target detection method based on cross-modal visual semantic injection, comprising the following steps: S1 Data Acquisition Steps: Acquire synchronously collected visual image data and millimeter-wave radar point cloud data; S2 Feature extraction steps: Extract visual semantic features from visual image data, and extract radar physical features from millimeter-wave radar point cloud data; S3 Spatial Contrast Learning Steps: Based on dynamic matching boundary constraints, define positive and negative sample pairs of radar physical features and visual object query features in the bird's-eye view plane, and calculate spatial contrast loss in the latent space. S4 Spatiotemporal Consistency Modeling Steps: Based on the vehicle motion compensation mechanism, cross-frame spatial alignment of historical visual object query features is performed and matched with the radar physical features of the current frame to calculate the temporal comparison loss; S5 Single-Modal Inference Deployment Steps: The radar network parameters are optimized by combining spatial contrast loss and temporal contrast loss. During the inference deployment phase, the vision branch is decoupled, and the 3D target detection results are output through the optimized radar network, relying solely on millimeter-wave radar point cloud data.

[0026] Furthermore, the specific content of S2 is as follows: Visual object query feature extraction steps: Input visual image data into a pre-trained visual teacher network with frozen parameters to extract and generate a visual object query set containing high-level category semantics and target spatial location information. ; Radar physical feature extraction steps: Input millimeter-wave radar point cloud data into the radar student network. The radar point cloud data includes three-dimensional spatial coordinates, Doppler velocity and radar cross section. The radar point cloud data is subjected to self-attention feature extraction through radar dense encoder to obtain radar physical feature map.

[0027] Furthermore, the specific content of S3 is as follows: The classification confidence score is greater than a set threshold. Visual object query; Target length predicted based on visual object query and width Adaptive calculation of dynamic matching radius: ; in, This is the preset scaling factor; The radar physical features are projected onto the two-dimensional bird's-eye view XY plane to calculate the radar points. Visual Inquiry Center The Euclidean distance; when the Euclidean distance satisfies If a match is successful, the corresponding radar features and the visual query are combined to form a set of positive sample pairs. Radar points that exceed the current dynamic matching radius are classified as negative samples. Using a nonlinear projection head, radar physical features and visual object queries are mapped to a shared latent space, yielding the mapped features respectively. and ; Calculate the InfoNCE spatial contrast loss function To maximize the mutual information of positive sample pairs, the formula is as follows:

[0028] in, This is the temperature scaling factor.

[0029] Furthermore, the specific content of S4 is as follows: Maintain a first-in-first-out historical memory queue for caching high-confidence visual object queries from previous frames; Obtain the relative pose matrix of the vehicle's odometer and locate the physical center point of the historical query in the historical memory queue. Perform a rigid body transformation to align it to the current frame coordinate system. The rigid body transformation formula is: ; in, Let be a rotation matrix. It is a translation vector; Calculate the cosine similarity between the radar physical features of the current frame and the aligned historical queries, and select those with similarity greater than a threshold. The matching pairs are used to construct a set of time-series positive sample pairs, and the time-series contrast loss is calculated accordingly. .

[0030] Furthermore, the specific details of S5 are as follows: After completing multimodal joint training, the visual teacher network, historical memory queue, and contrastive learning projector were removed from the network architecture. The radar student network that incorporates cross-modal semantic priors is retained and then fine-tuned on a defined subset of radar data with bounding box labels. It only receives single millimeter-wave radar point cloud inputs and directly outputs 3D bounding box prediction results containing position, size, heading angle and velocity through a finely tuned radar student network.

[0031] Furthermore, the radar network parameters are optimized by combining spatial contrast loss and temporal contrast loss, including: Construct the overall objective function for end-to-end joint optimization The calculation formula is as follows:

[0032] in, For object detection, classification, and regression loss; The weight hyperparameter for spatial contrast loss is... The weighting hyperparameters for time-series comparison loss.

[0033] and Figure 1 Corresponding to the method described above, this invention also discloses a millimeter-wave radar 3D target detection system based on cross-modal visual semantic injection, used to perform... Figure 1The method for 3D target detection using millimeter-wave radar based on cross-modal visual semantic injection, as shown, includes a data acquisition module, a feature extraction module, a spatial contrast learning module, a spatiotemporal consistency modeling module, and a single-modal inference deployment module connected in sequence; wherein, The data acquisition module is used to acquire synchronously collected visual image data and millimeter-wave radar point cloud data; The feature extraction module is used to extract visual semantic features from visual image data and radar physical features from millimeter-wave radar point cloud data. The spatial contrast learning module is used to define positive and negative sample pairs of radar physical features and visual object query features in the bird's-eye view plane based on dynamic matching boundary constraints, and to calculate spatial contrast loss in the latent space. The spatiotemporal consistency modeling module is used to perform cross-frame spatial alignment of historical visual object query features based on the vehicle motion compensation mechanism, and match them with the radar physical features of the current frame to calculate the temporal comparison loss. The single-modal inference deployment module is used to optimize radar network parameters by combining spatial contrast loss and temporal contrast loss. During the inference deployment phase, the vision branch is decoupled, and the three-dimensional target detection results are output through the optimized radar network based solely on millimeter-wave radar point cloud data.

[0034] See Figure 2 The diagram shows the overall framework of a millimeter-wave radar 3D target detection system based on cross-modal visual semantic injection disclosed in this invention. It illustrates the structure of the visual teacher network (offline distillation) and the radar student network (online learning), as well as the interaction relationships between modules such as spatial contrastive learning and temporal consistency modeling.

[0035] This invention discloses a method for three-dimensional target detection of millimeter-wave radar based on cross-modal visual semantic injection, which mainly includes four core stages: multimodal synchronous input and feature encoding, query-point spatial comparison learning based on dynamic boundaries, spatiotemporal consistency modeling based on motion compensation, and sensor decoupling and single-modal radar inference deployment. Figure 2 This is a schematic diagram of the overall system framework proposed in this invention. The following is in conjunction with... Figure 2 Each step is explained in detail.

[0036] Step S1: Synchronous input and feature encoding of multimodal data

[0037] like Figure 2 As shown on the left, during the offline training phase, the system synchronously collects data at the current moment. t Forward camera image stream and millimeter-wave radar point clouds .

[0038] (1) Visual Teacher Flow: The flow of images Input a pre-trained visual detector with frozen weights (such as a Transformer-based multi-view 3D detection network). The visual teacher network extracts high-level semantic and object geometric information from the image to generate high-quality visual object queries, which serve as semantic anchors and are denoted as the query set. .

[0039] (2) Radar Student Flow: Radar point cloud is defined as ,in N This refers to the number of valid radar points. Each radar point includes three-dimensional spatial coordinates, Doppler velocity compensated for by the vehicle's motion, and radar cross-section (RCS), i.e. .Will The input is a radar dense encoder (RDE) based on the U-Net architecture. The RDE utilizes a self-attention mechanism and skip connections to diffuse discrete radar points into a continuous grid, extracting the radar physical feature map, denoted as... .

[0040] Step S2: Construct a query-point space contrastive learning mechanism for dynamic matching boundaries

[0041] To avoid elevation errors caused by traditional pixel-level explicit alignment, this invention employs soft alignment in a three-dimensional latent space. The following section combines... Figure 3 The schematic diagram shown illustrates this step in detail. Specifically, it includes: (1) Filtering high-quality queries: Extracting the classification confidence score of each visual query. By setting a preset threshold (For example, take) =0.3) Filter out low-quality predictions and retain only those that meet the criteria. > High-quality visual queries prevent erroneous pseudo-labels from polluting the radar network.

[0042] (2) Define dynamic matching radius: For targets of different scales (such as large buses and pedestrians), the target length is predicted based on the visual query. and width Adaptive generation of dynamic matching radius The calculation formula is:

[0043] in, This is the scaling factor, with a preferred value of 0.5, which limits the matching radius to the physical diagonal of the target.

[0044] like Figure 3 As shown, the left side displays the target bounding box (including category, size, and center point) output by the visual teacher network, while the right side shows the radar point cloud projected onto the BEV plane. For targets of different scales (such as large and small vehicles), the system adaptively calculates the dynamic matching radius based on their length and width. This creates prototype matching regions of varying sizes. Radar points falling within these regions (such as solid circles) are defined as positive samples, while radar points outside the regions (such as crosses) are defined as negative samples.

[0045] (3) Defining positive and negative sample pairs: To avoid the physical defect of severely inaccurate Z-axis (height) measurement by millimeter-wave radar, this invention deliberately restricts spatial distance matching to a two-dimensional bird's-eye view (XY plane). The radar points are projected onto the XY plane to obtain... The visual query center projection obtained When the Euclidean distance between the two satisfies When a match is found, it is considered successful. (Positive sample pair set) The rigorous mathematical definition of is:

[0046] Radar points that exceed the dynamic boundary or do not match the category are classified as negative samples.

[0047] (4) Calculate the latent space InfoNCE contrast loss: through a learnable nonlinear projection head and By mapping radar physical features and visual queries to a shared latent space, we obtain... and The InfoNCE loss function is used to maximize the mutual information between positive sample pairs, thereby bridging the cosine distance between sparse physical signals and corresponding visual semantics in the latent space.

[0048] in, Temperature scaling factor, preferably =0.1.

[0049] Step S3: Spatiotemporal Consistency Modeling Based on Motion Compensation

[0050] To address the radar multipath effect and signal flicker problems caused by complex industrial environments (such as areas with dense metal scaffolding), this invention introduces temporal characteristics for implicit suppression. For example... Figure 2 As shown on the right, this step achieves spatiotemporal consistency modeling through historical semantic queues and bicycle motion compensation: (1) Historical semantic queue maintenance: The system maintains a first-in-first-out (FIFO) historical memory queue with a capacity of M (for example, M=3000) to cache high-confidence visual object queries of previous frames.

[0051] (2) Rigid body compensation of vehicle motion: The relative pose transfer matrix between adjacent frames is obtained by using sensor odometer. (Including rotation matrix) Translation vector ). Query the physical space center point from the historical queue. Perform a rigid body transformation to precisely align it with the current frame. t In the coordinate system:

[0052] This operation demonstrates the elimination of spatial misalignment caused by the vehicle's movement.

[0053] (3) Cross-frame semantic-geometric matching and joint optimization: Calculate the cosine similarity between the radar features of the current frame and the aligned historical queries, and select those with similarity greater than a threshold. (like Feature combinations with a feature value of 0.8 are used to construct time-series positive sample pairs. And calculate the time-series comparison loss. Finally, the overall objective function for end-to-end joint optimization is constructed as follows:

[0054] in, Classification and regression losses are fundamental for 3D object detection. and Empirical weight hyperparameters (preferred) ).

[0055] Step S4: Sensor decoupling and single-mode radar inference deployment

[0056] like Figure 2 As shown at the bottom, after completing the multimodal joint training, this invention removes the visual teacher branch, historical memory module, and contrastive learning projector from the network.

[0057] (1) Architecture pruning and fine-tuning: After multimodal joint training (distillation) is completed, the visual teacher branch, historical memory module and contrastive learning projection head in the network are pruned and fine-tuned. and All data is removed. Only the Radar Student network, which incorporates visual semantic knowledge, is retained. In a preferred embodiment, a small amount (e.g., 10%) of radar data with real 3D bounding box labels from the target scene is used to perform lightweight fine-tuning of the cropped radar network to achieve efficient adaptation of the feature domain.

[0058] (2) Independent perception by a single radar on the edge side: The finely tuned lightweight radar network is deployed on the embedded edge computing platform of the target vehicle or autonomous mobile robot. In actual operation, the system only receives a single millimeter-wave radar point cloud stream. The network relies on the cross-modal semantic prior and history alignment capabilities that are fixed in the parameters to directly output high-precision three-dimensional target bounding boxes end-to-end.

[0059] In a preferred embodiment, taking an intelligent building spraying or plastering robot environment as an example: Scene description: Inside a construction site, there is an extremely high concentration of suspended cement dust and high-pressure paint mist. The surrounding area is covered with metal scaffolding, which blinds the optical cameras and causes serious radar multipath interference.

[0060] Results: Visual sensor failure leads to the paralysis of traditional fusion networks. However, this invention, by completely decoupling from visual dependence during the inference phase, relies solely on the physical penetration of millimeter-wave radar and leverages its strong semantic priors acquired through contrastive learning during the training phase to successfully filter out multipath ghost noise. The system exhibited no trajectory jumps or missed detections, achieving high-precision real-time 3D obstacle detection with an average inference latency as low as 28.3ms. ms This meets the underlying safety and obstacle avoidance needs of robots in extremely degraded environments.

[0061] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements as follows: Figure 1 This paper presents a method for three-dimensional target detection using millimeter-wave radar based on cross-modal visual semantic injection.

[0062] An electronic device includes: a memory for storing computer programs; A processor is used to execute computer programs stored in memory to achieve, for example, Figure 1 This paper presents a method for three-dimensional target detection using millimeter-wave radar based on cross-modal visual semantic injection. Electronic devices are used in autonomous vehicles or autonomous mobile robots.

[0063] The processor, memory, and communication interface of an electronic device communicate with each other through a communication bus.

[0064] In a preferred scenario, the electronic device may specifically be an in-vehicle domain controller, an edge computing box (such as the NVIDIA Jetson series), etc., and may be mounted in an autonomous vehicle or an autonomous mobile robot (AMR) facing an extreme degradation environment.

[0065] To further verify the effectiveness and robustness of the method proposed in this invention, the following is combined with... Figure 4 , Figure 5a and Figure 5b A detailed quantitative analysis of the detection performance and speed estimation accuracy of the present invention is conducted.

[0066] like Figure 4 The figure shows a comparison of the detection performance of the algorithm of this invention and existing baseline algorithms in different distance ranges. The horizontal axis represents the distance range, and the vertical axis represents the average detection accuracy (mAP). In sparse regions far from the radar (e.g., 30-50m), the pure visual teacher network experiences a sharp drop in performance due to depth ambiguity; while the present invention (RCTrans-CD), with the inherent physical ranging advantage of millimeter-wave radar and the injected high-level semantic prior, significantly outperforms the visual teacher model and the traditional radar distillation baseline in long-distance detection accuracy, proving the strong robustness of the method in sparse and ambiguous boundary regions.

[0067] like Figure 5a The bar chart shown is a comparison of the detection accuracy of this invention under different motion states (static and dynamic objects). Traditional basic radar models suffer from extremely low detection accuracy on static objects due to the lack of visual texture semantics. The method of this invention, through cross-modal semantic distillation, enables the radar network to successfully inherit the powerful static target perception capability of the visual model, resulting in a significant leap in its static object mAP, breaking through the physical bottleneck of traditional millimeter-wave radar's inability to detect stationary obstacles.

[0068] like Figure 5b The figure shows a line graph comparing the speed estimation error (mAVE) of this invention under different vehicle speed conditions. As the target vehicle speed increases, the speed measurement error of the visual teacher network rises sharply due to the "motion blur" defect of the camera; while the traditional fusion baseline model, because it fails to separate the visual branch during inference, has its speed measurement accuracy contaminated by inferior visual features. The method of this invention, because it completely decouples the visual sensor during the inference stage and relies solely on a single-mode radar for end-to-end output, maintains its speed error at an extremely low level throughout the entire speed range. This proves that this invention perfectly preserves the inherent high-precision Doppler physical velocity measurement advantage of radar while endowing it with high-level semantics.

[0069] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for three-dimensional target detection using millimeter-wave radar based on cross-modal visual semantic injection, characterized in that, Includes the following steps: S1 Data Acquisition Steps: Acquire synchronously collected visual image data and millimeter-wave radar point cloud data; S2 Feature extraction steps: Extract visual semantic features from visual image data, and extract radar physical features from millimeter-wave radar point cloud data; S3 Spatial Contrast Learning Steps: Based on dynamic matching boundary constraints, define positive and negative sample pairs of radar physical features and visual object query features in the bird's-eye view plane, and calculate spatial contrast loss in the latent space. S4 Spatiotemporal Consistency Modeling Steps: Based on the vehicle motion compensation mechanism, cross-frame spatial alignment of historical visual object query features is performed and matched with the radar physical features of the current frame to calculate the temporal comparison loss; S5 Single-Modal Inference Deployment Steps: The radar network parameters are optimized by combining spatial contrast loss and temporal contrast loss. During the inference deployment phase, the vision branch is decoupled, and the 3D target detection results are output through the optimized radar network, relying solely on millimeter-wave radar point cloud data.

2. The millimeter-wave radar three-dimensional target detection method based on cross-modal visual semantic injection according to claim 1, characterized in that, The specific content of S2 is as follows: Visual object query feature extraction steps: Input visual image data into a pre-trained visual teacher network with frozen parameters to extract and generate a visual object query set containing high-level category semantics and target spatial location information; Radar physical feature extraction steps: Input millimeter-wave radar point cloud data into the radar student network. The radar point cloud data includes three-dimensional spatial coordinates, Doppler velocity and radar cross section. The radar point cloud data is subjected to self-attention feature extraction through radar dense encoder to obtain radar physical feature map.

3. The millimeter-wave radar three-dimensional target detection method based on cross-modal visual semantic injection according to claim 2, characterized in that, The specific content of S3 is as follows: The classification confidence score is greater than a set threshold. Visual object query; Target length predicted based on visual object query and width Adaptive calculation of dynamic matching radius: ； in, This is the preset scaling factor; The radar physical features are projected onto the two-dimensional bird's-eye view XY plane to calculate the radar points. Visual Inquiry Center The Euclidean distance; when the Euclidean distance satisfies If a match is successful, the corresponding radar features and the visual query are combined to form a set of positive sample pairs. Radar points that exceed the current dynamic matching radius are classified as negative samples. Using a nonlinear projection head, radar physical features and visual object queries are mapped to a shared latent space, yielding the mapped features respectively. and ; Calculate the InfoNCE spatial contrast loss function To maximize the mutual information of positive sample pairs, the formula is as follows: in, This is the temperature scaling factor.

4. The millimeter-wave radar three-dimensional target detection method based on cross-modal visual semantic injection according to claim 3, characterized in that, The specific content of S4 is as follows: Maintain a first-in-first-out historical memory queue for caching high-confidence visual object queries from previous frames; Obtain the relative pose matrix of the vehicle's odometer and locate the physical center point of the historical query in the historical memory queue. Perform a rigid body transformation to align it to the current frame coordinate system. The rigid body transformation formula is: ； in, Let be a rotation matrix. It is a translation vector; Calculate the cosine similarity between the radar physical features of the current frame and the aligned historical queries, and select those with similarity greater than a threshold. The matching pairs are used to construct a set of time-series positive sample pairs, and the time-series contrast loss is calculated accordingly. .

5. The millimeter-wave radar three-dimensional target detection method based on cross-modal visual semantic injection according to claim 4, characterized in that, The specific content of S5 is as follows: After completing multimodal joint training, the visual teacher network, historical memory queue, and contrastive learning projector were removed from the network architecture. The radar student network that incorporates cross-modal semantic priors is retained and then fine-tuned on a defined subset of radar data with bounding box labels. It only receives single millimeter-wave radar point cloud inputs and directly outputs 3D bounding box prediction results containing position, size, heading angle and velocity through a finely tuned radar student network.

6. The millimeter-wave radar three-dimensional target detection method based on cross-modal visual semantic injection according to claim 5, characterized in that, The radar network parameters are optimized by combining spatial contrast loss and temporal contrast loss, including: Construct the overall objective function for end-to-end joint optimization The calculation formula is as follows: in, For object detection, classification, and regression loss; The weight hyperparameter for spatial contrast loss is... The weighting hyperparameters for time-series comparison loss.

7. A millimeter-wave radar three-dimensional target detection system based on cross-modal visual semantic injection, characterized in that, A method for performing millimeter-wave radar 3D target detection based on cross-modal visual semantic injection as described in any one of claims 1-6, comprising, in sequence, a data acquisition module, a feature extraction module, a spatial contrast learning module, a spatiotemporal consistency modeling module, and a single-modal inference deployment module; wherein... The data acquisition module is used to acquire synchronously collected visual image data and millimeter-wave radar point cloud data; The feature extraction module is used to extract visual semantic features from visual image data and radar physical features from millimeter-wave radar point cloud data. The spatial contrast learning module is used to define positive and negative sample pairs of radar physical features and visual object query features in the bird's-eye view plane based on dynamic matching boundary constraints, and to calculate spatial contrast loss in the latent space. The spatiotemporal consistency modeling module is used to perform cross-frame spatial alignment of historical visual object query features based on the vehicle motion compensation mechanism, and match them with the radar physical features of the current frame to calculate the temporal comparison loss. The single-modal inference deployment module is used to optimize radar network parameters by combining spatial contrast loss and temporal contrast loss. During the inference deployment phase, the vision branch is decoupled, and the three-dimensional target detection results are output through the optimized radar network based solely on millimeter-wave radar point cloud data.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a millimeter-wave radar three-dimensional target detection method based on cross-modal visual semantic injection as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in a memory to implement a millimeter-wave radar three-dimensional target detection method based on cross-modal visual semantic injection as described in any one of claims 1 to 6; Electronic devices are used in autonomous vehicles or autonomous mobile robots.

Images