A three-dimensional environment perception method and device based on cross-modal bidirectional prior guidance

Abstract

This application discloses a three-dimensional environment perception method and device based on cross-modal bidirectional prior guidance, relating to the field of intelligent driving environment perception technology. The method includes: acquiring millimeter-wave radar echo data and visual image data in a driving scenario; generating virtual echoes of the radar echoes within the semantic target region based on the semantic segmentation results of the visual images to obtain a densed radar point cloud; encoding bird's-eye view features of the densed radar point cloud to generate radar bird's-eye view features; generating a spatial confidence map based on the radar bird's-eye view features, and using the spatial confidence map to modulate the viewpoint transformation process of the visual features to obtain corrected visual bird's-eye view features; fusing the radar bird's-eye view features and the corrected visual bird's-eye view features, and performing three-dimensional target detection based on the fused features. This application helps alleviate the problem of inconsistent bird's-eye view spatial features caused by sparse radar echoes and visual depth estimation bias, improving the stability and robustness of three-dimensional target detection in driving scenarios.

Description

Technical Field

[0001] This application relates to the field of intelligent driving environment perception technology, and in particular to a three-dimensional environment perception method and device based on cross-modal bidirectional prior guidance. Background Technology

[0002] With the rapid development of intelligent driving technology and intelligent transportation systems, environmental perception, as a key component of intelligent driving systems, directly affects the vehicle's decision-making and control capabilities. Among these technologies, 3D target detection technology can spatially locate and classify targets such as vehicles, pedestrians, and bicycles in traffic scenes, serving as a crucial foundation for achieving environmental perception in intelligent driving.

[0003] Currently, single-sensor-based 3D target detection methods have certain limitations. For example, while camera-based visual detection methods can acquire rich texture, color, and semantic information, they lack accurate depth information, making them prone to depth estimation errors in distant, small, and occluded scenarios, resulting in significant misalignment of spatial features from a bird's-eye view. On the other hand, millimeter-wave radar has strong distance perception capabilities and good all-weather performance, and can stably acquire target depth information even in complex environments such as rain, fog, and low light. However, millimeter-wave radar point clouds typically suffer from sparse point counts, discrete spatial distribution, and insufficient semantic representation, making it difficult to perform high-precision 3D target detection tasks independently.

[0004] Therefore, an increasing number of studies are adopting radar-vision fusion for 3D target detection, combining the rich semantic information of visual sensors with the stable geometric depth information of radar sensors to improve environmental perception performance. However, existing radar-vision fusion methods still have the following problems:

[0005] (1) The point cloud of millimeter-wave radar is sparse and has a lot of noise, resulting in an incomplete representation of the spatial structure of the target;

[0006] (2) Visual features are prone to depth ambiguity and spatial misalignment when transitioning from perspective to bird's-eye view;

[0007] (3) Existing multimodal fusion methods usually adopt simple splicing or weighted fusion, lacking an effective collaborative mechanism between modes, and making it difficult to make full use of the complementary information between radar and vision;

[0008] (4) Existing methods lack joint modeling of the relationship between radar spatial geometric information and visual semantic information, resulting in poor consistency and robustness of fused features.

[0009] Therefore, there is an urgent need for a radar visual 3D target detection method that can improve the spatial representation capability of radar point clouds and the consistency of visual bird's-eye view feature space, so as to improve the stability and robustness of 3D environment perception in complex traffic scenarios. Summary of the Invention

[0010] The purpose of this application is to provide a three-dimensional environment perception method, device, computer equipment, computer-readable storage medium, and computer program product based on cross-modal bidirectional prior guidance. It aims to at least alleviate the problems in existing radar-visual fusion three-dimensional perception, such as the spatial misalignment of bird's-eye view features caused by sparse millimeter-wave radar echoes and ambiguity in visual image depth estimation, as well as the difficulty of existing fusion strategies in fully utilizing the complementary relationship between radar geometric information and visual semantic information.

[0011] To achieve the above objectives, this application provides the following technical solution: Firstly, this application provides a three-dimensional environment perception method based on cross-modal bidirectional prior guidance, comprising: acquiring the next frame or multiple consecutive frames of millimeter-wave radar echo data and corresponding visual image data of the current driving scene, and synchronizing the millimeter-wave radar echo data and the visual image data in time based on a timestamp; unifying the millimeter-wave radar echo data and the visual image data to the same spatial coordinate system according to pre-calibrated camera intrinsic parameters, the extrinsic parameter transformation relationship between the radar and the camera, and the coordinate transformation relationship between the vehicle coordinate system and the sensor coordinate system; wherein, the millimeter-wave radar echo data includes one or more of the following: three-dimensional spatial coordinates of each echo point, radar cross-section, and radial velocity information;

[0012] The visual image data is segmented at the instance level to obtain a semantic mask that includes at least vehicle, pedestrian and cyclist categories; based on the semantic mask, the millimeter-wave radar echo data is semantically guided to generate virtual echoes, and the supplemented virtual echoes are merged and deduplicated with the foreground real echoes located in the semantic mask area to obtain a dense radar point cloud.

[0013] The densed radar point cloud is input into the radar coding network, and a radar bird's-eye view feature map is generated through channel feature extraction of real echo and virtual echo and cylinder scattering coding.

[0014] A multi-scale spatial confidence map is generated based on the radar bird's-eye view feature map. The visual image features are then subjected to a viewpoint transformation with radar geometric prior constraints using the multi-scale spatial confidence map to obtain a corrected visual bird's-eye view feature map.

[0015] The radar bird's-eye view feature map and the corrected visual bird's-eye view feature map are subjected to cross-modal feature fusion and channel adaptive recalibration to obtain a fused bird's-eye view feature map.

[0016] Based on the fused bird's-eye view feature map, 3D target detection is performed, and the target's category, location, size, and orientation information are output.

[0017] Secondly, this application provides a three-dimensional environment perception device based on cross-modal bidirectional prior guidance, comprising:

[0018] The data acquisition module is used to acquire multiple consecutive frames of millimeter-wave radar echo data and corresponding visual image data in the current driving scenario;

[0019] The semantic-guided radar enhancement module is used to perform instance-level semantic segmentation on the visual image data to obtain a semantic mask, and generate a semantically guided virtual echo on the millimeter-wave radar echo data based on the semantic mask. The generated virtual echo is then merged with the real radar echo points and deduplicated to obtain a denser radar point cloud.

[0020] The radar feature encoding module is used to generate a radar bird's-eye view feature map by extracting channel features of real echo and virtual echo and encoding column scattering from the dense radar point cloud.

[0021] The radar prior view correction module is used to generate a multi-scale spatial confidence map based on the radar bird's-eye view feature map, and to perform a view transformation of the visual image features with radar geometric prior constraints using the multi-scale spatial confidence map to obtain the corrected visual bird's-eye view feature map.

[0022] The cross-modal fusion module is used to perform cross-modal feature fusion and channel adaptive recalibration on the radar bird's-eye view feature map and the corrected visual bird's-eye view feature map to obtain a fused bird's-eye view feature map.

[0023] The target detection module is used to perform three-dimensional target detection based on the fused bird's-eye view feature map.

[0024] According to the specific embodiments provided in this application, this application has the following technical effects: This application guides the generation of virtual echoes of millimeter-wave radar through instance-level semantic masks of visual images, supplements virtual echo points in the semantic target area, and adaptively adjusts the spatial distribution of virtual echoes in combination with radar cross-section, thereby helping to improve the problem of insufficient bird's-eye view spatial geometry representation caused by sparse point clouds of millimeter-wave radar.

[0025] Meanwhile, this application uses a channel-separated coding mechanism for real and virtual echoes in the foreground to enable real physical echoes and semantically generated echoes to have distinguishable channel representations during the feature extraction stage, thereby reducing the risk of virtual echoes interfering with real radar response features.

[0026] This application also generates multi-scale spatial confidence maps based on radar bird's-eye view feature maps, and introduces radar geometric prior constraints in the process of converting visual image features into bird's-eye view space, so that the visual bird's-eye view features have better consistency with radar observation results in spatial location, which helps to reduce bird's-eye view spatial misalignment caused by visual depth estimation bias.

[0027] Furthermore, this application employs a two-way prior mechanism—image semantic-guided radar enhancement and radar geometric prior-guided visual correction—to enable targeted correction of both radar and visual modalities before fusion. Then, through cross-modal stitching, convolutional interaction, and channel adaptive recalibration, it achieves bird's-eye view spatial feature fusion, thereby improving the expressive power of the fused features for different traffic targets.

[0028] In some implementations, the semantically guided virtual echo generation module can be implemented as a preprocessing module, and the radar prior view correction module can be implemented using lightweight convolutional branches, thus improving the three-dimensional environment perception effect with limited computational complexity.

[0029] In some implementations, this application may also employ joint optimization of detection loss and bird's-eye view occupancy loss, so that the network can take into account both the target localization results and the geometric consistency of the bird's-eye view space in the 3D target detection task. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart illustrating a three-dimensional environment perception method based on cross-modal bidirectional prior guidance provided in an embodiment of this application;

[0032] Figure 2 A detailed flowchart illustrating semantic-guided radar echo enhancement, radar prior constraint perspective transformation, and cross-modal fusion is provided for another embodiment of this application.

[0033] Figure 3 A schematic diagram of the functional modules of a three-dimensional environment perception device based on cross-modal bidirectional prior guidance provided in an embodiment of this application;

[0034] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] Before describing the embodiments of this application, the following explanation is given for some of the terms involved in this application.

[0037] The millimeter-wave radar echo data described in this application may include a set of echo points acquired by millimeter-wave radar or four-dimensional imaging radar. The echo points include at least spatial coordinate information and may further include one or more of radar cross section, radial velocity, timestamp, or echo intensity.

[0038] The visual image data described in this application may include monocular images, multi-view images, or panoramic images captured by vehicle-mounted cameras. The semantic mask may be a semantic segmentation mask or an instance segmentation mask, used to characterize the image regions where vehicles, pedestrians, cyclists, or other traffic participants are located.

[0039] The virtual echo points mentioned in this application refer to auxiliary radar points generated based on real radar echo points, visual semantic masks, and sensor calibration relationships. These virtual echo points are used to supplement the spatial coverage of real radar echoes within the target area and do not represent the original physical echoes directly measured by radar sensors.

[0040] The bird's-eye view features or bird's-eye view feature maps mentioned in this application refer to the feature representation formed by mapping radar point cloud features, visual image features, or fused features to the bird's-eye view space. The spatial confidence map refers to a probability map or weighted map used to characterize the degree to which each spatial location in the bird's-eye view space has a reliable radar geometric prior.

[0041] In one exemplary embodiment, such as Figure 1 As shown, a three-dimensional environment perception method based on cross-modal bidirectional prior guidance is provided, including: Step 101: acquire the next frame or multiple frames of millimeter-wave radar echo data and visual image data of the current driving scene, and perform spatiotemporal synchronization and calibration parameter analysis to obtain cross-modal aligned radar echo data and visual image data.

[0042] In this embodiment, the spatiotemporal synchronization and calibration parameter parsing specifically include: reading the timestamps corresponding to the millimeter-wave radar echo data and visual image data respectively; using the acquisition time of the visual image data as a reference time; selecting one or more frames of millimeter-wave radar echo data that match the reference time within a preset time window; when using multiple frames of millimeter-wave radar echo data, performing self-motion compensation on the radar echo points at different acquisition times based on vehicle odometer information, inertial measurement unit information, or vehicle pose estimation results, so that the radar echo points of multiple frames are aligned to the same reference coordinate system.

[0043] Furthermore, based on the camera intrinsic parameter matrix, the extrinsic parameter matrix from the radar coordinate system to the camera coordinate system, and the projection relationship from the camera coordinate system to the image plane, the radar echo points are projected onto the visual image plane. Simultaneously, based on the back-projection relationship, the virtual sampling points on the image plane are transformed into a three-dimensional spatial coordinate system by combining the corresponding depth information. This provides a unified geometric basis for establishing the subsequent correspondence between semantic masks and radar echo points.

[0044] Specifically, the millimeter-wave radar echo data includes the three-dimensional spatial coordinates, radar cross-section, and radial velocity information of each echo point; the visual image data is an RGB image captured by an onboard camera.

[0045] Step 102: Perform instance-level semantic segmentation on the visual image data. Based on semantic masks, generate semantically guided virtual echoes from the millimeter-wave radar echo data. Merge and deduplicate the generated virtual echo points with real radar echo points to obtain a dense radar point cloud. Then, extract channel features from the real and virtual echoes and encode cylinder scattering from the dense radar point cloud to generate a radar bird's-eye view feature map. Specifically, the virtual echo points can inherit one or more of the following from the nearest neighbor real echo points: depth information, radar cross-section, radial velocity, timestamp, echo intensity, or semantic category information.

[0046] In one optional implementation, to reduce invalid noise introduced by virtual echo points, validity constraints are set for the generated virtual echo points. The validity constraints include at least one of the following: the two-dimensional projection position of the virtual echo point on the image plane is located within the corresponding semantic mask region; the depth value inherited by the virtual echo point is within a preset valid depth range; the two-dimensional projection distance between the virtual echo point and its nearest neighbor real echo point is less than a preset distance threshold; the three-dimensional spatial coordinates of the virtual echo point after backprojection are within a preset perception range; and the three-dimensional spatial distance between the virtual echo point and existing real echo points or existing virtual echo points is greater than a preset minimum interval.

[0047] If a candidate virtual echo point does not meet any of the above validity constraints, the candidate virtual echo point is discarded. When the distance between multiple virtual echo points in three-dimensional space is less than a preset overlap threshold, the virtual echo point that is closer to the real echo point or has a higher sampling probability is retained. Through the above constraints, virtual echo points can be mainly distributed within target areas with visual semantic support and radar geometric support, thereby reducing the impact of invalid virtual points on subsequent radar bird's-eye view feature encoding.

[0048] Step 103: Generate a multi-scale spatial confidence map based on the radar bird's-eye view feature map. Use the multi-scale spatial confidence map to perform a viewpoint transformation on the visual image features with radar geometric prior constraints to obtain a corrected visual bird's-eye view feature map. Then, perform cross-modal feature fusion and channel adaptive recalibration between the radar bird's-eye view feature map and the corrected visual bird's-eye view feature map to obtain a fused bird's-eye view feature map. Specifically, the viewpoint transformation uses residual modulation, selectively enhancing the visual feature response in areas with reliable radar observation, and maintaining the original visual feature response or reducing its contribution weight in subsequent fusion stages in areas with insufficient radar observation or unreliable radar priors.

[0049] Step 104: Perform 3D target detection based on the fused bird's-eye view feature map, and output the target's category, location, size, and orientation information. Specifically, the fused bird's-eye view feature map is input into an anchor-frame-based 3D detection head, and after confidence thresholding and non-maximum suppression, the final 3D target detection result is obtained.

[0050] In another exemplary embodiment of this application, the model training steps in the above-mentioned three-dimensional environment perception method based on cross-modal bidirectional prior guidance are described, specifically as follows: an end-to-end joint training strategy is adopted, using the real bounding boxes of three-dimensional object detection as supervision signals to calculate the detection loss function; the detection loss function includes classification loss, bounding box regression loss, and orientation angle regression loss; using the occupied area of ​​the real bounding boxes of three-dimensional object detection projected onto the bird's-eye view plane as supervision signals to calculate the bird's-eye view occupancy loss function; the detection loss function and the bird's-eye view occupancy loss function are weighted and summed to obtain the total loss function, and the network parameters participating in the training are updated with gradients based on the total loss function; wherein, the instance segmentation network, the visual backbone network, or other pre-trained modules can be set to a parameter frozen state or a trainable state according to actual needs.

[0051] In another exemplary embodiment of this application, the strategy for accumulating multi-frame millimeter-wave radar echo data in step 101 above is described.

[0052] Specifically, the number of radar echo accumulation frames is dynamically adjusted according to the scene: in static scenes or low-speed driving scenes (vehicle speed below the preset threshold), more frames (such as 5-10 frames) are used for accumulation to fully improve the radar point cloud density; in high-speed driving scenes (vehicle speed above the preset threshold), fewer frames (such as 2-3 frames) are used for accumulation to control the motion distortion introduced by the superposition of multiple frames.

[0053] The choice of the number of cumulative frames is also constrained by the original radar frame rate: when the radar frame rate is 10Hz, 5 cumulative frames correspond to a time window of about 500 milliseconds. Ego-motion compensation (such as based on vehicle odometer or inertial measurement unit) is needed to align the multiple radar echoes to eliminate point cloud misalignment caused by vehicle self-motion.

[0054] For implementations that lack self-motion compensation capabilities, a fixed-frame radar echo accumulation strategy can be adopted, such as 2-frame accumulation, to strike a trade-off between radar point cloud density and geometric consistency.

[0055] In another exemplary embodiment of this application, a processing strategy for the above method in scenarios of partial sensor failure or performance degradation is described.

[0056] Specifically, when the visual image quality is severely degraded due to heavy rain, dense fog, or strong glare, the semantic mask confidence score output by the instance segmentation network is generally lower than the preset effective threshold. At this time, the semantic-guided virtual echo generation module automatically downgrades: it stops generating virtual echoes based on low-confidence masks and only retains the original real radar echo points to participate in subsequent encoding and detection, so as to avoid introducing false virtual points by erroneous semantics.

[0057] In one optional implementation, a mask confidence threshold is set for each semantic mask instance. When the confidence of a semantic mask instance is lower than the mask confidence threshold, no virtual echo point is generated based on that semantic mask instance; when the number of valid semantic mask instances in the entire frame is lower than a preset number threshold, the semantically guided virtual echo generation module stops working and only the original real radar echo points are input into the subsequent radar coding network.

[0058] In another alternative implementation, the number of virtual echo points or the sampling weight can be adjusted according to the semantic mask confidence level, so that more virtual echo points are generated in high-confidence semantic regions, and fewer virtual echo points or no virtual echo points are generated in low-confidence semantic regions.

[0059] When the number of echoes from a millimeter-wave radar decreases sharply due to electromagnetic interference, multipath effects, or sensor malfunctions, and the radar echo density falls below a preset sparse threshold, the radar prior view correction module enters a degraded operating state. In this degraded operating state, the spatial confidence map can be set to a preset safety value, or the weight of the radar spatial confidence map in the visual bird's-eye view feature modulation process can be reduced, so that the corrected visual bird's-eye view feature map remains at or close to the initial visual bird's-eye view feature map.

[0060] Furthermore, in the cross-modal fusion stage, the fusion weights corresponding to the radar bird's-eye view feature map can be reduced, or the responses of low-confidence regions in the radar bird's-eye view feature map can be set to zero, so that the detection process mainly relies on visual bird's-eye view features while retaining usable radar geometric information. Through the above degradation strategies, the negative impact of unreliable radar priors on visual features and fused features can be reduced.

[0061] This application also provides a more specific embodiment to illustrate the above method in detail, such as... Figure 2 As shown, the specific steps include: Step 1: Intelligent driving scenario data acquisition and preprocessing. This involves inputting millimeter-wave radar echo data and visual image data into the spatiotemporal alignment engine to obtain cross-modal synchronization data.

[0062] S11: Read millimeter-wave radar echo data from the vehicle-mounted 4D imaging radar in batches at fixed intervals. Each cycle contains the three-dimensional spatial coordinates, radar cross section (RCS), and radial Doppler velocity of each echo point continuously acquired during that period, forming the radar echo data sequence corresponding to that period.

[0063] S12: Acquire corresponding visual image data at a fixed frame rate using an onboard forward-facing camera. The visual image data is an RGB color image, and a timestamp is recorded synchronously during acquisition.

[0064] S13: Inject timestamps based on a unified clock source into all acquired data. Due to the differences in acquisition frequency and data format between radar echo data and visual image data, time window matching is performed on the radar echo data based on the timestamps of the visual image data. Radar echo data within the corresponding time period is assigned to each frame of the visual image, generating cross-modal data sample pairs that are time-synchronized or approximately time-synchronized.

[0065] Step 2: Semantic-guided virtual echo generation. This step uses visual semantic guidance to generate virtual echoes from sparse radar echo data, resulting in a denser radar point cloud.

[0066] S21: Image semantic acquisition. Input the visual image into a pre-trained instance segmentation network to obtain instance-level semantic masks and their confidence scores that contain at least vehicle, pedestrian and cyclist categories.

[0067] S22: Radar echo projection. Using joint calibration parameters, radar echo points in the three-dimensional world coordinate system are projected onto the image plane to obtain the two-dimensional projected coordinates of each echo point.

[0068] S23: Hybrid Probability Map Construction. For each semantic mask instance containing foreground true echo points, construct a hybrid probability density distribution within its mask support domain.

[0069] S231: Construction of a two-dimensional Gaussian kernel. For each foreground true echo point, a two-dimensional Gaussian kernel is constructed with its two-dimensional projected coordinates as the center. The standard deviation of the Gaussian kernel is adaptively determined based on the reference standard deviation of the object category to which the echo point belongs and the radar cross section of the echo point.

[0070] S232: RCS weight calculation. Based on the normalized ratio of the radar cross section of each foreground true echo point to the maximum radar cross section of that category, and in conjunction with the category sensitivity parameter, the RCS weight of each foreground true echo point is calculated.

[0071] S233: Hybrid probability density map fusion. The semantic mask contour, the two-dimensional Gaussian kernel response of each foreground real echo point, and the RCS weight are superimposed pixel by pixel and normalized to obtain the hybrid probability density distribution.

[0072] S24: Hybrid Sampling. Importance sampling is performed based on the hybrid probability density distribution according to probability proportion, and uniform sampling is supplemented in sparse regions where the probability density is lower than a preset threshold, so that the total number of virtual echo points generated by each instance meets the preset quota.

[0073] S25: 3D Virtual Echo Generation. For each generated 2D virtual sampling point, the nearest real echo projection point is searched in its neighborhood. One or more of its depth information, radar cross section, radial velocity, echo intensity, or semantic category information are inherited. Then, it is back-projected to a unified 3D spatial coordinate system through calibration parameters to obtain the virtual echo point.

[0074] Step 3: Radar bird's-eye view feature encoding. This step encodes the densed radar point cloud into a radar bird's-eye view feature map.

[0075] S31: Column encoder processing. The dense radar point cloud is input into the dynamic column feature network. The channel identifiers of the real echo points and virtual echo points are marked and independent initial linear mappings are performed. They are concatenated into a unified column feature vector, and column encoding is performed through spatial sparse convolution.

[0076] S32: Multi-scale feature pyramid coding. Multi-scale spatial coding of columnar features is performed using a feature pyramid network to generate a radar bird's-eye view feature map.

[0077] Step 4: Radar Prior-Guided Visual Bird's-Eye Feature Space Correction. This step involves correcting the visual image features using radar geometric priors to achieve a viewpoint transformation, resulting in a corrected visual bird's-eye-view feature map.

[0078] S41: Multi-scale spatial confidence map generation. The radar bird's-eye view feature map is input into the multi-scale confidence map generation network. Through multiple sets of parallel convolutional dimensionality reduction branches and the Sigmoid probability normalization function, the network outputs spatial confidence maps at multiple scales.

[0079] S42: Visual bird's-eye view feature correction. Visual image features are mapped to three-dimensional voxel space through frustum voxel transformation and compressed in the height direction to obtain an initial visual bird's-eye view feature map; the spatial confidence maps at each scale are subjected to element-wise residual modulation with the initial visual bird's-eye view feature maps at the corresponding scales to obtain visual bird's-eye view feature maps corrected for each scale.

[0080] S43: Multi-scale fusion and dimensionality reduction. The visual bird's-eye view feature maps corrected at each scale are stitched together along the channel dimension, and the channel dimension is compressed through a dimensionality reduction convolutional network to obtain the corrected visual bird's-eye view feature map.

[0081] Step 5: Cross-modal fusion and 3D target detection. This step fuses the radar bird's-eye view feature map with the corrected visual bird's-eye view feature map and performs 3D target detection.

[0082] S51: Channel-dimensional stitching. The radar bird's-eye view feature map and the corrected visual bird's-eye view feature map are spatially and channel-aligned, and then stitched together along the channel dimension.

[0083] S52: Convolutional Fusion and SE Recalibration. The concatenated features are sequentially passed through a convolutional interaction layer and a channel attention recalibration module to perform local cross-modal interaction and channel adaptive weighting, resulting in a fused bird's-eye view feature map.

[0084] S53: Detection head output. The fused bird's-eye view feature map is input into the anchor-frame-based 3D detection head. After confidence thresholding and non-maximum suppression, the target's category, location, size, and orientation information are output.

[0085] This application also provides a more specific embodiment to illustrate the above method in detail. This embodiment is only used to illustrate an optional implementation of the technical solution of this application and does not constitute a limitation on the scope of protection of this application.

[0086] Step 201: Multimodal sensor data acquisition and spatiotemporal synchronization.

[0087] Step 2011: Taking an urban road scene as an example, the urban road scene may include intersections, main roads, low-light conditions, rain, fog, or obstructions. The vehicle is equipped with at least one onboard camera and millimeter-wave radar or 4D imaging radar. The camera acquires visual images at a preset frame rate, and the radar acquires radar echo data containing one or more of the following: three-dimensional spatial coordinates, radar cross-section, and radial velocity information, at a preset frequency. To alleviate the sparsity of single-frame radar echoes, one or more frames of radar echo data from the current moment and several previous historical moments can be accumulated.

[0088] Step 2012: Establish the projection mapping relationship between the radar coordinate system, camera coordinate system, and three-dimensional world coordinate system using the pre-calibrated camera intrinsic parameter matrix and radar-camera joint extrinsic parameter transformation matrix. Project the accumulated radar echoes from multiple frames onto the image plane to obtain the two-dimensional projected coordinates of each echo point.

[0089] Step 2013: Inject timestamps based on a unified clock source into the acquired data. Since there are differences in the acquisition frequency and data format between radar echo data and visual image data, the timestamps of the visual image data can be used as a benchmark to perform time window matching on the radar echo data. This assigns one or more frames of radar echo data within the corresponding time period to each frame of the visual image, generating time-synchronized or near-time-synchronized cross-modal data sample pairs.

[0090] Step 202: Semantic-guided virtual echo generation.

[0091] Step 2021: Image Semantic Acquisition. The visual image is input into a pre-trained semantic segmentation network or instance segmentation network to obtain semantic masks containing at least vehicle, pedestrian, and cyclist categories and their confidence scores. In one optional implementation, the instance segmentation network can employ Mask2Former, Mask R-CNN, or other networks capable of outputting instance-level semantic masks. The segmentation network can maintain its weights frozen during the training and inference phases of the detection network, or it can participate in joint training according to training requirements; this application does not limit this.

[0092] Step 2022: Radar Echo Projection and Foreground Extraction: Project one or more frames of accumulated radar echoes onto the image plane. For example, five frames of accumulated radar echoes can be projected onto the image plane to obtain the two-dimensional projection coordinates of each echo point. Traverse each semantic mask region and determine whether the two-dimensional projection coordinates of the radar echo points fall within the contour of a certain semantic mask. Filter out the true foreground echo points located within the semantic mask region.

[0093] Step 2023: Construction of Hybrid Probability Density Distribution. For each semantic mask instance containing at least one foreground true echo point, construct a hybrid probability density distribution within its mask support domain.

[0094] For the A two-dimensional Gaussian kernel is constructed centered on a foreground real echo point and its two-dimensional projected coordinates. In an optional implementation, the standard deviation of the Gaussian kernel is adaptively determined based on the object category and radar cross-section of the echo point, specifically expressed as:

[0095]

[0096] in, As the baseline widening factor, For category adaptive coefficients, Let be the radar cross section of the i-th foreground true echo point. In one example, You can take 7.0, and... Cut to Pixel range. For foreground true echo points with a larger radar cross-section, the adaptive standard deviation can be increased accordingly, thus having a wider spatial influence range on the image plane.

[0097] Simultaneously, weights are calculated based on the radar cross-section of each foreground real echo point. In one optional embodiment, the radar cross-section weights can be expressed as:

[0098]

[0099] in, This is the category sensitivity parameter. For small objects, such as pedestrians, a larger value can be used. To reduce the impact of differences in radar cross-section on the sampling distribution, thus preserving weak echo signals; for large objects, such as vehicles, a smaller cross-section can be used. To enhance the distinction between different echo points.

[0100] The semantic mask contour, the two-dimensional Gaussian kernel response of each foreground real echo point, and the radar cross section weight are superimposed pixel by pixel and normalized to obtain the mixed probability density distribution.

[0101] Step 2024: Hybrid Sampling and Virtual Echo Generation: Adjust parameters based on the instance area and density of each semantic mask region. Determine the virtual echo point quota that each instance needs to generate.

[0102] Importance sampling is performed on the mixed probability density distribution according to probability proportions to generate the first batch of virtual sampling points. In sparse regions of the mixed probability density distribution where the probability density is below a preset threshold, uniform supplementary sampling is performed to generate the second batch of virtual sampling points. The ratio of importance sampling to uniform sampling can be determined based on object category, semantic mask area, radar echo density, or a preset sampling strategy. For example, it can be set to a preset fixed ratio or an adaptive ratio to ensure that the total number of samples meets the virtual echo point quota.

[0103] For each generated two-dimensional virtual sampling point, the nearest real echo projection point is searched in the image plane. Its depth information, radar cross section, radial velocity, semantic category encoding (represented by an 8-dimensional one-hot vector) and mask confidence are inherited. Then, it is back-projected to the three-dimensional world coordinate system through calibration parameters to obtain the virtual echo point.

[0104] Step 2025: Dense Radar Point Cloud Construction: Merge the original real echo points (including foreground and background) with the virtual echo points, and remove duplicate points whose spatial distance is less than a preset overlap threshold to obtain a dense radar point cloud. The number of virtual echo points generated is jointly determined by density adjustment parameters, semantic instance area, and object category, so that the dense radar point cloud has more sufficient spatial coverage in the target area than the original radar echo.

[0105] Step 203: Online radar bird's-eye view feature encoding.

[0106] The densed radar point cloud obtained in step 202 is input into the radar bird's-eye view feature encoding network. In the dynamic cylinder feature network, channel identifiers are marked for real echo points and virtual echo points respectively: real echo points are marked as the first channel identifier, and virtual echo points are marked as the second channel identifier.

[0107] Independent initial linear mappings are performed on real and virtual echo points respectively, mapping real echo points to the first set of feature channels and virtual echo points to the second set of feature channels. The two sets of channels are then concatenated into a unified cylindrical feature vector. This channel-specific coding mechanism helps distinguish between real and virtual echo features and reduces the impact of noise that may be introduced by virtual echoes on the real radar response features.

[0108] Multi-scale spatial encoding is performed using a column feature extraction network (PFN), a spatially sparse convolutional encoder, and a feature pyramid network (FPN), ultimately generating a radar bird's-eye view feature map. In one example implementation, the bird's-eye view feature map is fused. The feature dimension can be ,in Indicates the batch size.

[0109] Step 204: Radar-guided visual bird's-eye view feature correction.

[0110] Step 2041: Input the visual image into the deep residual network to extract multi-scale image features. The deep residual network can maintain weight freeze during the training phase of the detection network, or it can participate in joint training according to training requirements; this application does not limit this.

[0111] The multi-scale image features are mapped to a three-dimensional voxel space through Frustum-to-Voxel multi-scale frustum transformation, and then compressed into an initial visual bird's-eye view feature map through height-direction average pooling.

[0112] Step 2042: Input the radar bird's-eye view feature map into the multi-scale confidence map generation network. Through multiple sets of parallel convolutional dimensionality reduction branches and probability normalization functions, output spatial confidence maps at multiple scales. Each pixel value in the spatial confidence map represents the confidence probability that a reliable radar geometric prior exists at the corresponding bird's-eye view location.

[0113] Step 2043: During the frustum transformation at each scale, perform element-wise residual modulation on the initial visual bird's-eye view feature map and the spatial confidence map. For the first... At each scale, the initial visual bird's-eye view feature map is denoted as... The corresponding spatial confidence map is The learnable scale adjustment parameter is The corrected visual bird's-eye view feature map It can be represented as:

[0114]

[0115] in, This represents element-wise multiplication. (When the spatial confidence graph...) When the residual modulation term approaches zero, the corrected visual bird's-eye view feature map remains the same as or close to the initial visual bird's-eye view feature map; when the spatial confidence map... When the value is high, the visual features of the area are selectively enhanced through radar geometric priors, which helps to reduce the impact of visual depth estimation bias on the distribution of aerial spatial features.

[0116] The modulated multi-scale features are refined by lightweight depthwise separable convolution, and then multi-scale convolution and dimensionality reduction convolution are performed to compress the channel dimensions to obtain the corrected visual bird's-eye view feature map. .

[0117] Step 205: Cross-modal fusion and 3D target detection.

[0118] Radar bird's-eye view feature map Compared with the corrected visual bird's-eye view feature map Perform spatial resolution alignment, and respectively through Channel alignment is performed on the convolutional layer.

[0119] The aligned two feature paths are concatenated along the channel dimension and then sequentially processed. Convolutional layers, batch normalization layers, and ReLU nonlinear activation layers perform local cross-modal interactions, enabling the convolutional kernel to learn the spatial correspondence between radar geometric features and visual semantic features in the local neighborhood.

[0120] The fused features after interaction are input into the channel attention recalibration module. The channel attention recalibration module first performs global average pooling on the fused features to generate channel-level statistics; then, it learns the importance weights of each channel through a fully connected layer; finally, it performs channel-by-channel weighting on the fused features based on these importance weights, dynamically highlighting modal channels that are more discriminative to the current scene, thus obtaining a fused bird's-eye view feature map. Its feature dimension is .

[0121] The fused bird's-eye view feature map is input into a 3D detection head, which can be an anchor-based detection head, such as AnchorHeadSingle, or an anchorless detection head. Through classification branch, bounding box regression branch, and orientation angle regression branch, the target's class probability, the center coordinates and size of the 3D bounding box, and the horizontal orientation angle are predicted, respectively. The candidate boxes output by the detection head are then subjected to confidence thresholding and non-maximum suppression to obtain the final 3D target detection result.

[0122] Step 206: Effect verification and applicability analysis.

[0123] In one example test, compared with radar vision fusion methods that do not employ bidirectional prior guidance, the method described in this application shows an improved trend in terms of radar point cloud spatial coverage of the target area, consistency of visual bird's-eye view feature space, and stability of three-dimensional target detection.

[0124] Specifically, semantically guided virtual echo generation can supplement radar echo points in the foreground semantic region, such as vehicles, pedestrians and cyclists, which helps to improve the problem of incomplete target spatial structure representation caused by sparse millimeter-wave radar echoes. It is especially suitable for scenarios with weak echoes, small-sized targets or distant targets.

[0125] Radar prior-guided visual bird's-eye view feature correction can introduce radar geometric constraints during the visual feature perspective transformation process, so that the distribution of visual bird's-eye view features in the bird's-eye view space maintains better consistency with the radar observation results, thereby reducing false responses or spatial misalignment caused by visual depth estimation errors.

[0126] Further experimental analysis shows that using either the semantically guided virtual echo generation module or the radar prior view correction module alone can improve the 3D perception results to some extent. When the two are used in combination, they can complement each other between radar geometric expression and visual semantic expression, which helps to improve the expressive ability of fused bird's-eye view features.

[0127] In terms of computational overhead, the semantically guided virtual echo generation module can be implemented as a preprocessing module for the detection network, and the radar prior view correction module can be implemented using lightweight convolutional branches. Therefore, the 3D environment perception effect can be improved while maintaining the deployability of the network structure. The specific computational overhead can vary depending on the selection of the instance segmentation network, radar coding network, visual backbone network, and detection head.

[0128] In one exemplary embodiment, such as Figure 3 As shown, a three-dimensional environment perception device based on cross-modal bidirectional prior guidance is provided, including: a data acquisition module, used to acquire one or more frames of millimeter-wave radar echo data and corresponding visual image data of the current driving scene, and perform spatiotemporal synchronization and calibration parameter analysis to obtain cross-modal aligned radar echo data and visual image data; the millimeter-wave radar echo data includes one or more of the following: three-dimensional spatial coordinates of each echo point, radar cross section, radial velocity, timestamp, or echo intensity; the visual image data is image data acquired by an onboard camera.

[0129] The semantic-guided radar enhancement module is used to perform instance-level semantic segmentation on the visual image data to obtain a semantic mask that includes at least vehicle, pedestrian and cyclist categories; based on the semantic mask, it performs semantic-guided virtual echo generation on the millimeter-wave radar echo data, and merges and deduplicates the generated virtual echo points with real radar echo points to obtain a dense radar point cloud.

[0130] The radar feature encoding module is used to input the dense radar point cloud into the radar encoding network and generate a radar bird's-eye view feature map by extracting channel features of real echo and virtual echo and encoding column scattering.

[0131] The radar prior view correction module is used to generate a multi-scale spatial confidence map based on the radar bird's-eye view feature map, and to perform a view transformation of the visual image features with radar geometric prior constraints using the multi-scale spatial confidence map to obtain the corrected visual bird's-eye view feature map.

[0132] The cross-modal fusion module is used to perform cross-modal feature fusion and channel adaptive recalibration on the radar bird's-eye view feature map and the corrected visual bird's-eye view feature map to obtain a fused bird's-eye view feature map.

[0133] The target detection module is used to perform three-dimensional target detection based on the fused bird's-eye view feature map and output the target's category, location, size, and orientation information.

[0134] In one optional implementation, the semantic-guided radar enhancement module is connected to the radar feature encoding module, and is used to output the real echo points, virtual echo points, and point type identifiers to the radar feature encoding module; the radar feature encoding module is connected to the radar prior view correction module, and is used to output a radar bird's-eye view feature map to the radar prior view correction module; the radar prior view correction module is connected to the cross-modal fusion module, and is used to output a corrected visual bird's-eye view feature map to the cross-modal fusion module; the cross-modal fusion module receives the radar bird's-eye view feature map and the corrected visual bird's-eye view feature map respectively, and outputs the fused bird's-eye view feature map to the target detection module.

[0135] In one optional embodiment, the device further includes a degradation control module, which is used to control the semantically guided radar enhancement module, the radar prior view correction module, or the cross-modal fusion module to enter a normal working state or a degraded working state based on one or more of the following: visual image quality, semantic mask confidence, radar echo quantity, or radar echo density.

[0136] In one exemplary embodiment, a computer device is provided, which may be a server, an in-vehicle edge computing terminal, or a cloud inference node, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores 3D environmental perception data, including millimeter-wave radar echo data, visual image data, semantic segmentation results, virtual echo point cloud parameters, radar bird's-eye view feature maps, visual bird's-eye view feature maps, fused bird's-eye view feature maps, and 3D target detection results. The I / O interfaces are used for exchanging information between the processor and external devices, including millimeter-wave radar sensors, vehicle-mounted cameras, and display terminals. The communication interface is used for communication with external terminals or roadside units via a network connection. When executed by a processor, this computer program implements a three-dimensional environment perception method based on cross-modal bidirectional prior guidance.

[0137] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method embodiments.

[0139] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method embodiments.

[0140] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method embodiments.

[0141] It should be noted that the data involved in this application, including but not limited to millimeter-wave radar echo data, visual image data, semantic segmentation results, virtual echo point cloud data, and 3D target detection results, can all be data that has been authorized for collection, de-identified, or complies with relevant regulations. The collection, storage, use, and processing of related data shall comply with applicable laws, regulations, and data security requirements.

[0142] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0143] The databases involved in the various embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, neural network processors, etc., and are not limited to these.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A three-dimensional environment perception method based on cross-modal bidirectional prior guidance, characterized in that, include: Acquire millimeter-wave radar echo data and visual image data in the current driving scenario, and extract features from the visual image data to obtain visual image features; Based on the semantic segmentation results of the visual image data, semantically guided virtual echo generation is performed on the millimeter-wave radar echo data to obtain a denser radar point cloud. The semantically guided virtual echo generation includes: projecting radar echoes onto an image plane and identifying foreground echo points located within a semantic mask region; constructing a probability density map based on the physical properties and geometric positions of the foreground echo points; and sampling based on the probability density map to generate virtual echo points. The densed radar point cloud is encoded into radar bird's-eye view features; A multi-scale spatial confidence map is generated based on the radar bird's-eye view features. The visual image features are then subjected to a radar prior-guided perspective transformation using the multi-scale spatial confidence map to obtain the corrected visual bird's-eye view features. The radar bird's-eye view features are fused with the corrected visual bird's-eye view features across modes to obtain fused bird's-eye view features. 3D target detection is performed based on the fused bird's-eye view features.

2. The method according to claim 1, characterized in that, The semantic segmentation result based on the visual image data is used to semantically guide the generation of virtual echoes from the millimeter-wave radar echo data, specifically including: The visual image data is segmented into instances to obtain a semantic mask; The millimeter-wave radar echo data is projected onto the image plane of the visual image data to obtain the projection coordinates of each echo point on the image plane. Based on the correspondence between the projected coordinates of each echo point and the semantic mask, the foreground echo points located within the semantic mask area are determined; For each foreground echo point, a two-dimensional Gaussian kernel is constructed with its projected coordinates as the center. The standard deviation of the two-dimensional Gaussian kernel is adaptively determined based on the object category and radar cross section corresponding to the foreground echo point. The semantic mask, the two-dimensional Gaussian kernel response of each foreground echo point, and the radar cross section weights are fused to construct a hybrid probability density map. Importance sampling is performed based on the hybrid probability density map, and uniform sampling is added to the sparse region to generate virtual echo points; The generated virtual echo points inherit the depth information of the nearest real echo points, and the virtual echo points are back-projected onto the three-dimensional spatial coordinate system based on the depth information; the virtual echo points obtained by back-projection are merged with the real radar echo points to obtain a dense radar point cloud.

3. The method according to claim 2, characterized in that, The standard deviation of the two-dimensional Gaussian kernel is determined as follows: Obtain the baseline standard deviation of the object category corresponding to the foreground echo point; Calculate the scale modulation factor based on the relationship between the radar cross-section of the foreground echo point and the preset reference radar cross-section. The standard deviation of the two-dimensional Gaussian kernel is determined based on the baseline standard deviation and the scale modulation factor. Among them, the two-dimensional Gaussian kernel corresponding to the foreground echo point with a larger radar cross section has a larger spatial influence range, while the two-dimensional Gaussian kernel corresponding to the foreground echo point with a smaller radar cross section has a smaller spatial influence range.

4. The method according to claim 2, characterized in that, The weighting based on the radar cross section of each foreground echo point includes: Obtain the radar cross section sensitivity parameter of the object category corresponding to the foreground echo point; The radar cross section weight of the foreground echo point is calculated based on the ratio of the radar cross section of the foreground echo point to the maximum radar cross section, combined with the sensitivity parameter. Specifically, a larger sensitivity parameter is used for small objects to retain weak echo signals, while a smaller sensitivity parameter is used for large objects to enhance the distinguishability of the radar cross section.

5. The method according to claim 2, characterized in that, The importance sampling based on the mixed probability density map, and the supplementation of uniform sampling in sparse regions, includes: Based on the number of target points and the density parameters of each object category, determine the number of virtual echo points that need to be generated for each instance; Importance sampling is performed based on the hybrid probability density map to generate the first batch of virtual echo points; In the sparse regions of the hybrid probability density map where the probability density is below a preset threshold, uniform sampling is performed to supplement virtual echo points, so that the total number of generated virtual echo points meets the target number.

6. The method according to claim 1, characterized in that, The process of generating a multi-scale spatial confidence map based on the radar bird's-eye view features, and using the multi-scale spatial confidence map to perform radar prior-guided perspective transformation on the visual image features, includes: The radar bird's-eye view features are input into multiple parallel convolutional network branches. Each branch contains cascaded convolutional layers, batch normalization layers, and activation function layers. The channel dimension decreases layer by layer. Finally, the spatial confidence map is output through the Sigmoid function to obtain radar confidence maps at multiple scales. The visual image features are projected into a three-dimensional voxel space through depth distribution estimation and aggregated into initial visual bird's-eye view features through depth direction pooling. The radar confidence maps at each scale are multiplied element-wise with the initial visual bird's-eye view features at the corresponding scale, and modulated by a learnable scaling factor to obtain the visual bird's-eye view features corrected for each scale. The corrected visual bird's-eye view features at each scale are stitched together, and the channel dimensions are compressed through a dimensionality-reducing convolutional network to obtain the corrected visual bird's-eye view features.

7. The method according to claim 6, characterized in that, The step of element-wise multiplying radar confidence maps at each scale with initial visual bird's-eye view features at the corresponding scale, and modulating them using a learnable scaling factor, includes: For the Each scale will represent the initial visual bird's-eye view features. Corresponding radar confidence map Perform element-wise multiplication to obtain the weighted features. ; The weighted features are combined with a learnable scaling factor After multiplication, it is combined with the initial visual bird's-eye view features. Add them together to get the first one. Visual bird's-eye view features after scale correction ,in , This indicates element-wise multiplication; Among them, the visual bird's-eye view features corresponding to areas with higher radar confidence map values ​​are enhanced through residual modulation, while the visual bird's-eye view features corresponding to areas with lower radar confidence map values ​​maintain the original response or are close to the original response.

8. The method according to claim 1, characterized in that, The cross-modal fusion of the radar bird's-eye view features and the corrected visual bird's-eye view features includes: The radar bird's-eye view features and the corrected visual bird's-eye view features are stitched together in the channel dimension; The concatenated features are input into a convolutional block for feature fusion to obtain fused intermediate features. The fused intermediate features are subjected to channel recalibration or spatial weighting to obtain the fused bird's-eye view features.

9. A three-dimensional environment perception device based on cross-modal bidirectional prior guidance, characterized in that, include: The data acquisition module is used to acquire millimeter-wave radar echo data and visual image data in the current driving scenario, and to extract features from the visual image data to obtain visual image features. The semantically guided radar enhancement module is used to generate a semantically guided virtual echo from the millimeter-wave radar echo data based on the semantic segmentation results of the visual image data, thereby obtaining a denser radar point cloud. The radar feature encoding module is used to encode the densed radar point cloud into radar bird's-eye view features; The radar prior view correction module is used to generate a multi-scale spatial confidence map based on the radar bird's-eye view features, and to perform radar prior-guided view transformation on the visual image features using the multi-scale spatial confidence map to obtain the corrected visual bird's-eye view features. A cross-modal fusion module is used to perform cross-modal fusion of the radar bird's-eye view features and the corrected visual bird's-eye view features to obtain fused bird's-eye view features; The target detection module is used to perform three-dimensional target detection based on the fused bird's-eye view features.

Images