BEV semantic prior-based 4D radar point cloud enhancement method

Abstract

The invention relates to the field of 4D radar point cloud enhancement in intelligent driving, in particular to a BEV semantic prior-based 4D radar point cloud enhancement method. According to the technical scheme, the method comprises the steps of collecting image data and 4D radar point cloud data; preprocessing the image data to obtain a BEV semantic graph and corresponding visual uncertainty; performing semantic context alignment on the 4D radar point cloud data and the BEV semantic graph to obtain a semantic enhanced radar point cloud; receiving the semantic enhanced radar point cloud, and classifying each point; and carrying out dynamic fusion on the BEV semantic graph, the corresponding visual uncertainty and the classified radar point cloud under different working conditions, and generating a final perception result. According to the invention, high-precision visual BEV perception influenced by weather and all-weather, sparse and loud-noise 4D radar perception are efficiently and robustly fused. The method is suitable for intelligent driving.

Description

Technical Field

[0001] This invention relates to the field of 4D radar point cloud enhancement in intelligent driving, specifically to a 4D radar point cloud enhancement method based on BEV (Bird's-Eye-View) semantic prior. Background Technology

[0002] In intelligent driving environment perception, camera-based bird's-eye view perception technology has become one of the mainstream solutions. This type of advanced pure vision BEV solution has achieved great success in building high-precision, semantically rich scene understanding through spatiotemporal fusion and attention mechanisms.

[0003] However, such purely visual solutions have inherent physical limitations: under extreme weather conditions (such as heavy rain, dense fog, and snow) or extreme lighting conditions (such as strong glare or tunnel exits), the performance of the camera will drop sharply, causing the perception system to temporarily "go blind".

[0004] The current mainstream solution in the industry is to introduce 4D millimeter-wave radar. 4D radar has the ability to work in all weather conditions (it can penetrate rain and fog), accurately measures speed information, and is cost-effective. However, how to efficiently and robustly integrate high-precision but weather-affected visual BEV perception with all-weather but sparse and noisy 4D radar perception has become a core technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a 4D radar point cloud enhancement method based on BEV semantic prior, which realizes the efficient and robust fusion of high-precision but weather-affected visual BEV perception with all-weather but sparse and noisy 4D radar perception.

[0006] This invention achieves the above objectives by adopting the following technical solution: This invention provides a 4D radar point cloud enhancement method based on BEV semantic prior, comprising:

[0007] S1. Acquire image data and 4D radar point cloud data;

[0008] S2. Preprocess the image data to obtain the BEV semantic map and the corresponding visual uncertainty;

[0009] S3. Align the 4D radar point cloud data with the BEV semantic graph to obtain a semantically enhanced radar point cloud.

[0010] S4. Receive semantically enhanced radar point cloud and classify each point;

[0011] S5. Dynamically fuse the BEV semantic map, the corresponding visual uncertainty, and the classified radar point cloud under different operating conditions to generate the final perception result.

[0012] Furthermore, step S1 specifically includes:

[0013] Multiple wide-angle cameras and 4D millimeter-wave radars deployed around the vehicle body are used to simultaneously collect image data and 4D radar point cloud data around the vehicle.

[0014] Furthermore, step S2 specifically includes:

[0015] Features are extracted from the acquired image data using ResNet or Swing Transformer, and a BEV semantic map is generated. At the same time, the visual uncertainty of the BEV semantic map is estimated.

[0016] Furthermore, step S3 specifically includes:

[0017] By using the vehicle's intrinsic and extrinsic parameter calibration matrix, the 4D radar point cloud is... Each point in Projected onto the BEV semantic graph In the two-dimensional raster coordinate system;

[0018] In BEV semantic graph Search The grid into which it falls Corresponding semantic feature vector The method is as follows:

[0019] ;

[0020] In the formula, Represents raster coordinates;

[0021] Then a semantically enhanced radar point cloud is generated.

[0022] Furthermore, step S4 specifically includes:

[0023] A lightweight point cloud processing network is employed, trained to understand the combination of physical and semantic relationships and distinguish clutter from obstacles, as detailed below:

[0024] Scene 1:

[0025] The characteristics of the input points are:

[0026] ;

[0027] ;

[0028] If the network identifies a combination pattern that is highly reflective, stationary, and located in a high-probability road area, it classifies it as static road clutter and suppresses or discards it during output.

[0029] Scene Two:

[0030] The characteristics of the input points are:

[0031] ;

[0032] ;

[0033] If the network identifies a combination pattern of high reflectivity, stationary location, and high probability of being in a vehicle area, it will classify it as a high-confidence obstacle and the point will be enhanced during output.

[0034] Third Scene:

[0035] The characteristics of the input points are:

[0036] ;

[0037] ;

[0038] If the physical reflection at that point is low, it is classified as a dynamic pedestrian.

[0039] Furthermore, step S5 specifically includes:

[0040] Receive the BEV semantic map, the corresponding visual uncertainty, and the classified radar point cloud;

[0041] The system determines whether the visual uncertainty exceeds the set threshold. If it does, the system determines that the vision has failed and then dynamically switches to the trust mode, which uses the original point cloud of the 4D radar as the basic perception source data. If not, the system determines that the vision is correct.

[0042] The high-confidence BEV semantic map is fused with the classified radar point cloud, and the final BEV perception result is generated by grid overlay.

[0043] The beneficial effects of this invention are as follows:

[0044] This invention abandons the blind voting at the result layer or the blind stitching at the feature layer of existing fusion schemes (such as post-fusion) or pre-fusion. Through BEV-radar semantic context alignment and radar point cloud cognitive network, this invention endows each 4D original radar point cloud with visual semantic context. Attached Figure Description

[0045] Figure 1 This is a flowchart of a 4D radar point cloud enhancement method based on BEV semantic prior provided by the present invention;

[0046] Figure 2 This is a flowchart of the semantic enhancement classification and discrimination of each radar point cloud provided by the present invention;

[0047] Figure 3 This is a flowchart for determining the working condition provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0049] This invention provides a 4D radar point cloud enhancement method based on BEV semantic prior, such as... Figure 1 As shown, it specifically includes:

[0050] S1. Acquire image data and 4D radar point cloud data;

[0051] Multiple wide-angle cameras and 4D millimeter-wave radars deployed around the vehicle body are used to simultaneously collect image data and 4D radar point cloud data around the vehicle.

[0052] S2. Preprocess the image data to obtain the BEV semantic map and the corresponding visual uncertainty;

[0053] Features are extracted from the acquired image data using ResNet and Swing Transformer, and a dense, semantically rich BEV semantic graph is generated. And at the same time, estimate the visual uncertainty of the BEV semantic map (e.g., the uncertainty is higher in dense fog).

[0054] The specific process is as follows:

[0055] Feature Extraction: First, the multi-view image data collected by the vehicle camera is input into the ResNet or Swin Transformer backbone network with shared weights to extract multi-scale 2D feature maps containing rich texture and semantic information.

[0056] View Transformation: Next, using the view transformation module (e.g., LSS-based depth estimation projection or Transformer-based attention mechanism), combined with the intrinsic and extrinsic parameter calibration matrices of each camera, the extracted 2D feature map is mapped from the perspective view to a three-dimensional geometric space, and further projected onto a unified BEV grid coordinate system to form the initial BEV features.

[0057] Feature Encoding & Decoding: The projected features are fused and context-enhanced using the BEV Encoder. Finally, the probability of each BEV raster belonging to different categories (such as roads, vehicles, and pedestrians) is predicted using the Segmentation Head to generate a BEV semantic map.

[0058] At the same time, the visual uncertainty of the current scene is estimated through uncertainty branching (e.g., calculating entropy or prediction variance).

[0059] S3. Align the 4D radar point cloud data with the BEV semantic graph to obtain a semantically enhanced radar point cloud.

[0060] Projection: Using the vehicle's intrinsic and extrinsic parameter calibration matrix, the 4D radar point cloud is projected. Each point in Projected onto the BEV semantic graph In the two-dimensional (x, y) grid coordinate system.

[0061] in, Indicates longitudinal distance, usually referring to the front of the vehicle; It indicates lateral distance, usually referring to the left-right direction of a vehicle; Indicates vertical height, usually referring to the direction perpendicular to the ground; This represents the Doppler velocity of the target, i.e., the radial relative velocity; (Radar Cross Section)

[0062] Meaning: In Search The grid into which it falls Corresponding semantic feature vector .

[0063] ;

[0064] In the formula, This represents the probability that the point cloud belongs to a road, that is, the likelihood that the system considers the current location to be a drivable road surface. If the value is 0.95, it is determined that this is a road; if the value is 0.01, it means that this is most likely not a road (it may be grass or obstacles). This represents the probability that a point cloud belongs to a vehicle, that is, the likelihood that the system believes that the current location is occupied by a vehicle. When the radar point hits a metal object and the BEV image recognizes that there is a vehicle there, this value will be very high (close to 1). This represents the probability that a point cloud belongs to a pedestrian, that is, the possibility that the system believes there is a pedestrian walking at this current location.

[0065] Output: Generate a semantically enhanced radar point cloud. Now, at every point It not only contains its physical properties, but also carries the visual semantic context of its location.

[0066] ;

[0067] S4. Receive semantically enhanced radar point cloud and classify each point;

[0068] like Figure 2 As shown, the specific process is as follows:

[0069] Input: Semantically enhanced radar point cloud

[0070] Network architecture: A lightweight point cloud processing network (such as PointNet++ or GNN) is used, which is trained to understand the combined relationship between physics and semantics.

[0071] Distinguishing between clutter and obstructions:

[0072] Scene 1 (Manhole Cover Clutter):

[0073] If input point The characteristics are:

[0074] ;

[0075]

[0076] If the network identifies a combination pattern that is highly reflective, stationary, and located in a high-probability road area, it classifies it as static road clutter and suppresses or discards it during output.

[0077] Second scenario (stationary vehicle):

[0078] If input point The characteristics are:

[0079] ;

[0080] ;

[0081] The probability that the point cloud falls into an area belonging to a road is low (e.g., less than 0.2), while the probability that it belongs to a vehicle is high (e.g., greater than 0.8). This indicates that visual perception strongly believes that a vehicle is present in that area.

[0082] If the network identifies a combination pattern of high reflectivity, stationary location, and high probability of being in a vehicle area, it will classify it as a high-confidence obstacle and the point will be enhanced during output.

[0083] Scene 3 (Sparse pedestrians):

[0084] If input point The characteristics are:

[0085] ;

[0086] ;

[0087] Despite the physical reflection at that point The probability is low, but visual semantics indicate a high probability that the area belongs to a pedestrian (e.g., greater than 0.6), thus classifying it as a valid target. Classifying it as a dynamic pedestrian via a network solves the problem of difficult classification due to sparse radar point clouds.

[0088] Finally, the output consists of cleaned and categorized radar point clouds.

[0089] S5. Dynamically fuse the BEV semantic map, the corresponding visual uncertainty, and the classified radar point cloud under different operating conditions to generate the final perception result.

[0090] like Figure 3 As shown, the process of determining the condition based on the working conditions (such as sunny day, dense fog, or obstruction) is as follows.

[0091] Dynamic Trust - Dense Fog / Heavy Rain:

[0092] The system determines whether the visual uncertainty exceeds a set threshold. If so, it identifies a visual failure and dynamically switches to a trust mode, which then fully utilizes the raw point cloud data from the 4D radar. If the visual data is not correct, the fusion result is output, which ensures that the vehicle retains the basic obstacle avoidance capability of the radar even when it is blind in extreme weather.

[0093] Fusion: Integrating cleaned radar point clouds (Including high-confidence real obstacles) and the original visual BEV semantic map Perform a final fusion (e.g., by overlaying BEV grids) to generate a robust, all-weather BEV occupancy map.

[0094] Dynamic Trust - Occlusion:

[0095] In the ghost peek-out scene A pedestrian with high confidence was output in a certain area, but The point cloud in this area is zero (due to occlusion).

[0096] Switching: During fusion, the high-confidence visual results are naturally trusted to ensure that targets obscured by radar are not missed.

[0097] This invention solves the "ghost brake" problem caused by radar clutter by eliminating noise at the source.

[0098] Specifically, this invention abandons the blind voting at the result layer of existing fusion schemes (such as post-fusion) or the blind stitching at the feature layer (such as pre-fusion). Through BEV-radar semantic context alignment and radar point cloud cognitive network, this invention endows each 4D original radar point cloud with visual semantic context.

[0099] This design enables the system to intelligently understand the true meaning of a radar point. For example, the system can identify a point with (v=0, RCS=high) + (context=road) as a manhole cover (clutter), while a point with (v=0, RCS=high) + (context=vehicle) is a stationary vehicle (obstacle). This intelligent noise reduction at the data source fundamentally solves the "ghost braking" problem caused by clutter misjudgment, significantly improving driving safety and comfort.

[0100] This invention solves the problem of sparse and difficult-to-classify point clouds in 4D radar, greatly improving the ability to perceive small targets.

[0101] Specifically, in response to the shortcomings of radar in detecting sparse point clouds and low radar cross-section (RCS) of targets such as pedestrians, the "radar point cloud cognitive network" of this invention is trained to simultaneously evaluate its "physical properties" and "semantic context".

[0102] A physically weak radar point (low RCS) if its semantic context ( The system's ability to identify pedestrians and bicycles as "real obstacles" is greatly enhanced when the object is located in the "pedestrian" zone. This "semantic guidance" enhancement mechanism effectively solves the problem of missed detections caused by sparse radar point clouds, making the system's perception of key traffic participants such as pedestrians and bicycles stronger and more reliable.

[0103] This invention addresses the shortcomings of existing fusion solutions in terms of poor robustness under extreme conditions, achieving true "all-weather" sensing.

[0104] Specifically, this invention incorporates a dynamic trust and all-weather fusion method to continuously monitor visual uncertainties. This design provides a robust "fail-safe" mechanism.

[0105] In dense fog / heavy rain, when the visual model When a failure occurs, the system detects its high uncertainty and automatically switches the trust focus, downgrading to a "pure radar perception" mode to ensure basic safety. This avoids the problem of "dirty" visual features contaminating radar features and causing model collapse, which is common in "pre-fusion" solutions.

[0106] In obstructed scenarios, when the radar is blocked but the visual field is clear, the system will trust the high confidence of the visual results to ensure that targets such as "ghost peeking out" are not missed.

[0107] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A 4D radar point cloud enhancement method based on BEV semantic prior, characterized in that, include: S1. Acquire image data and 4D radar point cloud data; S2. Preprocess the image data to obtain the BEV semantic map and the corresponding visual uncertainty; S3. Align the 4D radar point cloud data with the BEV semantic graph to obtain a semantically enhanced radar point cloud. S4. Receive semantically enhanced radar point cloud and classify each point; S5. Dynamically fuse the BEV semantic map, the corresponding visual uncertainty, and the classified radar point cloud under different operating conditions to generate the final perception result.

2. The 4D radar point cloud enhancement method based on BEV semantic prior as described in claim 1, characterized in that, Step S1 specifically includes: Multiple wide-angle cameras and 4D millimeter-wave radars deployed around the vehicle body are used to simultaneously collect image data and 4D radar point cloud data around the vehicle.

3. The 4D radar point cloud enhancement method based on BEV semantic prior as described in claim 2, characterized in that, Step S2 specifically includes: Features are extracted from the acquired image data using ResNet or Swing Transformer, and a BEV semantic map is generated. At the same time, the visual uncertainty of the BEV semantic map is estimated.

4. The 4D radar point cloud enhancement method based on BEV semantic prior as described in claim 3, characterized in that, Step S3 specifically includes: By using the vehicle's intrinsic and extrinsic parameter calibration matrix, the 4D radar point cloud is... Each point in Projected onto the BEV semantic graph In the two-dimensional raster coordinate system; in, Indicates vertical distance. Indicates horizontal distance. Indicates vertical height. Indicates the Doppler velocity of the target. Represents the radar cross section; In BEV semantic graph Search The grid into which it falls Corresponding semantic feature vector The method is as follows: ； In the formula, Represents raster coordinates, This represents the probability that a point cloud belongs to a road. This represents the probability that the point cloud belongs to a vehicle. This represents the probability that a point cloud belongs to a pedestrian. Then a semantically enhanced radar point cloud is generated.

5. The 4D radar point cloud enhancement method based on BEV semantic prior as described in claim 4, characterized in that, Step S4 specifically includes: A lightweight point cloud processing network is employed, trained to understand the combination of physical and semantic relationships and distinguish clutter from obstacles, as detailed below: Scene 1: If the features of the input point are: ； ； If the network identifies a combination pattern that is highly reflective, stationary, and located in a high-probability road area, it classifies it as static road clutter and suppresses or discards it during output. Scene Two: If the features of the input point are: ； ； If the network identifies a combination pattern of high reflectivity, stationary location, and high probability of being in a vehicle area, it will classify it as a high-confidence obstacle, and the input point will be enhanced during output. Third Scene: If the features of the input point are: ； ； The input point is then judged to have low physical reflection and classified as a dynamic pedestrian.

6. The 4D radar point cloud enhancement method based on BEV semantic prior as described in claim 5, characterized in that, Step S5 specifically includes: Receive the BEV semantic map, the corresponding visual uncertainty, and the classified radar point cloud; The system determines whether the visual uncertainty exceeds the set threshold. If it does, the system determines that the vision has failed and then dynamically switches to the trust mode, which uses the original point cloud of the 4D radar as the basic perception source data. If not, the system determines that the vision is correct. The high-confidence BEV semantic map is fused with the classified radar point cloud, and the final BEV perception result is generated by grid overlay.

Images