A 4D radar point cloud enhancement method based on BEV semantic prior

Abstract

The application relates to the field of 4D radar point cloud enhancement in intelligent driving, in particular to a 4D radar point cloud enhancement method based on BEV semantic priori. The technical scheme comprises the following steps: collecting image data and 4D radar point cloud data; pre-processing the image data to obtain a BEV semantic graph and corresponding visual uncertainty; aligning the 4D radar point cloud data and the BEV semantic graph in terms of semantic context to obtain a radar point cloud enhanced in semantics; receiving the radar point cloud enhanced in semantics and classifying each point; dynamically fusing the BEV semantic graph, the corresponding visual uncertainty and the classified radar point cloud under different working conditions to generate a final perception result. The application realizes efficient and robust fusion of high-precision but weather-affected visual BEV perception and all-weather but sparse and noisy 4D radar perception. The application is suitable for intelligent driving.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of 4D radar point cloud enhancement in intelligent driving, and in particular to a 4D radar point cloud enhancement method based on BEV (Bird's-Eye-View) semantic prior. BACKGROUND

[0002] In the intelligent driving environment perception, the bird's-eye view perception technology based on camera has become one of the mainstream solutions. This kind of advanced pure vision BEV scheme has achieved great success in building high-precision and rich semantic scene understanding through spatio-temporal fusion and attention mechanism.

[0003] However, this kind of pure vision scheme has inherent limitations in physics: in extreme weather (such as heavy rain, thick fog, snowy weather) or extreme light (such as strong glare, tunnel exit) conditions, the performance of the camera will decrease sharply, causing the perception system to be temporarily "blind".

[0004] The mainstream solution in the industry at present is to introduce 4D millimeter wave radar. The 4D radar has the ability to work all-weather (can penetrate rain and fog), measures speed information accurately, and the cost is controllable. However, how to efficiently and robustly fuse the high-precision but weather-affected visual BEV perception with the all-weather but sparse and noisy 4D radar perception has become a core technical problem to be solved in the field. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art, and to provide a 4D radar point cloud enhancement method based on BEV semantic prior, which realizes efficient and robust fusion of high-precision but weather-affected visual BEV perception with all-weather but sparse and noisy 4D radar perception.

[0006] The present application achieves the above-mentioned purpose by adopting the following technical scheme, and provides a 4D radar point cloud enhancement method based on BEV semantic prior, comprising:

[0007] S1, collecting image data and 4D radar point cloud data;

[0008] S2, pre-processing the image data to obtain a BEV semantic map and corresponding visual uncertainty;

[0009] S3, aligning the 4D radar point cloud data and the BEV semantic map in semantic context to obtain a semantically enhanced radar point cloud;

[0010] S4, receiving the semantically enhanced radar point cloud and classifying each point;

[0011] S5. Dynamically fuse the BEV semantic map, the corresponding visual uncertainty, and the classified radar point cloud under different operating conditions to generate the final perception result.

[0012] Furthermore, step S1 specifically includes:

[0013] Multiple wide-angle cameras and 4D millimeter-wave radars deployed around the vehicle body are used to simultaneously collect image data and 4D radar point cloud data around the vehicle.

[0014] Furthermore, step S2 specifically includes:

[0015] Features are extracted from the acquired image data using ResNet or Swing Transformer, and a BEV semantic map is generated. At the same time, the visual uncertainty of the BEV semantic map is estimated.

[0016] Furthermore, step S3 specifically includes:

[0017] By using the vehicle's intrinsic and extrinsic parameter calibration matrix, the 4D radar point cloud is... Each point in Projected onto the BEV semantic graph In the two-dimensional raster coordinate system;

[0018] In BEV semantic graph Search The grid into which it falls Corresponding semantic feature vector The method is as follows:

[0019] ;

[0020] In the formula, Represents raster coordinates;

[0021] Then a semantically enhanced radar point cloud is generated.

[0022] Furthermore, step S4 specifically includes:

[0023] A lightweight point cloud processing network is employed, trained to understand the combination of physical and semantic relationships and distinguish clutter from obstacles, as detailed below:

[0024] Scene 1:

[0025] The characteristics of the input points are:

[0026] ;

[0027] ;

[0028] Then the network determines that the combination of high reflection, static and located in the high probability road area is recognized, and is classified as static road clutter, and the output is suppressed or discarded;

[0029] The second scenario:

[0030] The characteristics of the input point are:

[0031] ;

[0032] ;

[0033] Then the network determines that the combination of high reflection, static and located in the high probability vehicle area is recognized, and is classified as a high-confidence obstacle, and the point will be enhanced in the output;

[0034] The third scenario:

[0035] The characteristics of the input point are:

[0036] ;

[0037] ;

[0038] Then the network determines that the physical reflection of the point is low, and is classified as a dynamic pedestrian.

[0039] Further, step S5 specifically comprises:

[0040] Receiving the BEV semantic graph, the corresponding visual uncertainty and the classified radar point cloud;

[0041] Working condition discrimination, judging whether the visual uncertainty is greater than the set threshold, if yes, determining that the vision is invalid, and then dynamically switching the trust mode, that is, completely using the original point cloud of the 4D radar as the basic perception source data, if not, determining that the vision is correct;

[0042] Fusing the high-confidence BEV semantic graph and the classified radar point cloud, and generating the final BEV perception result through grid superposition.

[0043] The beneficial effects of the present application are:

[0044] The present application discards the blind voting of the existing fusion scheme (such as post-fusion) in the result layer or the blind splicing of (such as pre-fusion) in the feature layer. Through the BEV-radar semantic context alignment and the radar point cloud cognition network, the present application gives each 4D radar original point cloud a visual semantic context. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a 4D radar point cloud enhancement method flowchart based on BEV semantic prior provided by the present application;

[0046] Figure 2 is a flow chart of the classification and discrimination of each radar point cloud enhanced by semantics provided by the present application;

[0047] Figure 3 is a working condition discrimination flow chart provided by the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0049] The present application provides a 4D radar point cloud enhancement method based on BEV semantic prior, as shown in Figure 1 , specifically comprising:

[0050] S1, collecting image data and 4D radar point cloud data;

[0051] Through the multiple wide-angle cameras and 4D millimeter wave radars deployed around the vehicle body, the image data and 4D radar point cloud data around the vehicle are synchronously collected.

[0052] S2, pre-processing the image data to obtain a BEV semantic map and corresponding visual uncertainty;

[0053] Through ResNet and Swin Transformer, features are extracted from the collected image data, and a dense and rich semantic BEV semantic map is generated , and the visual uncertainty of the BEV semantic map is estimated at the same time (for example, the uncertainty is higher in heavy fog).

[0054] The specific process is as follows:

[0055] Feature Extraction: First, the multi-view image data collected by the vehicle-mounted camera is input into the ResNet or Swin Transformer backbone network with shared weights, and multi-scale 2D feature maps containing rich texture and semantic information are extracted.

[0056] View Transformation: Then, using a view transformation module (such as a depth estimation projection based on LSS or an attention mechanism based on Transformer), combined with the internal and external parameter calibration matrix of each camera, the extracted 2D feature maps are mapped from the perspective view to the three-dimensional geometric space, and further projected to the unified BEV grid coordinate system, forming the initial BEV feature.

[0057] Feature Encoding & Decoding: The projected features are fused and contextually enhanced using a BEV encoder. Finally, a segmentation head is used to predict the probability of each BEV grid belonging to different classes (e.g., road, vehicle, pedestrian), generating a BEV semantic map.

[0058] At the same time, the visual uncertainty of the current scene is estimated through an uncertainty branch (e.g., calculating the entropy value or predicting the variance).

[0059] S3, align the 4D radar point cloud data with the BEV semantic map in semantic context to obtain a radar point cloud enhanced by semantics.

[0060] Projection: Through the intrinsic and extrinsic calibration matrix of the vehicle, each point in the 4D radar point cloud is projected into the two-dimensional (x, y) grid coordinate system of the BEV semantic map.

[0061] wherein represents the longitudinal distance, usually referring to the front of the vehicle; represents the lateral distance, usually referring to the left and right directions of the vehicle; represents the vertical height, usually referring to the vertical direction of the ground; represents the Doppler velocity of the target, i.e., the radial relative speed; (Radar Cross Section, Radar Cross Section).

[0062] Assignment: In , find the corresponding semantic feature vector of the grid where the point cloud falls. .

[0063] ;

[0064] In the formula, represents the probability of the point cloud belonging to the road, i.e., the possibility of the system considering the current position as a drivable road surface (Road). If the value is 0.95, it is judged as a road; if the value is 0.01, it is likely that it is not a road (may be grass or an obstacle); represents the probability of the point cloud belonging to the vehicle, i.e., the possibility of the system considering the current position as being occupied by a vehicle (Vehicle). When the radar point hits a metal object and the BEV image recognizes that there is a vehicle here, the value will be very high (close to 1); ​​​The probability that the point cloud belongs to a pedestrian, i.e. the likelihood that the system considers the current position as a walking pedestrian (Pedestrian).

[0065] Output: A semantically enhanced radar point cloud is generated Now, each point contains not only its physical properties, but also carries the visual semantic context of its location.

[0066] ;

[0067] S4, receive the semantically enhanced radar point cloud and classify each point;

[0068] As Figure 2 shown, the specific process is as follows:

[0069] Input: Semantically enhanced radar point cloud

[0070] Network architecture: A lightweight point cloud processing network (such as PointNet++ or GNN) is adopted, which is trained to understand the combined relationship between physics and semantics.

[0071] Distinguish clutter and obstacles:

[0072] First scenario (manhole cover clutter):

[0073] If the features of the input point are:

[0074] ;

[0075]

[0076] The network determines that it recognizes the combination of high reflection, static and located in the high probability road area, and classifies it as static road clutter, which is suppressed or discarded when output;

[0077] Second scenario (stationary vehicle):

[0078] If the features of the input point are:

[0079] ;

[0080] ;

[0081] That is, the probability that the point cloud falls into the area belongs to the road is low (for example, less than 0.2), while the probability that it belongs to the vehicle is high (for example, greater than 0.8). This indicates that visual perception strongly believes that there is a vehicle here.

[0082] Then the network identifies the combination of high reflection, static and in high probability vehicle area, and classifies it as high confidence obstacle, and outputs that this point will be enhanced;

[0083] The third scenario (sparse pedestrian):

[0084] If the input point has the following features:

[0085] ;

[0086] ;

[0087] Although the physical reflection of this point is low, the visual semantics show that the area has a high probability of being a pedestrian (e.g. greater than 0.6), so it is determined to be a valid target. It is classified as a dynamic pedestrian by the network, solving the problem of sparse radar point cloud classification.

[0088] Finally, output a cleaned and classified radar point cloud

[0089] S5, fuse the BEV semantic map, corresponding visual uncertainty and classified radar point cloud in different working conditions, and generate the final perception result.

[0090] As Figure 3 shown, according to the working condition (such as sunny, heavy fog, occlusion), the identification process is as follows.

[0091] Dynamic trust - heavy fog / heavy rain:

[0092] Working condition discrimination, judge whether the visual uncertainty is greater than the set threshold, if so, determine that the vision is invalid, then dynamically switch the trust mode, that is, completely use the original point cloud of 4D radar as the basic perception source data, if not, determine that the vision is correct, output the fusion result, which ensures that the vehicle still retains the basic obstacle avoidance ability of radar when it is blind in extreme weather.

[0093] Fusion: fuse the cleaned radar point cloud (containing high confidence real obstacles) with the original visual BEV semantic map to generate a robust, all-weather BEV occupancy map (e.g. by BEV grid stacking).

[0094] Dynamic trust - occlusion:

[0095] In the ghost head scenario, a high confidence pedestrian is output in a certain area, but the point cloud is zero in that area (due to occlusion).

[0096] Switching: During fusion, the high-confidence visual results are naturally trusted to ensure that targets obscured by radar are not missed.

[0097] This invention solves the "ghost brake" problem caused by radar clutter by eliminating noise at the source.

[0098] Specifically, this invention abandons the blind voting at the result layer of existing fusion schemes (such as post-fusion) or the blind stitching at the feature layer (such as pre-fusion). Through BEV-radar semantic context alignment and radar point cloud cognitive network, this invention endows each 4D original radar point cloud with visual semantic context.

[0099] This design enables the system to intelligently understand the true meaning of a radar point. For example, the system can identify a point with (v=0, RCS=high) + (context=road) as a manhole cover (clutter), while a point with (v=0, RCS=high) + (context=vehicle) is a stationary vehicle (obstacle). This intelligent noise reduction at the data source fundamentally solves the "ghost braking" problem caused by clutter misjudgment, significantly improving driving safety and comfort.

[0100] This invention solves the problem of sparse and difficult-to-classify point clouds in 4D radar, greatly improving the ability to perceive small targets.

[0101] Specifically, in response to the shortcomings of radar in detecting sparse point clouds and low radar cross-section (RCS) of targets such as pedestrians, the "radar point cloud cognitive network" of this invention is trained to simultaneously evaluate its "physical properties" and "semantic context".

[0102] A physically weak radar point (low RCS) if its semantic context ( The system's ability to identify pedestrians and bicycles as "real obstacles" is greatly enhanced when the object is located in the "pedestrian" zone. This "semantic guidance" enhancement mechanism effectively solves the problem of missed detections caused by sparse radar point clouds, making the system's perception of key traffic participants such as pedestrians and bicycles stronger and more reliable.

[0103] This invention addresses the shortcomings of existing fusion solutions in terms of poor robustness under extreme conditions, achieving true "all-weather" sensing.

[0104] Specifically, this invention incorporates a dynamic trust and all-weather fusion method to continuously monitor visual uncertainties. This design provides a robust "fail-safe" mechanism.

[0105] In dense fog / heavy rain, when the visual model When failure occurs, the system detects its high uncertainty and automatically switches the trust center, degrades to a "pure radar perception" mode, and ensures basic safety. This avoids the problem of "dirty" visual features polluting radar features and causing model collapse in the "pre-fusion" scheme.

[0106] In the occlusion scene, when the radar is occluded and the vision is clear, the system trusts the high confidence result of the vision to ensure that targets such as "ghost heads" are not missed.

[0107] The above only describes the preferred embodiments of the present application, and it should be understood that the present application is not limited to the forms disclosed herein, should not be considered as excluding other embodiments, and can be used in various other combinations, modifications and environments, and can be modified within the scope of the concepts described herein by the above teachings or related art or knowledge. Any modification and change made by those skilled in the art without departing from the spirit and scope of the present application shall be within the protection scope of the appended claims of the present application.

Claims

1. A 4D radar point cloud enhancement method based on BEV semantic prior, characterized in that, Comprise: S1, collect image data and 4D radar point cloud data; S2, pre-process the image data to obtain a BEV semantic map and corresponding visual uncertainty; S3, align the 4D radar point cloud data with the BEV semantic map in semantic context to obtain a semantic enhanced radar point cloud; Each point in the 4D radar point cloud is projected into the two-dimensional grid coordinate system in which the BEV semantic map resides by the intrinsic and extrinsic calibration matrix of the vehicle. ​​​ wherein, denotes the longitudinal distance, denotes the lateral distance, denotes the vertical height, denotes the Doppler velocity of the target, denotes the radar cross section area; In BEV semantic graph Search The grid into which it falls Corresponding semantic feature vector The method is as follows: ； wherein, represents a grid coordinate, represents a probability that the point cloud belongs to a road, represents a probability that the point cloud belongs to a vehicle, represents a probability that the point cloud belongs to a pedestrian; Then generate a semantic enhanced radar point cloud; S4, receive the semantic enhanced radar point cloud and classify each point; A lightweight point cloud processing network is used, which is trained to understand the combination of physical and semantic relationships, and to distinguish clutter and obstacles, as follows: First scenario: If the feature of the input point is: ； ； The network identifies the combination mode of high reflection, static and located in the high probability road area, and classifies it as static road clutter, and the output is suppressed or discarded; Second scenario: If the feature of the input point is: ； ； The network identifies the combination mode of high reflection, static and located in the high probability vehicle area, and classifies it as a high confidence obstacle, and the input point will be enhanced in the output; Third scenario: If the feature of the input point is: ； ； The input point is determined to have low physical reflection and is classified as a dynamic pedestrian. S5, dynamically fuse the BEV semantic map, corresponding visual uncertainty and classified radar point cloud under different working conditions, and generate the final perception result.

2. The 4D radar point cloud enhancement method based on BEV semantic prior according to claim 1, characterized in that, Step S1 specifically comprises: Through a plurality of wide-angle cameras and 4D millimeter wave radars deployed around the vehicle body, image data and 4D radar point cloud data around the vehicle are synchronously collected.

3. The 4D radar point cloud enhancement method based on BEV semantic prior of claim 2, wherein, Step S2 specifically comprises: Extract features from the collected image data through ResNet or Swin Transformer, generate a BEV semantic map, and estimate the visual uncertainty of the BEV semantic map at the same time.

4. The 4D radar point cloud enhancement method based on BEV semantic prior of claim 1, wherein, Step S5 specifically comprises: Receive the BEV semantic map, corresponding visual uncertainty and classified radar point cloud; Condition discrimination, judge whether the visual uncertainty is greater than the set threshold, if yes, determine that the vision is invalid, then dynamically switch the trust mode, that is, completely use the original point cloud of the 4D radar as the basic perception source data, if not, determine that the vision is correct; Fuse the high confidence BEV semantic map with the classified radar point cloud, and generate the final BEV perception result through grid superposition.

Images