4D millimeter-wave radar spatial occupancy probability estimation method based on lidar supervision

Abstract

The present invention belongs to the technical field of 4D millimeter-wave radar perception, and specifically relates to a 4D millimeter-wave radar spatial occupancy probability estimation method based on laser radar supervision, the method comprising constructing an environmental occupancy probability map as a supervision signal for the laser radar; performing feature extraction on the sparse point cloud data of the 4D millimeter-wave radar; performing cross-modal feature alignment of the supervision signal of the laser radar with the radar point cloud features to construct a one-stage occupancy probability model; using the same network and supervision method to obtain a reference occupancy probability model using the laser radar point cloud self-supervision, and using the reference occupancy probability model to supervise and fine-tune the one-stage occupancy probability model, and deploying it to downstream tasks. This application utilizes the geometric prior knowledge of the laser radar to constrain 4D millimeter-wave radar data learning, solves the problem of insufficient modeling accuracy caused by the sparsity, noise and penetration of the 4D millimeter-wave radar point cloud, and realizes low-cost, highly robust environmental geometric perception.

Description

Technical Field

[0001] The present invention belongs to the technical field of 4D millimeter-wave radar perception, and specifically relates to a 4D millimeter-wave radar space occupancy probability estimation method based on laser radar supervision. Background Art

[0002] In autonomous driving and robotic systems, accurate understanding of three-dimensional spatial structure is the core foundation for achieving environmental perception, path planning, and obstacle avoidance decisions. Occupancy grid, as a universal spatial representation method, can divide the scene into free areas, obstacle-occupied areas, and unknown areas, providing robots with key environmental understanding capabilities. However, existing occupancy grid estimation technology faces significant technical bottlenecks: First, current mainstream sensors have inherent defects. The performance of LiDAR drops sharply in severe weather such as rain, snow, fog, and haze. For example, laser beam scattering in dense fog causes the point cloud density to drop by more than 70%, making it impossible to effectively detect obstacle boundaries. Visual sensors are extremely sensitive to changes in lighting. In strong or dark light scenes, images are prone to overexposure or noise interference, making it difficult to extract stable features. At the same time, high-precision LiDAR equipment (such as 64-line radar) is expensive (about tens of thousands of dollars) and requires regular calibration and maintenance, which greatly limits large-scale commercial deployment. The insufficient resolution of LiDAR and cameras in long-distance scenarios is particularly prominent - the density of LiDAR point clouds decays exponentially with detection distance (for example, only sparse point clouds can be obtained for targets 100 meters away), and the camera's depth estimation accuracy also decreases significantly in long-distance scenarios, resulting in a sharp increase in the risk of missed obstacle detection.

[0003] Against this backdrop, 4D millimeter-wave radar (4D Radar) demonstrates significant technological potential. It transmits millimeter-wave signals to obtain target range, azimuth, elevation, and Doppler velocity information. It boasts advantages such as all-weather adaptability (maintaining an effective detection rate of over 80% even in dense fog), low cost (hardware cost is only 10%-20% of LiDAR), and motion sensing capabilities. However, 4D Radar's application to occupancy grid estimation still faces significant challenges. The number of point clouds per frame is less than 5% of that of LiDAR (typically only a few hundred points). 4D Radar is also affected by multipath reflections (e.g., false positioning caused by multiple reflections off glass walls in urban canyons) and harmonic interference, resulting in a large amount of noisy point clouds. The penetrating properties of millimeter-waves in non-metallic materials such as plastic and glass lead to geometric distortion (e.g., missing signals reflected from the surface of plastic traffic cones). Furthermore, when directly migrating traditional LiDAR algorithms (such as ray casting) to radar data, the sparsity of the point cloud and the penetration effect lead to misjudgment rates exceeding 40% for free space and occupied areas. Existing improvement plans also have obvious flaws. Multi-sensor fusion requires complex calibration with millimeter-level accuracy and cannot get rid of the real-time dependence on LiDAR; pure radar target detection methods are limited by undefined categories of obstacles (construction roadblocks, fallen trees) and weakly reflective objects (pedestrian miss detection rate reaches 30%); supervised learning based on deep learning is plagued by high labeling costs (single-frame 3D labeling takes tens of minutes) and labeling consistency problems (such as differences in the determination of occupied areas of penetrating point clouds).

[0004] In summary, current occupancy grid estimation technology is mired in a triple dilemma: poor environmental robustness, high hardware costs, and insufficient long-range resolution. Despite its all-weather adaptability and cost advantages, 4D Radar offers limited potential for replacing LiDAR for high-precision modeling due to point cloud sparsity, noise interference, penetration effects, and insufficient algorithm adaptability. Existing improvements either continue to rely on LiDAR or sacrifice the continuity (e.g., object detection methods ignore undefined obstacles) and integrity (e.g., penetration effects cause geometric distortion) of scene modeling. Therefore, there is an urgent need to develop a low-cost, self-supervised, and highly robust 4D Radar occupancy grid estimation method that can reconstruct continuous spatial structure from sparse, noisy point clouds and adapt to the real-time decision-making requirements in dynamic and complex environments, thereby breaking the vicious cycle between sensor performance limitations and algorithmic limitations. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a 4D millimeter-wave radar space occupancy probability estimation method based on lidar supervision.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In one aspect of the present invention, a method for estimating spatial occupancy probability of a 4D millimeter-wave radar based on laser radar supervision is provided, comprising the following steps:

[0008] Collect 3D point cloud data through LiDAR and construct an environment occupancy probability map as the supervision signal of LiDAR;

[0009] A lightweight convolutional neural network is used to extract features from the sparse point cloud data of the 4D millimeter wave radar to obtain radar point cloud features; the lightweight convolutional neural network includes a sparse encoder, a dense encoder and a prediction head connected in sequence; the sparse encoder includes more than three layers of 3D sparse convolution layers and a feature compression projection layer; the dense encoder includes stacked 2D convolution layers; the prediction head includes a parallel set of height regression branches and occupancy prediction branches, wherein the height regression branch is used to generate XY The height prediction map of the plane, the occupancy prediction branch is used to output the occupancy probability distribution map of the three-dimensional space;

[0010] Perform cross-modal feature alignment of the lidar supervision signal and the radar point cloud features, construct a highly aligned weak supervision loss function, iteratively optimize the weight parameters of the lightweight convolutional neural network, and obtain a one-stage occupancy probability model;

[0011] Using the same network structure and supervision method, the LiDAR point cloud is used for self-supervision to obtain a reference occupancy probability model, and the reference occupancy probability model is used to supervise and fine-tune the first-stage occupancy probability model to obtain a second-stage occupancy probability model.

[0012] Deploy the one-stage occupancy probability model or the two-stage occupancy probability model to downstream tasks.

[0013] As a preferred technical solution, the environment occupancy probability map includes a binary occupancy query map and a lidar height map;

[0014] The binary occupancy query graph is constructed by the following steps:

[0015] Assume that the point cloud set obtained by laser radar scanning is P ={ p i’ | i' =1,2,…, n},in p i’ ∈ R 3 Indicates the i' The three-dimensional coordinates of the point cloud, the center coordinates of the lidar sensor are c ∈ R 3 ;

[0016] For each point cloud pi’ Calculate the unit direction vector u i’ : ,||•|| represents Euclidean distance calculation;

[0017] According to the short-wave physical characteristics of the laser radar, two types of query point sets are generated: occupied query point set R i’ + ={ r i’ + | r i’ + = p i’ + r • u i’} represents the occupied area behind the object surface; unoccupied query point set R i’ - ={ r i’ - | r i’ - = p i’ - r • u i’} represents the free space in the sensor's line of sight; where r i’ + is a positive sampling point, r >0 is the preset backward offset, r i’ - is a negative sampling point;

[0018] Will{ R i’ + , R i’ - The combination is constructed into a binary occupancy query graph in three-dimensional space;

[0019] The lidar height map is constructed by the following steps:

[0020] The three-dimensional space along XY The plane is divided into a uniform two-dimensional grid with a grid resolution of Δ x ×Δ y ;

[0021] For each grid ( i , j ), calculate the height eigenvalueH ij : , where max means taking all points in the grid z Axis maximum value, ( x , y , z ) represents the coordinates, ( x i , y i ) is the coordinate of the lower left corner of the grid, P is the lidar point cloud;

[0022] 4D millimeter wave radar point cloud P k mmw Through the coordinate transformation matrix T lidar mmw Project to the lidar coordinate system and match the corresponding XY Plane grid position;

[0023] For grids with millimeter-wave radar points, the corresponding height feature values ​​are used as pseudo labels to construct the lidar height map. H LiDAR .

[0024] As a preferred technical solution, the sparse encoder includes a point cloud feature embedding module, a sparse feature pooling module, a feature dimension compression module, a three-plane feature generation module, and a cross-view feature alignment module connected in sequence;

[0025] The point cloud feature embedding module performs point-by-point feature learning on the input point cloud through a multi-layer perceptron, wherein the multi-layer perceptron comprises a plurality of linear layers and batch normalization layers stacked alternately;

[0026] The sparse feature pooling module adopts a three-dimensional sparse maximum pooling operation, performing grid division in three orthogonal dimensions of the XY plane, YZ plane, and ZX plane, where the pooling kernel size is based on the predefined segmentation parameters split= [ s 1, s 2, s 3] Dynamic determination;

[0027] The feature dimension compression module compresses the features output by the sparse feature pooling module to a preset dimension through a learnable linear projection layer, and uses the ReLU activation function for nonlinear transformation;

[0028] The three-plane feature generation module includes three processing branches working in parallel:

[0029] XY Plane branch: 3D sparse tensor along ZThe axis is max-pooled, reshaped into a two-dimensional feature map through transposition operation, and then generated by a dedicated multi-layer perceptron XY Plane view features; the dedicated multilayer perceptron includes alternately stacked linear layers and batch normalization layers;

[0030] YZ Plane branch: 3D sparse tensor along X The axis is max-pooled and reshaped into a two-dimensional feature map through dimension permutation operation, and then generated by a dedicated multi-layer perceptron YZ Plan view features;

[0031] ZX Plane branch: 3D sparse tensor along Y The axis is max-pooled, and after the channel rearrangement operation is reshaped into a two-dimensional feature map, a dedicated multi-layer perceptron is generated. ZX Plan view features;

[0032] The cross-view feature alignment module unifies the batch dimension so that XY flat, YZ flat, ZX The view features of the plane maintain the same spatial resolution, and the output dimension is [ B , C , H , W ]’s standard feature map format, B represents the batch size, C Indicates the number of channels, H Indicates height, W Indicates width.

[0033] As a preferred technical solution, the dense encoder adopts a neural network architecture based on bird's-eye view feature extraction, including a cascade downsampling module, a multi-level residual learning module, a feature pyramid fusion module and a multi-scale feature upsampling module connected in sequence;

[0034] The cascade downsampling module includes at least three convolution blocks connected in sequence, and the step sizes of the downsampling operations in the convolution blocks decrease in sequence;

[0035] The multi-level residual learning module includes at least ten groups of residual units connected in series, wherein the first, third, and fifth groups of residual units integrate downsampling operations, and the outputs of each group of residual units are connected to the subsequent feature pyramid fusion module through jump connections to establish a multi-scale feature transmission channel;

[0036] The feature pyramid fusion module is used to perform top-down feature fusion on the feature maps of different scales output by the multi-level residual learning module, including the following steps:

[0037] Upsample deep fine-grained features by a factor of 2;

[0038] Add the up-sampled result to the shallow coarse-grained features element by element;

[0039] Cross-channel feature calibration is achieved through 1×1 convolution kernel;

[0040] Apply batch normalization and ReLU activation to the fused feature map;

[0041] The multi-scale feature upsampling module uses a bilinear interpolation method to restore the feature map after feature fusion to the original input size and outputs three-view features.

[0042] As a preferred technical solution, the cross-modal feature alignment of the lidar supervision signal and the radar point cloud features is performed to construct a highly aligned weak supervision loss function, specifically:

[0043] Introducing height map constraints in the intermediate layers of the sparse encoder and the decoder of the dense encoder;

[0044] The height map constraint includes a supervisory signal generation module, a multimodal alignment prediction module, and an adaptive loss constraint module connected in sequence;

[0045] The supervisory signal generation module is used to perform point cloud height statistics and pseudo-label generation on the grid area covered by the millimeter wave radar point cloud; the point cloud height statistics specifically includes: calculating the mean or maximum value of the radar point height in a preset voxel space as a statistical value; the pseudo-label generation specifically includes: using the statistical value as a supervisory signal for height prediction to form a reference height map;

[0046] The multimodal alignment prediction module includes a parallel processing height feature extraction and nonlinear activation unit; the height feature extraction is specifically: generating an initial height prediction map through a learnable 1×1 convolution kernel on the BEV feature map; the nonlinear activation unit uses the ReLU function to perform a nonlinear transformation on the initial height prediction map and output a non-negative height estimate;

[0047] The adaptive loss constraint module optimizes the prediction accuracy by constructing an effective area mask and fusion of multi-source supervision. The effective area mask construction is specifically: generating a binary mask matrix based on the radar point distribution to identify the effective supervision area; the multi-source supervision fusion is specifically: calculating the non-negative height estimation value in the mask area H pred,i,j With reference height map H LiDAR,i,j The mean square error As a highly aligned weakly supervised loss function, M ( i , j ) is a binary mask matrix, i ,j The grid coordinates of the lidar height map.

[0048] As a preferred technical solution, the prediction head includes feature fusion and multi-task loss;

[0049] The feature fusion specifically comprises: unifying the three-view features output by the dense encoder using a multi-layer perceptron to output a final occupancy probability distribution map;

[0050] The multi-task loss is specifically defined as: ,in, is the highly aligned weakly supervised loss function in the sparse encoder, is the highly aligned weakly supervised loss function in the dense encoder, L occ is the binary cross entropy loss of occupancy query, ω1, ω2, ω3 are weight coefficients;

[0051] The binary cross entropy loss of the occupancy query includes a random sampling optimization module, a dynamic neighborhood construction module, a geometric perception loss calculation module, and a multi-scale loss aggregation module;

[0052] The dynamic neighborhood construction module performs the following operations on the query point cloud in each batch: k The nearest neighbor algorithm selects for each query point k Initial neighbor points are formed to form an initial adjacency relationship graph; the Euclidean distance between the query point and each initial neighbor point is calculated, and the Euclidean distance of the query point is dynamically selected to be less than the preset radius threshold. r The corresponding neighbor points of , generate an adaptive neighborhood graph;

[0053] The geometric perception loss calculation module includes the following parallel calculation processes:

[0054] The main loss calculation based on binary cross entropy is to calculate the standard cross entropy loss for the predicted logits and the true labels of each query point-neighbor point pair in the adaptive neighborhood graph.

[0055] Geometric consistency auxiliary loss: According to the spatial distribution characteristics of the initial neighbor points, geometric continuity constraints are imposed on the neighbor points in the adaptive neighborhood graph;

[0056] The random sampling optimization module specifically performs the following operations when the number of query points in a single batch in the geometric perception loss calculation module exceeds a preset threshold:

[0057] A number of sampling points are randomly selected according to uniform distribution as the support point set;

[0058] Construct a local neighborhood graph based only on the support point set to maintain the spatial distribution characteristics of the original point cloud;

[0059] The multi-scale loss aggregation module achieves cross-batch loss normalization in the following way:

[0060] Independently calculate the local neighborhood loss of each batch in the geometric perception loss calculation module and sum them up;

[0061] The total loss is averaged by batch size to generate the final optimization target.

[0062] As a preferred technical solution, the same network structure and supervision method are used to obtain a reference occupancy probability model through self-supervision of the lidar point cloud, and the reference occupancy probability model is used to supervise and fine-tune the first-stage occupancy probability model to obtain a second-stage occupancy probability model. Specifically, the following steps are included:

[0063] Using only the LiDAR point cloud, an initial occupancy model with the same structure as the first-stage occupancy probability model is trained to obtain a reference occupancy probability model.

[0064] The occupancy probability results generated by the reference occupancy probability model are used as supervisory signals. The cross-entropy loss is constructed with the output of the first-stage occupancy probability model and added to the original loss function of the first-stage occupancy probability model. Back-propagation is used to correct for noise and misjudgment areas in the radar prediction to obtain the second-stage occupancy probability model.

[0065] During the supervised fine-tuning process, an adaptive mask is introduced to filter out pseudo labels in areas not covered by the radar to achieve dynamic alignment.

[0066] As a preferred technical solution, the one-stage occupancy probability model or the two-stage occupancy probability model is deployed to downstream tasks, specifically:

[0067] Loading the model parameters of the one-stage occupancy probability model or the two-stage occupancy probability model, replacing the prediction head, and executing the corresponding downstream task;

[0068] The prediction head is replaced by the following modules connected in sequence:

[0069] A multi-scale voxel encoding module includes at least three voxel encoders; the downsampling factor multiples of the at least three voxel encoders increase sequentially, and are used to respectively extract feature maps of different resolutions in the XY plane, the YZ plane, and the ZX plane;

[0070] Feature dimension reduction convolution layer, used to compress and fuse the channel dimensions of feature maps of different resolutions in each plane;

[0071] An improved U-Net architecture is used to enhance the contextual information of the fused plane features. Specifically, the improved U-Net architecture uses depthwise separable convolution for feature compression in the encoding path, integrates an attention gating mechanism in the decoding path, and introduces a dynamic feature calibration module through skip connections.

[0072] A three-dimensional coordinate projection mechanism is used to map query points to each plane feature map and perform interpolation sampling of different plane features;

[0073] The scalable MLP output head is used to adjust the output dimension according to the requirements of downstream tasks.

[0074] As a preferred technical solution, the feature dimension reduction convolution layer uses a 1×1 convolution kernel to achieve cross-scale feature fusion, as shown in the following formula:

[0075] ;

[0076] in, is the output of the feature dimension reduction convolution layer, Conv 1×1 represents a 1×1 convolution kernel, [•] represents the channel dimension splicing operation, is the input of the feature dimension reduction convolution layer, plane ∈ XY , YZ , ZX Represents a three-dimensional space projection plane;

[0077] The scalable MLP output head achieves task adaptation through the following formula:

[0078] ;

[0079] in, Output For output, Adapt weights to tasks, is the hidden layer weight, σ represents the ReLU activation function, F xy 、 F yz 、 F zx Represents a three-dimensional projection plane XY , YZ , ZX characteristics; D For the output dimension, by setting the output dimension D Perform different downstream tasks.

[0080] Another aspect of the present invention further provides a 4D millimeter-wave radar space occupancy probability estimation system based on laser radar supervision, which is applied to the above-mentioned 4D millimeter-wave radar space occupancy probability estimation method based on laser radar supervision, including a laser radar supervision signal generation module, a millimeter-wave radar feature extraction network, a cross-modal joint training module, a self-supervised fine-tuning module, and a multi-task deployment module;

[0081] The laser radar supervision signal generation module is used to collect three-dimensional point cloud data through the laser radar and construct an environment occupancy probability map as a supervision signal of the laser radar;

[0082] The millimeter-wave radar feature extraction network uses a lightweight convolutional neural network to extract features from the sparse point cloud data of the 4D millimeter-wave radar to obtain radar point cloud features. The lightweight convolutional neural network includes a sparse encoder, a dense encoder, and a prediction head connected in sequence. The sparse encoder includes three or more 3D sparse convolution layers and a feature compression projection layer. The dense encoder includes stacked 2D convolution layers. The prediction head includes a height regression branch and an occupancy prediction branch arranged in parallel, wherein the height regression branch is used to generate a height prediction map in the XY plane, and the occupancy prediction branch is used to output an occupancy probability distribution map in the three-dimensional space.

[0083] The cross-modal joint training module is used to perform cross-modal feature alignment on the lidar supervision signal and the radar point cloud features, construct a highly aligned weak supervision loss function, iteratively optimize the weight parameters of the lightweight convolutional neural network, and obtain a one-stage occupancy probability model;

[0084] The self-supervised fine-tuning module is used to obtain a reference occupancy probability model using the same network and supervision method through self-supervision of the lidar point cloud, and to perform supervised fine-tuning on the first-stage occupancy probability model using the reference occupancy probability model to obtain a second-stage occupancy probability model.

[0085] The multi-task deployment module is used to deploy the one-stage occupancy probability model or the two-stage occupancy probability model to downstream tasks.

[0086] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0087] (1) This application uses the geometric prior knowledge of LiDAR to constrain 4D millimeter-wave radar data learning, solves the problem of insufficient modeling accuracy caused by the sparsity, noise and penetration of 4D millimeter-wave radar point clouds, and realizes low-cost and highly robust environmental geometric perception.

[0088] (2) This application only requires single millimeter-wave radar hardware to achieve occupancy grid estimation accuracy close to that of lidar, reducing system deployment costs;

[0089] (3) This application utilizes the anti-interference characteristics of 4D millimeter-wave radar to maintain stable performance in adverse weather conditions such as rain, snow, fog, and haze, and has all-weather robustness;

[0090] (4) This application avoids large-scale manual labeling through lidar self-supervised learning and improves the scalability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 This is an algorithm flow chart of a method for estimating spatial occupancy probability of a 4D millimeter-wave radar based on laser radar supervision according to an embodiment of the present invention;

[0092] Figure 2 is an algorithm flow chart of a sparse encoder based on three-plane projection according to an embodiment of the present invention;

[0093] Figure 3 is an algorithm flow chart of a dense encoder for bird's-eye view feature extraction according to an embodiment of the present invention;

[0094] Figure 4 is a flow chart of a voxel occupancy loss calculation algorithm based on spatial consistency constraints according to an embodiment of the present invention;

[0095] Figure 5 This is a flow chart of an algorithm for 3D segmentation and BEV segmentation model based on three-plane views (TPV) according to an embodiment of the present invention. DETAILED DESCRIPTION

[0096] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0097] Example 1:

[0098] like Figure 1 As shown, this embodiment provides a 4D millimeter-wave radar space occupancy probability estimation method based on lidar supervision. By synergizing multimodal data of lidar and 4D millimeter-wave radar, high-precision three-dimensional perception of the autonomous driving environment is achieved, including the following steps:

[0099] S1. Collect 3D point cloud data through LiDAR and construct an environment occupancy probability map as the supervision signal of LiDAR.

[0100] In one or more preferred embodiments, the environment occupancy probability map includes a binary occupancy query map and a lidar height map, and is constructed by the following steps:

[0101] (1.1) Construct a binary occupancy query graph:

[0102] (1.1.1) Assume that the point cloud set obtained by laser radar scanning is P ={ p i’ | i' =1,2,…, n},in p i’ ∈ R 3 Indicates the i' The three-dimensional coordinates of the point cloud, the center coordinates of the lidar sensor are c ∈ R 3 ;

[0103] (1.1.2) For each point cloud p i Calculate the unit direction vector u i’ : ,||•|| represents Euclidean distance calculation;

[0104] (1.1.3) Based on the shortwave physical characteristics of the lidar, two types of query point sets are generated: occupied query point set R i’ + ={ r i’ + | r i’ + = p i’ + r • u i’} represents the occupied area behind the object surface; unoccupied query point set R i’ - ={ r i’ - | r i’ - = p i’ - r • u i’} represents the free space in the sensor's line of sight; where r i’ + is a positive sampling point, r >0 is the preset backward offset, r i’- is a negative sampling point;

[0105] (1.1.4) Replace { R i’ + , R i’ -}The combination is constructed into a binary occupancy query graph in three-dimensional space.

[0106] (1.2) Constructing the LiDAR height map:

[0107] (1.2.1) The three-dimensional space is XY The plane is divided into a uniform two-dimensional grid with a grid resolution of Δ x ×Δ y , in particular, in this embodiment Δ x =Δ y =0.4m;

[0108] (1.2.2) For each grid ( i , j ), calculate the height eigenvalue H ij : , where max means taking all points in the grid z Axis maximum value, ( x , y , z ) represents the coordinates, ( x i , y i ) is the coordinate of the lower left corner of the grid, P is the lidar point cloud;

[0109] (1.2.3) 4D millimeter wave radar point cloud P k mmw Through the coordinate transformation matrix T lidar mmw Project to the lidar coordinate system and match it XY Plane grid position;

[0110] (1.2.4) For grids with millimeter-wave radar points, use the corresponding height feature values ​​as pseudo-labels to construct a lidar height map H LiDAR .

[0111] S2. Extracting features from the sparse point cloud data of the 4D millimeter-wave radar. Specifically, a lightweight convolutional neural network is used to perform multi-scale feature encoding to obtain radar point cloud features. The lightweight convolutional neural network includes a sparse encoder, a dense encoder, and a prediction head connected in sequence.

[0112] In one or more preferred embodiments, the sparse encoder extracts geometric features from the input radar point cloud data through a multi-level (three or more) 3D sparse convolutional layer, and compresses and projects the features into a two-dimensional view space through a feature compression projection layer, specifically including two parts: three-view projection and height map constraint.

[0113] (2.1) Three-view projection: It includes a point cloud feature embedding module, a sparse feature pooling module, a feature dimension compression module, a three-plane feature generation module, and a cross-view feature alignment module, which are connected in sequence.

[0114] (2.1.1) Point Cloud Feature Embedding Module: This module performs point-by-point feature learning on the input point cloud using a multi-layer perceptron (MLP). The MLP consists of several (preferably four in this embodiment) alternating linear layers and batch normalization layers. The output dimension is increased from the input channels to 256 dimensions.

[0115] (2.1.2) Sparse feature pooling module: Using three-dimensional sparse maximum pooling operation, respectively XY flat, YZ flat, ZX The plane is meshed in three orthogonal dimensions, where the pooling kernel size is determined by the predefined segmentation parameters split =[ s 1, s 2, s 3] Dynamically determine and realize multi-scale sparse feature extraction.

[0116] (2.1.3) Feature Dimension Compression Module: Compresses the pooled features of the sparse feature pooling module from 256 dimensions to a preset dimension through a learnable linear projection layer. fea _ compre , and the ReLU activation function is used for nonlinear transformation.

[0117] (2.1.4) Three-plane feature generation module: It includes three processing branches working in parallel:

[0118] XY Plane branch: 3D sparse tensor along Z The axis is max-pooled, reshaped into a two-dimensional feature map through transposition operation, and then generated by a dedicated MLP XY Plan view features;

[0119] YZPlane branch: 3D sparse tensor along X The axis is max-pooled, and after being reshaped into a two-dimensional feature map through dimension permutation, it is generated by a dedicated MLP. YZ Plan view features;

[0120] ZX Plane branch: 3D sparse tensor along Y The axis is max-pooled, and after the channel rearrangement operation is reshaped into a two-dimensional feature map, a dedicated MLP is generated. ZX Plan view features;

[0121] The specialized MLP consists of alternating stacks of linear and batch normalization layers.

[0122] (2.1.5) The cross-view feature alignment module unifies the batch dimension so that XY flat, YZ flat, ZX The view features of the plane maintain the same spatial resolution, and the output dimension is [ B , C , H , W ]’s standard feature map format, B represents the batch size, C Indicates the number of channels, H Indicates height, W Indicates width.

[0123] (2.2) Height graph constraint: It includes a dynamic supervision signal generation module, a multimodal alignment prediction module, and an adaptive loss constraint module, which are connected in sequence.

[0124] (2.2.1) The dynamic supervisory signal generation module is used to perform point cloud height statistics and pseudo-label generation on the grid area covered by the millimeter wave radar point cloud;

[0125] The point cloud height statistics specifically include: calculating the mean or maximum value of the radar point height in a preset voxel space as a statistical value;

[0126] The pseudo-label generation is specifically as follows: using the statistical value as a supervisory signal for height prediction to form a reference height map; the reference height map is used to align with the features output by the sparse encoder and the dense encoder and calculate the loss.

[0127] (2.2.2) The multimodal alignment prediction module includes parallel processing of highly feature extraction and nonlinear activation units;

[0128] The height feature extraction is specifically as follows: generating an initial height prediction map through a learnable 1×1 convolution kernel on the BEV feature map;

[0129] The nonlinear activation unit specifically uses the ReLU function to perform nonlinear transformation on the initial height prediction map and outputs a non-negative height estimation value.

[0130] (2.2.3) The adaptive loss constraint module optimizes prediction accuracy by constructing effective region masks and fusing multi-source supervision;

[0131] The effective area mask construction is specifically as follows: generating a binary mask matrix based on radar point distribution to identify the effective supervision area;

[0132] The multi-source supervision fusion is specifically as follows: calculating non-negative height estimation values ​​in the mask area H pred,i,j With reference height map H LiDAR,i,j The mean square error As a highly aligned weakly supervised loss function, M ( i , j ) is a binary mask matrix, i , j The grid coordinates of the lidar height map.

[0133] In one or more preferred embodiments, the dense encoder utilizes stacked 2D convolutional layers to extract and enhance multi-scale dense features, including TPV feature reconstruction and depth supervision.

[0134] (2.3) TPV feature reconstruction: Using a neural network architecture based on bird’s-eye view (BEV) feature extraction, the multi-scale features of BEV, FV, and SV are gradually aggregated to generate the refined features T of the three views. BEV 、T FV 、T SV ; Among them, the neural network architecture based on bird's-eye view feature extraction includes a cascade downsampling module, a multi-level residual learning module, a feature pyramid fusion module and a multi-scale feature upsampling module connected in sequence.

[0135] (2.3.1) Cascade downsampling module: includes at least three convolution blocks connected in sequence, and the step size of the downsampling operation in the convolution blocks decreases in sequence.

[0136] Furthermore, in this embodiment, the cascade downsampling module is set as:

[0137] The first convolution block uses a 3×3 convolution with a stride of 2 for spatial downsampling;

[0138] The second convolution block uses 3×3 convolution with a stride of 1 to enhance local features;

[0139] The third convolution block uses a 3×3 convolution with a stride of 1 to complete the channel dimension expansion.

[0140] (2.3.2) Multi-level residual learning module: It includes alternating standard residual blocks and downsampling residual blocks, where the downsampling residual blocks halve the feature map size through 1×1 convolution.

[0141] Furthermore, in this embodiment, a multi-level residual learning module is provided including at least ten groups of residual units connected in series, wherein the 1st, 3rd, and 5th groups of residual units integrate downsampling operations, and the outputs of each group of residual units establish a multi-scale feature transmission channel with the subsequent feature pyramid fusion module through a jump connection.

[0142] (2.3.3) Feature Pyramid Fusion Module: Performs top-down feature fusion on the feature maps of different scales output by the multi-level residual learning module.

[0143] Furthermore, in this embodiment, the feature pyramid fusion module implements multi-level feature aggregation through the following steps:

[0144] Upsample deep fine-grained features by a factor of 2;

[0145] Add the up-sampled result to the shallow coarse-grained features element by element;

[0146] Cross-channel feature calibration is achieved through 1×1 convolution kernel;

[0147] Batch normalization and ReLU activation are applied to the fused feature maps.

[0148] (2.3.4) Multi-scale feature upsampling module: The bilinear interpolation method is used to restore the feature map after feature fusion to the original input size.

[0149] (2.4) Deep Supervision: Introducing height map constraints in the middle layer of the decoder of the dense encoder to obtain loss , strengthen the vertical geometric constraints.

[0150] Furthermore, the introduction of height map constraints in the middle layer of the decoder is the same as the implementation method in (2.2) height map constraints, that is, height map constraints are used for alignment in both the sparse encoder and the dense encoder to improve stability.

[0151] In one or more preferred embodiments, the prediction head includes a height regression branch and an occupancy prediction branch arranged in parallel, wherein the height regression branch is used to generate XY The height prediction map of the plane, the occupancy prediction branch is used to output the occupancy probability distribution map of the three-dimensional space.

[0152] The prediction head includes feature fusion and multi-task loss.

[0153] (2.5) Feature fusion: A multi-layer perceptron is used to unify the three-view features output by the dense encoder and output the final occupancy probability distribution map.

[0154] (2.6) Multi-task loss: The total loss function is defined as: ,in, is the highly aligned weakly supervised loss function in the sparse encoder, is the highly aligned weakly supervised loss function in the dense encoder, L occ is the binary cross entropy loss of occupancy query, ω1, ω2, ω3 are weight coefficients.

[0155] Furthermore, the binary cross entropy loss of the occupancy query includes a random sampling optimization module, a dynamic neighborhood construction module, a geometry-aware loss calculation module, and a multi-scale loss aggregation module.

[0156] (2.6.1) Dynamic Neighborhood Construction Module: For each query point cloud in a batch, perform the following operations:

[0157] pass k The KNN algorithm selects a k Initial neighbor points to form an initial adjacency graph;

[0158] Calculate the Euclidean distance between the query point and each initial neighbor point, based on the preset radius threshold r’ Perform dynamic filtering to retain the Euclidean distance less than the preset radius threshold r’ The corresponding neighbor points of , generate an adaptive neighborhood graph;

[0159] (2.6.2) Geometric Perception Loss Calculation Module: This module includes the following parallel computation processes:

[0160] Main loss calculation based on binary cross entropy: the standard cross entropy loss is calculated for each query point-neighbor point pair in the adaptive neighborhood graph, with the predicted logits value and the true label; the predicted logits value refers to the value after sigmoid normalization of the model;

[0161] Geometric consistency auxiliary loss: According to the spatial distribution characteristics of all initial neighbor points, the radius threshold r’ The valid neighbor points within the grid impose geometric continuity constraints.

[0162] (2.6.3) Random Sampling Optimization Module: When the number of query points in a single batch in the Geometry Perception Loss Calculation Module exceeds the preset threshold, the following operations are performed:

[0163] Randomly select from uniform distribution sample_size sampling points as the support point set;

[0164] The local neighborhood graph is constructed only based on the support point set, maintaining the spatial distribution characteristics of the original point cloud.

[0165] (2.6.4) Multi-scale loss aggregation module: Cross-batch loss normalization is achieved through:

[0166] Independently calculate the local neighborhood loss of each batch in the geometric perception loss calculation module and sum them up;

[0167] The total loss is averaged by batch size to generate the final optimization target.

[0168] S3. Perform cross-modal feature alignment on the lidar supervision signal and the radar point cloud features, construct a highly aligned weak supervision loss function, iteratively optimize the weight parameters of the lightweight convolutional neural network, and obtain a one-stage occupancy probability model.

[0169] Specifically, height map constraints are introduced in the middle layers of the sparse encoder and the decoder of the dense encoder.

[0170] S4. Use the same network and supervision method to obtain a reference occupancy probability model using lidar point cloud self-supervision, and use the reference occupancy probability model to supervise and fine-tune the first-stage occupancy probability model to obtain a second-stage occupancy probability model.

[0171] In one or more preferred embodiments, step S4 specifically includes the following steps:

[0172] (4.1) Self-supervised LiDAR pre-training:

[0173] Only the lidar point cloud is used to train an initial occupancy model with the same structure as the first-stage occupancy probability model to obtain a reference occupancy probability model and learn a high-precision geometric prior.

[0174] (4.2) Radar model fine-tuning:

[0175] The occupancy probability result generated by the reference occupancy probability model is used as the supervision signal. The cross entropy loss is constructed with the output of the one-stage occupancy probability model and added to the original loss function of the one-stage occupancy probability model. The noise and misjudgment areas in the radar prediction are corrected through back propagation to obtain the two-stage occupancy probability model.

[0176] (4.3) Dynamic alignment mechanism:

[0177] During the supervised fine-tuning process, an adaptive mask is introduced to filter out pseudo labels in areas not covered by the radar to avoid interference from invalid supervisory signals.

[0178] S5. Deploy the one-stage occupancy probability model or the two-stage occupancy probability model to downstream tasks, including but not limited to the BEV segmentation network and the 3D occupancy segmentation network, to generate multi-dimensional perception results of the autonomous driving environment.

[0179] In one or more preferred embodiments, taking the BEV segmentation network and the 3D occupancy segmentation network as examples, step S5 specifically includes the following steps:

[0180] (5.1) BEV Segmentation Network and / or 3D Occupancy Segmentation Load the model parameters of the one-stage occupancy probability model or the two-stage occupancy probability model and replace the prediction head to implement BEV segmentation and / or 3D occupancy segmentation.

[0181] The prediction head is replaced by the following modules connected in sequence:

[0182] (5.1.1) Multi-scale voxel encoding module, including at least three voxel encoders with different downsampling factors, used to extract XY flat, YZ flat, ZX Planar multi-resolution feature maps.

[0183] Furthermore, the multi-scale voxel encoding module in step (5.1.1) is specifically:

[0184] The first-level encoder performs voxel feature aggregation using the original resolution;

[0185] The second-stage encoder extracts coarse-grained spatial features by downsampling by a factor of 2;

[0186] The fourth-stage encoder captures global context information by 4x downsampling.

[0187] (5.1.2) Feature dimensionality reduction convolution layer, used to compress and fuse the channel dimension of feature maps of different scales in each plane.

[0188] Furthermore, step (5.1.2) is as follows:

[0189] A 1×1 convolution kernel is used to achieve cross-scale feature fusion, and its mathematical form is:

[0190] ;

[0191] in, is the output of the feature dimension reduction convolution layer, Conv 1×1 represents a 1×1 convolution kernel, [•] represents the channel dimension splicing operation, is the input of the feature dimension reduction convolution layer, plane ∈ XY , YZ ,ZX Represents a three-dimensional projection plane.

[0192] (5.1.3) A lightweight codec network is used to enhance the contextual information of the fused plane features. Specifically:

[0193] Adopting an improved U-Net architecture, including:

[0194] The encoding path uses depthwise separable convolution for feature compression;

[0195] The decoding pathway integrates an attention gating mechanism;

[0196] The skip connection introduces a dynamic feature calibration module.

[0197] (5.1.4) A three-dimensional coordinate projection mechanism is used to map query points to each plane feature map and perform multi-plane feature interpolation sampling.

[0198] (5.1.5) Scalable MLP output head for adjusting output dimensions according to downstream task requirements, enabling end-to-end fine-tuning of BEV segmentation or 3D semantic segmentation.

[0199] Furthermore, step (5.1.5) is as follows:

[0200] The scalable MLP output head achieves task adaptation through the following formula:

[0201] ;

[0202] in, Output For output, is the hidden layer weight, Adapt weights to tasks, σ represents the ReLU activation function, F xy 、 F yz 、 F zx Represents a three-dimensional projection plane XY , YZ , ZX Features, D is the output dimension, when the output dimension D When set to single channel, the model performs the BEV drivable area segmentation task; when the output dimension is set to D When is the number of semantic categories, the model performs 3D voxel-level semantic segmentation tasks.

[0203] Example 2:

[0204] This embodiment provides a 4D millimeter-wave radar space occupancy probability estimation system based on laser radar supervision, which can be applied to the 4D millimeter-wave radar space occupancy probability estimation method based on laser radar supervision described in Example 1, including a laser radar supervision signal generation module, a millimeter-wave radar feature extraction network, a cross-modal joint training module, a self-supervised fine-tuning module, and a multi-task deployment module.

[0205] 1. The LiDAR supervisory signal generation module is used to collect three-dimensional point cloud data through the LiDAR and construct an environment occupancy probability map as the LiDAR supervisory signal, specifically:

[0206] Use Ouster 128-line LiDAR or LiVOX LiDAR equipment to collect high-density point cloud data at a scanning frequency of 10Hz. Geometric supervision is achieved by constructing a binary occupancy query map: for each LiDAR point p i’ , calculate its relative to the sensor center c The unit direction vector , generate an occupancy query point set with a backward offset of 0.5m R i’ + ={ r i’ + | r i’ + = p i’ +0.5• u i’} and a set of unoccupied query points with a forward offset of 0.3m R i’ + ={ r i’ + | r i’ + = p i’ +0.3• u i’}. At the same time, establish XY The plane grid resolution is 0.4m×0.4m height map, each grid ( i , j )Record maximum Z Axis coordinate value .

[0207] 2. The millimeter wave radar feature extraction network is used to extract features from the sparse point cloud data of the 4D millimeter wave radar to obtain radar point cloud features; it includes a sparse encoder, a dense encoder and a prediction head connected in sequence; the sparse encoder includes more than three layers of 3D sparse convolution layers and a feature compression projection layer; the dense encoder includes stacked 2D convolution layers; the prediction head includes a height regression branch and an occupancy prediction branch set in parallel, wherein the height regression branch is used to generate XY The height prediction map of the plane, the occupancy prediction branch is used to output the occupancy probability distribution map of the three-dimensional space; specifically, the millimeter wave radar feature extraction network is equipped with the Aoku millimeter wave radar to perform coordinate system transformation on the original point cloud.

[0208] (1) The sparse encoder achieves the compressed representation of 3D point cloud data through multi-stage feature transformation. The workflow is as follows: Figure 2 As shown: First, the input point cloud is spatially discretized, each 3D point is bound to a preset grid coordinate system, and a 4D index tuple ( batch_id , x , y , z ) establishes a spatial mapping relationship between point clouds and regularized voxels. In this process, a dynamic random shuffling mechanism is introduced to optimize data distribution, and an inverse index mapping algorithm is used to achieve efficient association between sparse features and dense grids.

[0209] The feature extraction stage utilizes a four-level fully connected network to construct a point-level feature learning module. Local geometric features are captured through a progressive dimensionality expansion strategy (3→64→128→256→256). Each network level incorporates a standard structure of linear transformation, batch normalization, and nonlinear activation. A competitive max-pooling layer is designed to aggregate multiple point features within the same voxel unit. A dimensionality reduction module with an adjustable compression ratio compresses the 256-dimensional features to a preset dimension, forming a sparse feature tensor with spatial resolution.

[0210] The three-plane projection conversion stage uses a heterogeneous pooling strategy to construct orthogonal feature planes: XY Plane implementation Z Axis layered pooling, through 1×1× Kz The kernel size preserves the two-dimensional spatial resolution; YZ Plane Kx ×1×1 slice pooling kernel along X Axial compression characteristics; ZX Graphic Design 1× Ky ×1 channel pooling kernel along Y Axis dimensionality reduction. Each projection plane is configured with an independent feature enhancement network, which goes through a three-step process of feature expansion, bilinear transformation, and activation reorganization to finally output a standardized three-dimensional representation tensor. TPV_xyPlane maintenance H × W ] spatial dimensions, TPV_yz Plane Preservation D × H ]Structural information, TPV_zx Plane load[ W × D ] channel features to form a complementary three-view feature expression system.

[0211] The sparse encoder is flexible and adaptable through configurable parameters, where split Parameters control the granularity of feature splitting on each axis. fea_compre Parameters adjust the feature compression ratio, grid_size and voxel_size Parameters support spatial resolution configuration for different scenarios. A specially designed origin offset compensation mechanism corrects for differences in sensor coordinate systems, and debugging flags enable visual verification of feature dimensions throughout the entire process.

[0212] (2) The workflow of the dense encoder is as follows Figure 3 As shown in Figure 2, efficient spatial representation learning is achieved through multi-scale feature fusion and a dynamic convolution mechanism. The network architecture employs a three-stage "encoding-enhancement-aggregation" pipeline. First, an initial convolutional module spatially downsamples the input features. A 3×3 convolution kernel with a stride of 2 is used to reduce the feature map resolution to 1 / 4 of its original size. Subsequently, two sets of standard convolutional layers enhance local features to form a basic feature representation. Batch normalization and nonlinear activation are introduced throughout this process to ensure stable feature distribution and diverse representations.

[0213] The feature enhancement stage utilizes ten levels of residual learning units, with odd-numbered levels integrating a spatial downsampling mechanism. A 3×3 convolution kernel with a stride of 2 is used to achieve a step-by-step reduction in feature map resolution. Each residual unit employs a dynamically adjustable convolution kernel for feature transformation. A channel attention mechanism is introduced after the convolution operation, generating channel weight vectors through global average pooling. These are then compressed and expanded in a fully connected layer and gated and weighted with the original features. A specially designed skip connection structure preserves original feature information, and random neuron dropout with a probability of 0.1 is applied before the output of each residual unit level to enhance model robustness.

[0214] The multi-scale feature fusion stage adopts a three-level feature pyramid architecture, selecting the outputs of the 2nd, 4th, and 10th layer residual units as the multi-level feature sources. Deep features are upsampled to the mid-level resolution using bilinear interpolation, and feature aggregation is achieved through element-by-element addition. The aggregated features are progressively refined using three levels of 3×3 convolution kernels and ultimately restored to the target spatial dimensions through fourfold upsampling. This architecture maintains feature consistency in the channel dimension, simultaneously capturing local details and global context through cross-level information fusion. The dynamic convolution mechanism enables the network to adaptively adjust the receptive field based on input features, and the channel attention module strengthens the weight allocation of key feature channels.

[0215] The dense encoder innovatively integrates dynamic convolution and compression excitation mechanisms within the residual unit. This allows for adaptive input adjustment of convolution kernel weights through learnable parameters, while also improving the representation efficiency of key information through channel-level feature recalibration. The feature pyramid network utilizes a lightweight design, effectively integrating multi-scale information through only three levels of feature fusion. Those skilled in the art can balance model performance and computational resource consumption by adjusting parameters such as the number of residual layers and channel compression ratio.

[0216] 3. The cross-modal joint training module is used to perform cross-modal feature alignment between the lidar supervision signal and the radar point cloud features, construct a highly aligned weak supervision loss function, iteratively optimize the weight parameters of the lightweight convolutional neural network, and obtain a one-stage occupancy probability model.

[0217] In the cross-modal joint training phase, spatial consistency is used to construct a composite supervision function including binary cross entropy loss and height mean square error loss for the millimeter wave network output and lidar constraints. , Adam optimizer (initial learning rate 2e-4, batch size 16) was used for 50 rounds of iterative training.

[0218] The voxel occupancy loss calculation method based on spatial consistency constraints is adopted, such as Figure 4 As shown. It improves the accuracy of three-dimensional occupancy prediction by jointly optimizing multi-scale neighborhood relationship modeling and feature reconstruction. The loss calculation process includes three stages: spatial neighborhood sampling, consistency constraint calculation and multi-task supervision fusion. First, dynamic query point sampling is implemented for each batch of input data. A random permutation index strategy is used to extract a preset number of samples from the original query point set. k The nearest neighbor algorithm constructs a local spatial relationship graph, calculates the Euclidean distance matrix between the query point and the neighboring points, and uses the radius threshold to filter out invalid adjacency relationships that exceed the preset spatial range, forming a local relationship subgraph after topological connection sparse processing.

[0219] During the loss calculation phase, a binary cross-entropy loss is applied to the retained valid adjacent nodes. The predicted logical value of the query point is compared with the actual occupied labels of its neighboring points. Gradient backpropagation is used to optimize the network's ability to model local spatial consistency. A dual-path feature reconstruction supervision mechanism is also constructed. Pixel-level feature vectors are extracted from the shallow and deep feature maps of the three-plane encoder, respectively. Bilinear interpolation is used to spatially align the predicted feature map with the true label map. Consistency constraints on feature distribution are enforced through mean squared error calculation. The shallow reconstruction loss emphasizes matching geometric details, while the deep reconstruction loss strengthens semantic feature alignment.

[0220] The proposed voxel occupancy loss calculation method based on spatial consistency constraints innovatively introduces an adaptive neighborhood sampling strategy, dynamically adjusting the perceptual range of the spatial consistency constraint through configurable radius and number of neighbors parameters. The multi-task loss fusion module uses a 0.05:1 weighting ratio to balance the primary occupancy prediction and auxiliary feature reconstruction tasks. The deep and shallow reconstruction features output by the feature decoder are spatially projected using independent convolutional kernels to ensure deep matching of the supervisory signal with the network layer. This loss calculation method demonstrated significant advantages in NTU dataset testing, improving the occupancy prediction CD metric by 30% compared to the traditional cross-entropy loss.

[0221] 4. The self-supervised fine-tuning module is used to obtain a reference occupancy probability model using the same network and supervision method using the lidar point cloud self-supervision, and use the reference occupancy probability model to supervise fine-tune the first-stage occupancy probability model to obtain a second-stage occupancy probability model.

[0222] The self-supervised fine-tuning module inputs the lidar point cloud into the same network architecture to generate a reference model, and constructs the BCE loss and adds it to the original loss function for self-supervised pre-training. In the fine-tuning stage, the learning rate is reduced to 1e-4 for 20 rounds of training.

[0223] 5. The multi-task deployment module is used to deploy the one-stage occupancy probability model or the two-stage occupancy probability model to the BEV segmentation network and / or the 3D occupancy segmentation network for segmentation, such as Figure 5 shown.

[0224] The 3D and BEV segmentation models based on the three-plane view (TPV) utilize multi-scale feature fusion and a lightweight design to efficiently process point cloud data and predict the occupancy status of query points. The following describes the implementation details.

[0225] The input point cloud first passes through three TPV encoders (TPV_encoder_1, TPV_encoder_2, TPV_encoder_4) with different downsampling rates to extract multi-scale features. Each encoder voxelizes the point cloud to generate orthogonal XY 、 YZ、 ZX Planar feature maps, where TPV_encoder_1 retains the original resolution to capture details, and TPV_encoder_4 downsamples by a factor of 4 to expand the receptive field to capture global information. The features output by the three encoders are concatenated in the channel dimension, compressed using a 1×1 convolution, and fused to form the final three-planar feature representation.

[0226] The fused features are further enhanced by the lightweight ULite module. ULite adopts a simplified UNet structure, which preserves multi-scale information by downsampling in the encoding path and upsampling in the decoding path, combined with skip connections. For example, [[ID= After the planar features are processed by ULite, the output enhanced feature map has a dimension of 512×40×64, corresponding to a bird's-eye view space with a resolution of 0.4 meters. ​ and ​ The plane generates feature maps of 40×512×64 and 512×40×64 respectively, completely covering the vertical and horizontal directions of the three-dimensional space.

[0227] For feature extraction of query points, the model converts the voxel index input from the outside into 2D coordinates on three planes. ​ ) as an example, ​ The plane takes its first two coordinates ( ​ ), ​ The plane takes the middle two dimensions ( ​ ), the ZX plane takes the last two dimensions ( ​ ) and permutes the dimensions to match the feature map structure. After extracting features from each plane using bilinear interpolation, the feature vectors of the three planes are concatenated along the channel dimension, forming a 192-dimensional feature vector that is fed into the MLP network. This MLP consists of two fully connected layers. The first layer maps the features to a 64-dimensional space and outputs the final prediction after ReLU activation. When used for 3D semantic segmentation, the output dimension can be adjusted to the number of categories, directly generating a semantic label for each query point.

[0228] The model defines its perception space using a 128×128×32 voxel grid, corresponding to an actual range of 51.2m×51.2m×12.8m, covering the near-field perception requirements of autonomous vehicles. Its lightweight design also ensures a real-time inference speed of 30 FPS. Experiments demonstrate that the model effectively identifies road boundaries, vehicles, and pedestrians in complex urban scenes. Its bird's-eye view features can be directly used for drivable area segmentation, while its 3D features support obstacle height and volume estimation, providing dense environment understanding for path planning.

[0229] In the final deployment phase, the optimized model was integrated into a bird's-eye view segmentation network and a 3D occupancy grid prediction network (voxel resolution 0.4m×0.4m×0.4m, sliding window real-time update latency ≤50ms). Verified on the MSC dataset, it achieved an inference speed of 33.6 FPS on a performance platform equipped with an NVIDIA 4060 GPU.

[0230] It should be noted here that the system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above. The system is a 4D millimeter-wave radar space occupancy probability estimation method based on lidar supervision applied to the above embodiment.

[0231] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0232] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A 4D millimeter-wave radar space occupancy probability estimation method based on lidar supervision, characterized in that: The steps include: Collect 3D point cloud data through LiDAR and construct an environment occupancy probability map as the supervision signal of LiDAR; A lightweight convolutional neural network is used to extract features from the sparse point cloud data of the 4D millimeter wave radar to obtain radar point cloud features; the lightweight convolutional neural network includes a sparse encoder, a dense encoder and a prediction head connected in sequence; the sparse encoder includes more than three layers of 3D sparse convolution layers and a feature compression projection layer; the dense encoder includes stacked 2D convolution layers; the prediction head includes a parallel set of height regression branches and occupancy prediction branches, wherein the height regression branch is used to generate XY The height prediction map of the plane, the occupancy prediction branch is used to output the occupancy probability distribution map of the three-dimensional space; Perform cross-modal feature alignment of the lidar supervision signal and the radar point cloud features, construct a highly aligned weak supervision loss function, iteratively optimize the weight parameters of the lightweight convolutional neural network, and obtain a one-stage occupancy probability model; Using the same network structure and supervision method, the LiDAR point cloud is used for self-supervision to obtain a reference occupancy probability model, and the reference occupancy probability model is used to supervise and fine-tune the first-stage occupancy probability model to obtain a second-stage occupancy probability model. Deploy the one-stage occupancy probability model or the two-stage occupancy probability model to downstream tasks.

2. The 4D millimeter-wave radar space occupancy probability estimation method based on laser radar supervision according to claim 1 is characterized in that: The environmental occupancy probability map includes a binary occupancy query map and a lidar height map; The binary occupancy query graph is constructed by the following steps: Assume that the point cloud set obtained by laser radar scanning is P ={ p i’ | i' =1,2,…, n },in p i’ ∈ R 3 Indicates the i' The three-dimensional coordinates of the point cloud, the center coordinates of the lidar sensor are c ∈ R 3 ; For each point cloud p i’ Calculate the unit direction vector u i’ : ,||•|| represents Euclidean distance calculation; According to the short-wave physical characteristics of the laser radar, two types of query point sets are generated: occupied query point set R i’ + ={ r i’ + | r i’ + = p i’ + r • u i’ } represents the occupied area behind the object surface; unoccupied query point set R i’ - ={ r i’ - | r i’ - = p i’ - r • u i’ } represents the free space in the sensor's line of sight; where r i’ + is a positive sampling point, r >0 is the preset backward offset, r i’ - is a negative sampling point; Will{ R i’ + , R i’ - The combination is constructed into a binary occupancy query graph in three-dimensional space; The lidar height map is constructed by the following steps: The three-dimensional space along XY The plane is divided into a uniform two-dimensional grid with a grid resolution of Δ x ×Δ y ; For each grid ( i , j ), calculate the height eigenvalue H ij : , where max means taking all points in the grid z Axis maximum value, ( x , y , z ) represents the coordinates, ( x i , y i ) is the coordinate of the lower left corner of the grid, P is the lidar point cloud; 4D millimeter wave radar point cloud P k mmw Through the coordinate transformation matrix T lidar mmw Project to the lidar coordinate system and match the corresponding XY Plane grid position; For grids with millimeter-wave radar points, the corresponding height feature values ​​are used as pseudo labels to construct the lidar height map. H LiDAR .

3. The method for estimating spatial occupancy probability of 4D millimeter-wave radar based on laser radar supervision according to claim 1 is characterized in that: The sparse encoder includes a point cloud feature embedding module, a sparse feature pooling module, a feature dimension compression module, a three-plane feature generation module, and a cross-view feature alignment module connected in sequence; The point cloud feature embedding module performs point-by-point feature learning on the input point cloud through a multi-layer perceptron, wherein the multi-layer perceptron comprises a plurality of linear layers and batch normalization layers stacked alternately; The sparse feature pooling module adopts a three-dimensional sparse maximum pooling operation, performing grid division in three orthogonal dimensions: XY plane, YZ plane, and ZX plane, where the pooling kernel size is determined according to the predefined segmentation parameters. split= [ s 1, s 2, s 3] Dynamic determination; The feature dimension compression module compresses the features output by the sparse feature pooling module to a preset dimension through a learnable linear projection layer, and uses the ReLU activation function for nonlinear transformation; The three-plane feature generation module includes three processing branches working in parallel: XY Plane branch: 3D sparse tensor along Z The axis is max-pooled, reshaped into a two-dimensional feature map through transposition operation, and then generated by a dedicated multi-layer perceptron XY Plane view features; the dedicated multi-layer perceptron includes alternately stacked linear layers and batch normalization layers; YZ Plane branch: 3D sparse tensor along X The axis is max-pooled and reshaped into a two-dimensional feature map through dimension permutation operation, and then generated by a dedicated multi-layer perceptron YZ Plan view features; ZX Plane branch: 3D sparse tensor along Y The axis is max-pooled, and after the channel rearrangement operation is reshaped into a two-dimensional feature map, a dedicated multi-layer perceptron is generated. ZX Plan view features; The cross-view feature alignment module unifies the batch dimension so that XY flat, YZ flat, ZX The view features of the plane maintain the same spatial resolution, and the output dimension is [ B , C , H , W ]’s standard feature map format, B represents the batch size, C Indicates the number of channels, H Indicates height, W Indicates width.

4. The method for estimating spatial occupancy probability of 4D millimeter-wave radar based on laser radar supervision according to claim 1 is characterized in that: The dense encoder adopts a neural network architecture based on bird's-eye view feature extraction, including a cascaded downsampling module, a multi-level residual learning module, a feature pyramid fusion module and a multi-scale feature upsampling module connected in sequence; The cascade downsampling module includes at least three convolution blocks connected in sequence, and the step sizes of the downsampling operations in the convolution blocks decrease in sequence; The multi-level residual learning module includes at least ten groups of residual units connected in series, wherein the first, third, and fifth groups of residual units integrate downsampling operations, and the outputs of each group of residual units are connected to the subsequent feature pyramid fusion module through jump connections to establish a multi-scale feature transmission channel; The feature pyramid fusion module is used to perform top-down feature fusion on the feature maps of different scales output by the multi-level residual learning module, including the following steps: Upsample deep fine-grained features by a factor of 2; Add the up-sampled result to the shallow coarse-grained features element by element; Cross-channel feature calibration is achieved through 1×1 convolution kernel; Apply batch normalization and ReLU activation to the fused feature map; The multi-scale feature upsampling module uses a bilinear interpolation method to restore the feature map after feature fusion to the original input size and outputs three-view features.

5. The method for estimating spatial occupancy probability of 4D millimeter-wave radar based on laser radar supervision according to claim 1 is characterized in that: The cross-modal feature alignment of the lidar supervision signal and the radar point cloud features is performed to construct a highly aligned weak supervision loss function, specifically: Introducing height map constraints in the intermediate layers of the sparse encoder and the decoder of the dense encoder; The height map constraint includes a supervisory signal generation module, a multimodal alignment prediction module, and an adaptive loss constraint module connected in sequence; The supervisory signal generation module is used to perform point cloud height statistics and pseudo-label generation on the grid area covered by the millimeter wave radar point cloud; the point cloud height statistics specifically includes: calculating the mean or maximum value of the radar point height in a preset voxel space as a statistical value; the pseudo-label generation specifically includes: using the statistical value as a supervisory signal for height prediction to form a reference height map; The multimodal alignment prediction module includes a parallel processing height feature extraction and nonlinear activation unit; the height feature extraction is specifically: generating an initial height prediction map through a learnable 1×1 convolution kernel on the BEV feature map; the nonlinear activation unit uses the ReLU function to perform a nonlinear transformation on the initial height prediction map and output a non-negative height estimate; The adaptive loss constraint module optimizes the prediction accuracy by constructing an effective area mask and fusion of multi-source supervision. The effective area mask construction is specifically: generating a binary mask matrix based on the radar point distribution to identify the effective supervision area; the multi-source supervision fusion is specifically: calculating the non-negative height estimation value in the mask area H pred,i,j With reference height map H LiDAR,i,j The mean square error As a highly aligned weakly supervised loss function, M i,j is a binary mask matrix, i , j The grid coordinates of the lidar height map.

6. The method for estimating spatial occupancy probability of 4D millimeter-wave radar based on laser radar supervision according to claim 1, characterized in that: The prediction head includes feature fusion and multi-task loss; The feature fusion specifically comprises: unifying the three-view features output by the dense encoder using a multi-layer perceptron to output a final occupancy probability distribution map; The multi-task loss is specifically defined as: ,in, is the highly aligned weakly supervised loss function in the sparse encoder, is the highly aligned weakly supervised loss function in the dense encoder, L occ is the binary cross entropy loss of occupancy query, ω1, ω2, ω3 are weight coefficients; The binary cross entropy loss of the occupancy query includes a random sampling optimization module, a dynamic neighborhood construction module, a geometric perception loss calculation module, and a multi-scale loss aggregation module; The dynamic neighborhood construction module performs the following operations on the query point cloud in each batch: k The nearest neighbor algorithm selects for each query point k Initial neighbor points are formed to form an initial adjacency relationship graph; the Euclidean distance between the query point and each initial neighbor point is calculated, and the Euclidean distance of the query point is dynamically selected to be less than the preset radius threshold. r’ The corresponding neighbor points of , generate an adaptive neighborhood graph; The geometric perception loss calculation module includes the following parallel calculation processes: The main loss calculation based on binary cross entropy is to calculate the standard cross entropy loss for the predicted logits and the true labels of each query point-neighbor point pair in the adaptive neighborhood graph. Geometric consistency auxiliary loss: According to the spatial distribution characteristics of the initial neighbor points, geometric continuity constraints are imposed on the neighbor points in the adaptive neighborhood graph; The random sampling optimization module specifically performs the following operations when the number of query points in a single batch in the geometric perception loss calculation module exceeds a preset threshold: A number of sampling points are randomly selected according to uniform distribution as the support point set; Construct a local neighborhood graph based only on the support point set to maintain the spatial distribution characteristics of the original point cloud; The multi-scale loss aggregation module achieves cross-batch loss normalization in the following way: Independently calculate the local neighborhood loss of each batch in the geometric perception loss calculation module and sum them up; The total loss is averaged by batch size to generate the final optimization target.

7. The method for estimating spatial occupancy probability of 4D millimeter-wave radar based on laser radar supervision according to claim 1, characterized in that: The same network structure and supervision method are used to obtain a reference occupancy probability model through self-supervision of the lidar point cloud, and the reference occupancy probability model is used to supervise and fine-tune the first-stage occupancy probability model to obtain a second-stage occupancy probability model. Specifically, the following steps are included: Using only the LiDAR point cloud, an initial occupancy model with the same structure as the first-stage occupancy probability model is trained to obtain a reference occupancy probability model. The occupancy probability results generated by the reference occupancy probability model are used as supervisory signals. The cross-entropy loss is constructed with the output of the first-stage occupancy probability model and added to the original loss function of the first-stage occupancy probability model. Back-propagation is used to correct for noise and misjudgment areas in the radar prediction to obtain the second-stage occupancy probability model. During the supervised fine-tuning process, an adaptive mask is introduced to filter out pseudo labels in areas not covered by the radar to achieve dynamic alignment.

8. The method for estimating spatial occupancy probability of 4D millimeter-wave radar based on laser radar supervision according to claim 1 is characterized in that: The deployment of the one-stage occupancy probability model or the two-stage occupancy probability model to downstream tasks is specifically as follows: Loading the model parameters of the one-stage occupancy probability model or the two-stage occupancy probability model, replacing the prediction head, and executing the corresponding downstream task; The prediction head is replaced by the following modules connected in sequence: A multi-scale voxel encoding module includes at least three voxel encoders; the downsampling factor multiples of the at least three voxel encoders increase sequentially, and are used to respectively extract feature maps of different resolutions in the XY plane, the YZ plane, and the ZX plane; Feature dimension reduction convolution layer, used to compress and fuse the channel dimensions of feature maps of different resolutions in each plane; An improved U-Net architecture is used to enhance the contextual information of the fused plane features. Specifically, the improved U-Net architecture uses depthwise separable convolution for feature compression in the encoding path, integrates an attention gating mechanism in the decoding path, and introduces a dynamic feature calibration module through skip connections. A three-dimensional coordinate projection mechanism is used to map query points to each plane feature map and perform interpolation sampling of different plane features; The scalable MLP output head is used to adjust the output dimension according to the requirements of downstream tasks.

9. The method for estimating spatial occupancy probability of 4D millimeter-wave radar based on laser radar supervision according to claim 8, characterized in that: The feature dimension reduction convolution layer uses a 1×1 convolution kernel to achieve cross-scale feature fusion, as shown in the following formula: ； in, is the output of the feature dimension reduction convolution layer, Conv 1×1 represents a 1×1 convolution kernel, [•] represents the channel dimension splicing operation, is the input of the feature dimension reduction convolution layer, plane ∈ XY , YZ , ZX Represents a three-dimensional space projection plane; The scalable MLP output head achieves task adaptation through the following formula: ； in, Output For output, Adapt weights to tasks, is the hidden layer weight, σ represents the ReLU activation function, F xy 、 F yz 、 F zx Represents a three-dimensional projection plane XY , YZ , ZX characteristics; D For the output dimension, by setting the output dimension D Perform different downstream tasks.

10. 4D millimeter-wave radar space occupancy probability estimation system based on laser radar supervision, characterized by: A 4D millimeter-wave radar space occupancy probability estimation method based on lidar supervision applied to any one of claims 1-9, comprising a lidar supervision signal generation module, a millimeter-wave radar feature extraction network, a cross-modal joint training module, a self-supervised fine-tuning module, and a multi-task deployment module; The laser radar supervision signal generation module is used to collect three-dimensional point cloud data through the laser radar and construct an environment occupancy probability map as a supervision signal of the laser radar; The millimeter-wave radar feature extraction network uses a lightweight convolutional neural network to extract features from the sparse point cloud data of the 4D millimeter-wave radar to obtain radar point cloud features. The lightweight convolutional neural network includes a sparse encoder, a dense encoder, and a prediction head connected in sequence. The sparse encoder includes three or more 3D sparse convolution layers and a feature compression projection layer. The dense encoder includes stacked 2D convolution layers. The prediction head includes a height regression branch and an occupancy prediction branch arranged in parallel, wherein the height regression branch is used to generate a height prediction map in the XY plane, and the occupancy prediction branch is used to output an occupancy probability distribution map in the three-dimensional space. The cross-modal joint training module is used to perform cross-modal feature alignment on the lidar supervision signal and the radar point cloud features, construct a highly aligned weak supervision loss function, iteratively optimize the weight parameters of the lightweight convolutional neural network, and obtain a one-stage occupancy probability model; The self-supervised fine-tuning module is used to obtain a reference occupancy probability model using the same network and supervision method through self-supervision of the lidar point cloud, and to perform supervised fine-tuning on the first-stage occupancy probability model using the reference occupancy probability model to obtain a second-stage occupancy probability model. The multi-task deployment module is used to deploy the one-stage occupancy probability model or the two-stage occupancy probability model to downstream tasks.

Images