System and method for three-dimensional space occupancy and target detection based on radar and vision fusion

Through the three-dimensional space occupation and target detection method of lightning vision fusion, combined with 4D millimeter wave radar and image data, features are extracted and fused to generate three-dimensional occupancy grid points and frames, solving the problem of the existing technology that has a great impact on light and occlusion and target recognition Poor problems achieve more accurate and robust dynamic and static target perception.

CN118584465BActive Publication Date: 2024-11-05SHANGHAI GEOMETRICAL PERCEPTION & LEARNING CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411074405.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-11-05
Estimated Expiration
2044-08-07

Smart Images

  • Figure CN118584465B_ABST
    Figure CN118584465B_ABST
Patent Text Reader

Abstract

This invention discloses a system and method for 3D spatial occupancy and target detection based on radar-visual fusion. Compared to single-task target detection, the joint detection and occupancy of this invention can provide more accurate and dense scene description information. In terms of data selection, this invention utilizes a self-developed 4D millimeter-wave radar as the point cloud data input, which has a high signal-to-noise ratio and a data volume of approximately 6000, providing excellent spatial and velocity information of target reflection points, and is visible at night and unaffected by adverse weather conditions such as wind and rain. Multi-channel surround-view images provide very rich category semantic information about the vehicle's surroundings, but are affected by resolution and weather conditions. Therefore, effectively combining these two methods, utilizing their respective data characteristics, and fusing multi-sensor data in a certain way can achieve complementary advantages, improve single-modal perception performance, and enhance the accuracy and robustness of the detection system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a system and method for three-dimensional space occupation and target detection. Background Art

[0002] Due to the sparsity of observation point clouds, traditional 3D millimeter radars only play an auxiliary role in target detection solutions as an additional attribute of multi-source sensor fusion. The introduction of 4D imaging millimeter wave radars and rich point cloud images make it fully feasible to perform pixel-level pre-fusion of 4D imaging millimeter wave radar point cloud images with visual images and laser images, which can greatly improve the performance of network models when single modality is used as input.

[0003] Existing technologies have certain limitations in space occupancy and target detection. Traditional image-based target detection methods are easily affected by factors such as lighting and occlusion, while radar-based methods perform poorly in target classification and recognition. Therefore, a method that can combine the advantages of radar and images is needed to improve the accuracy, real-time and robustness of space occupancy and target detection.

[0004] Patent CN202410030567.7 proposes a 4D millimeter-wave radar target detection and semantic segmentation model that integrates graphs and grids. This method uses the data of a new 4D millimeter-wave radar sensor as the only data source input to the target detection network, and selects the graph method encoding among the three methods based on points, grids, and graphs in the perception task, hoping that the graph structure can utilize the topological relationship between point clouds to improve the detection effect. This method ignores the problem of difficulty in extracting semantic information due to the sparsity and unevenness of millimeter waves when there is a single data source.

[0005] Patent CN202410184532.9 proposes a probabilistic target detection method based on image and millimeter wave radar fusion. This method uses image and millimeter wave radar multimodal data as the data input for the detection task, converts the millimeter wave radar data into image coordinates and generates a 2D pseudo feature map that is consistent with the image size, and outputs the 2D information of the detected target on the image. In this task, the category information of the target can be obtained, but the problem of scene applicability such as spatial positioning is ignored.

[0006] Patent CN202410404069.4 proposes a 4D millimeter-wave radar and vision adaptive fusion target recognition system. This method uses 4D millimeter-wave radar features as query vectors, sets images as access and alignment vectors, interactively fuses information at the BEV feature map level, and finally performs target detection on the reshaped BEV features based on the anchor frame detection head. In the article, the author classified the failure conditions of two types of sensors and made network switching operations. In this case, we use shielded branch input in the network instead of switching to the traditional depth estimation method, which can be equivalent to data enhancement learning, which has more flexible and adaptive learning model capabilities.

[0007] In the above patents, 3D target detection using 4D millimeter-wave radar and image fusion appears and achieves good detection effects on dynamic targets. However, it cannot cope well with the cases where there are no categories in the detection task and obvious obstacles on the road. Therefore, how to achieve effective semantic perception expression within the scene range is very critical. Summary of the invention

[0008] The purpose of the present invention is to propose a joint learning training of two types of tasks, occupancy and detection, to optimize the detection task and improve the perception and performance of dynamic and static targets.

[0009] In order to achieve the above object, the technical solution of the present invention is to provide a method for three-dimensional space occupancy and target detection based on radar and vision fusion, which is characterized by comprising the following steps:

[0010] Step 1, receiving 4D millimeter wave radar point cloud and image data;

[0011] Step 2: Preprocess the 4D millimeter wave radar point cloud and image data;

[0012] Step 3: The occupancy and detection joint model performs three-dimensional space occupancy and target detection on the preprocessed 4D millimeter wave radar point cloud and image data, specifically including the following steps:

[0013] Step 301: The occupancy and detection joint model extracts radar features and image features from the pre-processed 4D millimeter wave radar point cloud and image data;

[0014] Step 302, fusing the extracted radar features and image features to obtain three-dimensional space occupancy information, includes the following steps:

[0015] Stitching radar features and image features along the channel;

[0016] Obtain feature maps of different sizes for information interaction;

[0017] Step 303: According to the information interaction between feature maps of different sizes, high-level semantic information and low-level spatial position information are integrated to detect the target and generate three-dimensional occupied grid points and boxes, including occupation tasks and target detection tasks, wherein:

[0018] In the inference stage, the occupation task uses the first feature map F0 obtained in step 302. After the feature map F0 is processed by CRC convolution and dimension reshaping, the feature maps 𝛰 of 5 categories are obtained. 𝑜𝑐𝑐,𝑐l𝑠 And the regression feature map of 3 speeds 𝑂 𝑜𝑐𝑐,𝑟𝑒𝑔 ; Feature map 𝛰 𝑜𝑐𝑐,𝑐l𝑠 After the Sigmoid activation function and confidence filtering with a category threshold of 0.1, the feature map Ο occ,cls,sig and feature map occ,mask , combined with the clipping range pcr, the grid size of the occupied points and the channel order, it is converted into a discrete point occupied point O with real-world coordinates and categories cls , and combined with the feature map occ,mask Obtain the vehicle speed and target speed, and then combine the output of the category and speed prediction to obtain the complete occupied point output;

[0019] In the inference stage, the target detection task uses the third, fourth, and fifth feature maps obtained in step 302, and merges these feature maps to obtain feature map F cat,det ; Feature map F cat,det The predicted box attribute set is obtained through the CRC structure; the number of grids grid_det, the world range point_range, and the grid size voxel_det are combined, and the fixed grid space is expanded in a certain order to obtain equally spaced grid scatter points. Then, each predicted box attribute in the predicted box attribute set is expanded in the same order to obtain the original output of the predicted box; the original output of the predicted box is filtered by NMS to obtain the final available box set;

[0020] The reasoning results of the joint occupancy task and the target detection task are output to output the target's 3D occupied grid points and boxes, as well as the background's occupied points.

[0021] Preferably, in step 1, the 4D millimeter wave radar point cloud and image data have completed the alignment of timestamps.

[0022] Preferably, in step 2, preprocessing the 4D millimeter wave radar point cloud includes the following steps:

[0023] The 4D millimeter wave radar point cloud is cropped based on a preset cropping range pcr;

[0024] Convert the discrete points after range clipping into a dense and continuous grid;

[0025] Add, concatenate, and reshape feature dimensions to the mesh.

[0026] Preferably, in step 2, preprocessing the image data includes the following steps:

[0027] Crop each channel of image data to select a fixed area;

[0028] Randomly perform data augmentation operations on the cropped images;

[0029] The images of all roads after data augmentation are concatenated along the channel dimension.

[0030] Preferably, in step 301, the occupancy and detection joint model extracts the image features using the following method:

[0031] The image data preprocessed in step 2 is recorded as the input image, the input image is passed through the network layers of the first four modules of ResNet50 to obtain the feature map C4, and the input image is passed through the network layers of the first three modules of ResNet50 to obtain the feature map C3;

[0032] The feature map C4 is upsampled by 2 times and convolved with a double-layer kernel of 1, and then merged with the feature map C3 after convolution with a double-layer kernel of 1 to obtain a usable image feature map I after the image features are downsampled by 8 times;

[0033] Combine the clipping range pcr and the grid size size to build the 3D space voxel center point tensor G 3D , and the 3D space voxel center point tensor G 3D The camera internal and external parameters are projected and constrained to the range of the two-dimensional image feature map of the same size as the available image feature map I; then, the 3D space voxel center point tensor G 3D Normalize the coordinates to get the reference image G with depth information img ;

[0034] Each image feature in the available image feature map I is combined with the reference image G in turn. img , and sampled by bilinear sampling to a tensor G with the same size as the center point of the 3D space voxel 3D Spatial feature map F with consistent grid size s superior;

[0035] Merge the feature maps F of each camera in turn s , and then compressed into the BEV space through dimensional reshaping to obtain the image features.

[0036] Preferably, when training the joint occupancy and detection model, the truth data is preprocessed, and the occupancy heat map and the truth box heat map are preprocessed respectively using the occupancy scattered truth values ​​of the same timestamp and the annotated truth boxes available in the current frame:

[0037] The preprocessing of the occupancy heat map includes the following steps:

[0038] The occupied scatter point truth values ​​are clipped to limit the occupied scatter point truth value range so as to adapt to the division of the fixed grid;

[0039] The cropped ground truth points are relatively voxelized and converted into dense tensors;

[0040] Generates the ground truth of the occupancy Gaussian heatmap with the target distribution on the tensor;

[0041] The preprocessing of the ground truth box heat map includes the following steps:

[0042] The labeled truth box is cropped to limit the range of the labeled truth box;

[0043] According to the middle box position of the cropped annotated true value box, the position index from the BEV perspective is generated to be used for the regression task to convert it into a dense mask to participate in the prediction value loss calculation.

[0044] Preferably, when training the occupancy and detection joint model, seven feature maps of four different sizes after interaction are obtained through step 302 as inputs of step 303; in the reasoning stage of the occupancy and detection joint model, the detection head feature map F0 for occupancy task reasoning and the detection head feature map {F2, F3, F4} for target detection task are obtained through step 302 as inputs of step 303.

[0045] Preferably, the training of the occupation task comprises the following steps:

[0046] Seven feature maps of four different sizes are obtained after interaction. After each feature map is processed by CRC convolution, a feature map F of five categories in the order of channels is obtained. occ,cls,j And the regression feature map F of the three speeds occ,reg,j ; The feature map F occ,cls,j And regression feature map F occ,reg,j The channel dimension C and the height dimension D are separated and reshaped into a 5-dimensional tensor, and then the different sizes after preprocessing occupy the true value set. The loss is calculated for the true value of the same size of the index; different loss calculation methods are used for classification and regression tasks. In the classification loss calculation, a loss function is used to reduce the weight of easy-to-classify samples and increase the weight of difficult-to-classify samples, and the classification loss is recorded as Locc,cls ; In the regression loss calculation, the L1 loss function is used for regression supervision, and the regression loss is recorded as L occ,reg ; Then the total loss L of the training of the occupied task occ =L occ,cls +2×L occ,reg .

[0047] Preferably, the training of the target detection task comprises the following steps:

[0048] After obtaining three feature maps of preset sizes from the seven feature maps of four different sizes, these feature maps are merged by channel, and the key quantities are predicted based on the merged feature maps. For the two different tasks of regression and classification, the target detection task uses Focal Loss and L1 loss functions for supervision calculation, and the position index of the labeled true value box needs to be substituted in the regression task, so the loss in the det task is recorded as L det =(L det,cls +L det,dim +L det,dir +L det,vel +L det,z +L det,offset )×5, where L det,cls is the category loss of the detection task, L det,dim is the box size loss for the detection task, L det,dir is the box orientation angle loss for the detection task, L det,vel is the box speed loss for the detection task, L det,z is the box height direction loss for the detection task, L det,offset The box center point offset loss for the detection task;

[0049] Combined with the training of the occupation task and the training of the object detection task, the total loss of the occupation and detection joint model is recorded as L = L occ +L det , and update the weights of the entire model through gradient back-propagation.

[0050] Another technical solution of the present invention is to provide a system for three-dimensional space occupancy and target detection based on radar and vision fusion, which is used to implement the above method and includes:

[0051] Heterogeneous data module for receiving 4D millimeter wave radar point cloud and image data;

[0052] Preprocessing module, used to preprocess 4D millimeter wave radar point cloud and image data;

[0053] Joint occupancy and detection model.

[0054] Compared with single-task target detection, the combined detection and occupancy of the present invention can provide more accurate and dense scene description information. In terms of data selection, the present invention uses self-developed 4D millimeter radar as point cloud data input, which has a high signal-to-noise ratio and a data volume of up to about 6000. It can provide very good spatial information and speed information of target reflection points, and is visible at night and not afraid of bad weather such as wind and rain; multi-way surround view images provide very rich category semantic information around the vehicle, but are affected by resolution, weather, etc. Therefore, effectively combining the two, utilizing their respective data characteristics, and fusing multi-sensor data in a certain way can achieve complementary advantages, improve single-modal perception performance, and improve the accuracy and robustness of the detection system.

[0055] Occupancy and detection tasks often complement each other in practical applications. For example, in autonomous driving, object detection is used to identify pedestrians and vehicles, while scene occupancy analysis is used to understand the entire traffic environment, provide more accurate real-world obstacle conditions, and better support subsequent downstream tasks, such as planning safe paths. In occupancy and detection tasks, object detection usually focuses on identifying and locating specific targets, while scene occupancy analysis focuses more on understanding and describing the entire scene. In detection tasks, we focus more on locating known categories in images or scenes, and the output is the spatial position, size, orientation, and speed estimation of dynamic targets. In occupancy tasks, we focus not only on object detection, but also on the description of the entire scene, analyzing the structure and layout of the scene, understanding how targets are distributed in space, including the distribution of static and dynamic targets, and outputting the position, category, and speed of the perceived three-dimensional occupied grid points in the entire scene.

[0056] The present invention effectively designs a system and method for three-dimensional space occupancy and target detection based on radar and vision fusion through an innovative approach, providing a highly robust method for achieving all-weather driving scene perception and detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 The network framework of the present invention is illustrated. DETAILED DESCRIPTION

[0058] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0059] Combination Figure 1 One aspect of an embodiment of the present invention is to provide a method for three-dimensional space occupancy and target detection based on radar and vision fusion, comprising the following steps:

[0060] Step 1: Receive 4D millimeter wave radar point cloud and image data.

[0061] When the 4D millimeter wave radar point cloud and image data are input into the occupation and detection joint model of the present invention, the alignment of the timestamps of the 4D millimeter wave radar point cloud and image data has been completed in the front-end hardware. For the 4D millimeter wave radar data points and image data with the same timestamp t, they can be respectively expressed as and , where: N is the number of points of the 4D millimeter wave radar; M is the number of pictures at each perspective; the nth point dimension information is expressed as ,and , x n ,y n 、z n , …, * n is the multi-dimensional attribute of the point in the point cloud, i.e., the front, left, top and other optional attributes in space; I m is the mth camera image I.

[0062] Step 2: Preprocess the 4D millimeter wave radar point cloud and image data obtained in step 1, including input data enhancement, true value encoding and other operations, specifically including:

[0063] 1) Preprocessing of input data

[0064] In the following it will appear:

[0065] size is the grid size of the 4D millimeter wave radar in the three directions of x, y and z. The coordinate system uses the laser radar coordinate system.

[0066] point_range_clound (pcr) is the range setting centered on the vehicle.

[0067] 1) The preprocessing of 4D millimeter wave radar point cloud includes the following steps:

[0068] Step 2A01, cropping: 4D millimeter wave radar points The range of restrictions, set the reduction range pcr to: , , , the operation is recorded as Crop(), then:

[0069]

[0070] In the formula, The discrete points after the range is clipped.

[0071] Step 2A02, voxelization: discrete points after clipping the range Convert to a dense and continuous grid Combined with the grid size, the process can be expressed as the following formula, and the whole operation can be recorded as Voxel().

[0072]

[0073] In the formula, The spatial point is voxelized to become the tensor coordinate position. x, y, and z are the three-dimensional coordinate information of the point in the laser radar coordinate system. size x 、size y 、size z Preset the grid size in three-dimensional space respectively;

[0074] The above operation can be recorded as Voxel(), which is equivalent to:

[0075]

[0076] Step 2A03, data encoding: Add, concatenate, and reshape the feature dimensions. The overall operation can be recorded as Trans():

[0077]

[0078]

[0079] Where B is the number of sample dimensions of the 4D millimeter wave radar feature tensor (Batch Size), C is in_r Channel is the input channel dimension of the 4D millimeter wave radar feature tensor, D in_r is the height (z direction) dimension of the 4D millimeter wave radar feature tensor, W in_r is the width (y direction) dimension of the 4D millimeter wave radar feature tensor, H in_r It is the length (x direction) dimension of the 4D millimeter wave radar feature tensor.

[0080] 2) The preprocessing of image data includes the following steps:

[0081] Step 2B01: image data from several cameras To crop the image, a fixed area is selected as the input of the model image branch. The overall cropping process can be recorded as CropImg():

[0082]

[0083] In the formula, is the cropped image;

[0084] Step 2B02: crop the image Randomly perform data enhancement operations, such as brightness, contrast, adding noise, etc. The whole process can be recorded as Augment():

[0085]

[0086] In the formula, The image after data augmentation.

[0087] Step 2B03: the images processed in step 2B02 The concatenation is performed along the channel dimension as shown below:

[0088]

[0089]

[0090] In the formula, Concat() represents the concatenation process.

[0091] 2) Preprocessing of ground truth data

[0092] The preprocessing stage of true value data only exists in the model training stage.

[0093] Same as step 1, the occ (occupancy) scatter point truth value with the same timestamp t needs to be used The ground truth box available for the current frame .in: , ,cls k is the integer representation of the category; , , l k is the length of the current truth box k in the x direction, w k is the width of the current truth box k in the y direction, h k is the height of the current truth box k in the z direction, r k is the overall rotation angle of the current truth box k, v_c k is the speed of the vehicle at the previous true value frame k, v_t k The target vehicle speed of the previous truth frame k.

[0094] 1) occ-heatmap (occupancy heatmap), including the following steps:

[0095] Step 2C01, the same Crop operation as described in step 2A01 above, limits the range of the scattered point true value of occ to adapt to the division of the fixed grid, expressed as:

[0096]

[0097] In the formula, is the true value point after clipping.

[0098] Step 2C02: Perform the Voxel operation described in step 2A02 above, and then Perform relative voxelization and convert it into a dense tensor , expressed as:

[0099]

[0100] Step 2C03, generate Gaussian heatmap by true value encoding. Combined with the range-restricted PCR, grid size, and annotation box The xy position information and the two-dimensional Gaussian function Gaussian in the tensor Generate the true value of the OCC Gaussian heat map based on the target distribution , for subsequent loss calculation (i.e. loss calculation). The overall data transformation process can be simplified as:

[0101]

[0102] In the formula, C occ is the channel dimension size of the tensor, H occ is the height dimension size of the tensor, W occ is the tensor width dimension size, L occ The length dimension of the tensor.

[0103] 2) Box-heatmap (truth box heatmap), including the following steps:

[0104] Step 2D01: To adapt to a certain perception range, the range of the true value box needs to be limited, which is the same as the Crop operation described in step 2A01 above, expressed as:

[0105]

[0106] In the formula, is the original truth box (original), It is the true value box after clipping according to the range.

[0107] Step 2D02: Similar to the heatmap generation of occ, the heatmap of Box also needs to encode some regressors separately, especially according to The middle frame position of the BEV is used to generate the position index from the BEV perspective. For regression tasks, it is converted into a dense mask to participate in the prediction value loss calculation. The process can be divided into 6 key quantities, namely , , , , , ,in, , other key quantities are , W det is the width (y) dimension of the thermal map from the BEV perspective, H det is the height (x) dimension of the heat map from the BEV perspective, C det are the dimensions of different regression key quantities, which are 3, 2, 2, 1, and 1 in the above order.

[0108] Step 3: Extract radar features and image features from the preprocessed 4D millimeter-wave radar point cloud and image data. For the 4D millimeter-wave radar point cloud, the target’s distance, speed, angle and other information can be extracted; for the image data, the target’s color, texture, shape and other information can be extracted.

[0109] 1) Radar feature extraction, including the following steps:

[0110] After the data preprocessing in step 2, the 4D millimeter wave radar data input is obtained , perform additional learning of channel weights to obtain radar features , the whole process can be recorded as conv_radar():

[0111] .

[0112] 2) Image feature extraction, including the following steps:

[0113] Step 301: Image feature pre-extraction. After the multi-channel image pre-processing process in step 2, the transformed input , after passing through the network layers of the first four modules of ResNet50, the feature map C4 is downsampled by 16 times, and then the feature map C4 is upsampled by 2 times and convolved with a double-layer kernel of 1 to integrate the channel dimension information, and then combined with the feature map C3 (input The image features are obtained through the network layers of the first three modules of ResNet50) and merged as the available image feature map I after the image features are downsampled 8 times.

[0114] Step 302: Introduce depth information. Combine pcr and grid size to create a 3D space voxel center point tensor G 3D (Grid), this generation process can be recorded as Generate_3DGrid(), and the tensor G 3DThe camera internal and external parameters are projected and constrained to the range of the two-dimensional image feature map of the same size as the image feature map I in step 301. This process can be recorded as 3DTo2D(); then the coordinates are normalized to obtain the reference image G with depth information img This process can be recorded as normalize_grid2d (); then each image feature in the image feature map I in step 301 is combined with G img And through bilinear sampling, the spatial feature map with the same size as the 3D space voxel center point tensor G grid size is sampled F s This process can be recorded as sampling_grid(). The feature map I of the image after feature extraction contains a new feature map of three-dimensional spatial position information. F s , and then merge the feature maps of each camera in turn F s , and then compressed to BEV space by dimension reshaping to obtain the BEV feature map of the final image F I , this process can be recorded as reshape(). In summary, the process of introducing depth information can be recorded as follows:

[0115] G = Generate_3DGrid(pcr,size)

[0116] G img = normalize_grid2d(3DTo2D(G 3D ))

[0117]

[0118]

[0119] Step 4: Fuse the extracted radar features and image features to obtain three-dimensional space occupancy information. Fusion can be performed using weighted averaging, feature concatenation, feature splicing, etc. to improve detection accuracy.

[0120] In step 3, the radar features after 4D millimeter wave radar point cloud encoding are obtained respectively and image features , and then splice the two along the channel. The process is recorded as:

[0121]

[0122] In the formula, concat() is the feature concatenation operation.

[0123] Then, a method similar to FPN is adopted to make feature maps of different scales interact with each other, which is recorded as:

[0124]

[0125] During the training phase, seven feature maps of four different scales will be simultaneously derived as the feature map input of the detection head, denoted as ; In the inference phase, the OCC detection head feature map will be derived And det detection head feature map .

[0126] Step 5: Based on the information interaction between feature maps of different sizes, high-level semantic information and low-level spatial position information are integrated to detect the target and generate three-dimensional occupied grid points and boxes. Object detection algorithms such as deep learning and multi-target tracking can be used to determine the location, size and category of the target.

[0127] The detection head uses the CRC structure of the CeterHead standard calculation paradigm by default, that is, the convolution processing flow of Conv1x1, ReLu, Conv1x1, which acts on the feature map F* series derived in step 4 to obtain the original output of classification and regression. In particular, in the det (detection task) task, the center position index of the true value box is involved , which comes from the true value data preprocessing in step 2.

[0128] 1) Training phase

[0129] 1) OCC training, including the following steps:

[0130] Step 5A01: 7 feature maps obtained from step 4 , that is, 7 corresponding loss detection heads are used for multi-layer supervision calculation loss. Each detection head is processed by CRC convolution to obtain feature maps of 5 categories (including background) in channel order. And the regression feature map of 3 speeds , where B is the number of samples (Batch size), D is the size of the height (z direction) dimension, W is the size of the width (y direction), and H is the size of the length (x direction).

[0131] Step 5A02: Generally, and The channel dimension C and the height dimension D are separated and reshaped into a 5-dimensional tensor , which can be uniformly expressed as , and then the occ true value sets of different sizes after preprocessing The loss is calculated using the true occ value of the same size indexed by j in .

[0132] Step 5A03: Different loss calculation methods are used for classification and regression tasks. In the classification loss calculation, a loss function (FocalLoss) is used to reduce the weight of easy-to-classify samples and increase the weight of difficult-to-classify samples. The classification loss can be expressed as ; In the regression calculation, the L1 loss function is used for regression supervision, denoted as .

[0133] Step 5A04, in the multi-layer supervision loss of OCC, the default .

[0134] 2) DET training

[0135] Step 5B01, 7 feature maps obtained from step 4 , using s feature maps with preset sizes , and use the feature map after channel merging As the input of the det detection head, .

[0136] Step 5B02: In the det detection head, there are also two tasks, classification and regression, and both use CRC. Perform convolution processing on the detection head to obtain C=4 categories of BEV feature maps in channel order , C=3 prediction box size feature map , C = 2 orientation angle , C=2 ego vehicle and prediction box speed , Z-axis value of C=1 and the center offset of C=1 Above, there are a total of 6 predictions of key quantities.

[0137] Step 5B03, similar to the calculation of OCC loss, the classification and regression tasks of DET use Focal Loss and L1 loss functions for supervision calculation respectively. In the regression task, the position index 𝕀 of the true value box needs to be substituted, which generates an additional logical mask. The loss in the DET task can be recorded as ,in, is the category loss of the detection task (loss, detection, class), is the box size loss (loss, detection, dimension) for the detection task, is the box orientation angle loss (loss, detection, direction) for the detection task, is the box velocity loss (loss, detection, velocity) for the detection task, is the box height (z-axis) direction loss of the detection task (loss, detection, z-axis), is the box center point offset loss for the detection task (loss, detection, offset).

[0138] The classification loss function is the binary cross entropy loss function BCELoss, and the remaining is the numerical regression task, and the function used is the L1 loss function MAELoss, which can be expressed as:

[0139]

[0140]

[0141] Among them, 𝑝 𝑖 is the predicted quantity, 𝑦 𝑖 is the corresponding truth value.

[0142] 3) Gradient Update

[0143] Combined with the OCC training task and the DET training task, the total loss can be recorded as , and update the weights of the entire model through gradient back-propagation.

[0144] II) Reasoning Stage

[0145] In the inference phase, OCC prediction only uses the first feature map derived in step 4. , det prediction is the same as the feature map after channel merging in the training stage .

[0146] 1) Reasoning of the OCC perception task, including the following steps:

[0147] Step 5C01, from the OCC training task, we know that After CRC convolution processing and dimension reshaping, the feature maps of 5 categories (including background) are obtained. And the regression feature map of 3 speeds .

[0148] Step 5C02, After the Sigmoid activation function and confidence filtering using a category threshold of 0.1, and , and then combined with point_range (i.e., pcr in step 2), voxel_occ (grid size of occupied points) and channel order to transform into discrete occupied points with real-world coordinates and categories , where N mask is the number of occupied points after threshold filtering. Similarly, additionally combined Get the vehicle speed and target speed , where v_own i is the speed of the vehicle (the same vehicle), v_tar i is to predict the speed of occupied point i. Therefore, the output of combined category and speed prediction can get the complete occ occupied point output, which is recorded as .

[0149] 2) Reasoning of the det perception task, including the following steps

[0150] Step 5D01, from the det training task, we know that The detection head output is obtained through the CRC structure , which is the set of predicted box attributes. Generally speaking, , where B is the batch, defaults to 1, and W and H depend on the grid_det setting of the det task.

[0151] Step 5D02, combine the number of grids grid_det, the world range point_range, and the grid size voxel_det, and expand the fixed grid space in a certain order 𝒯 to obtain equally spaced grid points, and then expand the position offset 𝐹d𝑒𝑡,offset in the same order 𝒯 to obtain the final BEV space position x and y. Similarly, reshape the remaining prediction box attributes in the same expansion order 𝒯 to obtain the original output of the prediction box .

[0152] Step 5D03: After NMS filtering, the final available box set is obtained , denoted as 𝑫.

[0153] 3) Inference output

[0154] The reasoning results of the combined OCC perception task and the DET perception task can output the target's three-dimensional occupied grid points and boxes, as well as the background's occupied points.

[0155] Another aspect of the embodiments of the present invention is to provide a system for three-dimensional space occupancy and target detection based on radar and visual fusion for implementing the above method, comprising:

[0156] A heterogeneous data module, used to receive the 4D millimeter wave radar point cloud and image data described in step 1 above;

[0157] A preprocessing module, used to implement the data preprocessing described in step 2 above;

[0158] The joint occupancy and detection model further includes:

[0159] A feature extraction module, used to extract radar features and image features as described in step 3 above;

[0160] A fusion module, used to realize the fusion of radar features and image features described in step 4 above;

[0161] The output module is used to implement the fused feature information obtained by the fusion module as described in step 5 above, detect the target and generate three-dimensional occupied grid points and boxes.

[0162] The above technical solution uses the self-developed 4D millimeter radar sensor to output point cloud data and multi-channel image data. After pre-coding by their respective branch convolutional networks, the millimeter wave radar feature map and the image feature map after dual sampling of additional spatial information are spliced ​​along the channel dimension. The fused feature map is subjected to a feature reuse operation similar to a feature pyramid to fully integrate and interact with the semantic and position information of each size, and at the same time, multiple feature maps of different sizes are induced. After passing through the detection head preset in the task, the regressors required for occupation and detection are output.

Claims

1. A method for three-dimensional space occupancy and target detection based on radar and vision fusion, characterized in that: The following steps are involved: Step 1, receiving 4D millimeter wave radar point cloud and image data; Step 2, preprocessing the 4D millimeter wave radar point cloud and image data; Step 3, using the occupation and detection joint model to perform three-dimensional space occupancy and target detection on the preprocessed 4D millimeter wave radar point cloud and image data, specifically including the following steps: Step 301, the occupation and detection joint model extracts radar features and image features from the preprocessed 4D millimeter wave radar point cloud and image data; Step 302, the extracted radar features and image features are fused to obtain three-dimensional space occupancy information, including the following steps: splicing radar features and image features along the channel; obtaining feature maps of different sizes for information interaction; Step 303, according to the information interaction between feature maps of different sizes, high-level semantic information and low-level spatial position information are fused to detect targets and generate three-dimensional occupied grid points and boxes, including occupation tasks and target detection tasks, wherein: in the reasoning stage, the occupation task uses the first feature map F0 obtained in step 302, and the feature map F0 is processed by CRC convolution and dimension reshaping to obtain 5 categories of feature maps 𝛰 𝑜𝑐𝑐,𝑐l𝑠 And the regression feature map of 3 speeds 𝑂 𝑜𝑐𝑐,𝑟𝑒𝑔 ; Feature map 𝛰 𝑜𝑐𝑐,𝑐l𝑠 After the Sigmoid activation function and confidence filtering with a category threshold of 0.1, the feature map Ο occ,cls,sig and feature map occ,mask , combined with the clipping range pcr, the grid size of the occupied points and the channel order, it is converted into a discrete point occupied point O with real-world coordinates and categories cls , and combined with the feature map occ,mask The vehicle speed and the target speed are obtained, and thus, the output of the combined category and speed prediction is used to obtain the complete occupied point output; in the inference stage, the target detection task uses the third, fourth, and fifth feature maps obtained in step 302, and these feature maps are merged to obtain the feature map F cat,det ; Feature map F cat,det The predicted box attribute set is obtained through the CRC structure; the number of grids grid_det, the world range point_range, and the grid size voxel_det are combined, and the fixed grid space is expanded in a certain order to obtain equally spaced grid scatter points, and then each predicted box attribute in the predicted box attribute set is expanded in the same order to obtain the original output of the predicted box; the original output of the predicted box is filtered by NMS to obtain the final available box set; the reasoning results of the joint occupation task and the target detection task are combined to output the target three-dimensional occupied grid points and boxes, as well as the occupied points of the background; In step 301, the occupancy and detection joint model extracts the image features by the following method: the image data preprocessed in step 2 is recorded as the input image, the input image is passed through the network layer of the first four modules of ResNet50 to obtain the feature map C4, and the input image is passed through the network layer of the first three modules of ResNet50 to obtain the feature map C3; the feature map C4 is upsampled by 2 times and convolved with a double-layer kernel of 1, and then merged with the feature map C3 after convolution with a double-layer kernel of 1 to obtain the available image feature map I after the image features are downsampled by 8 times; combined with the cropping range pcr and the grid size size, a 3D space voxel center point tensor G is established 3D , and the 3D space voxel center point tensor G 3D The camera internal and external parameters are projected and constrained to the range of the two-dimensional image feature map of the same size as the available image feature map I; then, the 3D space voxel center point tensor G 3D Normalize the coordinates to get the reference image G with depth information img ; Combine each image feature in the available image feature map I with the reference image G in turn img , and sampled by bilinear sampling to a tensor G with the same size as the center point of the 3D space voxel 3D Spatial feature map F with consistent grid size s Above: Merge the feature maps F of each camera in turn s , and then compressed into the BEV space through dimensional reshaping to obtain the image features.

2. A method for three-dimensional space occupancy and target detection based on radar and visual fusion as claimed in claim 1, characterized in that: In step 1, the 4D millimeter wave radar point cloud and image data have completed the alignment of timestamps.

3. The method for three-dimensional space occupancy and target detection based on radar and visual fusion as claimed in claim 1, characterized in that: In step 2, the preprocessing of the 4D millimeter-wave radar point cloud includes the following steps: cropping the 4D millimeter-wave radar point cloud based on a preset cropping range PCR; converting the discrete points after range cropping into a dense and continuous grid; adding, splicing and reshaping the grid in terms of feature dimensions.

4. The method for three-dimensional space occupancy and target detection based on radar and visual fusion as claimed in claim 1, characterized in that: In step 2, the preprocessing of the image data includes the following steps: cropping each channel of the image data to select a fixed area; randomly performing data enhancement operations on the cropped image; and splicing the images of all channels after data enhancement along the channel dimension.

5. The method for three-dimensional space occupancy and target detection based on radar and visual fusion as claimed in claim 1, characterized in that: When training the joint occupancy and detection model, the truth data is preprocessed, and the occupancy heat map and the truth box heat map are preprocessed respectively using the occupancy scatter point truth values ​​of the same timestamp and the annotated truth boxes available in the current frame: the preprocessing of the occupancy heat map includes the following steps: cropping the occupancy scatter point truth values ​​to limit the range of the occupancy scatter point truth values ​​to adapt to the division of the fixed grid; relatively voxelizing the cropped truth points and converting them into a dense tensor; generating an occupancy Gaussian heat map truth with target distribution on the tensor; the preprocessing of the truth box heat map includes the following steps: cropping the annotated truth box to limit the range of the annotated truth box; generating a position index under the BEV perspective based on the middle box position of the cropped annotated truth box, so as to convert the regression task into a dense mask to participate in the prediction value loss calculation.

6. The method for three-dimensional space occupancy and target detection based on radar and visual fusion as claimed in claim 1, characterized in that: When training the occupancy and detection joint model, seven feature maps of four different sizes after interaction are obtained through step 302 as inputs of step 303; in the reasoning stage of the occupancy and detection joint model, the detection head feature map F0 for occupancy task reasoning and the detection head feature map {F2, F3, F4} for target detection task are obtained through step 302 as inputs of step 303.

7. A method for three-dimensional space occupancy and target detection based on radar and visual fusion as claimed in claim 6, characterized in that: The training of the occupation task includes the following steps: obtaining seven feature maps of four different sizes after interaction, each feature map is processed by CRC convolution to obtain a feature map F of five categories in the order of channels. occ,cls,j And the regression feature map F of the three speeds occ,reg,j ; The feature map F occ,cls,j And regression feature map F occ,reg,j The channel dimension C and the height dimension D are separated and reshaped into a 5-dimensional tensor, and then the loss is calculated with the occupied truth values ​​of the same size indexed by j in the set of occupied truth values ​​of different sizes after preprocessing; for the two different tasks of classification and regression, different loss calculation methods are used respectively. In the classification loss calculation, a loss function is used to reduce the weight of easy-to-classify samples and increase the weight of difficult-to-classify samples, and the classification loss is recorded as L occ,cls ; In the regression loss calculation, the L1 loss function is used for regression supervision, and the regression loss is recorded as L occ,reg ; Then the total loss L of the training of the occupied task occ =L occ,cls +2×L occ,reg .

8. The method for three-dimensional space occupancy and target detection based on radar and visual fusion as claimed in claim 7, characterized in that: The training of the target detection task includes the following steps: after obtaining three feature maps of preset sizes from seven feature maps of four different sizes, these feature maps are merged by channel, and the key quantities are predicted based on the merged feature maps. For the two different tasks of regression and classification, the target detection task uses Focal Loss and L1 loss functions for supervised calculation, and the position index of the labeled true value box needs to be substituted in the regression task, so the loss in the det task is recorded as L det =(L det,cls +L det,dim +L det,dir +L det,vel +L det,z +L det,offset )×5, where L det,cls is the category loss of the detection task, L det,dim is the box size loss for the detection task, L det,dir is the box orientation angle loss for the detection task, L det,vel is the box speed loss for the detection task, L det,z is the box height direction loss for the detection task, L det,offset is the box center point offset loss of the detection task; combined with the training of the occupation task and the training of the target detection task, the total loss of the occupation and detection joint model is recorded as L=L occ +L det , and update the weights of the entire model through gradient back-propagation.

9. A system for three-dimensional space occupancy and target detection based on radar and vision fusion, characterized in that: Used to implement the method described in claim 1, characterized in that it includes: a heterogeneous data module for receiving 4D millimeter wave radar point cloud and image data; a preprocessing module for preprocessing 4D millimeter wave radar point cloud and image data; and an occupancy and detection joint model.