A three-dimensional target detection system and method based on multi-task fusion

Abstract

The present invention relates to a three-dimensional target detection system and method based on multi-task fusion, the detection system includes an RGB camera, a laser radar and a fusion algorithm perception module, and the detection method includes: collecting a traffic scene image in front of a vehicle by an RGB camera, collecting three-dimensional point cloud data around the vehicle by a laser radar; transmitting the traffic scene image in front of the vehicle to an image processing unit to perform semantic segmentation on the traffic scene image in front of the vehicle, and outputting corresponding semantic information to a point cloud processing unit; transmitting the three-dimensional point cloud data around the vehicle to a point cloud processing unit to add corresponding semantic information to the three-dimensional point cloud data around the vehicle, and performing three-dimensional target detection on the point cloud data with added semantic information, and outputting three-dimensional target detection results. Compared with the prior art, the present invention can effectively improve the accuracy of three-dimensional target detection in complex low-light environments.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a three-dimensional target detection system and method based on multi-task fusion. Background Art

[0002] 3D object detection, the core foundation of autonomous driving perception systems, is a crucial prerequisite for path planning, motion prediction, collision avoidance, and other functions. As an upstream link in autonomous driving, its detection performance directly impacts subsequent autonomous driving processes, such as planning and decision-making.

[0003] In recent years, camera- and lidar-based object recognition and classification technologies have made significant progress. However, current industry research and testing of this technology primarily focuses on scenarios with good visibility. In poor lighting conditions, such as at night and in fog, camera- and lidar-based object recognition technologies perform poorly for detecting dynamic objects such as vehicles and pedestrians.

[0004] One of the current mainstream methods for fusing LiDAR and cameras is to perform semantic segmentation on images captured by the camera, augmenting the LiDAR point cloud with semantic information, and then using advanced pure point cloud detection methods for detection. However, this fusion method is overly simplistic, making the accuracy of subsequent network detection highly dependent on the quality of the output results of the preceding semantic segmentation network. In complex lighting environments, the signal-to-noise ratio of camera images decreases, inevitably leading to a significant decline in semantic segmentation quality, which in turn can lead to significant deviations in subsequent detection. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a three-dimensional target detection system and method based on multi-task fusion to improve the accuracy of three-dimensional target detection in complex low-light environments.

[0006] The objectives of the present invention can be achieved through the following technical solutions: a three-dimensional target detection system based on multi-task fusion, including an RGB camera and a laser radar, the RGB camera and the laser radar are respectively connected to a fusion algorithm perception module, the RGB camera is used to collect traffic scene images in front of the vehicle, the laser radar is used to collect three-dimensional point cloud data around the vehicle, and the fusion algorithm perception module is used to perform fusion algorithm processing on the traffic scene images in front of the vehicle and the three-dimensional point cloud data around the vehicle to output three-dimensional target detection information in the current traffic scene.

[0007] Furthermore, the fusion algorithm perception module includes an image processing unit and a point cloud processing unit connected in sequence, the image processing unit is connected to the RGB camera, and the point cloud processing unit is connected to the laser radar;

[0008] The image processing unit is used to perform semantic segmentation on the traffic scene image in front of the vehicle to output corresponding semantic information to the point cloud processing unit;

[0009] The point cloud processing unit is used to add corresponding semantic information to the three-dimensional point cloud data around the vehicle, perform three-dimensional target detection on the point cloud data with added semantic information, and output the three-dimensional target detection result.

[0010] Furthermore, the RGB camera and laser radar are both installed on the top of the vehicle, and the horizontal field of view of the laser radar is 360°.

[0011] A three-dimensional object detection method based on multi-task fusion includes the following steps:

[0012] S1, collects traffic scene images in front of the vehicle through RGB camera, and collects 3D point cloud data around the vehicle through LiDAR;

[0013] S2. The traffic scene image in front of the vehicle is transmitted to the image processing unit to perform semantic segmentation on the traffic scene image in front of the vehicle and output corresponding semantic information to the point cloud processing unit;

[0014] S3. The three-dimensional point cloud data around the vehicle is transmitted to the point cloud processing unit to add corresponding semantic information to the three-dimensional point cloud data around the vehicle, perform three-dimensional target detection on the point cloud data with added semantic information, and output the three-dimensional target detection result.

[0015] Furthermore, the image processing unit in step S2 is specifically a semantic segmentation network based on deep learning to perform semantic segmentation on the traffic scene image in front of the vehicle.

[0016] Furthermore, the step S3 specifically includes the following steps:

[0017] S31, the point cloud processing unit first performs foreground segmentation to segment the laser point cloud belonging to the foreground object, that is, obtains the foreground laser points, and then performs a fusion colorization strategy based on discarding boundary points, distance, and semantic segmentation uncertainty to add corresponding semantic information to the three-dimensional point cloud data around the vehicle;

[0018] S32. The point cloud processing unit performs three-dimensional target detection on the point cloud data with added semantic information based on the target detection network architecture of deep learning, and outputs the three-dimensional target detection results.

[0019] Furthermore, the coloring strategy based on discarding boundary points in step S31 specifically discards all object boundary points during fusion: first, it is determined whether the point cloud projected onto the image plane belongs to the object boundary point. If it is not a boundary point, the point cloud coloring operation is performed; if it is a boundary point, the point cloud coloring operation is not performed, but all boundary points are given a separate category, wherein the boundary point refers to a point with a different semantic segmentation category from the eight surrounding pixel points on the image plane.

[0020] Furthermore, the distance-based coloring strategy in step S31 is specifically as follows: a distance range is set, and a coloring operation is performed on point clouds classified as vehicles outside the set distance range, and no coloring operation is performed on point clouds classified as vehicles within the set distance range.

[0021] Furthermore, the coloring strategy based on semantic segmentation uncertainty in step S31 is specifically as follows: using the Monte Carlo sampling method to obtain semantic segmentation uncertainty, outputting the category of each pixel point in the semantic segmentation stage, and outputting the uncertainty of the semantic segmentation of each pixel point at the same time, and then using the semantic segmentation category and uncertainty together to expand the three-dimensional point cloud information.

[0022] Furthermore, the step S32 specifically includes the following steps:

[0023] S321, performing point cloud feature extraction on the point cloud data with added semantic information to obtain a point cloud feature vector;

[0024] S322. For the point cloud feature vector, simultaneously perform the point cloud semantic segmentation task and the 3D target detection task, combine the loss functions of the two tasks to obtain a multi-task loss function, complete the three-dimensional target detection, and output the three-dimensional target detection result.

[0025] Furthermore, in step S322, the loss function of the point cloud semantic segmentation task is:

[0026]

[0027] Among them, N pos is the number of foreground laser points, M is the total number of categories of point cloud semantic segmentation, s ij is the probability that the i-th foreground laser point belongs to the j-th semantic category, For s ij The predicted value of

[0028] The 3D object detection task uses a regression loss based on the reference unit bin to estimate the 3D bounding box of the object: each foreground laser point is placed in a series of reference units along the X and Z axes. The localization loss along the X and Z axes consists of two parts: the first part is the classification loss of each center point in multiple reference units in the sub-coordinate system, and the other part is the regression loss of the offset in the reference unit:

[0029]

[0030]

[0031]

[0032] Among them, x (p) ,y (p) ,z (p) is the coordinate value of the foreground laser point, x p ,y p ,z p is the center coordinate of the corresponding object, and is the true value bin distribution along the X-axis and Z-axis, and is the true value of the position offset within the reference unit, C is the length of each reference unit after standardization, S is the search range of each canonical coordinate system, δ is the unit length of each reference unit, u is the unified representation of the x and z coordinates, that is, u∈{x,z}, for the definition of the yaw angle θ and Similarly, for the definition of bounding box dimensions (h,w,l) and similar;

[0033] Therefore, the 3D regression box loss can be summarized as

[0034]

[0035]

[0036]

[0037] Among them, N pos is the number of foreground laser points, F cls is the cross entropy loss, F reg is the smooth L1 regression loss, is the sum of the cross entropy loss and regression loss for the x, z, and θ dimensions, is the predicted bin distribution, is the predicted position bias, is the sum of the regression losses for y, h, w, and l dimensions, L regis the average loss of all foreground laser points;

[0038] Get the loss function L for the 3D object detection task refine for:

[0039]

[0040] Among them, β is the set of generated target 3D candidate boxes, β pos is the set of positive samples obtained by regression, prob i Predict confidence for the category of the candidate box, label i is the corresponding category truth value.

[0041] Comprehensive L Seg and L refine , get the multi-task loss function L total for:

[0042]

[0043] Among them, σ1 and σ2 are the homoscedastic uncertainties corresponding to the 3D object detection task and the point cloud semantic segmentation task, respectively.

[0044] Compared with the existing technology, the present invention constructs a fusion algorithm perception module. The fusion algorithm perception module first uses the image processing unit to perform semantic segmentation on the image, then inputs the semantic information of the image into the point cloud processing unit, uses a fusion strategy based on discarding boundary points, distance, and semantic uncertainty to expand the semantic information of the point cloud, and then realizes 3D target detection through the designed multi-task pure point cloud detection framework. The present invention innovatively proposes a colorization fusion strategy based on uncertainty, distance, and discarding boundary points. Compared with traditional point cloud colorization, the vehicle detection accuracy of the vehicle detection network integrated with the semantic information network will be effectively improved in the case of high dependence on semantic segmentation quality and reduced vehicle detection accuracy. At the same time, the detection accuracy of pedestrians and cyclists remains at a high level.

[0045] In addition, the present invention designs the point cloud semantic segmentation task and the 3D object detection task to be performed in parallel to realize a multi-task learning architecture, which can effectively improve the accuracy of three-dimensional object detection and the generalization effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the system structure of the present invention;

[0047] Figure 2 Schematic diagram of the method flow of the present invention;

[0048] Figure 3 Schematic diagram of the application process of the embodiment;

[0049] Figure 4This is a diagram of the PointRCNN three-dimensional target detection framework in the embodiment;

[0050] Description of the marks in the figure:

[0051] 1. RGB camera, 2. LiDAR, 3. Fusion algorithm perception module, 301. Image processing unit, 302. Point cloud processing unit. DETAILED DESCRIPTION

[0052] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Example

[0054] like Figure 1 As shown, a three-dimensional target detection system based on multi-task fusion includes an RGB camera 1 and a laser radar 2. The RGB camera 1 and the laser radar 2 are respectively connected to a fusion algorithm perception module 3. The fusion algorithm perception module 3 includes an image processing unit 301 and a point cloud processing unit 302 connected in sequence. The image processing unit 301 is connected to the RGB camera 1, and the point cloud processing unit 302 is connected to the laser radar 2.

[0055] Among them, the RGB camera 1 is used to collect images of the traffic scene in front of the vehicle, the laser radar 2 is used to collect three-dimensional point cloud data around the vehicle, and the fusion algorithm perception module 3 is used to perform fusion algorithm processing on the traffic scene image in front of the vehicle and the three-dimensional point cloud data around the vehicle to output three-dimensional target detection information in the current traffic scene. Specifically, the image processing unit 301 is used to perform semantic segmentation on the traffic scene image in front of the vehicle to output corresponding semantic information to the point cloud processing unit 302;

[0056] The point cloud processing unit 302 is used to add corresponding semantic information to the three-dimensional point cloud data around the vehicle, perform three-dimensional target detection on the point cloud data with added semantic information, and output the three-dimensional target detection result.

[0057] In actual applications, the RGB camera 1 and the lidar 2 can input the data they collect into the fusion algorithm perception module 3 via a data cable. Both the RGB camera 1 and the lidar 2 are installed on the top of the vehicle. The lidar 2 has a horizontal field of view of 360°, thereby ensuring that the RGB camera 1 can capture the traffic scene in front of the vehicle and the lidar 2 can collect three-dimensional point cloud data around the vehicle.

[0058] Apply the above system to realize a 3D target detection method based on multi-task fusion, such as Figure 2 As shown, the following steps are included:

[0059] S1, collects traffic scene images in front of the vehicle through RGB camera, and collects 3D point cloud data around the vehicle through LiDAR;

[0060] S2. The image of the traffic scene in front of the vehicle is transmitted to the image processing unit to perform semantic segmentation on the image of the traffic scene in front of the vehicle, and output corresponding semantic information to the point cloud processing unit. In this embodiment, a semantic segmentation network based on deep learning is used to perform semantic segmentation on the image of the traffic scene in front of the vehicle;

[0061] S3. The three-dimensional point cloud data around the vehicle is transmitted to the point cloud processing unit, which adds corresponding semantic information to the three-dimensional point cloud data around the vehicle, performs three-dimensional target detection on the point cloud data with added semantic information, and outputs the three-dimensional target detection results. Specifically:

[0062] S31. The point cloud processing unit first performs foreground segmentation to segment the laser point cloud belonging to the foreground object, that is, to obtain the foreground laser point. Then, a fusion coloring strategy based on discarding boundary points, distance, and semantic segmentation uncertainty is performed to add corresponding semantic information to the three-dimensional point cloud data around the vehicle. The coloring strategy based on discarding boundary points specifically discards all object boundary points during fusion: first, it is determined whether the point cloud projected onto the image plane belongs to the object boundary point (a boundary point refers to a point with a semantic segmentation category different from that of the point in the eight surrounding pixel points on the image plane). If it is not a boundary point, the point cloud coloring operation is performed; if it is a boundary point, the point cloud coloring operation is not performed, but all boundary points are given a separate category.

[0063] The distance-based coloring strategy is as follows: set a distance range, perform coloring on point clouds with vehicle categories outside the set distance range, and do not perform coloring on point clouds with vehicle categories within the set distance range;

[0064] The colorization strategy based on semantic segmentation uncertainty specifically uses the Monte Carlo sampling method to obtain semantic segmentation uncertainty. During the semantic segmentation stage, the category of each pixel is output, and the uncertainty of the semantic segmentation of each pixel is output at the same time. The category and uncertainty of the semantic segmentation are then used together to expand the 3D point cloud information.

[0065] S32, the point cloud processing unit performs three-dimensional target detection on the point cloud data with added semantic information based on the target detection network architecture of deep learning, and outputs the three-dimensional target detection result. First, point cloud feature extraction is performed on the point cloud data with added semantic information to obtain a point cloud feature vector;

[0066] Then, for the point cloud feature vector, the point cloud semantic segmentation task and the 3D object detection task are performed simultaneously. The loss functions of the two tasks are combined to obtain a multi-task loss function to complete the 3D object detection and output the 3D object detection results. Among them, the loss function of the point cloud semantic segmentation task is:

[0067]

[0068] Where N pos is the number of foreground laser points, M is the total number of categories of point cloud semantic segmentation, s ij is the probability that the i-th foreground laser point belongs to the j-th semantic category, For s ij The predicted value of

[0069] The 3D object detection task uses a regression loss based on the base unit bin to estimate the 3D bounding box of the object: each foreground laser point is placed in a series of base units along the X and Z axes. The localization loss along the X and Z axes consists of two parts: the first part is the classification loss of each center point in multiple base units in the sub-coordinate system, and the other part is the regression loss of the offset in the base unit:

[0070]

[0071]

[0072]

[0073] Where x (p) ,y (p) ,z (p) is the coordinate value of the foreground laser point, x p ,y p ,z p is the center coordinate of the corresponding object, and is the true value bin distribution along the X-axis and Z-axis, and is the true value of the position offset within the reference unit, C is the length of each reference unit after standardization, S is the search range of each canonical coordinate system, δ is the unit length of each reference unit, u is the unified representation of the x and z coordinates, that is, u∈{x,z}, for the definition of the yaw angle θ and Similarly, for the definition of bounding box dimensions (h,w,l) and similar;

[0074] Therefore, the 3D regression box loss can be summarized as

[0075]

[0076]

[0077]

[0078] Where N pos is the number of foreground laser points, F cls is the cross entropy loss, F regis the smooth L1 regression loss, is the sum of the cross entropy loss and regression loss for the x, z, and θ dimensions, is the predicted bin distribution, is the predicted position bias, is the sum of the regression losses for y, h, w, and l dimensions, L reg is the average loss of all foreground laser points;

[0079] Get the loss function L for the 3D object detection task refine for:

[0080]

[0081] Among them, β is the set of generated target 3D candidate boxes, β pos is the set of positive samples obtained by regression, prob i Predict confidence for the category of the candidate box, label i is the corresponding category truth value.

[0082] Comprehensive L seg and L refine , get the multi-task loss function L total for:

[0083]

[0084] Where σ1 and σ2 are the homoscedastic uncertainties corresponding to the 3D object detection task and the point cloud semantic segmentation task, respectively.

[0085] In summary, the present invention proposes a multi-task fusion detection method based on point cloud semantic colorization. The present invention can complete the detection of 3D targets through an RGB camera and a multi-line laser radar. The detection method architecture is as follows Figure 3 As shown, it can be roughly divided into Figure 3 The two stages shown in the figure are as follows: in the first stage, semantic segmentation is performed on the image, and semantic coloring is performed on the point cloud based on the strategy of discarding boundary points, distance, and uncertainty according to the semantics of the image; in the second stage, the point cloud semantic segmentation task and the 3D object detection task are set using the point cloud with additional semantic information, and 3D object detection is performed using a multi-task learning architecture.

[0086] This technical solution takes into account the fact that the second-stage detection network in the point cloud semantic colorization detection method is highly dependent on the quality of the first-stage semantic segmentation, resulting in no significant improvement in vehicle detection accuracy compared to the pure point cloud detection method, and even lower detection accuracy in low-light scenarios than the pure point cloud detection method. During fusion, a suitable fusion strategy is formulated in combination with uncertainty to improve the accuracy of vehicle detection in traffic scenarios. This embodiment uses PointRCNN to construct a three-dimensional target detection framework. Its main process includes: in the first stage, a three-dimensional candidate box is generated from the original point cloud from top to bottom, and in the second stage, the three-dimensional candidate box is refined in the regular coordinate system. Specifically:

[0087] (1) Use RGB cameras and lidar to collect perception images and three-dimensional point cloud data around the vehicle in traffic scenes.

[0088] (2) The image data is input into the image processing unit, and the image semantic segmentation is completed through the trained image semantic segmentation network.

[0089] (3) The image semantic information and the original three-dimensional point cloud data are input into the point cloud processing unit, and the semantic information in the corresponding image is given to the point cloud based on the fusion strategy of discarding boundary points, distance, and uncertainty.

[0090] (4) Figure 4 As shown in the figure, the point cloud with attached semantic information is input into PointRCNN. First, the point cloud feature vector is extracted using PointNet++ as the backbone network.

[0091] (5) The point cloud feature vector is used to simultaneously perform foreground point segmentation and bin-based 3D bounding box generation. The ground truth mask of the foreground point is provided by the ground truth 3D bounding box of the target. Focal Loss is used to address the class imbalance problem caused by too few foreground points in outdoor scenes. The 3D bounding box generation branch uses foreground point regression to generate a 3D bounding box proposal for the target, predicting the target center position, target orientation, and target size.

[0092] (6) After obtaining the 3D bounding box proposal, optimize the position and orientation of the box. Pool each laser point and its features based on the position of each 3D proposal. Zoom in on the bounding box by 1.2 times, and retain the points and features within the bounding box. Then, use a segmentation mask to distinguish foreground and background points within the zoomed box, and eliminate proposals without internal points.

[0093] (7) The pooled points and their related features are fed into the second stage sub-network to optimize the position of the 3D bounding box and the credibility of the foreground object. Figure 4The canonical coordinate transformation shown in the figure above better learns the local spatial features of each proposal. The optimization subnetwork then combines the transformed local spatial points with the global semantic features from the first stage to optimize bounding boxes and confidences. A similar loss based on bin-based regression is also used to optimize object proposals. Finally, the 3D object detection results are output.

[0094] Compared to other 3D object detection methods, this technical solution, which incorporates a semantic information network, significantly improves vehicle detection accuracy compared to the original pure point cloud detection network, while maintaining a high level of pedestrian and cyclist detection accuracy.

[0095] By designing a network multi-task learning architecture for the detection task in the second stage of sequential fusion, the detection accuracy and model generalization effect are improved;

[0096] It has better feasibility and leading edge for vehicle detection in challenging low-light scenarios.

Claims

1. A three-dimensional object detection method, applied to a three-dimensional object detection system based on multi-task fusion, characterized in that: The three-dimensional object detection system comprises an RGB camera (1) and a laser radar (2), wherein the RGB camera (1) and the laser radar (2) are respectively connected to a fusion algorithm perception module (3), wherein the RGB camera (1) is used to collect a traffic scene image in front of a vehicle, and the laser radar (2) is used to collect three-dimensional point cloud data around the vehicle, and the fusion algorithm perception module (3) is used to perform fusion algorithm processing on the traffic scene image in front of the vehicle and the three-dimensional point cloud data around the vehicle, so as to output three-dimensional object detection information in the current traffic scene; The three-dimensional object detection method comprises the following steps: S1, collect traffic scene images in front of the vehicle through RGB camera, and collect 3D point cloud data around the vehicle through LiDAR; S2, transmitting the traffic scene image in front of the vehicle to the image processing unit to perform semantic segmentation on the traffic scene image in front of the vehicle, and outputting corresponding semantic information to the point cloud processing unit; S3, transmitting the three-dimensional point cloud data around the vehicle to the point cloud processing unit, so as to add corresponding semantic information to the three-dimensional point cloud data around the vehicle, perform three-dimensional target detection on the point cloud data with added semantic information, and output the three-dimensional target detection result; The step S3 specifically comprises the following steps: S31, the point cloud processing unit first performs foreground segmentation to segment the laser point cloud belonging to the foreground object, that is, obtains the foreground laser point, and then performs a fusion coloring strategy based on discarding boundary points, distance, and semantic segmentation uncertainty to add corresponding semantic information to the three-dimensional point cloud data around the vehicle; S32. The point cloud processing unit uses a target detection network architecture based on deep learning to perform three-dimensional target detection on point cloud data with added semantic information, and outputs the three-dimensional target detection results.

2. A three-dimensional target detection method according to claim 1, characterized in that: The fusion algorithm perception module (3) comprises an image processing unit (301) and a point cloud processing unit (302) which are connected in sequence, wherein the image processing unit (301) is connected to the RGB camera (1), and the point cloud processing unit (302) is connected to the laser radar (2); The image processing unit (301) is used to perform semantic segmentation on the traffic scene image in front of the vehicle, so as to output corresponding semantic information to the point cloud processing unit (302); The point cloud processing unit (302) is used to add corresponding semantic information to the three-dimensional point cloud data around the vehicle, perform three-dimensional target detection on the point cloud data with the semantic information added, and output the three-dimensional target detection result.

3. A three-dimensional target detection method according to claim 1, characterized in that: The image processing unit in step S2 is specifically a semantic segmentation network based on deep learning to perform semantic segmentation on the traffic scene image in front of the vehicle.

4. A three-dimensional target detection method according to claim 1, characterized in that: The coloring strategy based on discarding boundary points in step S31 is specifically to discard all object boundary points during fusion: first determine whether the point cloud projected onto the image plane belongs to the object boundary point. If it is not a boundary point, perform the point cloud coloring operation; if it is a boundary point, do not perform the point cloud coloring operation, but give all boundary points a separate category, wherein the boundary point refers to a point in the eight surrounding pixel points on the image plane that has a semantic segmentation category different from that of the point.

5. A three-dimensional target detection method according to claim 1, characterized in that: The distance-based coloring strategy in step S31 is specifically as follows: setting a distance range, performing a coloring operation on point clouds of vehicle category outside the set distance range, and not performing a coloring operation on point clouds of vehicle category within the set distance range.

6. A three-dimensional target detection method according to claim 1, characterized in that: The coloring strategy based on semantic segmentation uncertainty in step S31 is specifically as follows: using the Monte Carlo sampling method to obtain semantic segmentation uncertainty, outputting the category of each pixel point in the semantic segmentation stage, and outputting the uncertainty of the semantic segmentation of each pixel point at the same time, and then using the category and uncertainty of the semantic segmentation together to expand the three-dimensional point cloud information.

7. A three-dimensional target detection method according to claim 1, characterized in that: The step S32 specifically includes the following steps: S321, extracting point cloud features from the point cloud data with added semantic information to obtain a point cloud feature vector; S322. For the point cloud feature vector, the point cloud semantic segmentation task and the 3D target detection task are performed simultaneously, and the loss functions of the two tasks are combined to obtain a multi-task loss function, so as to complete the three-dimensional target detection and output the three-dimensional target detection result.

8. A three-dimensional target detection method according to claim 7, characterized in that: In step S322, the loss function of the point cloud semantic segmentation task is: Among them, N pos is the number of foreground laser points, M is the total number of categories of point cloud semantic segmentation, and s ij is the probability that the i-th foreground laser point belongs to the j-th semantic category, For ij The predicted value of The 3D object detection task uses a regression loss based on the reference unit bin to estimate the 3D bounding box of the object: each foreground laser point is placed in a series of reference units along the X-axis and Z-axis. The positioning loss along the X-axis and Z-axis consists of two parts. The first part is the classification loss of each center point in multiple reference units in the sub-coordinate system, and the other part is the regression loss of the offset in the reference unit: Among them, x (p) ,y (p) ,z (p) is the coordinate value of the foreground laser point, x p ,y p ,z p is the center coordinate of the corresponding object, and is the true value bin distribution along the X-axis and Z-axis, and is the true value of the position offset in the reference unit, C is the length of each reference unit after standardization, S is the search range of each canonical coordinate system, δ is the unit length of each reference unit, u is the unified representation of the x and z coordinates, that is, u∈{x,z}, for the definition and Similarly, for the definition of bounding box dimensions (h,w,l) and similar; Therefore, the 3D regression box loss can be summarized as Among them, N pos is the number of foreground laser points, F cls is the cross entropy loss, F reg is the smooth L1 regression loss, is the sum of the cross entropy loss and regression loss for the x, z, and θ dimensions, is the predicted bin distribution, is the predicted position bias, is the sum of the regression losses for the y, h, w, and l dimensions, L reg is the average loss of all foreground laser points; Get the loss function L for the 3D object detection task refine for: Among them, β is the set of generated target 3D candidate boxes, β pos is the set of positive samples obtained by regression, prob i Predict confidence for the category of the candidate box, label i is the corresponding category truth value; Comprehensive L Sea and L refine , we get the multi-task loss function L total for: Among them, σ1 and σ2 are the homoscedastic uncertainties corresponding to the 3D object detection task and the point cloud semantic segmentation task, respectively.

Images