A 3D target detection model training method and a 3D target detection method

Abstract

The application provides a 3D target detection model training method and a 3D target detection method. The training method comprises the following steps: obtaining sample label information of a sample 3D space; dividing 3D occupancy grids of the sample 3D space into a plurality of down-sampling regions; wherein each down-sampling region comprises a plurality of 3D occupancy grids; fusing the category labels of the plurality of 3D occupancy grids of each down-sampling region to determine the down-sampling label of the down-sampling region, and updating the sample label information of the sample 3D space based on the down-sampling label; constructing a loss function and training a 3D target detection model based on the updated sample label information of the sample 3D space. The training method can improve the down-sampling label accuracy in the model training process, thereby improving the model accuracy of the 3D target detection model.

Description

Technical Field

[0001] This application relates to the field of 3D object detection, and more specifically, to a training method for a 3D object detection model and a 3D object detection method. Background Technology

[0002] With the development of artificial intelligence technology, autonomous driving technology is becoming increasingly mature. Autonomous driving-related technologies can be broadly categorized into perception, prediction, localization, decision-making, and planning and control. Perception typically refers to using algorithms to analyze data from sensors (such as LiDAR, optical cameras, and ultrasonic radar) to create a real-time model of the surrounding environment. This modeled world is then input into downstream decision planning (PnC) for path planning (e.g., planning within decision planning). Among all perception tasks, 3D object detection is one of the most important. The recently popular 3D occupancy mesh is a more fine-grained 3D object detection method. Specifically, the 3D occupancy mesh discretizes the 3D space into voxels and then uses a model to output the category of each voxel, thereby more accurately segmenting the detected objects in the 3D space. From an algorithmic perspective, the 3D occupancy mesh is a 3D extension of 2D image semantic segmentation; therefore, it also utilizes image segmentation techniques. During the training of 3D object detection models, the resolution of 3D samples often differs from the resolution required by the model, and sometimes sample enhancement is necessary. Therefore, downsampling of 3D samples is frequently required. However, traditional methods use nearest neighbor sampling to process the corresponding labels when downsampling 3D samples. Since the resolution of 2D images is generally large (e.g., 1024×512), using nearest neighbor sampling to process image labels generally does not cause significant problems. However, the resolution of 3D grids is generally low (e.g., 200×200×30), and simply using nearest neighbor sampling will lead to significant ambiguity. In other words, the labels of the 3D grids of the downsampled 3D samples are ambiguous. This results in low accuracy of the trained 3D object detection model when using the sample 3D data, leading to low accuracy in detecting 3D objects. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a training method for a 3D object detection model and a 3D object detection method, which can improve the downsampling label accuracy during the model training process, thereby improving the model accuracy of the 3D object detection model.

[0004] This application provides a method for training a 3D object detection model, the training method comprising:

[0005] Obtain sample label information in the 3D space of the sample; wherein, the label information includes the category label of each 3D grid in the 3D space of the sample, and the 3D space of the sample is constructed based on sensor data during the autonomous driving process;

[0006] Based on a preset downsampling rule, the 3D occupancy grid of the sample 3D space is divided into multiple downsampling regions; wherein each downsampling region includes multiple 3D occupancy grids;

[0007] The category labels of multiple 3D occupied grids in each downsampled region are merged to determine the downsampled label of the downsampled region, and the sample label information of the sample 3D space is updated based on the downsampled label;

[0008] Based on the updated sample label information in the 3D space, a loss function is constructed and a 3D object detection model is trained to obtain a well-trained 3D object detection model.

[0009] In some embodiments, the training method for the 3D object detection model, fusing the class labels of multiple 3D occupied grids in each downsampled region to determine the downsampled label of the downsampled region, includes:

[0010] Determine the proportion of each category label in the downsampling region among all category labels;

[0011] Based on the proportion of each category label, the downsampling label of the downsampling region is determined; the downsampling label represents the probability that the 3D occupied grid after the downsampling region is merged belongs to each category label of the downsampling region.

[0012] In some embodiments, the training method for the 3D object detection model, based on a preset downsampling rule, divides the 3D occupancy grid of the sample 3D space into multiple downsampling regions, including:

[0013] Based on the requirements of the 3D target detection model and the attribute information of the 3D occupied grid in the sample 3D space, the downsampling sampling ratio is determined.

[0014] Based on the downsampling ratio, the 3D occupied grid of the sample 3D space is divided into multiple downsampling regions.

[0015] In some embodiments, the training method for the 3D object detection model, based on the downsampling ratio, divides the 3D occupied grid of the sample 3D space into multiple downsampling regions, including:

[0016] Based on the downsampling ratio, the sampling interval k in the 3D space of the sample is determined;

[0017] The adjacent k in the 3D space of the sample3 Each 3D occupied grid is defined as the sampling area.

[0018] In some embodiments, in the training method of the 3D object detection model, before obtaining the sample label information in the 3D space of the sample, the label information in the 3D space of the sample is determined by the following method:

[0019] Acquire various sensor data in the 3D space of a sample during autonomous driving; the sensor data includes: lidar data, optical camera data, and ultrasonic radar data;

[0020] By fusing the data from the multiple sensors, the 3D space of the sample is discretized into multiple 3D occupancy grids based on the fusion result of the multiple sensor data;

[0021] Label each 3D occupied grid with the corresponding category label to determine the sample label information in the 3D space of the sample.

[0022] In some embodiments, a 3D target detection method is also provided, the detection method comprising:

[0023] Obtain sample label information in the 3D space of the sample; wherein, the label information includes the category label of each 3D grid in the 3D space of the sample, and the 3D space of the sample is constructed based on sensor data during the autonomous driving process;

[0024] Based on a preset downsampling rule, the 3D occupancy grid of the sample 3D space is divided into multiple downsampling regions; wherein each downsampling region includes multiple 3D occupancy grids;

[0025] The category labels of multiple 3D occupied grids in each downsampled region are merged to determine the downsampled label of the downsampled region, and the sample label information of the sample 3D space is updated based on the downsampled label;

[0026] Based on the updated sample label information in the 3D space, a loss function is constructed and a 3D object detection model is trained to obtain a well-trained 3D object detection model.

[0027] Real-time acquisition of sensor data during vehicle operation, and construction of a target 3D space including multiple 3D occupancy grids based on the sensor data;

[0028] The target 3D space is input into the trained 3D target detection model to detect 3D targets within the target 3D space.

[0029] In some embodiments, the 3D target detection method, fusing the category labels of multiple 3D occupied grids in each downsampled region to determine the downsampled label of the downsampled region, includes:

[0030] Determine the proportion of each category label in the downsampling region among all category labels;

[0031] Based on the proportion of each category label, the downsampling label of the downsampling region is determined, and the downsampling label guarantees the probability that the 3D occupied grid in the downsampling region belongs to each category label of the downsampling region.

[0032] In some embodiments, the 3D target detection method divides the 3D occupancy grid of the sample 3D space into multiple downsampling regions based on a preset downsampling rule, including:

[0033] Based on the requirements of the 3D target detection model and the attribute information of the 3D occupied grid in the sample 3D space, the downsampling sampling ratio is determined.

[0034] Based on the downsampling ratio, the 3D occupied grid of the sample 3D space is divided into multiple downsampling regions.

[0035] In some embodiments, the 3D target detection method divides the 3D occupied grid of the sample 3D space into multiple downsampling regions based on the downsampling sampling ratio, including:

[0036] Based on the downsampling ratio, the sampling interval k in the 3D space of the sample is determined;

[0037] In the sample 3D space, the adjacent k 3 Each 3D occupied grid is defined as the sampling area.

[0038] In some embodiments, the 3D target detection method involves acquiring sensor data during vehicle movement in real time and constructing a target 3D space including multiple 3D occupancy grids based on the sensor data, including:

[0039] The system acquires multiple sensor data in the target 3D space in real time during autonomous driving; the sensor data includes: lidar data, optical camera data, and ultrasonic radar data.

[0040] By fusing the data from the various sensors, the target 3D space is discretized into multiple 3D occupancy grids based on the fusion result of the data from the various sensors.

[0041] This application provides a training method for a 3D object detection model and a 3D object detection method. The training method acquires sample label information in a sample 3D space. The label information includes the category label of each 3D occupied grid in the sample 3D space, which is constructed based on sensor data from autonomous driving. Based on a preset downsampling rule, the 3D occupied grid of the target 3D space is divided into multiple downsampling regions. Each downsampling region includes multiple 3D occupied grids. The category labels of the multiple 3D occupied grids in each downsampling region are fused to determine the downsampling label of that downsampling region, and the sample label information of the sample 3D space is updated based on the downsampling label. Based on the updated sample 3D space, a loss function is constructed and a 3D object detection model is trained to obtain a trained 3D object detection model. In this way, during the downsampling process, the fused label replaces the one-hot encoded hard label, solving the ambiguity problem in the downsampling process. This results in a trained 3D object detection model free from ambiguity and with higher accuracy, allowing for more accurate 3D object detection. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 The original RGB image described in the embodiments of this application is shown;

[0044] Figure 2 The labels for semantic segmentation corresponding to the original RGB image described in the embodiments of this application are shown;

[0045] Figure 3 This illustration shows a schematic diagram of the nearest neighbor sampling problem for image semantic segmentation labels as described in an embodiment of this application;

[0046] Figure 4 A flowchart illustrating the training method of the 3D object detection model described in an embodiment of this application is shown.

[0047] Figure 5 This document illustrates a flowchart of a method for dividing a 3D occupancy grid of a target 3D space into multiple downsampling regions, as described in an embodiment of this application.

[0048] Figure 6 A flowchart of the 3D target detection method according to an embodiment of this application is shown;

[0049] Figure 7 A schematic diagram of the structure of the training device for the 3D target detection model described in an embodiment of this application is shown;

[0050] Figure 8 A schematic diagram of the structure of the electronic device described in an embodiment of this application is shown. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0052] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0053] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0054] With the development of artificial intelligence technology, autonomous driving technology is becoming increasingly mature. Autonomous driving-related technologies can be broadly categorized into perception, prediction, localization, decision-making, and planning and control. Perception typically refers to using algorithms to analyze data from sensors (such as LiDAR, optical cameras, and ultrasonic radar) to create a real-time model of the surrounding environment. This modeled world is then input into downstream decision planning (PnC) for path planning (e.g., planning within decision planning). Among all perception tasks, 3D object detection is one of the most important. The recently popular 3D occupancy mesh is a more fine-grained 3D object detection method. Specifically, the 3D occupancy mesh discretizes the 3D space into voxels and then uses a model to output the category of each voxel, thereby more accurately segmenting the detected objects in the 3D space. From an algorithmic perspective, the 3D occupancy mesh is a 3D extension of 2D image semantic segmentation; therefore, it also utilizes image segmentation techniques. During the training of 3D object detection models, the resolution of 3D samples often differs from the resolution required by the model, and sometimes sample enhancement is necessary. Therefore, downsampling of 3D samples is frequently required. However, traditional methods use nearest neighbor sampling to process the corresponding labels when downsampling 3D samples. Since the resolution of 2D images is generally large (e.g., 1024×512), using nearest neighbor sampling to process image labels generally does not cause significant problems. However, the resolution of 3D grids is generally low (e.g., 200×200×30), and simply using nearest neighbor sampling will lead to significant ambiguity. In other words, the labels of the 3D grids of the downsampled 3D samples are ambiguous. This results in low accuracy of the trained 3D object detection model when using the sample 3D data, leading to low accuracy in detecting 3D objects.

[0055] The following example, using labels for two-dimensional images, illustrates the ambiguity of labels for 3D-occupied grids during the downsampling process.

[0056] Please refer to Figure 1 , Figure 1 The original RGB image described in the embodiments of this application is shown; please refer to Figure 2 , Figure 2 The labels for semantic segmentation corresponding to the original RGB image described in the embodiments of this application are shown.

[0057] Image semantic segmentation aims to determine the category of each pixel in an image. Figure 1This demonstrates the task: identifying the category of each pixel in a given image. During training, it's often necessary to downsample the labels (e.g., multi-scale image enhancement, relay supervision of feature maps). Common nearest-neighbor sampling label processing methods (and other schemes based on mode voting) assign the same category to pixels across all regions. Figure 2 The problem was visualized. In fact, assigning either 0 or 1 to the downsampled "hyperlabel" (where one position represents multiple original labels) is "reasonable," and this ambiguity introduces significant ambiguity into model training. These downsampled labels are one-hot encoded hard labels.

[0058] Specifically, please refer to Figure 3 , Figure 3 This illustration shows a schematic diagram of the nearest neighbor sampling problem for image semantic segmentation labels according to an embodiment of this application. Figure 3 As shown, nearest neighbor sampling is a commonly used downsampling method. Figure 3 The downsampling region shown is a 4×4 area of ​​16 pixels. Nearest neighbor sampling assigns the same value to all 16 pixels in the 4×4 area. Figure 3 For example, it can be assigned the value 1 or 0, which is unreasonable in principle.

[0059] However, because the resolution of two-dimensional images is generally large (e.g., 1024×512), using nearest neighbor sampling to process image labels generally does not cause particularly obvious accuracy problems; but the resolution of 3D grids is generally low (e.g., 200×200×30), and simply using nearest neighbor sampling will cause obvious ambiguity problems, affecting the accuracy of semantic segmentation.

[0060] Based on this, this application provides a training method for a 3D object detection model and a 3D object detection method. The training method acquires sample label information in a sample 3D space. The label information includes the category label of each 3D occupied grid in the sample 3D space, which is constructed based on sensor data from autonomous driving. Based on a preset downsampling rule, the 3D occupied grid of the target 3D space is divided into multiple downsampling regions. Each downsampling region includes multiple 3D occupied grids. The category labels of the multiple 3D occupied grids in each downsampling region are fused to determine the downsampling label of that downsampling region, and the sample label information of the sample 3D space is updated based on the downsampling label. Based on the updated sample 3D space, a loss function is constructed and a 3D object detection model is trained to obtain a trained 3D object detection model. In this way, during the downsampling process, the fused label replaces the one-hot encoded hard label, solving the ambiguity problem in the downsampling process. This ensures that the trained 3D object detection model is free from ambiguity and has higher accuracy, allowing for more accurate 3D object detection using the trained 3D object detection model.

[0061] Please refer to Figure 4 , Figure 4 This paper illustrates a flowchart of a method for training a 3D object detection model according to an embodiment of this application. Specifically, the training method includes the following steps S401-S404:

[0062] S401. Obtain sample label information in the sample 3D space; wherein, the label information includes the category label of each 3D grid in the sample 3D space, and the sample 3D space is constructed based on sensor data during the autonomous driving process;

[0063] S402. Based on a preset downsampling rule, the 3D occupancy grid of the target 3D space is divided into multiple downsampling regions; wherein each downsampling region includes multiple 3D occupancy grids;

[0064] S403. Merge the category labels of multiple 3D occupied grids in each downsampling region, determine the downsampling label of the downsampling region, and update the sample label information of the sample 3D space based on the downsampling label;

[0065] S404. Based on the updated sample label information in the sample 3D space, construct a loss function and train a 3D object detection model to obtain a trained 3D object detection model.

[0066] This application provides a training method for a 3D object detection model. The training method acquires sample label information in a sample 3D space. The label information includes the category label of each 3D occupied grid in the sample 3D space, which is constructed based on sensor data from autonomous driving. Based on a preset downsampling rule, the 3D occupied grid of the target 3D space is divided into multiple downsampling regions. Each downsampling region includes multiple 3D occupied grids. The category labels of the multiple 3D occupied grids in each downsampling region are fused to determine the downsampling label of that downsampling region, and the sample label information of the sample 3D space is updated based on the downsampling label. Based on the updated sample label information of the sample 3D space, a loss function is constructed and a 3D object detection model is trained to obtain a trained 3D object detection model. In this way, during the downsampling process, the fused label replaces the one-hot encoded hard label, solving the ambiguity problem in the downsampling process. This results in a trained 3D object detection model free from ambiguity and with higher accuracy, allowing for more accurate 3D object detection.

[0067] In step S401, sample label information of the sample 3D space is obtained; wherein, the label information includes the category label of each 3D occupied grid in the sample 3D space, and the sample 3D space is constructed based on sensor data during the autonomous driving process.

[0068] Here, the 3D space is set based on the sensor performance parameters of the autonomous vehicle. For example, the area with a horizontal dimension of 40m, a vertical dimension of 60m, and a height of 2m in front of the autonomous vehicle is set as the 3D space, with a size of 40m×60m×2m. The 3D occupancy grid of the 3D space is obtained by discretizing the 3D space. For example, the 40m×60m×2m 3D space is discretized into a 20cm×20cm×20cm 3D occupancy grid.

[0069] The 3D space can also be called three-dimensional space; the 3D occupancy grid can also be called three-dimensional occupancy grid.

[0070] It should be noted that different sample 3D spaces may be constructed based on sensor data from different types of vehicles. Different types of vehicles have different sensor performance and collect different ranges of data. Therefore, in some embodiments, the properties of different sample 3D spaces and the properties of their 3D occupancy grids will differ. Specifically, the sample set includes sample 3D spaces of various sizes, and each sample 3D space includes 3D occupancy grids of various sizes. For example, sample 3D spaces with dimensions of 40m × 60m × 2m, sample 3D spaces with dimensions of 30m × 60m × 2m, etc. The sample 3D spaces in the sample set are discretized into 20cm × 20cm × 20cm 3D occupancy grids, 30cm × 30cm × 30cm 3D occupancy grids, etc.

[0071] In this embodiment of the application, before obtaining the sample label information in the 3D space of the sample, the label information in the 3D space of the sample is determined by the following method:

[0072] Acquire various sensor data in the 3D space of a sample during autonomous driving; the sensor data includes: lidar data, optical camera data, and ultrasonic radar data;

[0073] By fusing the data from the multiple sensors, the 3D space of the sample is discretized into multiple 3D occupancy grids based on the fusion result of the multiple sensor data;

[0074] Label each 3D occupied grid with the corresponding category label to determine the label information of the sample's 3D space.

[0075] Here, the category label corresponding to each 3D occupied grid is used to determine the sample label information in the 3D space of the sample. This is existing technology and will not be described in detail here.

[0076] In step S402, based on a preset downsampling rule, the 3D occupancy grid of the target 3D space is divided into multiple downsampling regions; wherein each downsampling region includes multiple 3D occupancy grids.

[0077] For details, please refer to Figure 5 Based on a preset downsampling rule, the 3D occupancy grid of the target 3D space is divided into multiple downsampling regions, including:

[0078] S501. Based on the requirements of the 3D target detection model and the attribute information of the 3D occupied grid in the sample 3D space, determine the downsampling ratio;

[0079] S502. Based on the downsampling ratio, the 3D occupied grid of the sample 3D space is divided into multiple downsampling regions.

[0080] Based on the downsampling ratio, the 3D occupied grid of the sample 3D space is divided into multiple downsampling regions, including:

[0081] Based on the downsampling ratio, the sampling interval k in the 3D space of the sample is determined;

[0082] The adjacent k in the 3D space of the sample 3 Each 3D occupied grid is defined as the sampling area.

[0083] Here, the adjacent k 3 A 3D occupancy grid, that is, a 3D occupancy grid within a k×k×k region on a three-dimensional scale.

[0084] In layman's terms, based on the resolution required by the 3D object detection model and the resolution of the sample 3D space, the downsampling ratio is determined. For example, if the resolution of the sample 3D space is 200×200×200 and the resolution required by the 3D object detection model is 100×100×100, then the sampling interval k of the sample 3D space is 2, and the adjacent 2×2×2 3D occupied grids in the sample 3D space are determined as the downsampling region.

[0085] In step S403, the category labels of multiple 3D occupied grids of each downsampling region are fused to determine the downsampling label of the downsampling region, and the sample label information of the sample 3D space is updated based on the downsampling label.

[0086] In other words, before fusion, the downsampling region corresponds to multiple 3D occupied grids, and also to each category label, in order to Figure 3 The two-dimensional downsampling region shown illustrates the category label of the 3D downsampling region.

[0087] For details, please refer to Figure 3 , Figure 3 The two-dimensional downsampling region in the image is 4×4, with a total of 16 pixels, corresponding to 16 category labels. Among them, 6 category labels are 1 and 10 category labels are 0. Traditional methods select label 1 or label 0 as the downsampling label of the two-dimensional downsampling region, which leads to ambiguity.

[0088] In this application, the six category labels "1" and ten category labels "0" are merged to obtain the downsampling labels of the two-dimensional downsampling region.

[0089] Specifically, the category labels of multiple 3D occupied grids in each downsampling region are merged to determine the downsampling label for that downsampling region. This downsampling label is a probability label, representing the probability corresponding to each category label in the downsampling region. Please refer to... Figure 3In the probabilistic labeling scheme, the "downsampling label" has a probability of 0 in 10 / 16 and a probability of 1 in 6 / 16. That is, the probabilistic label obtained by the label fusion step under this application is (10 / 16, 6 / 16), rather than (0, 1) or (1, 0).

[0090] Similarly, the labels of 3D occupied meshes will have the same problem during downsampling. Currently, the mainstream method for k× downsampling directly applies labels to spatially adjacent k... 3 Setting the label of each 3D occupancy grid to a fixed value will lead to the same ambiguity problem in image semantic segmentation. Therefore, in processing the downsampling label of the 3D occupancy grid, this application specifically sets the adjacent k... 3 In a 3D occupied grid, there are c cells of category i. i Similarly, the final "downsampling label" is set to (c1 / k). 3 c1 / k 3 …c i / k 3 ).

[0091] The downsampling step described in this application involves sampling adjacent k... 3 When merging the labels of 3D occupancy grids, the final "downsampled label" is not actually composed entirely of the downsampled labels of a single 3D occupancy grid, but rather includes the downsampled labels of the merged k grids. 3 Information about the category labels of each 3D occupied grid.

[0092] Specifically, in this embodiment of the application, the category labels of multiple 3D occupied grids in each downsampling region are fused to determine the downsampling label of the downsampling region, including:

[0093] Determine the proportion of each category label in the downsampling region among all category labels;

[0094] Based on the proportion of each category label, the downsampling label of the downsampling region is determined. The downsampling label represents the probability that the 3D occupied grid after the downsampling region is merged belongs to each category label of the downsampling region.

[0095] In step S404, a loss function is constructed and a 3D object detection model is trained based on the updated sample label information in the sample 3D space to obtain a trained 3D object detection model.

[0096] Here, the trained 3D target detection model is used to process sensor data in the target 3D space during the autonomous driving process and detect 3D targets in the target 3D space.

[0097] Specifically, detecting 3D targets in the target 3D space involves semantic segmentation of those targets. More specifically, this involves selecting the region containing the 3D target. It should be noted that when semantic segmentation is not performed using a 3D grid method, the detected 3D target is typically a rectangle containing the target; whether it's a car or a triangular road sign, the final semantic segmentation result is always a rectangle. However, with a 3D grid, especially after training the 3D target detection model using the method described in this application, the shape of the 3D target can be accurately segmented, for example, the semantically segmented region can be a car shape, a triangle shape, etc.

[0098] Here, the 3D target detection model can also be called a 3D target recognition model, a 3D semantic segmentation model, a three-dimensional target detection model, a three-dimensional target recognition model, etc.

[0099] The loss function of the 3D object detection model is composed of the CrossEntropy function, specifically...

[0100]

[0101] Where, p∈R H×W×D×N , H, W, and D represent the dimensions of the three dimensions of the 3D occupied grid V, respectively; N represents the number of categories for each downsampled region, also known as the number of semantic categories; C represents the number of categories of voxels to be predicted, where c represents the value of the predicted voxel category. For example, if there are only five categories: sky, car, ground, tree branches, and others, then C is 5, and the value of c is [0, 1, 2, 3, 4]; p represents the probability distribution of the 3D occupied grid prediction. The true probability distribution characterizing the 3D occupied grid.

[0102] For one-hot encoding It is a one-hot vector (i.e., only one position is 1, and the rest are 0). Obviously, when Convergence occurs However, in the embodiments of this application It is not a one-hot vector, therefore the proof still holds under this condition.

[0103] because The calculation for each 3D occupancy grid is independent; therefore, this embodiment only considers an arbitrary 3D occupancy grid and sets p... c =p(h,w,d,c), Since p is the result of the model after soft-max, we have ∑ c pc =1 and p c ≥0, where p c The predicted value representing the probability that a 3D occupied grid at position (h,w,d) belongs to class c. The true value representing the probability that a 3D occupied grid at position (h,w,d) belongs to class c.

[0104] This leads to a conditional optimization problem, as follows:

[0105]

[0106] st,

[0107] ∑ c p c =1;

[0108] Using the Lagrange operator, we can have:

[0109]

[0110] in, λ represents the augmented Lagrange operator, and λ represents the Lagrange factor;

[0111] For p c Differentiate and set equal to 0, then

[0112] Considering ∑ c p c =1 and It's very easy to have

[0113] In this embodiment of the application, the updated sample label information in the sample 3D space is specifically a probability label. Based on the above discussion, a loss function is constructed and a 3D object detection model is trained based on the updated sample label information in the sample 3D space, which enables the 3D object detection model to be trained to convergence and obtain a trained 3D object detection model.

[0114] Based on the updated sample label information in the sample 3D space, a loss function is constructed and a 3D object detection model is trained. Specifically, based on the 3D occupancy grid information in the sample 3D space and the corresponding sample label information, a loss function is constructed and a 3D object detection model is trained. Here, the 3D occupancy grid information in the sample 3D space meets the resolution requirements of the 3D object detection model.

[0115] Based on the same inventive concept, this application also provides a 3D object detection method corresponding to the training method of the 3D object detection model. Since the principle of the 3D object detection method in this application is similar to the training method of the 3D object detection model in this application, the implementation of the 3D object detection method can refer to the implementation of the training method of the 3D object detection model, and the repeated parts will not be described again.

[0116] Please refer to Figure 6 , Figure 6 A flowchart of the 3D target detection method according to an embodiment of this application is shown. The detection method includes the following steps S601-S606:

[0117] S601. Obtain sample label information in the sample 3D space; wherein, the label information includes the category label of each 3D grid in the sample 3D space, and the sample 3D space is constructed based on sensor data during the autonomous driving process;

[0118] S602. Based on a preset downsampling rule, the 3D occupancy grid of the target 3D space is divided into multiple downsampling regions; wherein each downsampling region includes multiple 3D occupancy grids;

[0119] S603. Merge the category labels of multiple 3D occupied grids in each downsampling region, determine the downsampling label of the downsampling region, and update the sample label information of the sample 3D space based on the downsampling label;

[0120] S604. Based on the updated sample label information in the sample 3D space, construct a loss function and train a 3D object detection model to obtain a trained 3D object detection model.

[0121] S605. Acquire sensor data during vehicle operation in real time, and construct a target 3D space including multiple 3D occupancy grids based on the sensor data.

[0122] S606. Input the target 3D space into the trained 3D target detection model to detect the 3D target in the target 3D space.

[0123] This application provides a 3D target detection method. During training, the 3D target detection method first acquires sample label information in a sample 3D space. This label information includes the category label of each 3D occupied grid in the sample 3D space, which is constructed based on sensor data from autonomous driving. Based on a preset downsampling rule, the 3D occupied grid of the target 3D space is divided into multiple downsampling regions. Each downsampling region includes multiple 3D occupied grids. The category labels of the multiple 3D occupied grids in each downsampling region are fused to determine the downsampling region. The sample labels are downsampled, and the sample label information in the 3D space of the sample is updated based on the downsampled labels. Based on the updated sample label information in the 3D space of the sample, a loss function is constructed and a 3D object detection model is trained to obtain a trained 3D object detection model. Since the hard labels of one-hot encoding are replaced with fused labels during the downsampling process, the ambiguity problem in the downsampling process is solved, so that the trained 3D object detection model does not have the problem of ambiguity and has higher accuracy. Therefore, the trained 3D object detection model can be used to perform 3D object detection more accurately, thereby improving the detection accuracy of the 3D object detection method.

[0124] In step S606, the target 3D space is input into the trained 3D target detection model to detect 3D targets within the target 3D space, including:

[0125] Based on the requirements of the 3D target detection model, the 3D occupancy grid of the target 3D space is downsampled to obtain the downsampled target 3D space.

[0126] The downsampled target 3D space is input into the trained 3D target detection model to detect 3D targets within the target 3D space.

[0127] Here, based on the requirements of the 3D target detection model, the 3D occupancy grid of the target's 3D space is downsampled, including:

[0128] Based on the requirements of the 3D target detection model and the properties of the target's 3D space, the sampling interval k is determined;

[0129] The k neighboring elements in the target 3D space 3 The 3D occupancy grids are merged into one 3D occupancy grid.

[0130] In some embodiments, the 3D target detection method involves acquiring sensor data during vehicle movement in real time and constructing a target 3D space including multiple 3D occupancy grids based on the sensor data, including:

[0131] The system acquires multiple sensor data in the target 3D space in real time during autonomous driving; the sensor data includes: lidar data, optical camera data, and ultrasonic radar data.

[0132] By fusing the data from the various sensors, the target 3D space is discretized into multiple 3D occupancy grids based on the fusion result of the data from the various sensors.

[0133] In some embodiments, the 3D target detection method divides the 3D occupancy grid of the sample 3D space into multiple downsampling regions based on a preset downsampling rule, including:

[0134] Based on the requirements of the 3D target detection model and the attribute information of the 3D occupied grid in the sample 3D space, the downsampling sampling ratio is determined.

[0135] Based on the downsampling ratio, the 3D occupied grid of the sample 3D space is divided into multiple downsampling regions.

[0136] In some embodiments, the 3D target detection method divides the 3D occupied grid of the sample 3D space into multiple downsampling regions based on the downsampling sampling ratio, including:

[0137] Based on the downsampling ratio, the sampling interval k in the 3D space of the sample is determined;

[0138] The adjacent k in the 3D space of the sample 3 Each 3D occupied grid is defined as the sampling area.

[0139] In some embodiments, the 3D target detection method, fusing the category labels of multiple 3D occupied grids in each downsampled region to determine the downsampled label of the downsampled region, includes:

[0140] Determine the proportion of each category label in the downsampling region among all category labels;

[0141] Based on the proportion of each category label, the downsampling label of the downsampling region is determined, and the downsampling label guarantees the probability that the 3D occupied grid in the downsampling region belongs to each category label of the downsampling region.

[0142] Based on the same inventive concept, this application also provides a training device for a 3D target detection model corresponding to the training method of the 3D target detection model. Since the principle of the device in this application is similar to the training method of the 3D target detection model described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0143] For details, please refer to Figure 7 ,Figure 7 This application illustrates a training apparatus for a 3D object detection model according to an embodiment of the present application. The training apparatus includes:

[0144] The acquisition module 701 is used to acquire sample label information in the sample 3D space; wherein, the label information includes the category label of each 3D grid in the sample 3D space, and the sample 3D space is constructed based on sensor data during the autonomous driving process;

[0145] The partitioning module 702 is used to divide the 3D occupancy grid of the sample 3D space into multiple downsampling regions based on a preset downsampling rule; wherein each downsampling region includes multiple 3D occupancy grids;

[0146] The fusion module 703 is used to fuse the category labels of multiple 3D occupied grids in each downsampling region, determine the downsampling label of the downsampling region, and update the sample label information of the sample 3D space based on the downsampling label;

[0147] Training module 704 is used to construct a loss function and train a 3D object detection model based on the updated sample label information in the sample 3D space to obtain a trained 3D object detection model.

[0148] Based on the same inventive concept, this application also provides an electronic device corresponding to the training method of the 3D target detection model. Since the principle of solving the problem by the electronic device in this application is similar to the training method of the 3D target detection model described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0149] Please refer to Figure 8 , Figure 8 The diagram shows the structure of the electronic device according to an embodiment of this application. Specifically, the electronic device 800 includes a processor 802, a memory 801, and a bus. The memory 801 stores machine-readable instructions that can be executed by the processor 802. When the electronic device 800 is running, the processor 802 communicates with the memory 801 through the bus. When the machine-readable instructions are executed by the processor 802, the steps of the training method of the 3D target detection model are performed.

[0150] Based on the same inventive concept, this application also provides a computer-readable storage medium corresponding to the training method of the 3D target detection model. Since the principle of the computer-readable storage medium in this application is similar to the training method of the 3D target detection model in this application, the implementation of the computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be described again.

[0151] A computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the training method for the 3D object detection model.

[0152] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0153] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0155] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, platform server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0156] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a 3D object detection model, characterized in that, The training method includes: Obtain sample label information in the 3D space of the sample; wherein, the sample label information includes the category label of each 3D grid occupied in the 3D space of the sample, and the 3D space of the sample is constructed based on sensor data during the autonomous driving process; the sensor data includes: LiDAR data, optical camera data and ultrasonic radar data; Based on the downsampling ratio, determine the sampling interval k in the 3D space of the sample; then, in the 3D space of the sample, select adjacent k... 3 Each 3D occupied grid is defined as a downsampling region; Determine the proportion of each category label in the downsampling region among all category labels; based on the proportion of each category label, determine the downsampling label of the downsampling region; the downsampling label represents the probability that the merged 3D occupied grid of the downsampling region belongs to each category label of the downsampling region, and update the sample label information of the sample 3D space based on the downsampling label; the downsampling label is a probability label, and the probability label represents the probability corresponding to each category label in the downsampling region; Based on the updated sample label information in the sample 3D space, a loss function is constructed and a 3D object detection model is trained to detect 3D objects in the target 3D space.

2. The training method for the 3D object detection model according to claim 1, characterized in that, Based on a preset downsampling rule, the 3D occupancy grid of the sample 3D space is divided into multiple downsampling regions, including: Based on the requirements of the 3D target detection model and the attribute information of the 3D occupied grid in the sample 3D space, the downsampling sampling ratio is determined. Based on the downsampling ratio, the 3D occupied grid of the sample 3D space is divided into multiple downsampling regions.

3. The training method for the 3D object detection model according to claim 1, characterized in that, Before obtaining the sample label information in the 3D space of the sample, the sample label information in the 3D space of the sample is determined in the following way: Acquire multiple sensor data in the 3D space of the sample during autonomous driving; By fusing the data from the multiple sensors, the 3D space of the sample is discretized into multiple 3D occupancy grids based on the fusion result of the multiple sensor data; Label each 3D occupied grid with the corresponding category label to determine the sample label information in the 3D space of the sample.

4. A 3D target detection method, characterized in that, The detection method includes: Obtain sample label information in the 3D space of the sample; wherein, the sample label information includes the category label of each 3D grid occupied in the 3D space of the sample, and the 3D space of the sample is constructed based on sensor data during the autonomous driving process; the sensor data includes: LiDAR data, optical camera data and ultrasonic radar data; Based on the downsampling ratio, determine the sampling interval k in the 3D space of the sample; then, in the 3D space of the sample, select adjacent k... 3 Each 3D occupancy grid is defined as a downsampling region; the proportion of each category label in the downsampling region among all category labels is determined; based on the proportion of each category label, the downsampling label of the downsampling region is determined; the downsampling label represents the probability that the merged 3D occupancy grid belongs to each category label of the downsampling region, and the sample label information of the sample 3D space is updated based on the downsampling label; the downsampling label is a probability label, and the probability label represents the probability corresponding to each category label in the downsampling region; Based on the updated sample label information in the 3D space, a loss function is constructed and a 3D object detection model is trained to obtain a well-trained 3D object detection model. Real-time acquisition of sensor data during vehicle operation, and construction of a target 3D space including multiple 3D occupancy grids based on the sensor data; The target 3D space is input into the trained 3D target detection model to detect 3D targets within the target 3D space.

5. The 3D target detection method according to claim 4, characterized in that, Based on a preset downsampling rule, the 3D occupancy grid of the sample 3D space is divided into multiple downsampling regions, including: Based on the requirements of the 3D target detection model and the attribute information of the 3D occupied grid in the sample 3D space, the downsampling sampling ratio is determined. Based on the downsampling ratio, the 3D occupied grid of the sample 3D space is divided into multiple downsampling regions.

6. The 3D target detection method according to claim 4, characterized in that, Real-time acquisition of sensor data during vehicle operation, and construction of a target 3D space including multiple 3D occupancy grids based on the sensor data, including: Real-time acquisition of multiple sensor data in the target 3D space during autonomous driving; By fusing the data from the various sensors, the target 3D space is discretized into multiple 3D occupancy grids based on the fusion result of the data from the various sensors.

Images