Training method and device, target detection method and device, and electronic device

Abstract

This disclosure relates to training methods and apparatus, object detection methods and apparatus, electronic devices, and computer-storable media. The training method includes: acquiring training point cloud data and label data of multiple 3D objects; generating, using an object detection model, object detection boxes representing the predicted spatial location of each 3D object and object category information representing the predicted classification of each 3D object based on the training point cloud data; determining a weight value corresponding to each object detection box, the weight value being positively correlated with the spatial proportion of the observation point of the corresponding 3D object within the labeled box; determining a target loss value for multiple 3D objects using the object detection model based on multiple object detection boxes, multiple labeled boxes, object category information of multiple 3D objects, labeled category information, and weight values ​​corresponding to multiple object detection boxes; and training the object detection model based on the target loss value to obtain the trained object detection model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to training methods and apparatus, target detection methods and apparatus, electronic devices, and computer-storable media. Background Technology

[0002] Point cloud-based 3D object detection is one of the fundamental topics in the field of computer vision. It is a technique that uses the spatial distribution of point clouds obtained by LiDAR scanning to identify the category and spatial location of objects. Due to occlusion, distance from the sensor, and other reasons, much point cloud data cannot completely cover the entire object to be detected, resulting in unsatisfactory detection results for objects with sparse observation points in the point cloud data.

[0003] In related technologies, the method of adding points is used to increase the number of observation points in point cloud data in order to improve the outline of the object. Summary of the Invention

[0004] Related technologies, such as point supplementation to solve the sparsity problem of point clouds, are not stable enough and introduce a large amount of computation.

[0005] To address the aforementioned technical problems, this disclosure proposes a solution that can improve the stability of the trained target detection model, thereby improving the detection accuracy of the target detection model.

[0006] According to a first aspect of this disclosure, a method for training an object detection model is provided, comprising: acquiring training point cloud data and label data of multiple three-dimensional objects, wherein the training point cloud data includes observation points of each three-dimensional object, and the label data includes bounding boxes representing the actual spatial position of each three-dimensional object and label category information representing the actual classification of each three-dimensional object; generating object detection boxes representing the predicted spatial position of each three-dimensional object and target category information representing the predicted classification of each three-dimensional object using an object detection model based on the training point cloud data; determining a weight value corresponding to each object detection box based on the bounding boxes corresponding to each object detection box, wherein the weight value is positively correlated with the spatial proportion of the observation points of the corresponding three-dimensional object within the bounding box; determining a target loss value of the multiple three-dimensional objects using the object detection model based on the multiple object detection boxes of the multiple three-dimensional objects, the multiple bounding boxes corresponding to the multiple object detection boxes, the target category information of the multiple three-dimensional objects, the label category information, and the weight value corresponding to the multiple object detection boxes; and training the object detection model based on the target loss value to obtain a trained object detection model.

[0007] In some embodiments, determining the weight value corresponding to each target detection box includes: for each target detection box, determining a minimum bounding box including the observation point of the corresponding 3D object within the corresponding labeled box, wherein the minimum bounding box represents the actual spatial position of the observation point of the corresponding 3D object within the labeled box corresponding to each target detection box; for each target detection box, determining the overlap rate between the minimum bounding box and the labeled box based on the corresponding minimum bounding box and the labeled box; and determining the weight value corresponding to each target detection box based on the overlap rate corresponding to each target detection box.

[0008] In some embodiments, determining the weight value corresponding to each target detection box based on the overlap rate corresponding to each target detection box includes: determining the total number of target detection boxes with an overlap rate greater than or equal to an overlap rate threshold; determining the sum of multiple overlap rates corresponding to the multiple target detection boxes; and for each target detection box, if the corresponding overlap rate is greater than or equal to the overlap rate threshold, determining the weight value corresponding to each target detection box based on the overlap rate corresponding to each target detection box, the total number, and the sum.

[0009] In some embodiments, the weight value is positively correlated with both the corresponding overlap rate and the total quantity, and negatively correlated with the sum.

[0010] In some embodiments, determining the weight value corresponding to each target detection box based on the overlap rate corresponding to each target detection box, the total number, and the sum includes: determining the ratio between the overlap rate corresponding to each target detection box and the sum; and determining the weight value corresponding to each target detection box based on the ratio and the total number, wherein the weight value is positively correlated with both the ratio and the total number.

[0011] In some embodiments, determining the weight value corresponding to each target detection box based on the ratio and the total number includes: determining the weight value corresponding to each target detection box based on the product between the ratio and the total number.

[0012] In some embodiments, determining the weight value corresponding to each target detection box based on the overlap rate corresponding to each target detection box includes: for each target detection box, if the corresponding overlap rate is less than the overlap rate threshold, determining the weight value corresponding to each target detection box as a preset value.

[0013] In some embodiments, the overlap ratio includes the intersection-union ratio of the minimum bounding box and the label box.

[0014] In some embodiments, the object detection model includes a first sub-model and a second sub-model. Generating object detection boxes representing the predicted spatial location of each 3D object and object category information representing the predicted classification of each 3D object includes: generating initial detection boxes representing the predicted spatial location of each 3D object and initial category information representing the predicted classification of each 3D object using the first sub-model based on the training point cloud data; adjusting the initial detection boxes and initial category information of each 3D object using the second sub-model to obtain corresponding object detection boxes and object category information; determining the target loss value of the plurality of 3D objects includes: determining the target loss value using a loss function corresponding to the second sub-model based on the plurality of object detection boxes of the plurality of 3D objects, the plurality of labeled boxes corresponding to the plurality of object detection boxes, the object category information of the plurality of 3D objects, the labeled category information, and the weight values ​​corresponding to the plurality of object detection boxes; training the object detection model based on the target loss value to obtain the trained object detection model includes: training the second sub-model based on the target loss value to obtain the trained object detection model.

[0015] In some embodiments, the target loss value includes a first target loss value and a second target loss value, and the loss function includes a first loss function and a second loss function. Determining the target loss value of the plurality of three-dimensional objects includes: determining the first target loss value using the first loss function based on the plurality of target detection boxes of the plurality of three-dimensional objects, the plurality of labeled boxes corresponding to the plurality of target detection boxes, and the weight values ​​corresponding to the plurality of target detection boxes; and determining the second target loss value using the second loss function based on the target category information, labeled category information, and the weight values ​​corresponding to the plurality of target detection boxes of the plurality of three-dimensional objects.

[0016] In some embodiments, determining the first target loss value using the first loss function includes: determining a first loss value corresponding to the 3D object corresponding to each target detection box and its corresponding annotation box using the first loss function; weighting the first loss values ​​corresponding to the plurality of 3D objects using weight values ​​corresponding to the plurality of target detection boxes to obtain the first target loss value; determining the second target loss value using a second loss function corresponding to the second sub-model includes: determining a second loss value corresponding to each 3D object using the second loss function based on the target category information of each 3D object and its corresponding annotation category information; weighting the second loss values ​​corresponding to the plurality of 3D objects using weight values ​​corresponding to the plurality of target detection boxes to obtain the second target loss value.

[0017] In some embodiments, there are multiple second sub-models, which are connected in series. The input of the first second sub-model is the initial detection box and the initial category information. The output of each sub-model is the corresponding target detection box and target category information. The output of the last second sub-model is the output of the target detection model. Each second sub-model corresponds to a target loss value. Training the second sub-model includes: for each second sub-model, training each second sub-model according to the corresponding target loss value.

[0018] According to a second aspect of this disclosure, a target detection method is provided, comprising: acquiring point cloud data of an environment to be detected, the environment to be detected including one or more three-dimensional objects to be detected; and generating, based on the point cloud data of the environment to be detected, a target detection box representing the predicted spatial position of each three-dimensional object to be detected and target category information representing the predicted classification of each three-dimensional object to be detected, using a target detection model trained based on the training method described in any of the above embodiments.

[0019] In some embodiments, the target detection model includes a first sub-model and a second sub-model. Generating a target detection box representing the predicted spatial location of each 3D object to be detected and target category information representing the predicted classification of each 3D object to be detected includes: generating an initial detection box representing the predicted spatial location of each 3D object to be detected and initial category information representing the predicted classification of each 3D object to be detected using the first sub-model based on the point cloud data of the environment to be detected; and adjusting the initial detection box and the initial category information of each 3D object to be detected using the second sub-model to obtain the corresponding target detection box and target category information.

[0020] In some embodiments, there are multiple second sub-models, which are connected in series. The input of the first second sub-model is the initial detection box and the initial category information, and the output of the last second sub-model is the target detection box and target category information, which are the output of the target detection model.

[0021] According to a third aspect of this disclosure, a training apparatus for an object detection model is provided, comprising: an acquisition module configured to acquire training point cloud data and label data of a plurality of three-dimensional objects, wherein the training point cloud data includes observation points of each three-dimensional object, and the label data includes bounding boxes representing the actual spatial position of each three-dimensional object and label category information representing the actual classification of each three-dimensional object; a generation module configured to, based on the training point cloud data and using an object detection model, generate object detection boxes representing the predicted spatial position of each three-dimensional object and object category information representing the predicted classification of each three-dimensional object; and a first determination module configured to, based on the object detection data and the object detection model, generate object detection boxes representing the predicted spatial position of each three-dimensional object and object category information representing the predicted classification of each three-dimensional object; and a first determination module configured to, based on the object detection data and the object detection model, determine the object detection object's predicted spatial position of each three-dimensional object. The first module is configured to determine the weight value corresponding to each target detection box, wherein the weight value is positively correlated with the spatial proportion of the observation point of the corresponding 3D object within the label box; the second module is configured to determine the target loss value of the multiple 3D objects using the target detection model based on the multiple target detection boxes of the multiple 3D objects, the multiple label boxes corresponding to the multiple target detection boxes, the target category information of the multiple 3D objects, the label category information, and the weight value corresponding to the multiple target detection boxes; and the third module is configured to train the target detection model based on the target loss value to obtain the trained target detection model.

[0022] According to a fourth aspect of this disclosure, a target detection apparatus is provided, comprising: an acquisition module configured to acquire point cloud data of a target environment, the target environment including one or more target 3D objects; and a generation module configured to, based on the point cloud data of the target environment, generate target detection boxes representing the predicted spatial position of each target 3D object and target category information representing the predicted classification of each target 3D object using a target detection model trained based on the training method described in any of the above embodiments.

[0023] According to a fifth aspect of this disclosure, an electronic device is provided comprising: a memory; and a processor coupled to the memory, the processor being configured to execute a training method or a target detection method as described in any of the above embodiments based on instructions stored in the memory.

[0024] According to a sixth aspect of this disclosure, a computer-storeable medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the training method or target detection method described in any of the above embodiments.

[0025] In the above embodiments, the stability of the trained target detection model can be improved, thereby improving the detection accuracy of the target detection model. Attached Figure Description

[0026] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0027] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0028] Figure 1 This is a flowchart illustrating a training method according to some embodiments of the present disclosure;

[0029] Figure 2 This is a schematic diagram illustrating a training method according to some embodiments of the present disclosure;

[0030] Figure 3 This is a flowchart illustrating the determination of weight values ​​corresponding to each detection box according to some embodiments of the present disclosure;

[0031] Figure 4 This is a flowchart illustrating the determination of a weight value corresponding to each target detection box based on the overlap rate corresponding to each target detection box according to some embodiments of the present disclosure;

[0032] Figure 5 This is a flowchart illustrating a target detection method according to some embodiments of the present disclosure;

[0033] Figure 6 This is a block diagram illustrating a training apparatus according to some embodiments of the present disclosure;

[0034] Figure 7 This is a block diagram illustrating a target detection apparatus according to some embodiments of the present disclosure;

[0035] Figure 8 This is a block diagram illustrating an electronic device according to some embodiments of the present disclosure;

[0036] Figure 9 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure. Detailed Implementation

[0037] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0038] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0039] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0040] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0041] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0042] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0043] The following will combine Figure 1 , 2 The training methods in some embodiments of this disclosure are described in detail below.

[0044] Figure 1 This is a flowchart illustrating a training method according to some embodiments of the present disclosure.

[0045] Figure 2 This is a schematic diagram illustrating a training method according to some embodiments of the present disclosure.

[0046] like Figure 1 As shown, the training method for the object detection model includes steps S1-S5.

[0047] In step S1, training point cloud data and label data for multiple 3D objects are acquired. The training point cloud data includes the observation points of each 3D object. The label data includes bounding boxes representing the actual spatial location of each 3D object and label category information representing the actual classification of each 3D object.

[0048] In some embodiments, the annotation box is represented by the three-dimensional spatial coordinates of its center point, the length, width, and height of the annotation box, and its spatial rotation angle. For example, a certain annotation box (i.e., the actual position of each three-dimensional object) is represented as (x, y, z, l, w, h, r), where x, y, and z are three-dimensional spatial coordinate values, l, w, and h are the length, width, and height of the annotation box, respectively, and r is the spatial rotation angle of the annotation box in three-dimensional space.

[0049] In some embodiments, the category information includes the actual category to which each 3D object belongs and the confidence level of belonging to that actual category. The confidence level represents the probability that each 3D object belongs to the actual category.

[0050] In step S2, based on the training point cloud data, the target detection model is used to generate target detection boxes representing the predicted spatial location of each 3D object and target category information representing the predicted classification of each 3D object.

[0051] In some embodiments, the object detection model includes a first sub-model and a second sub-model. In this case, generating object detection boxes representing the predicted spatial location of each 3D object and object category information representing the predicted classification of each 3D object includes the following steps.

[0052] First, based on the training point cloud data, the first sub-model is used to generate initial detection boxes representing the predicted spatial location of each 3D object and initial category information representing the predicted classification of each 3D object.

[0053] In some embodiments, the first sub-model includes a point cloud feature extraction network, a bird's-eye view feature generation module, and a detection box generation network.

[0054] In some embodiments, training point cloud data with dimensions W (width), H (height), and D (depth) along the X, Y, and Z axes, respectively, are meshed, dividing the training point cloud data into uniformly sized three-dimensional voxels, resulting in... Figure 2 The gridded voxel data shown. (The following is a continuation of the previous sentence, likely due to an error in the Figure 2 The gridded voxel data shown is input into the point cloud feature extraction network to obtain the three-dimensional features of the training point cloud data.

[0055] In some embodiments, such as Figure 2 The point cloud feature extraction network shown stacks multiple 3D sparse convolutional layers. These layers progressively abstract voxel data to high-level 3D features, i.e., the 3D features of the training point cloud data. Each convolutional layer has a stride of 2 to reduce the resolution of the point cloud. Taking a three-layer cascaded 3D sparse convolutional layer as an example, the resolution of the point cloud features obtained by each convolutional layer is respectively a fraction of the resolution of the input gridded voxel data (i.e., the point cloud data). Finally, the point cloud feature extraction network performs feature fusion by adding a sparse convolutional layer with a sliding stride of 1 after the three-layer 3D sparse convolutional layers to obtain the 3D features of the entire training point cloud data.

[0056] In the above embodiments, with a resolution of Point cloud features are used as the basis for feature fusion, resulting in the fastest processing speed. Moreover, a resolution of 1 / 8 is sufficient to meet the required accuracy for generating initial detection boxes and initial category information. That is, this disclosure selects the fastest resolution while meeting the accuracy requirements. By connecting a sparse convolutional layer to the 1 / 8 resolution features, information from different channels can be fused, thereby transforming the height of the point cloud into a multi-channel structure.

[0057] In some embodiments, using, such as Figure 2The bird's-eye view generation module shown stitches together the 3D features of the training point cloud data along the Z-axis to obtain the bird's-eye view features of the training point cloud data. That is, features at the same X and Y positions are stitched together, so the resulting features are only related to the X and Y positions and are independent of the height Z. The bird's-eye view feature is a two-dimensional feature map without height information.

[0058] In some embodiments, such as Figure 2 The detection box generation network shown consists of a multi-layered two-dimensional convolutional neural network and two 1×1 convolutional layers connected after the two-dimensional convolutional neural network (i.e., Figure 2 (The regression and classification branches are shown). A multi-layer 2D convolutional neural network fuses features from the bird's-eye view along the X and Y axes. The last two 1×1 convolutional layers are connected after the 2D convolutional neural network to generate initial detection boxes (also called pre-selected boxes) representing the predicted spatial location of each 3D object and initial category information representing the predicted classification of each 3D object. The difference between the spatial location of each labeled sample and the preset anchor boxes is calculated as the target value for prediction. Labeled samples are manually labeled object location information and categories. This is because training the neural network requires manually labeled samples for the network to learn from the manually labeled data. Anchor boxes are fixed-size candidate boxes uniformly placed throughout the space.

[0059] Then, using the second sub-model, the initial detection box and initial category information of each 3D object are adjusted (optimized) to obtain the corresponding target detection box and target category information. In some embodiments, there are multiple second sub-models, which are connected in series. The input of the first second sub-model is the initial detection box and initial category information, the output of each sub-model is the corresponding target detection box and target category information, and the output of the last second sub-model is the output of the target detection model.

[0060] In some embodiments, the second sub-model is a detector. Multiple cascaded second sub-models constitute a cascaded detector. For example, a cascaded detector includes... Figure 2 The diagram shows the first-stage detector, the second-stage detector, and the third-stage detector. The first detector utilizes... Figure 2 The detection box generation network shown has generated initial detection boxes, and employs... Figure 2 The RoI (Region of Interest) pooling method shown extracts the bounding box features of each initial detection box. Figure 2 The detection frame grid points, annotation boxes, and voxel center points are also shown.

[0061] For example, N×N×N virtual points can be generated based on the initial detection box and evenly placed within it. Then, for each virtual point, its nearest neighbor is found in the 3D features of the training point cloud data. The features of all nearest neighbors are averaged and pooled to obtain the features of the virtual point. The features of all virtual points are then concatenated to obtain the features of the initial detection box. For example, assuming there are K virtual points, each with a feature dimension of C, the concatenated feature box will have a feature dimension of K×C.

[0062] The first detector also utilizes Figure 2 The multilayer perceptron shown performs high-level fusion processing on the bounding box features of each extracted initial detection box, and connects two linear layers (also called fully connected layers) on top of the multilayer perceptron to obtain the target detection box and target category information output by the detector. For example, the two linear layers are... Figure 2 The diagram shows the regression and classification branches (also called confidence branches) of the first-stage detector. For example, the output of the first detector is the box displacement, which is the adjustment information for the initial detection box. Other detectors process their inputs in a similar way. Figure 2 As shown, in addition to the detection frame grid points, annotation boxes, and voxel center points, the second-stage detector and the third-stage detector also show the target detection frame generated by the previous-stage detector.

[0063] In some embodiments, the cascaded detector comprises T (T is a positive integer) stages of detectors. Each stage of the detector includes a regression branch. and a confidence branch t∈{1,2,…,T}. We denote the boxes generated during the initial detection box generation stage as the 0th stage. And let the frame of the t-th stage be denoted as r represents the bounding box feature. Here, M represents the number of detected bounding boxes in all corresponding stages. In the t-th stage, RoI pooling is used to extract each bounding box. Features F 3D These are three-dimensional features obtained from a three-dimensional sparse convolutional network.

[0064] For each box feature Confidence branch and regression branch The predicted confidence scores and optimized bounding box positions are as follows: and The multi-stage detector iterates until the final stage, yielding the output of the object detection model: the bounding box and object category information output by the final stage detector. For example, the confidence prediction value in the object category information output by the final stage detector is the average confidence score across multiple stages, denoted as... The target detection box output by the last detector is the final output result of the target detection model.

[0065] In some embodiments, before adjusting the initial bounding boxes and initial category information of each 3D object using the second sub-model, non-maximum suppression can be used to remove the initial bounding boxes and their initial category information that have high repetition and low confidence.

[0066] In step S3, a weight value is determined for each target detection box based on the corresponding bounding box. The weight value is positively correlated with the spatial proportion of the observation point of the corresponding 3D object within the bounding box.

[0067] In some embodiments, it can be done as follows: Figure 3 The steps shown accomplish step S3 above.

[0068] Figure 3 This is a flowchart illustrating the determination of weight values ​​corresponding to each detection box according to some embodiments of the present disclosure.

[0069] like Figure 3 As shown, determining the weight value corresponding to each detection box includes steps S31-S33.

[0070] In step S31, for each target detection box, a minimum bounding box is determined that includes the observation point of the corresponding 3D object within the corresponding labeled box. The minimum bounding box represents the actual spatial position of the observation point of the corresponding 3D object within the labeled box corresponding to each target detection box.

[0071] In step S32, for each target detection box, the overlap rate between the minimum bounding box and the label box is determined based on the corresponding minimum bounding box and label box. For example... Figure 2 In the point cloud integrity module shown, for a given target detection box, the labeled box is B, and the minimum bounding box is A. Label box B marks the actual spatial location of a vehicle, and the minimum bounding box A represents the actual spatial location of the vehicle's observation point. Based on this point cloud integrity model, the overlap rate between the minimum bounding box and the labeled box corresponding to each target detection box can be determined.

[0072] In some embodiments, the overlap ratio includes the intersection-union ratio of the minimum bounding box to the label box. For example, this intersection-union ratio can be defined as a point integrity metric.

[0073] Generally, objects with a large number of observed points are easier to detect and can be considered high-quality samples. From this perspective, a direct way to measure sample sparsity is to count the number of observed points for each sample. However, there are also cases where the number of points and the completeness of the point cloud do not match. For example, some samples may have many observed locations, but these points are concentrated in a small area, thus constituting an incomplete point cloud. For object detection models, performing object detection based on incomplete point clouds is a significant challenge. Therefore, this disclosure proposes using the ratio of the space occupied by observed points to the entire object space as a point completeness index.

[0074] Suppose we are given a... Figure 2 The spatial position (i.e., the annotation frame itself) of the labeled 3D object "vehicle" is B = (x, y, z, l, w, h, r). This annotation frame contains N observation points P = {p1, p2, ..., pN}. (x, y, z) represent the spatial coordinates of the annotation frame, (l, w, h) are the magnitudes of the 3D object within the annotation frame along the X, Y, and Z axes, respectively, and r represents the spatial rotation angle of the 3D object within the annotation frame along the Z-axis. Definitions are as follows... Figure 2 shown If the smallest bounding box including all observation points P in the same direction as B is considered, then the point integrity index is the IoU (Intersection over Union) between A and B. The IoU is expressed as...

[0075] In step S33, a weight value corresponding to each target detection box is determined based on the overlap rate corresponding to each target detection box. In some embodiments, based on Figure 2 The overlap rate calculated from the point cloud integrity model can determine the weight value corresponding to each object detection box, and use it as... Figure 2 The input to the detector at each stage.

[0076] In some embodiments, it can be achieved through, as shown in Figure 4 The steps shown achieve step S33 above.

[0077] Figure 4 This is a flowchart illustrating the determination of a weight value corresponding to each target detection box based on the overlap rate corresponding to each target detection box according to some embodiments of the present disclosure.

[0078] like Figure 4 As shown, determining the weight value corresponding to each target detection box based on the overlap rate corresponding to each target detection box includes steps S331-S333.

[0079] In step S331, the total number of target detection boxes with an overlap rate greater than or equal to the overlap rate threshold is determined.

[0080] In step S332, the sum of multiple overlap rates corresponding to multiple target detection boxes is determined.

[0081] For each target detection box, if the corresponding overlap rate is greater than or equal to the overlap rate threshold, step S333 is executed. In step S333, a weight value corresponding to each target detection box is determined based on the overlap rate, the total number, and the sum. For example, the overlap rate threshold is 0.5.

[0082] In some embodiments, the weight values ​​are positively correlated with the corresponding overlap rate and the total number, and negatively correlated with the sum.

[0083] In some embodiments, the ratio between the overlap rate of each target detection box and the total number of overlaps is determined; based on the ratio and the total number of overlaps, a weight value is determined for each target detection box, wherein the weight value is positively correlated with both the ratio and the total number of overlaps. In some embodiments, determining the weight value for each target detection box based on the ratio and the total number of overlaps includes: determining the weight value for each target detection box based on the product of the ratio and the total number of overlaps.

[0084] For example, consider a scenario where there are multiple second sub-models connected in series, with each sub-model representing a stage. For the m-th object detection box in stage t... Its corresponding overlap rate Greater than or equal to the overlap rate threshold. This disclosure uses a linear normalization function (to... Linear normalization to Obtain the target detection box Corresponding weight value The overlap ratio is the total number of target detection boxes (number of positive samples) with an overlap ratio greater than or equal to the overlap ratio threshold. M is the total number of target detection boxes in stage t. The purpose of using a linear normalization function is to ensure that the sum of weights for all tasks remains consistent. This linear normalization function can be replaced with an exponential normalization function to obtain similar detection results.

[0085] In some embodiments, determining the weight value corresponding to each target detection box based on the overlap rate further includes: for each target detection box, if the corresponding overlap rate is less than an overlap rate threshold, determining the weight value corresponding to each target detection box as a preset value. In some embodiments, the preset value is 1. That is, the weight value remains unchanged.

[0086] For example, for all object detection boxes, the weight function used to calculate the weight values ​​is defined as follows: in, This is the set of target detection boxes with an overlap rate greater than or equal to the overlap rate threshold.

[0087] return Figure 1 In step S4, based on multiple target detection boxes of multiple three-dimensional objects, multiple annotation boxes corresponding to the multiple target detection boxes, target category information of multiple three-dimensional objects, annotation category information, and weight values ​​corresponding to the multiple target detection boxes, the target loss value of multiple three-dimensional objects is determined using the target detection model.

[0088] In some embodiments, taking the target detection model as an example including a first sub-model and a second sub-model, the target loss value of multiple three-dimensional objects can be determined through the following steps.

[0089] Based on multiple target detection boxes of multiple 3D objects, multiple labeled boxes corresponding to the multiple target detection boxes, target category information of multiple 3D objects, labeled category information, and weight values ​​corresponding to the multiple target detection boxes, the target loss value is determined using the loss function corresponding to the second sub-model.

[0090] In some embodiments, the target loss value includes a first target loss value and a second target loss value, and the loss function includes a first loss function and a second loss function. Determining the target loss value for multiple 3D objects includes the following steps.

[0091] First, based on multiple target detection boxes for multiple 3D objects, multiple corresponding annotation boxes, and weight values ​​for those boxes, a first target loss value is determined using a first loss function. This first loss function is the loss function corresponding to the target detection boxes. For example, if the second sub-model includes a regression branch, the first loss function is a Smooth-L1 based regression loss function.

[0092] In some embodiments, based on each target detection box and its corresponding annotation box, a first loss value corresponding to the 3D object corresponding to each target detection box is determined using a first loss function; the first loss values ​​corresponding to multiple 3D objects are weighted using weight values ​​corresponding to multiple target detection boxes to obtain a first target loss value. In some embodiments, for... Figure 2 The first loss value corresponding to a target detection box generated by the first-stage detector shown is denoted as: The corresponding weight value is w 1 After weighted processing, it becomes The representation of other stage detectors is similar and will not be repeated here.

[0093] Then, based on the target category information, labeled category information, and weight values ​​corresponding to the multiple target detection boxes for the multiple 3D objects, a second target loss value is determined using a second loss function. The second loss function is the loss function corresponding to the target category information. For example, the second sub-model also includes a classification branch, where the second loss function is the IoU cross-entropy loss function.

[0094] In some embodiments, based on the target category information and corresponding annotation category information of each 3D object, a second loss value corresponding to each 3D object is determined using the second loss function; the second loss values ​​corresponding to multiple 3D objects are weighted using weight values ​​corresponding to multiple target detection boxes to obtain a second target loss value. In some embodiments, for... Figure 2 The second loss value corresponding to a target detection box generated by the first-stage detector shown is denoted as: The corresponding weight value is w 1 After weighted processing, it becomes The representation of other stage detectors is similar and will not be repeated here.

[0095] Generally, during training, the loss values ​​corresponding to the first and second loss functions are simply summed together. However, this approach ignores the imbalance caused by the sparsity of point clouds between different objects. This inevitably leads to low-quality samples generating larger loss functions, masking the contribution of high-quality samples to the training process and causing instability. This disclosure obtains the weight value corresponding to each object detection box based on the integrity of the point cloud, and weights the loss value of each object detection box to reshape the overall training objective, reducing the contribution rate of low-quality samples and thus increasing the weight of high-quality samples during training. Since some low-quality samples are difficult for humans to distinguish, using them to train the neural network will introduce significant noise into the gradient, affecting the overall training results. Reducing the contribution rate of low-quality samples improves the accuracy of the object detection model.

[0096] In step S5, the target detection model is trained based on the target loss value to obtain the trained target detection model. Taking the existence of multiple second sub-models connected in series as an example, each second sub-model corresponds to a target loss value. In this case, training the second sub-model includes: for each second sub-model, training it based on the corresponding target loss value. Taking a multi-stage detector as an example, by calculating the loss value for the detection results of each stage and backpropagating the regression, richer gradient information can be provided, thereby improving the detection accuracy of the trained target detection model. Taking a multi-stage detector as an example, during training, the IoU between the target detection box and the labeled box is used as the prediction target for the confidence branch. The difference between the target detection box and the labeled box (dx, dy, dz, dl, dw, dh, dr) is used as the prediction target for the regression branch.

[0097] In some embodiments, taking the object detection model as an example, which includes a first sub-model and a second sub-model, the object detection model can be trained based on the object loss value through the following steps.

[0098] Based on the target loss value, a second sub-model is trained to obtain the trained object detection model. Since the process of adjusting the initial detection boxes and initial category information in the object detection model is an optimization process and a more crucial part, training the second sub-model using weight values ​​can improve training efficiency.

[0099] In some embodiments, taking the target loss value as an example including a first target loss value and a second target loss value, during the process of training the second sub-model using the target loss value, the first target loss value and the second target loss value are summed before the second sub-model is trained.

[0100] In the above embodiments, a weight value is determined based on the annotation box corresponding to the target detection box of each 3D object. This weight value is positively correlated with the spatial proportion of the observation point of the corresponding 3D object within the annotation box. Since this weight value is positively correlated with the spatial proportion of the observation point of the corresponding 3D object within the annotation box, the target loss value of multiple 3D objects determined by this weight value can reduce the contribution rate of low-quality point clouds in the training process and increase the weight of high-quality point clouds. This makes the training process of the target detection model more stable, thereby improving the detection performance of the target detection model, such as improving detection accuracy.

[0101] Figure 5 This is a flowchart illustrating a target detection method according to some embodiments of the present disclosure.

[0102] like Figure 5 As shown, the target detection method includes steps S51 to S52.

[0103] In step S51, point cloud data of the environment to be detected is acquired. The environment to be detected includes one or more 3D objects to be detected.

[0104] In step S52, based on the point cloud data of the environment to be detected, a target detection box representing the predicted spatial position of each 3D object to be detected and target category information representing the predicted classification of each 3D object to be detected are generated using the target detection model trained based on the training method described in the foregoing embodiments.

[0105] In some embodiments, the object detection model includes a first sub-model and a second sub-model. Based on the point cloud data of the environment to be detected, the first sub-model generates an initial detection box representing the predicted spatial location of each 3D object to be detected and initial category information representing the predicted classification of each 3D object to be detected. The second sub-model adjusts the initial detection box and initial category information of each 3D object to be detected to obtain the corresponding object detection box and object category information. Refer to the steps in the training process for details.

[0106] In some embodiments, there are multiple second sub-models, which are connected in series. The input of the first second sub-model is the initial detection box and initial category information, and the output of the last second sub-model is the target detection box and target category information, which is the output of the target detection model. Referring to the training process, each second sub-model will output the corresponding target detection box and target category information, but in the target detection process, only the target detection box and target category information output by the last second sub-model are output as the final target detection result.

[0107] For example, the training method and object detection method proposed in this disclosure can be applied to products such as vision-based sorting robotic arms in logistics scenarios. That is, by using point cloud data collected by a 3D camera mounted on the sorting robotic arm, each item can be accurately located and identified, thereby helping the robotic arm to sort items one by one.

[0108] Figure 6 This is a block diagram illustrating a training apparatus according to some embodiments of the present disclosure.

[0109] like Figure 6 As shown, the training device 6 for the target detection model includes an acquisition module 61, a generation module 62, a first determination module 63, a second determination module 64, and a training module 65.

[0110] The acquisition module 61 is configured to acquire training point cloud data and label data of multiple 3D objects, for example, by performing actions such as... Figure 1Step S1 is shown. The training point cloud data includes observation points for each 3D object, and the label data includes bounding boxes representing the actual spatial location of each 3D object and label category information representing the actual classification of each 3D object.

[0111] The generation module 62 is configured to, based on the training point cloud data and using an object detection model, generate object detection boxes representing the predicted spatial location of each 3D object and object category information representing the predicted classification of each 3D object, for example, by performing... Figure 1 Step S2 is shown.

[0112] The first determining module 63 is configured to determine a weight value corresponding to each target detection box based on the annotation box corresponding to each target detection box, for example, by performing the following: Figure 1 Step S3 is shown. The weight value is positively correlated with the spatial proportion of the observation point of the corresponding 3D object within the annotation box.

[0113] The second determining module 64 is configured to determine the target loss value of the plurality of three-dimensional objects using the target detection model based on the plurality of target detection boxes of the plurality of three-dimensional objects, the plurality of labeled boxes corresponding to the plurality of target detection boxes, the target category information of the plurality of three-dimensional objects, the labeled category information, and the weight values ​​corresponding to the plurality of target detection boxes, for example, by performing the following: Figure 1 Step S4 is shown.

[0114] Training module 65 is configured to train the object detection model based on the target loss value to obtain a trained object detection model, for example, by performing the following: Figure 1 Step S5 is shown.

[0115] Figure 7 This is a block diagram illustrating a target detection apparatus according to some embodiments of the present disclosure.

[0116] like Figure 7 As shown, the target detection device 7 includes an acquisition module 71 and a generation module 72.

[0117] The acquisition module 71 is configured to acquire point cloud data of the environment to be detected, for example, by performing the following: Figure 5 Step S51 is shown. The environment to be detected includes one or more three-dimensional objects to be detected.

[0118] The generation module 72 is configured to generate, based on the point cloud data of the environment to be detected and using the target detection model trained by the training method described in the foregoing embodiments, target detection boxes representing the predicted spatial location of each 3D object to be detected and target category information representing the predicted classification of each 3D object to be detected, for example, by performing the following... Figure 5 Step S52 is shown.

[0119] Figure 8 This is a block diagram illustrating an electronic device according to some embodiments of the present disclosure.

[0120] like Figure 8 As shown, the electronic device 8 includes a memory 81 and a processor 82 coupled to the memory 81. The memory 81 is used to store instructions for executing embodiments of training methods or object detection methods. The processor 82 is configured to execute training methods or object detection methods in any of the embodiments of this disclosure based on the instructions stored in the memory 81.

[0121] Figure 9 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0122] like Figure 9 As shown, the computer system 90 can be represented in the form of a general computing device. The computer system 90 includes a memory 910, a processor 920, and a bus 900 connecting different system components.

[0123] The memory 910 may include, for example, system memory, non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs. The system memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage media may store, for example, instructions for executing corresponding embodiments of at least one of the training method and the object detection method. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0124] The processor 920 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the decision module and the determination module, can be implemented by executing instructions in the central processing unit (CPU) memory to perform the corresponding steps, or by implementing dedicated circuitry to perform the corresponding steps.

[0125] Bus 900 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.

[0126] The computer system 90 may also include an input / output interface 930, a network interface 940, and a storage interface 950. These interfaces 930, 940, and 950, as well as the memory 910 and processor 920, can be connected via a bus 900. The input / output interface 930 provides a connection interface for input / output devices such as a monitor, mouse, and keyboard. The network interface 940 provides a connection interface for various networked devices. The storage interface 950 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0127] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.

[0128] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.

[0129] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.

[0130] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0131] The training method and apparatus, target detection method and apparatus, electronic device, and computer storage medium described in the above embodiments can improve the stability of the trained target detection model, thereby improving the detection accuracy of the target detection model.

[0132] The training methods and apparatus, target detection methods and apparatus, electronic devices, and computer-storable media according to this disclosure have been described in detail above. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

Claims

1. A method for training an object detection model, comprising: Acquire training point cloud data and label data of multiple three-dimensional objects. The training point cloud data includes observation points of each three-dimensional object, and the label data includes bounding boxes representing the actual spatial location of each three-dimensional object and label category information representing the actual classification of each three-dimensional object. Based on the training point cloud data, using the target detection model, target detection boxes representing the predicted spatial location of each 3D object and target category information representing the predicted classification of each 3D object are generated. Based on the annotation box corresponding to each target detection box, a weight value corresponding to each target detection box is determined, including: for each target detection box, determining a minimum bounding box including the observation point of the corresponding 3D object within the corresponding annotation box, wherein the minimum bounding box represents the actual spatial position of the observation point of the corresponding 3D object within the annotation box corresponding to each target detection box; for each target detection box, determining the overlap rate between the minimum bounding box and the annotation box based on the corresponding minimum bounding box and the annotation box; and determining a weight value corresponding to each target detection box based on the overlap rate, wherein the weight value is positively correlated with the spatial proportion of the observation point of the corresponding 3D object within the annotation box within the annotation box. Based on the multiple target detection boxes of the multiple three-dimensional objects, the multiple labeled boxes corresponding to the multiple target detection boxes, the target category information of the multiple three-dimensional objects, the labeled category information, and the weight values ​​corresponding to the multiple target detection boxes, the target loss value of the multiple three-dimensional objects is determined using the target detection model. Based on the target loss value, the target detection model is trained to obtain the trained target detection model.

2. The training method of claim 1, wherein, The weight value corresponding to each target detection box is determined based on the overlap rate corresponding to each target detection box, including: Determine the total number of target detection boxes whose overlap rate is greater than or equal to the overlap rate threshold; Determine the sum of multiple overlap rates corresponding to the multiple target detection boxes; For each target detection box, if the corresponding overlap rate is greater than or equal to the overlap rate threshold, the weight value corresponding to each target detection box is determined based on the overlap rate corresponding to each target detection box, the total number, and the sum.

3. The training method of claim 2, wherein, The weight value is positively correlated with the corresponding overlap rate and the total quantity, and negatively correlated with the sum.

4. The training method of claim 3, wherein, The weight value corresponding to each target detection box is determined based on the overlap rate corresponding to each target detection box, the total number, and the sum, including: Determine the ratio between the overlap rate of each target detection box and the sum; Based on the ratio and the total number, a weight value corresponding to each target detection box is determined, and the weight value is positively correlated with both the ratio and the total number.

5. The training method of claim 4, wherein, Based on the ratio and the total number, the weight value corresponding to each target detection box is determined as follows: The weight value corresponding to each target detection box is determined by multiplying the ratio by the total number.

6. The training method of claim 2, wherein, The weight value corresponding to each target detection box is determined based on the overlap rate corresponding to each target detection box, including: For each target detection box, if the corresponding overlap rate is less than the overlap rate threshold, the weight value corresponding to each target detection box is determined to be a preset value.

7. The training method according to any one of claims 1-6, wherein, The overlap ratio includes the intersection-union ratio of the minimum bounding box and the label box.

8. The training method according to claim 1, wherein, The target detection model includes a first sub-model and a second sub-model. The generated object detection bounding boxes characterizing the predicted spatial location of each 3D object and the object category information characterizing the predicted classification of each 3D object include: Based on the training point cloud data, the first sub-model is used to generate an initial detection box representing the predicted spatial location of each 3D object and initial category information representing the predicted classification of each 3D object. Using the second sub-model, the initial detection box and the initial category information of each 3D object are adjusted to obtain the corresponding target detection box and target category information; Determining the target loss value of the plurality of three-dimensional objects includes: Based on the multiple target detection boxes of the multiple three-dimensional objects, the multiple labeled boxes corresponding to the multiple target detection boxes, the target category information of the multiple three-dimensional objects, the labeled category information, and the weight values ​​corresponding to the multiple target detection boxes, the target loss value is determined using the loss function corresponding to the second sub-model; Based on the target loss value, the target detection model is trained to obtain the trained target detection model, which includes: Based on the target loss value, the second sub-model is trained to obtain the trained target detection model.

9. The training method according to claim 8, wherein, The target loss value includes a first target loss value and a second target loss value, and the loss function includes a first loss function and a second loss function. Determining the target loss value of the plurality of three-dimensional objects includes: Based on the multiple target detection boxes of the multiple three-dimensional objects, the multiple annotation boxes corresponding to the multiple target detection boxes, and the weight values ​​corresponding to the multiple target detection boxes, the first target loss value is determined using the first loss function; Based on the target category information, annotation category information, and weight values ​​corresponding to the multiple target detection boxes of the multiple three-dimensional objects, the second target loss value is determined using the second loss function.

10. The training method according to claim 9, wherein, Determining the first target loss value using the first loss function includes: Based on each target detection box and its corresponding annotation box, the first loss function is used to determine the first loss value corresponding to the three-dimensional object corresponding to each target detection box; Using the weight values ​​corresponding to the plurality of target detection boxes, the first loss values ​​corresponding to the plurality of three-dimensional objects are weighted to obtain the first target loss value; Using the second loss function corresponding to the second sub-model, the second target loss value is determined as follows: Based on the target category information and its corresponding labeled category information of each 3D object, the second loss value corresponding to each 3D object is determined using the second loss function; The second target loss value is obtained by weighting the second loss value corresponding to the multiple target detection boxes using the weight values ​​corresponding to the multiple target detection boxes.

11. The training method according to claim 8, wherein, There are multiple second sub-models, which are connected in series. The input of the first second sub-model is the initial detection box and the initial category information. The output of each second sub-model is the corresponding target detection box and target category information. The output of the last second sub-model is the output of the target detection model. Each second sub-model corresponds to a target loss value. Training the second sub-model includes: For each second sub-model, train each second sub-model according to the corresponding target loss value.

12. A target detection method, comprising: Acquire point cloud data of the environment to be detected, wherein the environment to be detected includes one or more three-dimensional objects to be detected; Based on the point cloud data of the environment to be detected, using the target detection model trained according to any one of the training methods described in claims 1-11, a target detection box representing the predicted spatial location of each 3D object to be detected and target category information representing the predicted classification of each 3D object to be detected are generated.

13. The target detection method according to claim 12, wherein, The target detection model includes a first sub-model and a second sub-model, generating target detection boxes representing the predicted spatial location of each 3D object to be detected and target category information representing the predicted classification of each 3D object to be detected, including: Based on the point cloud data of the environment to be detected, the first sub-model is used to generate an initial detection box representing the predicted spatial position of each 3D object to be detected and initial category information representing the predicted classification of each 3D object to be detected. Using the second sub-model, the initial detection box and the initial category information of each 3D object to be detected are adjusted to obtain the corresponding target detection box and target category information.

14. The target detection method according to claim 13, wherein, There are multiple second sub-models, which are connected in series. The input of the first second sub-model is the initial detection box and the initial category information, and the output of the last second sub-model is the target detection box and the target category information, which are the output of the target detection model.

15. A training device for an object detection model, comprising: The acquisition module is configured to acquire training point cloud data and label data of multiple three-dimensional objects. The training point cloud data includes observation points of each three-dimensional object, and the label data includes bounding boxes representing the actual spatial position of each three-dimensional object and label category information representing the actual classification of each three-dimensional object. The generation module is configured to generate, based on the training point cloud data and using the target detection model, a target detection box representing the predicted spatial location of each 3D object and target category information representing the predicted classification of each 3D object. The first determining module is configured to determine a weight value corresponding to each target detection box based on the annotation box corresponding to each target detection box, including: for each target detection box, determining a minimum bounding box including the observation point of the corresponding 3D object within the corresponding annotation box, wherein the minimum bounding box represents the actual spatial position of the observation point of the corresponding 3D object within the annotation box corresponding to each target detection box; for each target detection box, determining the overlap rate between the minimum bounding box and the annotation box based on the corresponding minimum bounding box and the annotation box; and determining a weight value corresponding to each target detection box based on the overlap rate, wherein the weight value is positively correlated with the spatial proportion of the observation point of the corresponding 3D object within the annotation box within the annotation box. The second determining module is configured to determine the target loss value of the multiple three-dimensional objects based on the multiple target detection boxes of the multiple three-dimensional objects, the multiple labeled boxes corresponding to the multiple target detection boxes, the target category information of the multiple three-dimensional objects, the labeled category information, and the weight values ​​corresponding to the multiple target detection boxes, using the target detection model. The training module is configured to train the target detection model based on the target loss value, thereby obtaining the trained target detection model.

16. A target detection device, comprising: The acquisition module is configured to acquire point cloud data of the environment to be detected, the environment to be detected including one or more 3D objects to be detected; The generation module is configured to generate, based on the point cloud data of the environment to be detected, a target detection box representing the predicted spatial location of each 3D object to be detected and target category information representing the predicted classification of each 3D object to be detected, using a target detection model trained according to any one of the training methods described in claims 1-11.

17. An electronic device comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the training method as described in any one of claims 1 to 11 or the object detection method as described in any one of claims 12 to 14, based on instructions stored in the memory.

18. A computer-storeable medium having stored thereon computer program instructions that, when executed by a processor, implement the training method as claimed in any one of claims 1 to 11 or the target detection method as claimed in any one of claims 12 to 14.

Images