Large vehicle target detection methods, devices, electronic equipment, and autonomous vehicles

Abstract

This disclosure relates to the field of autonomous driving technology, specifically to the field of computer vision technology, and particularly to methods, devices, electronic devices, and autonomous vehicles for large vehicle target detection. The specific implementation scheme is as follows: acquiring point cloud data of the target object; predicting a first target detection box and multiple visible corner points based on the point cloud data; adjusting the vertex positions of the first target detection box based on the positions of the visible corner points to obtain a second target detection box; and performing recognition based on the second target detection box to obtain the target detection result. By predicting the target detection box and visible corner points separately using point cloud data, and then adjusting the target detection box using the visible corner points, the target detection box becomes more accurate because corner points are located at the edges of objects and contain richer semantic information. This helps improve subsequent tasks such as vehicle speed prediction and vehicle tracking.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, specifically to the field of computer vision technology, and in particular to methods, devices, electronic devices, media, and autonomous vehicles for detecting large vehicle targets. Background Technology

[0002] During autonomous vehicle operation, it is crucial to accurately identify the position, size, orientation, and speed of surrounding vehicles to make accurate driving predictions, thereby improving driving safety and passenger experience. For example, an autonomous vehicle might detect a car to its left front and predict its position, size, and orientation to determine its intention to change lanes to the right. In this case, the autonomous vehicle can slow down in advance to avoid the risk of a rear-end collision. Existing 3D object detection technologies typically utilize features near the object's center to predict the coordinates, length, width, height, and angle of the object's center point to determine its position, size, and orientation. However, this method is not ideal for LiDAR-based vehicle detection, especially for large vehicles (typically those longer than 10 meters). This is because the LiDAR point cloud reflects off the vehicle's surface, causing semantic information to be concentrated at the vehicle's edges rather than its center. While this problem can be mitigated by increasing the detection receptive field, for large vehicles, the gain from increasing the detection receptive field is limited due to the excessive distance between the center and the ends, thus easily leading to inaccurate position, size, and orientation detection. Furthermore, due to the excessive length of large vehicles, the front and rear of the vehicle will amplify the orientation error, thereby affecting subsequent vehicle speed tracking and autonomous vehicle driving planning. Summary of the Invention

[0003] This disclosure provides a method, apparatus, training method for target detection model, training device electronic equipment, storage medium, and autonomous vehicle for large vehicle target detection.

[0004] According to a first aspect of this disclosure, a target detection method is provided, comprising:

[0005] Acquire point cloud data of the target object;

[0006] Based on the point cloud data, a first target detection box and multiple visible corner points are predicted respectively: semantic features of the point cloud data are extracted; corner features in the semantic features are extracted through the corner heatmap, and the positions of multiple visible corner points are predicted based on the corner features; target detection features are extracted based on the semantic features, and the first target detection box is constructed based on the target detection features;

[0007] The second target detection box is obtained by adjusting the vertex position of the first target detection box based on the position of the visible corner point;

[0008] The target detection result of the target object is obtained by recognizing the target based on the second target detection box.

[0009] According to a second aspect of this disclosure, a target detection apparatus is provided, comprising:

[0010] The acquisition module is configured to acquire point cloud data of the target object.

[0011] The prediction module is configured to predict the first target detection box and multiple visible corner points based on the point cloud data.

[0012] The prediction module includes:

[0013] The feature extraction unit is configured to extract semantic features from the point cloud data;

[0014] The corner prediction unit is configured to extract corner features from the semantic features through a corner heatmap, and predict the positions of multiple visible corners based on the corner features;

[0015] The target detection box prediction unit is configured to extract target detection features based on the semantic features and construct the first target detection box based on the target detection features;

[0016] The adjustment module is configured to adjust the vertex position of the first target detection box based on the position of the visible corner point to obtain the second target detection box;

[0017] The target detection module is configured to identify the target object based on the second target detection box to obtain the target detection result.

[0018] According to a third aspect of this disclosure, a method for training an object detection model is provided, comprising:

[0019] Acquire point cloud data of the target object;

[0020] Obtain multiple visible corner points as training labels;

[0021] The point cloud data is input into a neural network to extract semantic features, and based on the semantic features, a predicted target detection box and multiple predicted visible corner points are predicted.

[0022] Calculate the loss between the predicted visible corner point and the training label, and adjust the weights of the neural network based on the loss to obtain the target detection model.

[0023] According to a fourth aspect of this disclosure, a training apparatus for an object detection model is provided, comprising:

[0024] The point cloud acquisition module is configured to acquire point cloud data of the target object.

[0025] The training label generation module is configured to obtain multiple visible corner points as training labels;

[0026] The training module is configured to input the point cloud data into a neural network to extract semantic features, and to predict the target detection box and multiple predicted visible corner points based on the semantic features.

[0027] The calculation module is configured to calculate the loss between the predicted visible corner point and the training label, and adjust the weights of the neural network based on the loss to obtain the target detection model.

[0028] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in any one of the above technical solutions.

[0029] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method according to any one of the above-described technical solutions.

[0030] According to a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to any one of the above-described technical solutions.

[0031] According to the eighth aspect of this disclosure, an autonomous vehicle is provided, including the electronic devices described above.

[0032] This disclosure provides a method, apparatus, electronic device, medium, and autonomous vehicle for large vehicle target detection. It predicts target detection boxes and visible corner points using point cloud data, and then adjusts the target detection boxes using the visible corner points. Since corner points are located at the edges of objects and contain richer semantic information, the target detection boxes after corner point adjustment are more accurate, which helps to improve subsequent tasks such as vehicle speed prediction and vehicle tracking.

[0033] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0034] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0035] Figure 1 This is a schematic diagram of the steps of the large vehicle target detection method in the embodiments of this disclosure;

[0036] Figure 2 This is a framework diagram of the target detection model in the embodiments of this disclosure;

[0037] Figure 3 This is a schematic diagram illustrating the relationship between the point cloud coordinate system and the heatmap in an embodiment of this disclosure;

[0038] Figure 4 This is a schematic diagram of the corner-optimized target detection box in an embodiment of this disclosure;

[0039] Figure 5a This is a schematic diagram of the ground-truth heatmap in an embodiment of this disclosure;

[0040] Figure 5b This is a schematic diagram of the predicted heatmap obtained after training in an embodiment of this disclosure;

[0041] Figure 6 This is a schematic diagram illustrating the selection principle of visible corner points from the BEV perspective in this embodiment of the disclosure;

[0042] Figure 7 This is a schematic block diagram of the large vehicle target detection device in the embodiments of this disclosure;

[0043] Figure 8a This is a flowchart illustrating the training process of the target detection model in this embodiment of the disclosure;

[0044] Figure 8b This is a flowchart of the target detection model in this embodiment of the disclosure;

[0045] Figure 9 This is a schematic diagram of the training device for the target detection model in this embodiment of the present disclosure;

[0046] Figure 10 This is a schematic block diagram of an example electronic device in an embodiment of this disclosure. Detailed Implementation

[0047] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0048] To address the problem of inaccurate detection of large vehicles using existing methods, this disclosure provides a method for detecting large vehicle targets, such as... Figure 1 As shown, it includes:

[0049] Step S101: Acquire point cloud data of the target object. The point cloud data can be obtained through radar acquisition. For example... Figure 2 The target detection flowchart shown illustrates the input of point cloud data collected by radar into a convolutional neural network.

[0050] Step S102: Based on the point cloud data, a first target detection box and multiple visible corner points are predicted. The first target detection box is a 3D target detection box. Visible corner points can be object corner points from a BEV (bird's eye of view) perspective. Corner points can be defined from two different angles: a corner point is the intersection of two edges; a corner point is a feature point in its neighborhood with two principal directions. The region where the corner point is located is usually a stable and information-rich region in the image. These regions may have certain properties, such as rotation invariance, scale invariance, affine invariance, and illumination invariance. Visible corner points refer to the minimum distance l from the vertices of the 3D target detection box. i Corner points are identified only if their distance is less than a distance threshold θ; otherwise, they are considered invisible. First, semantic features of the point cloud data are extracted using a shared convolutional neural network. Then, two branches of the convolutional neural network predict the 3D bounding box of the object and the visible corner points of the object from the BEV perspective, respectively. The branch predicting the 3D bounding box uses a common 3D object detection structure, while the branch predicting the visible corner points uses... Figure 3 The structure of the heatmap shown.

[0051] Step S103: Adjust the vertex positions of the first target detection box based on the positions of the visible corner points to obtain the second target detection box. By using the predicted corner point positions to fine-tune the vertex coordinates of the 3D target detection box, and given the rigidity of the 3D target detection box, adjusting its coordinates also fine-tunes the target box's orientation.

[0052] Step S104: Based on the second target detection box, the target detection result is obtained by identification of the target object. For example... Figure 4 As shown, the first target detection box (solid line box) composed of the original vertices A, B, C, and D is adjusted by the predicted visible corner points E and F to obtain the second target detection box (dashed line box) with E and F as vertices. The side of the target detection box facing the main vehicle O is fine-tuned. Since the corner points are located at the edges of the object and contain richer semantic information, the target detection box after corner point adjustment is more accurate, which helps to improve subsequent vehicle speed prediction, vehicle tracking, and other tasks.

[0053] As an optional implementation, predicting the first target detection box and visible corner points based on point cloud data includes: extracting semantic features from the point cloud data; extracting corner features from the semantic features using a corner heatmap; predicting the positions of multiple visible corner points based on the corner features; extracting target detection features based on the semantic features; and constructing the first target detection box based on the target detection features. The point cloud information is processed by a shared convolutional neural network to extract point cloud features, which are then passed through two smaller branches to predict the 3D target box of the object and the visible corner points of the object from the BEV perspective. The branch related to corner point prediction is actually a heatmap, which is used to determine whether each position is a visible corner point and to specify the position of the visible corner point. Figure 3 This demonstrates the relationship between point clouds and heatmaps from a BEV (Browser Active Vehicle) perspective. The heatmap is a two-dimensional matrix with the following shape: Where H and W are the length and width of the visible point cloud, respectively, and n is the scaling factor from the visible point cloud size to the heatmap size. At a certain location on the heatmap... The element is represented as a confidence level, which indicates the position (W) on the origin cloud. i H i ) is the probability of a visible corner point. Figure 5(a) shows a ground-truth heatmap generated based on the visible corner point labels, where the response at the corner point is the highest, and the response in the corner's neighborhood decreases Gaussian. Figure 5(b) shows the predicted heatmap obtained after training. Ideally, the response near the corner point is the highest, and the response at other locations approaches zero. During training, is set as the prediction confidence at the location on the heatmap, and is set as the label value at the corresponding location on the ground-truth heatmap. The designed loss function is:

[0054]

[0055] Where N is the number of objects in the input image, and α and β are two weight parameters.

[0056] As an optional implementation, before extracting corner features from semantic features through corner heatmap and predicting information of multiple visible corners based on corner features, the method further includes: predicting a set of compensation vectors for each corner to compensate for the information lost when each corner is mapped from the point cloud coordinate system to the corner heatmap.

[0057] In this embodiment, to more accurately predict corner positions, a set of compensation vectors (offsets) needs to be predicted for each corner. Offsets represent the information lost when the corner coordinates are mapped from the point cloud coordinate system to the heatmap. When the corner coordinates are mapped back from the heatmap to the original point cloud coordinate system, the offsets can compensate for this lost information. First, corner prediction is performed on the heatmap. The visible area of ​​the point cloud coordinate system is scaled proportionally to the size of the heatmap. Information loss occurs when the input image is mapped to the heatmap. For example, when the visible area of ​​the point cloud coordinate system is scaled four times the size of the heatmap, the position (102, 102) in the original point cloud coordinate system maps to the following coordinates on the heatmap: Without compensation, these coordinates, when mapped back to the input image, become (25×4, 25×4) = (100, 100). Therefore, the network needs to predict offsets to compensate for the lost information. For example, if offset = (Δx, Δy), then the coordinates mapped back to the input image become [(25+Δx)×4, (25+Δy)×4]. During training, the network is trained to predict offsets using Smooth L1 loss:

[0058]

[0059] in and represents the x and y coordinates of the k-th ground-truth corner point, respectively, and n represents the scaling factor from the visible area of ​​the point cloud coordinate system to the heatmap size, i.e., the step size. This is the predicted offset value.

[0060] As an optional implementation, corner features are extracted from semantic features through corner heatmaps, and the positions of multiple visible corners are predicted based on the corner features. The method also includes: predicting a vector pointing from each visible corner to the center of the target object based on the corner features, and using the vector to determine whether the visible corners belong to the same object.

[0061] In this embodiment, to avoid ambiguity between two corner points that are close to each other but belong to different objects, a vector pointing to the center of their respective object is also predicted at each corner point. Smooth L1 loss is used to constrain this during training.

[0062]

[0063] in ce x and ce yThese represent the coordinates of the center point pointed to by the k-th ground-truth corner point.

[0064] As an optional implementation, step S103, adjusting the vertex positions of the first target detection box based on the positions of visible corner points to obtain the second target detection box, includes: obtaining the positions of multiple vertices of the first target detection box facing the main vehicle. For each vertex, traversing all visible corner points, selecting two visible corner points that meet preset conditions, and adjusting the positions of the vertices corresponding to the first target detection box to obtain the second target detection box. The first target detection box is a 3D target detection box, such as... Figure 4 As shown, for each first object detection box, its vertices a, b, and c facing the main vehicle O are obtained. After adjusting the predicted visible corner points e and f, a second object detection box (dashed box) with e and f as vertices is obtained. The side of the object detection box facing the main vehicle O is fine-tuned. Since the corner points are located at the edges of the object and contain richer semantic information, the object detection box after corner point adjustment is more accurate, which helps to improve subsequent vehicle speed prediction, vehicle tracking, and other tasks. Since the object detection box is a rigid body, adjusting the coordinates of the object detection box also serves to fine-tune the orientation of the object detection box. Because the prediction of the corner points is more accurate, the object detection box after corner point fine-tuning is also more accurate, especially for large vehicles, the effect is more obvious.

[0065] Furthermore, the predicted visible corner points are based on the following preset conditions: (1) the visible corner point has a minimum distance from its corresponding vertex, and the minimum distance is less than a preset distance threshold; (2) the prediction confidence of the visible corner point is greater than a set confidence threshold; and (3) the visible corner point points to the center of its object. The branch related to the bounding box regression predicts a set of 3D object detection boxes, while the branch related to corner point detection predicts a set of corner point positions. First, the coordinates of the 3D object detection boxes under the BEV view are obtained. For each object detection box, three vertices facing the main vehicle are obtained. For each vertex v i (i = 0, 1, 3), iterate through all predicted corner points. If all three conditions are met, the selected corner point is retained; otherwise, it is deleted. If three corner points that meet the conditions can be selected from the object's three vertices, select the two corner points closest to the main vehicle and align the two vertices corresponding to the first target detection box with these two corner points to achieve fine-tuning of the predicted target box. If two corner points that meet the conditions can be selected from the object's three vertices, use these two corner points to fine-tune the first target detection box. If fewer than two corner points that meet the conditions are selected from the object's three vertices, the predicted first target detection box is not fine-tuned. Figure 8bThe diagram shown is a flowchart of the object detection process in this embodiment. Point cloud data is input into a pre-trained 3D object detection model. The 3D object detection model includes two branches: a heatmap and a 3D object detection structure. Visible corner points are predicted using the heatmap, and a 3D object detection bounding box (the first object detection box) is predicted using the 3D object detection structure. The 3D object detection bounding box is then optimized using the visible corner points. During corner point optimization, the height of the 3D object detection bounding box can be removed, converting it into a 2D object detection bounding box. The vertices of the 2D object detection bounding box are then... Figure 4 The adjustments shown, followed by adding height to the adjusted 2D object detection box to convert it into a 3D object detection box, can reduce the computational load during corner optimization.

[0066] The above technical solution involves fine-tuning the target detection bounding box by selecting visible corner points that meet preset conditions. Since corner points are located at the edges of objects and contain richer semantic information, and heatmap-based methods can predict corner point positions more accurately than regression-based methods, the face of the object facing the main vehicle after corner point optimization will be more accurate. This will help improve subsequent tasks such as vehicle speed prediction and vehicle tracking. Furthermore, since the predicted target bounding box is a rigid body, it can also be used for fine-tuning angles.

[0067] This disclosure also provides a target detection device 700, such as Figure 7 As shown, it includes:

[0068] The acquisition module 701 is configured to acquire point cloud data of the target object. The point cloud data can be obtained through radar acquisition. For example... Figure 2 The target detection flowchart shown illustrates the input of point cloud data collected by radar into a convolutional neural network.

[0069] The prediction module 702 is configured to predict a first target detection box and multiple visible corner points based on point cloud data. The first target detection box is a 3D target detection box, and the visible corner points can be object corner points from a BEV (bird's eye of view) perspective. Corner points can be defined from two different angles: a corner point is the intersection of two edges; a corner point is a feature point in its neighborhood with two principal directions. The region where the corner point is located is usually a stable and information-rich region in the image. These regions may have certain properties, such as rotation invariance, scale invariance, affine invariance, and illumination invariance. Here, a visible corner point refers to the minimum distance l from the vertex of the 3D target detection box. iCorner points are identified only if their distance is less than a distance threshold θ; otherwise, they are considered invisible. First, semantic features of the point cloud data are extracted using a shared convolutional neural network. Then, two branches of the convolutional neural network predict the 3D bounding box of the object and the visible corner points of the object from the BEV perspective, respectively. The branch predicting the 3D bounding box uses a common 3D object detection structure, while the branch predicting the visible corner points uses... Figure 3 The structure of the heatmap shown.

[0070] The adjustment module 703 is configured to adjust the vertex positions of the first target detection box based on the positions of visible corner points to obtain the second target detection box. By using the predicted corner point positions to fine-tune the vertex coordinates of the 3D target detection box, and since the 3D target detection box is a rigid body, adjusting the coordinates of the 3D target detection box also serves to fine-tune the orientation of the target box.

[0071] The target detection module 704 is configured to identify the target object based on the second target detection box to obtain the target detection result. For example... Figure 4 As shown, the first target detection box (solid line box) composed of the original vertices A, B, C, and D is adjusted by the predicted visible corner points E and F to obtain the second target detection box (dashed line box) with E and F as vertices. The side of the target detection box facing the main vehicle O is fine-tuned. Since the corner points are located at the edges of the object and contain richer semantic information, the target detection box after corner point adjustment is more accurate, which helps to improve subsequent vehicle speed prediction, vehicle tracking, and other tasks.

[0072] As an optional implementation, the prediction module 702 includes: a feature extraction unit configured to extract semantic features from point cloud data; a corner prediction unit configured to extract corner features from the semantic features using a corner heatmap, and predict the positions of multiple visible corners based on the corner features; and a target detection box prediction unit configured to extract target detection features based on the semantic features, and construct a first target detection box based on the target detection features. Point cloud information is processed by a shared convolutional neural network to extract point cloud features, and then two smaller branches are used to predict the 3D target box of the object and the visible corners of the object from the BEV perspective, respectively. The branch related to corner prediction is actually a heatmap, which is used to determine whether each position is a visible corner and to specify the position of the visible corner. Figure 3 This demonstrates the relationship between point clouds and heatmaps from a BEV (Browser Active Vehicle) perspective. The heatmap is a two-dimensional matrix with the following shape: Where H and W are the length and width of the visible point cloud, respectively, and n is the scaling factor from the visible point cloud size to the heatmap size. At a certain location on the heatmap... The element is represented as a confidence level, which indicates the position (W) on the origin cloud.i H i ) is the probability of a visible corner point. Figure 5(a) shows a ground-truth heatmap generated based on the visible corner point labels, where the response at the corner point is the highest, and the response in the corner's neighborhood decreases Gaussian. Figure 5(b) shows the predicted heatmap obtained after training. Ideally, the response near the corner point is the highest, and the response at other locations approaches zero. During training, is set as the prediction confidence at the location on the heatmap, and is set as the label value at the corresponding location on the ground-truth heatmap. The designed loss function is:

[0073]

[0074] Where N is the number of objects in the input image, and α and β are two weight parameters.

[0075] As an optional implementation, before the corner prediction unit extracts corner features from the semantic features through the corner heatmap and predicts information of multiple visible corners based on the corner features, it also includes: predicting a set of compensation vectors for each corner to compensate for the information lost when each corner is mapped from the point cloud coordinate system to the corner heatmap.

[0076] In this embodiment, to more accurately predict corner positions, a set of compensation vectors (offsets) needs to be predicted for each corner. Offsets represent the information lost when the corner coordinates are mapped from the point cloud coordinate system to the heatmap. When the corner coordinates are mapped back from the heatmap to the original point cloud coordinate system, the offsets can compensate for this lost information. First, corner prediction is performed on the heatmap. The visible area of ​​the point cloud coordinate system is scaled proportionally to the size of the heatmap. Information loss occurs when the input image is mapped to the heatmap. For example, when the visible area of ​​the point cloud coordinate system is scaled four times the size of the heatmap, the position (102, 102) in the original point cloud coordinate system maps to the following coordinates on the heatmap: Without compensation, these coordinates, when mapped back to the input image, become (25×4, 25×4) = (100, 100). Therefore, the network needs to predict offsets to compensate for the lost information. For example, if offset = (Δx, Δy), then the coordinates mapped back to the input image become [(25+Δx)×4, (25+Δy)×4]. During training, the network is trained to predict offsets using Smooth L1 loss:

[0077]

[0078] in and represents the x and y coordinates of the k-th ground-truth corner point, respectively, and n represents the scaling factor from the visible area of ​​the point cloud coordinate system to the heatmap size, i.e., the step size. This is the predicted offset value.

[0079] As an optional implementation, the corner prediction unit extracts corner features from semantic features through a corner heatmap, predicts the positions of multiple visible corners based on the corner features, and further includes: predicting a vector pointing from each visible corner to the center of the target detection object based on the corner features, the vector being used to determine whether the visible corners belong to the same object.

[0080] In this embodiment, to avoid ambiguity between two corner points that are close to each other but belong to different objects, a vector pointing to the center of their respective object is also predicted at each corner point. Smooth L1 loss is used to constrain this during training.

[0081]

[0082] in ce x and ce y These represent the coordinates of the center point pointed to by the k-th ground-truth corner point.

[0083] As an optional implementation, the adjustment module 703 adjusts the vertex positions of the first target detection box based on the positions of visible corner points to obtain the second target detection box. This includes: acquiring the positions of multiple vertices of the first target detection box facing the main vehicle; traversing all visible corner points for each vertex; selecting two visible corner points that meet preset conditions; adjusting the positions of the vertices corresponding to the first target detection box; and obtaining the second target detection box. The first target detection box is a 3D target detection box, such as... Figure 4 As shown, for each first object detection box, its vertices a, b, and c facing the main vehicle O are obtained. After adjusting the predicted visible corner points e and f, a second object detection box (dashed box) with e and f as vertices is obtained. The side of the object detection box facing the main vehicle O is fine-tuned. Since the corner points are located at the edges of the object and contain richer semantic information, the object detection box after corner point adjustment is more accurate, which helps to improve subsequent vehicle speed prediction, vehicle tracking, and other tasks. Since the object detection box is a rigid body, adjusting the coordinates of the object detection box also serves to fine-tune the orientation of the object detection box. Because the prediction of the corner points is more accurate, the object detection box after corner point fine-tuning is also more accurate, especially for large vehicles, the effect is more obvious.

[0084] Furthermore, the predicted visible corner points are based on the following preset conditions: (1) the visible corner point has a minimum distance from its corresponding vertex, and the minimum distance is less than a preset distance threshold; (2) the prediction confidence of the visible corner point is greater than a set confidence threshold; and (3) the visible corner point points to the center of its object. The branch related to the bounding box regression predicts a set of 3D object detection boxes, while the branch related to corner point detection predicts a set of corner point positions. First, the coordinates of the 3D object detection boxes under the BEV view are obtained. For each object detection box, three vertices facing the main vehicle are obtained. For each vertex v i (i = 0, 1, 3), iterate through all predicted corner points. If all three conditions are met, the selected corner point is retained; otherwise, it is deleted. If three corner points that meet the conditions can be selected from the object's three vertices, select the two corner points closest to the main vehicle and align the two vertices corresponding to the first target detection box with these two corner points to achieve fine-tuning of the predicted target box. If two corner points that meet the conditions can be selected from the object's three vertices, use these two corner points to fine-tune the first target detection box. If fewer than two corner points that meet the conditions are selected from the object's three vertices, the predicted first target detection box is not fine-tuned.

[0085] The above technical solution involves fine-tuning the target detection bounding box by selecting visible corner points that meet preset conditions. Since corner points are located at the edges of objects and contain richer semantic information, and heatmap-based methods can predict corner point positions more accurately than regression-based methods, the face of the object facing the main vehicle after corner point optimization will be more accurate. This will help improve subsequent tasks such as vehicle speed prediction and vehicle tracking. Furthermore, since the predicted target bounding box is a rigid body, it can also be used for fine-tuning angles.

[0086] This disclosure provides a training method for an object detection model, such as Figure 8a As shown, it includes:

[0087] Obtain point cloud data of the target object. During training, point cloud data is needed as training samples.

[0088] Multiple visible corner points are acquired as training labels. Visible corner points that meet the minimum distance condition (i.e., distance to a vertex is less than a distance threshold) are selected as training labels. To predict the position of visible corner points, training labels for these corner points must first be created. Given the 3D bounding box of an object, the coordinates of the four corner points of the 2D bounding box (i.e., the coordinates of the four vertices of the rectangle, A, B, C, and D) in the BEV view (top view) are first obtained. Simultaneously, the point cloud coordinates are also mapped to the BEV view, as shown below. Figure 6 As shown. Then, a distance threshold θ is set for each corner point c. i =(xi ,y i Find the nearest point cloud coordinates, with a minimum distance of l. i If l i If the angle is less than or equal to θ, it is a visible corner point (A and B are visible angles); otherwise, it is an invisible corner point (C and D are invisible angles). Visible corner points are found using this method and used as training labels to train a target detection model that can predict visible corner points.

[0089] Point cloud data is input into a neural network to extract semantic features, and based on the semantic features, the predicted target detection box and multiple predicted visible corner points are predicted.

[0090] The loss between predicted visible corner points and training labels is calculated, and the weights of the neural network are adjusted based on the loss to obtain the object detection model. The model training is supervised. The visible corner point labels can be calculated from the labels of the target boxes. The labels are not used in training; they are calculated based on the labels and the output results after the model outputs the results. The model weights are then adjusted based on the calculated loss.

[0091] Specifically, the training process in this embodiment includes: 1) Data processing: The input is 3D point cloud data. Manual features of the point cloud under the BEV perspective are generated through the geometric relationship between the point clouds and the reflection intensity of the point clouds. 2) Feature extraction: Semantic information of the point cloud is extracted through a shared convolutional layer. This feature contains semantic information for 3D object detection and visible corner prediction. 3) 3D object detection module branch and visible corner prediction module: Each module has an independent convolutional layer to extract semantic information useful to its own module, and finally predicts the 3D object box and visible corner. 4) Loss function: Each module has a corresponding loss function for optimization. The 3D object detection module optimizes the predicted category of the object box through a cross-entropy loss classification function and predicts the coordinates of the object box through a Smooth L1 regression loss. For the visible corner prediction module, the prediction of the corner is jointly optimized by formulas (1), (2) and (3).

[0092] This disclosure provides a training apparatus for an object detection model, such as... Figure 9 As shown, it includes:

[0093] The point cloud acquisition module 901 is configured to acquire point cloud data of the target object. During training, point cloud data is required as training samples.

[0094] The training label generation module 902 is configured to acquire multiple visible corner points as training labels. Visible corner points that meet the minimum distance condition (i.e., distance to a vertex less than a distance threshold) are selected as training labels. To predict the position of visible corner points, training labels for these corner points must first be created. Given the 3D bounding box annotation of an object, the coordinates of the four corner points (i.e., the coordinates of the four vertices of the rectangle) of the 2D bounding box in the BEV view (i.e., top view) are first obtained. Simultaneously, the point cloud coordinates are also mapped to the BEV view, such as... Figure 6 As shown. Then, a distance threshold θ is set for each corner point c. i =(x i ,y i Find the nearest point cloud coordinates, with a minimum distance of l. i If l i If the angle is less than or equal to θ, it is a visible corner point; otherwise, it is an invisible corner point. Visible corner points are identified using this method and used as training labels to train a target detection model that can predict visible corner points.

[0095] The training module 903 is configured to input point cloud data into a neural network to extract semantic features, and then predict the target detection box and multiple predicted visible corner points based on the semantic features.

[0096] The computation module 904 is configured to calculate the loss between predicted visible corner points and training labels, and adjust the weights of the neural network based on the loss to obtain the object detection model. The model training is supervised; the visible corner point labels can be calculated from the labels of the target boxes. The labels are not used in training; they are calculated based on the labels and the output results after the model outputs the results, and the model weights are adjusted based on the calculated loss.

[0097] Specifically, the training process in this embodiment includes: 1) Data processing: The input is 3D point cloud data. Manual features of the point cloud under the BEV perspective are generated through the geometric relationship between the point clouds and the reflection intensity of the point clouds. 2) Feature extraction: Semantic information of the point cloud is extracted through a shared convolutional layer. This feature contains semantic information for 3D object detection and visible corner prediction. 3) 3D object detection module branch and visible corner prediction module: Each module has an independent convolutional layer to extract semantic information useful to its own module, and finally predicts the 3D object box and visible corner. 4) Loss function: Each module has a corresponding loss function for optimization. The 3D object detection module optimizes the predicted category of the object box through a cross-entropy loss classification function and predicts the coordinates of the object box through a Smooth L1 regression loss. For the visible corner prediction module, the prediction of the corner is jointly optimized by formulas (1), (2) and (3).

[0098] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0099] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0100] Figure 10 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0101] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded into random access memory (RAM) 1003 from storage unit 1008. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0102] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] The computing unit 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as object detection methods. For example, in some embodiments, the object detection method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the object detection method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform object detection methods by any other suitable means (e.g., by means of firmware).

[0104] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0107] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0108] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0109] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0110] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0111] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A large vehicle target detection method, comprising: obtaining point cloud data of a target detection object; predicting a first target detection box and a plurality of visible corner points based on the point cloud data, respectively: extracting semantic features of the point cloud data; extracting corner point features in the semantic features through a corner point heat map, and predicting positions of the plurality of visible corner points based on the corner point features; extracting target detection features based on the semantic features, and constructing the first target detection box based on the target detection features; adjusting vertex positions of the first target detection box based on the positions of the visible corner points to obtain a second target detection box; performing identification based on the second target detection box to obtain a target detection result of the target detection object; wherein the adjusting of the vertex positions of the first target detection box based on the positions of the visible corner points to obtain the second target detection box comprises: obtaining positions of a plurality of vertices of the first target detection box facing the host vehicle; for each of the vertices, traversing all the visible corner points to screen positions of two visible corner points meeting a preset condition to adjust positions of corresponding vertices of the first target detection box to obtain the second target detection box; the preset condition comprises: the visible corner points have a minimum distance with the corresponding vertices, and the minimum distance is less than a preset distance threshold; a prediction confidence of the visible corner points is greater than a set confidence threshold; the visible corner points point to a center of the object. 2.The method of claim 1, before the extracting of the corner point features in the semantic features through the corner point heat map and the predicting of the positions of the plurality of visible corner points based on the corner point features, further comprising: predicting a set of compensation vectors for each corner point to compensate for lost information of each of the corner points when mapping from a point cloud coordinate system to the corner point heat map.

3. The method of claim 1 or 2, wherein, the extracting of the corner point features in the semantic features through the corner point heat map and the predicting of the positions of the plurality of visible corner points based on the corner point features further comprises: predicting, based on the corner point features, a vector of each of the visible corner points pointing to a center of the target detection object, the vector being used to determine whether the visible corner point belongs to the same object. 4.A large vehicle target detection apparatus, comprising: an obtaining module configured to obtain point cloud data of a target detection object; a predicting module configured to predict a first target detection box and a plurality of visible corner points based on the point cloud data, respectively; the predicting module comprises: a feature extraction unit configured to extract semantic features of the point cloud data; a corner point prediction unit configured to extract corner point features in the semantic features through a corner point heat map, and predict positions of the plurality of visible corner points based on the corner point features; a target detection box prediction unit configured to extract target detection features based on the semantic features, and construct the first target detection box based on the target detection features; an adjusting module configured to adjust vertex positions of the first target detection box based on the positions of the visible corner points to obtain a second target detection box; a target detection module configured to perform identification based on the second target detection box to obtain a target detection result of the target detection object; The adjusting module adjusts the vertex position of the first target detection frame based on the position of the visible corner point to obtain a second target detection frame, which comprises: Obtaining the position of the vertex of the first target detection frame facing the host vehicle; For each of the vertices, all the visible corner points are traversed to screen out the positions of two visible corner points meeting the preset condition to adjust the position of the corresponding vertex of the first target detection frame to obtain the second target detection frame; Wherein, the preset condition comprises: The visible corner point and the corresponding vertex have a minimum distance, and the minimum distance is less than a preset distance threshold; The prediction confidence of the visible corner point is greater than a set confidence threshold; The visible corner point points to the center of the object.

5. The apparatus of claim 4, wherein, Before the corner point prediction unit extracts the corner point feature in the semantic feature based on the corner point feature to predict the information of the plurality of visible corner points, it further comprises: A set of compensation vectors is predicted for each corner point to compensate for the information lost when each of the corner points is mapped from the point cloud coordinate system to the corner point heat map.

6. The apparatus of claim 4 or 5, wherein, Before the corner point prediction unit extracts the corner point feature in the semantic feature based on the corner point feature to predict the position of the plurality of visible corner points, it further comprises: Based on the corner point feature, a vector pointing to the center of the target detection object is predicted for each of the visible corner points, and the vector is used to determine whether the visible corner point belongs to the same object.

7. A target detection model training method, comprising: obtaining point cloud data of a target detection object; obtaining a plurality of visible corner points as training labels, the visible corner points meeting a minimum distance condition, the minimum distance condition being that the distance from the vertex is less than a distance threshold; inputting the point cloud data into a neural network for feature extraction to obtain semantic features, and respectively predicting a predicted target detection frame and a plurality of predicted visible corner points based on the semantic features; calculating the loss between the predicted visible corner points and the training labels, and adjusting the weights of the neural network based on the loss to obtain a target detection model.

8. A target detection model training device, comprising: a point cloud acquisition module configured to obtain point cloud data of a target detection object; a training label generation module configured to obtain a plurality of visible corner points as training labels, the visible corner points meeting a minimum distance condition, the minimum distance condition being that the distance from the vertex is less than a distance threshold; a training module configured to input the point cloud data into a neural network for feature extraction to obtain semantic features, and respectively predict a predicted target detection frame and a plurality of predicted visible corner points based on the semantic features; a calculation module configured to calculate the loss between the predicted visible corner points and the training labels, and adjust the weights of the neural network based on the loss to obtain a target detection model.

9. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3.

10. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are for causing the computer to perform the method according to any one of claims 1-3.

11. A computer program product comprising computer program which, when executed by a processor, implements the method according to any one of claims 1-3.

12. An autonomous vehicle comprising the electronic device of claim 9.

Images