Three-dimensional target detection method and device, automatic driving equipment and storage medium

Abstract

Embodiments of the present application provide a three-dimensional target detection method and device, an automatic driving device and a computer storage medium, wherein the three-dimensional target detection method comprises: acquiring a three-dimensional point cloud of an environment in which an automatic driving device is located; performing three-dimensional voxel feature extraction based on the three-dimensional point cloud to obtain corresponding three-dimensional voxel features; projecting the three-dimensional voxel features into a two-dimensional pseudo image, and performing sparse convolution on the two-dimensional pseudo image to obtain corresponding two-dimensional image features; performing heat peak value calculation according to the two-dimensional image features, and performing three-dimensional target detection according to a calculation result and the two-dimensional image features. Through the embodiments of the present application, the real-time and accuracy requirements of the three-dimensional target detection scene can be effectively balanced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a three-dimensional target detection method, apparatus, autonomous driving equipment, and computer storage medium. Background Technology

[0002] Autonomous driving technology is a technology that relies on computers and artificial intelligence to control corresponding equipment to drive autonomously; it is also known as driverless technology.

[0003] Environmental perception is a crucial component of autonomous driving technology. It primarily involves using various types of sensors to collect data on the surrounding environment of autonomous driving devices (such as vehicles, robots, and aircraft), generating quantitative descriptions, and outputting them in different data formats. The results directly impact the quality and effectiveness of backend processes such as localization and control. Among the various sensors used for data collection, LiDAR (Light Detection and Ranging) is widely used in environmental perception scenarios due to its high detection accuracy and ability to accurately perceive the 3D information of the environment surrounding autonomous driving devices. The aim is to obtain relatively accurate target detection results by using point cloud data collected by LiDAR and 3D target detection models, thereby achieving accurate environmental perception.

[0004] However, due to the large amount of point cloud data acquired by LiDAR, 3D target detection based on this data and using a 3D target detection model places high demands on the computing power of autonomous driving equipment. This is especially true in the context of in-vehicle embedded platforms, where the computing power is limited but the real-time requirements for 3D target detection are high. This makes it extremely difficult to achieve 3D target detection that combines real-time performance with accuracy in such scenarios. Summary of the Invention

[0005] In view of this, embodiments of this application provide a three-dimensional target detection scheme to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of the embodiments of this application, a three-dimensional target detection method is provided, comprising: acquiring a three-dimensional point cloud of the environment in which an autonomous driving device is located; extracting three-dimensional voxel features based on the three-dimensional point cloud to obtain corresponding three-dimensional voxel features; projecting the three-dimensional voxel features into a two-dimensional pseudo-image and performing sparse convolution on the two-dimensional pseudo-image to obtain corresponding two-dimensional image features; calculating thermal peaks based on the two-dimensional image features; and performing three-dimensional target detection based on the calculation results and the two-dimensional image features.

[0007] According to a second aspect of the embodiments of this application, a three-dimensional target detection device is provided, comprising: a first acquisition module, configured to acquire a three-dimensional point cloud of the environment in which an autonomous driving device is located; a second acquisition module, configured to extract three-dimensional voxel features based on the three-dimensional point cloud to acquire corresponding three-dimensional voxel features; a third acquisition module, configured to project the three-dimensional voxel features into a two-dimensional pseudo-image and perform sparse convolution on the two-dimensional pseudo-image to obtain corresponding two-dimensional image features; and a detection module, configured to perform thermal peak calculation based on the two-dimensional image features, and perform three-dimensional target detection based on the calculation result and the two-dimensional image features.

[0008] According to a third aspect of the embodiments of this application, an autonomous driving device is provided, including: a three-dimensional point cloud acquisition device and a processor; wherein: the three-dimensional point cloud acquisition device is used to acquire a three-dimensional point cloud of the environment in which the autonomous driving device is located; the processor is used to perform three-dimensional target detection as described in the first aspect based on the three-dimensional point cloud.

[0009] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0010] According to the 3D object detection scheme provided in the embodiments of this application, on the one hand, for the 2D pseudo-image projected from 3D voxel features, the traditional ordinary 2D convolution method is no longer used; instead, a sparse convolution method is adopted to better fit the actual point cloud description of the 3D object and obtain more accurate 2D image features. On the other hand, the traditional anchor-based detection head is replaced with a detection head based on heatmap peaks without anchors to avoid wasting computing power due to inaccurate or expanded anchor candidate ranges. Furthermore, based on the heatmap peak form, the bounding boxes of each category of 3D objects are regressed by combining 2D image features, which not only saves hardware computing power and improves the real-time performance of the calculation, but also greatly improves the accuracy of bounding box detection, ensuring the accuracy of the detection results. When the scheme of the embodiments of this application is applied to 3D object detection scenarios, especially in 3D object detection scenarios of automotive embedded platforms, the saving of computing power and the improvement of detection accuracy effectively balance the real-time and accuracy requirements of 3D object detection scenarios. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0012] Figure 1A This is a schematic diagram of the detection process of a pillar-based 3D target detection model in related technologies;

[0013] Figure 1B This is a schematic diagram of a pillar-based three-dimensional target detection model in an embodiment of this application;

[0014] Figure 2A This is a flowchart illustrating the steps of a three-dimensional target detection method according to Embodiment 1 of this application;

[0015] Figure 2B for Figure 2A A schematic diagram of a detection process in the illustrated embodiment;

[0016] Figure 3A This is a flowchart illustrating the steps of a three-dimensional target detection method according to Embodiment 2 of this application;

[0017] Figure 3B for Figure 3A A schematic diagram of a training process in the illustrated embodiment;

[0018] Figure 4 This is a structural block diagram of a three-dimensional target detection device according to Embodiment 3 of this application;

[0019] Figure 5 This is a schematic diagram of the structure of an autonomous driving device according to Embodiment 4 of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0021] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0022] To facilitate understanding of the solutions in the embodiments of this application, the detection process of a pillar-based 3D target detection model in the related art will first be illustrated by way of example, such as... Figure 1A As shown.

[0023] Depend on Figure 1AAs can be seen, the three-dimensional target detection model includes a voxelization part, a 3D backbone part mainly formed by a voxel feature encoder, a 2D backbone part mainly including a pillar scatter part and a 2D encoder, and a detection head part mainly implemented by a PRN (Region Proposal Network) head.

[0024] Based on this 3D object detection model, the 3D point cloud data (hereinafter referred to as 3D point cloud) is first voxelized by the voxelization part of the 3D object detection model to obtain voxel partitions; then, shallow 3D voxel feature extraction is performed by the voxel feature encoder to obtain the corresponding 3D voxel features; then, according to the preset voxel grid size, the extracted 3D voxel features are projected into a 2D pseudo-image (also called a 2D pseudo-image) through the pillar dispersion part; then, the 2D encoder is used to extract features from the 2D pseudo-image to obtain the corresponding 2D image features; finally, the PRN detection head is used to perform 3D object detection, to perform bounding box regression and object classification for one or more 3D object objects, and obtain the detection results. Figure 1A In this diagram, the detection result is simply illustrated as a bounding box. However, those skilled in the art should understand that the result obtained through 3D target detection includes not only the bounding box but also the category, size, orientation, and speed of the object defined by the bounding box. Of course, those skilled in the art can add more detection result information according to actual needs.

[0025] In the above process, the voxelization part mainly voxels the 3D point cloud data (x, y, z, r). Then, the 3D voxel feature extraction also extracts features from this voxelized data to obtain feature data corresponding to (x, y, z, r). Here, (x, y, z) represents the 3D coordinates of the point cloud, and r represents the reflectivity of the point cloud. The 2D encoder is typically formed by ordinary 2D convolutional layers to extract features from 2D pseudo-images. Furthermore, the PRN detection head is usually implemented based on an anchor-based approach, using anchor boxes to define 3D target objects and achieve the detection of these objects.

[0026] However, in the above process, to more accurately describe the information of the 3D target object, the voxelization part of this application embodiment has been improved. In addition to voxelizing the 3D point cloud data (x,y,z,r), distance features and rotation angle features of the 3D point cloud in cylindrical coordinates are added, so that the extracted 3D voxel features carry richer and more comprehensive feature information. Furthermore, the 2D encoder part has been improved, using a sparse convolution encoder instead of a traditional encoder using ordinary convolution, to better fit the actual 3D object described by the point cloud, obtain more accurate 2D image features, and reduce computational load. In addition, for the detection head part, an anchor-free heatmap peak form is used instead of the RPN detection head to save computing power, improve the real-time performance of the calculation, and enhance detection accuracy.

[0027] Based on the above improvements, a pillar-based 3D target detection model in one embodiment of this application is as follows: Figure 1B As shown.

[0028] Depend on Figure 1B As can be seen, the three-dimensional target detection model includes: a three-dimensional voxel feature extraction part, a two-dimensional image feature extraction part, and a detection head part.

[0029] The 3D voxel feature extraction section is used to extract 3D voxel features based on the 3D point cloud, obtaining the corresponding 3D voxel features. For example, the 3D voxel feature extraction section may include a voxelization section and a 3D voxel feature encoder section. The voxelization section is used to voxelize the 3D point cloud and calculate the distance of the 3D point cloud from the horizontal plane in the cylindrical coordinate system and the rotation angle of the 3D point cloud around the vertical axis in the cylindrical coordinate system. This section is as follows: Figure 1B The improved voxelization section shown is used to extract three-dimensional voxel features from the three-dimensional point cloud based on the voxelization result, the distance, and the rotation angle, thereby obtaining the corresponding three-dimensional voxel features. This section is as follows: Figure 1B The 3D backbone shown is formed by the improved Voxel Feature Encoder.

[0030] The 2D image feature extraction part projects the 3D voxel features into a 2D pseudo-image and performs sparse convolution on the pseudo-image to obtain the corresponding 2D image features, which are sparse features. This 2D image feature extraction part includes a projection part and a 2D encoder part. The projection part projects the 3D voxel features into a 2D pseudo-image; the 2D encoder part performs sparse convolution on the pseudo-image to obtain the corresponding 2D image features. This part is as follows... Figure 1BThe diagram shows a 2D backbone consisting mainly of a PillarScatter section and an improved 2D encoder employing sparse convolutions.

[0031] The detection head section is used to calculate thermal peaks based on two-dimensional image features, and then perform three-dimensional target detection based on the calculation results and the two-dimensional image features. For example, this detection head section can be implemented as a center thermal map detection head. This section is as follows... Figure 1B The head shown is a portion of the detection head implemented by the Center Heatmap Head.

[0032] The following describes the implementation of the above-described embodiments through multiple examples. Figure 1B The process of 3D target detection using the 3D target detection model shown is explained.

[0033] Example 1

[0034] Reference Figure 2A The diagram shows a flowchart of a three-dimensional target detection method according to Embodiment 1 of this application.

[0035] The three-dimensional target detection method in this embodiment includes the following steps:

[0036] Step S102: Obtain the 3D point cloud of the environment where the autonomous driving device is located.

[0037] In this embodiment, the autonomous driving device can be a device with autonomous driving function, such as an autonomous vehicle, robot, or aircraft. When these autonomous driving devices are in a real physical environment, the 3D point cloud acquisition device installed in the device, such as a lidar, will acquire the 3D point cloud in that physical environment. Typically, the acquired 3D point cloud has attribute data of (x, y, z, r).

[0038] Step S104: Extract three-dimensional voxel features based on the three-dimensional point cloud to obtain the corresponding three-dimensional voxel features.

[0039] A voxel is a combination of a pixel, a volume, and an element; it can be simply understood as a pixel in 3D space. Voxelization is the process of assigning points in a 3D point cloud to voxels. In one feasible embodiment of this application, the 3D space is partitioned according to a cylindrical coordinate system, generating voxels in cylindrical slice shapes. If no point is assigned to a voxel, it is determined as a zero voxel; if at least one point is assigned to it, it is determined as a non-zero voxel. Typically, point-to-voxel mapping can be achieved through any of the following methods: fixed voxelization, dynamic voxelization, and hybrid scale voxelization.

[0040] Furthermore, three-dimensional voxel features can be extracted based on the obtained voxel data.

[0041] However, in traditional use such as Figure 1A The model shown primarily extracts 3D voxel features from 3D point clouds by starting from the geometric center of the voxel and describing the relationship between the point cloud within the voxel and the voxel's geometric center. However, actual 3D target object detection is based on the viewpoint of a LiDAR radar. Therefore, in one feasible embodiment of this application, a method is adopted... Figure 1B The three-dimensional target detection model shown in the figure performs voxelization on the three-dimensional point cloud, and also calculates the distance in the xy plane in the cylindrical coordinate system and the rotation angle around the z axis, so as to be passed into the Voxel Feature Encoder as an extension of the voxelfeature.

[0042] The calculation of the above distance and rotation angle can be expressed by the following formula:

[0043] A point in a 3D point cloud is represented as: p(x,y,z);

[0044] The xy-plane distance is:

[0045] The rotation angle around the z-axis is:

[0046] Therefore, the voxelized data passed to the Voxel Feature Encoder can be represented as (x,y,z,r,dist) x-y , Furthermore, features are extracted from the voxel feature encoder, resulting in the obtained 3D voxel features including the distance of the 3D point cloud from the horizontal plane in cylindrical coordinates and the rotation angle of the 3D point cloud around the vertical axis in cylindrical coordinates.

[0047] That is, the extraction of 3D voxel features based on 3D point clouds can optionally be implemented as follows: The 3D point cloud is voxelized, and the distance of the 3D point cloud from the horizontal plane in a cylindrical coordinate system and the rotation angle of the 3D point cloud around the vertical axis in the same cylindrical coordinate system are calculated. Based on the voxelization result, the distance, and the rotation angle, 3D voxel features are extracted from the 3D point cloud to obtain 3D voxel features including the distance feature of the 3D point cloud from the horizontal plane in a cylindrical coordinate system and the rotation angle feature of the 3D point cloud around the vertical axis in the same cylindrical coordinate system. In this embodiment, in addition to traditional coordinate features and reflectivity features, the distance and rotation angle features of the 3D point cloud in a cylindrical coordinate system are extracted to describe the positional relationship between the 3D point cloud and the acquisition device that acquired the 3D point cloud. This expands the descriptive dimension of the environment in which the autonomous driving device is located, providing a foundation for obtaining richer and more comprehensive 3D voxel features and subsequent feature processing.

[0048] When adopting such Figure 1B When implementing this step using the 3D target detection model shown, the above operation can be achieved through the 3D voxel feature extraction part of the 3D target detection model. Specifically, the 3D point cloud is voxelized by the voxelization part of the 3D voxel feature extraction part, and the distance of the 3D point cloud from the horizontal plane in the cylindrical coordinate system and the rotation angle of the 3D point cloud around the vertical axis in the cylindrical coordinate system are calculated. Based on the voxelization result, the distance, and the rotation angle, the 3D voxel feature encoder part of the 3D voxel feature extraction part extracts 3D voxel features from the 3D point cloud, obtaining 3D voxel features including the distance feature of the 3D point cloud from the horizontal plane in the cylindrical coordinate system and the rotation angle feature of the 3D point cloud around the vertical axis in the cylindrical coordinate system.

[0049] Therefore, the cylindrical coordinate features based on the lidar coordinate system, namely the distance between the 3D target object and the lidar in the xy plane, and the rotation angle of the 3D target object and the lidar around the z-axis, can be added to the 3D voxel features. The addition of these two types of features can more specifically and in detail describe the distance and positional relationship between each 3D target object and the lidar corresponding to the 3D point cloud, thereby expanding the model's dimension of environmental perception.

[0050] Step S106: Project the three-dimensional voxel features into a two-dimensional pseudo-image, and perform sparse convolution on the two-dimensional pseudo-image to obtain the corresponding two-dimensional image features.

[0051] Points in three-dimensional space can be projected onto a 2D plane through projection transformation, becoming a 2D pseudo-image (i.e., a two-dimensional pseudo-image). Specifically, in this embodiment, feature points in a three-dimensional voxel feature can be projected to generate a corresponding two-dimensional pseudo-image. This two-dimensional pseudo-image can be a front view (FV) or a bird's-eye view (BEV). This embodiment does not limit the specific form of the two-dimensional pseudo-image or the specific method of projecting the three-dimensional voxel feature into a two-dimensional pseudo-image. After obtaining the two-dimensional pseudo-image, feature extraction can be performed to obtain the corresponding two-dimensional image features.

[0052] To avoid using, such as Figure 1A When the 3D object detection model shown performs this step, it uses ordinary 2D convolution for extraction. This causes the originally sparse features in the 2D pseudo-image to become dense in the final high-dimensional features, resulting in the extracted features failing to accurately match the actual object. In this embodiment, when using... Figure 1B When the 3D object detection model shown performs this step, the projection part of the 2D image feature extraction part projects the 3D voxel features into a 2D pseudo-image; the 2D encoder part of the 2D image feature extraction part performs sparse convolution on the 2D pseudo-image to obtain the corresponding 2D image features. The 2D encoder part includes multiple 2D sparse convolutional layers; the 2D encoder part performs sparse convolution on the 2D pseudo-image through multiple 2D sparse convolutional layers to obtain the corresponding 2D image features. Optionally, the multiple 2D sparse convolutional layers are multiple sparse convolutional layers in the ResNet18 model that employs sparse convolution.

[0053] Because the 3D voxel feature projection of a 3D point cloud into a 2D pseudo-image results in a sparse data distribution, with most positions within a voxelsize*voxel size region being 0, this type of data is considered sparse data. However, as... Figure 1A The 2D encoder in the 3D object detection model shown uses ordinary convolution operation, which performs convolution operation on every position of the entire region of voxel size * voxel size. However, in sparse data, since most of the data is 0, there is a lot of computational redundancy. Sparse convolution operation only calculates non-zero regions, so the computing power is fully utilized. Furthermore, the features extracted by sparse convolution are not affected by 0 and lose their representativeness. Therefore, the features extracted by 2D sparse convolution are more in line with the sparse characteristics of point clouds and are more representative.

[0054] Based on this, in one feasible embodiment of this application, ResNet18, a 2D sparse convolution operator, is used as the main structure in the 2D Encoder. On the one hand, this allows for a more accurate extraction of sparse feature descriptions of the two-dimensional pseudo-images; on the other hand, using ResNet18, which performs well in visual detection, as the backbone network for feature extraction of two-dimensional pseudo-images can effectively balance running speed and detection performance while improving feature extraction and classification.

[0055] Step S108: Calculate the thermal peak value based on the two-dimensional image features, and perform three-dimensional target detection based on the calculation results and the two-dimensional image features.

[0056] In one feasible approach, this step can be implemented as follows: determining the corresponding heatmap based on the feature map corresponding to the two-dimensional image features; performing peak calculation on the heatmap to obtain at least one detection center point; performing regression calculation based on the at least one detection center point and the feature map to determine the detection box corresponding to each of the at least one detection center point; and determining the three-dimensional detection result based on the detection box.

[0057] The detection head in traditional 3D object detection models, such as Figure 1A Most of the detection heads shown use an anchor-based approach. However, this approach, with its single anchor size for different categories, cannot meet the scale diversity requirements of various object categories. Although it is possible to satisfy the multi-scale size variations of different categories by expanding the candidate range of anchors for each category, expanding the candidate range of anchors would increase a lot of useless computation, wasting limited computing power.

[0058] Therefore, in this embodiment, an anchor-free Center Heatmap Head is used as the detection head. The center of each category's bounding box is regressed by calculating the heatmap peak based on the feature map corresponding to the features extracted from the 2D pseudo-image. Then, each edge of the bounding box is regressed based on the center point and the feature map. That is, the Center Heatmap Head is used to perform regression calculations on the center point and each edge of the bounding box. Specifically, regression calculations can be performed using 2D convolution operations based on the elements of the predicted output (predict box) from the detection head: center (xy coordinates of the box center point), center_z (z coordinate of the box center point), dim (length, width, and height of the box), and rot (orientation of the box), ultimately determining each edge of the bounding box. This allows for targeted regression for different object categories, significantly improving the accuracy of the bounding box; furthermore, this method effectively utilizes and saves hardware computing power.

[0059] The following is based on Figure 1B The 3D target detection model shown is an example to illustrate the above detection process. Figure 2B As shown.

[0060] Depend on Figure 2B As can be seen, the 3D point cloud is first voxelized through the voxelization part of the 3D target detection model to obtain voxels containing (x, y, z, r, dist). x-y , The process involves voxel partitioning of the information; then, a shallow 3D voxel feature extraction is performed by a 3D voxel feature encoder to obtain corresponding 3D voxel features, including the distance features of the 3D point cloud in the horizontal direction and the rotation angle features of the 3D point cloud around the vertical axis in the cylindrical coordinate system. Based on the voxel grid size, the extracted 3D voxel features are projected into a 2D pseudo-image (also called a 2D pseudo-image) through a pillar-dispersed part. Next, a 2D encoder is used to perform sparse convolution on the 2D pseudo-image to extract features, obtaining the corresponding 2D image features. Finally, Center HeatmapHead is used to perform 3D target detection based on the 2D image features to determine the detection boxes and classifications of one or more 3D target objects, thus obtaining the detection results, which are simply illustrated in the figure as detection boxes.

[0061] It should be noted that, unless otherwise specified, in the embodiments of this application, "multiple", "various" and other quantities related to "multiple" all refer to two or more.

[0062] Furthermore, based on the detection results, autonomous driving equipment can also perform corresponding downstream tasks, such as navigation rules, driving decisions, obstacle avoidance, and so on.

[0063] As can be seen, through this embodiment, on the one hand, for the two-dimensional pseudo-image projected from three-dimensional voxel features, the traditional two-dimensional convolution method is no longer used; instead, a sparse convolution method is adopted to better fit the actual point cloud description of the three-dimensional object and obtain more accurate two-dimensional image features. On the other hand, the traditional anchor-based detection head is replaced with a detection head based on heatmap peaks without anchor boxes. This avoids wasting computational power due to inaccurate or expanded anchor box candidate ranges. Furthermore, based on the heatmap peak form, the bounding boxes of each category of three-dimensional objects are regressed by combining two-dimensional image features. This not only saves hardware computing power and improves the real-time performance of the calculation, but also greatly improves the accuracy of bounding box detection, ensuring the accuracy of the detection results. When the solution of this embodiment is applied to three-dimensional object detection scenarios, especially in three-dimensional object detection scenarios of automotive embedded platforms, the saving of computing power and the improvement of detection accuracy effectively balance the real-time and accuracy requirements of three-dimensional object detection scenarios.

[0064] Example 2

[0065] This embodiment focuses on illustrating the training process of the three-dimensional target detection model in the above embodiments.

[0066] Reference Figure 3A The diagram shows a flowchart of a three-dimensional target detection method according to Embodiment 2 of this application.

[0067] The three-dimensional target detection method in this embodiment includes the following steps:

[0068] Step S202: Obtain 3D point cloud training samples.

[0069] In practical applications, those skilled in the art can select appropriate 3D point cloud training samples according to the actual model application requirements. These training samples include both the 3D point cloud data itself and its corresponding label data. This label data can be generated manually or automatically through a corresponding labeling model or algorithm. In this embodiment, the label data, in addition to the bounding box, also includes: type data of the autonomous driving device (indicating the specific type of the autonomous driving device, such as a sedan, SUV, van, minibus, truck, etc.), size information of the autonomous driving device, and label data for each face of the autonomous driving device (e.g., indicating the "front" of the vehicle, the "rear" of the vehicle, the "left" of the vehicle's left side, the "right" of the vehicle's right side, the "top" of the vehicle's roof, the "bottom" of the vehicle's underside, etc.). This label data can be used as supervisory information during training and can also be used for data augmentation processing of the training samples.

[0070] Step S204: Perform data augmentation processing on the 3D point cloud training samples.

[0071] Data augmentation can make the data in the 3D point cloud training samples more representative, reduce the impact of invalid data on subsequent model training, and make the 3D point cloud training samples more closely match the point cloud data in the actual model application scenario.

[0072] In this embodiment of the application, at least one of the following data augmentation processes is used to perform data augmentation processing on the 3D point cloud training samples, taking point cloud frames as the unit:

[0073] Method 1: Perform point cloud deletion processing on the object surface of the target object in the 3D point cloud training sample.

[0074] This method, also known as point cloud discarding, primarily simulates occlusion of target objects (mainly autonomous driving equipment). For example, simulating occlusion of autonomous driving equipment from the perspective of a LiDAR scanner typically involves the four sides of the device: front, rear, left, and right. Based on this, a side can be selected with equal probability, and the points on that side can be queued. Then, different degrees of occlusion can be simulated, with different discard ratios selected accordingly. For example, severe occlusion can randomly discard 90%-100% of the points; moderate occlusion can randomly discard 40%-80% of the points; and slight occlusion can randomly discard 10%-20% of the points, and so on.

[0075] By deleting points on one or more faces of an autonomous driving device object with a certain probability, the occlusion of the autonomous driving device object can be effectively simulated, providing training samples under such conditions for model training, which facilitates the model's subsequent learning of 3D object detection under such conditions.

[0076] Method 2: Based on the types of multiple target objects in the 3D point cloud training samples, perform point cloud swapping processing on target objects of the same type.

[0077] This method, also known as point cloud swapping, primarily expands the diversity of the sample by randomly combining autonomous driving devices with similar surfaces. For example, a frame of point cloud data typically contains multiple autonomous driving devices. Based on the annotation information, the size of each autonomous driving device can be obtained, and vehicle objects of similar sizes are grouped together. The point clouds of each facet of the bounding box (here, the annotation box) of the autonomous driving devices in the same group are compared for similarity, where the similarity can be the similarity between the contours formed by the point clouds. The point clouds of different autonomous driving devices with similar contours are swapped to expand a new frame of point cloud data, thereby expanding the diversity of the training samples.

[0078] By exchanging the current point cloud frame with other autonomous driving devices that have similar surfaces on a certain side with a certain probability, the diversity of object samples can be increased, which facilitates the subsequent 3D object detection learning of the model.

[0079] Method 3: Sparsify the point cloud of at least one target object in the 3D point cloud training samples.

[0080] This method, also known as point cloud sparsification, primarily simulates the distance relationship between autonomous driving device samples, reducing the detection impact of distance. Because LiDAR-acquired 3D point clouds are denser for nearby objects and sparser for distant objects, the detection accuracy for distant targets is lower under the same model parameters. Therefore, to balance the effect on objects at different distances, sparsification is applied to the training samples. For example, annotation information can be used to select one or more sides of the autonomous driving device object close to the LiDAR, reducing the point cloud density on those sides while maintaining the basic outline integrity; and / or, sparsification can be achieved by rasterizing one or more sides of the autonomous driving device sample and randomly discarding points in the raster.

[0081] By performing this randomized sparse point cloud operation, the point cloud changes at different distances of the same object can be effectively simulated, providing training samples for model training under such conditions, which facilitates the model's subsequent learning of 3D object detection under these conditions.

[0082] It should be noted that the 3D point clouds acquired by LiDAR are large-scale sparse data. Under the same environment, the point cloud becomes sparser with increasing acquisition distance. Therefore, the number of point clouds for the same object varies significantly at different distances. Furthermore, even for the same object at the same distance, the number of point clouds on different surfaces of the object differs considerably. Typically, when preparing training sample datasets, it is impossible to acquire the same object at different distances and angles, resulting in a limited number of actual training samples. However, the data augmentation method for training samples described in this application embodiment, especially when augmenting the same object in a single point cloud frame, uses bounding boxes (indicated by the indicator box) as the representation method for objects. Therefore, each object in the acquired data is divided into six surfaces: front, back, left, right, top, and bottom. Few objects expose the point cloud below the bounding box. Therefore, in actual data augmentation processing, only the point clouds of the other five surfaces (excluding the bottom) need to be augmented.

[0083] By augmenting the data of the training samples, the point cloud on a certain face can be discarded, swapped, or sparsified with a certain probability based on the characteristics of the objects in the current point cloud frame. This makes the objects in the training samples more diverse and improves the generalization ability of the model.

[0084] Step S206: Train the 3D target detection model using the data-augmented 3D point cloud training samples.

[0085] In this embodiment of the application, the three-dimensional target detection model is a pillar-based three-dimensional target detection model, for example, as shown below. Figure 1B The image shows a 3D object detection model. The following section uses this model as an example, combined with... Figure 3B The training of the 3D target detection model is explained.

[0086] like Figure 3B As shown, an existing point cloud dataset is first obtained, and some or all of the point cloud frames in it are used as training samples. The training samples have corresponding label information, including but not limited to: the bounding box of the autonomous driving device, the size information of the autonomous driving device, the type data of the autonomous driving device, and the label data of each face of the autonomous driving device.

[0087] During the data preparation phase, the training samples are augmented using a data preprocessor. As mentioned earlier, this data augmentation can employ any one or more of the methods described in steps one through three. In this example, all three data augmentation methods are used to effectively expand the training samples. Subsequently, the 3D object detection model is trained based on the data-augmented training samples.

[0088] Taking a single frame of 3D point cloud training sample as an example, firstly, the data-augmented current 3D point cloud training sample is input into the 3D object detection model. The model voxelizes the 3D point cloud in the current 3D point cloud training sample and simultaneously calculates the distance (dist) of the 3D point cloud in the training sample from the horizontal plane in cylindrical coordinates. x-y The rotation angle of the 3D point cloud around the vertical axis in cylindrical coordinate system. Obtain the voxel block corresponding to the training sample of this frame. Each point in the voxel block contains at least (x, y, z, r, dist) x-y , The three-dimensional voxel feature encoder then extracts three-dimensional voxel features based on voxel blocks. Correspondingly, the obtained three-dimensional voxel features also contain the feature information corresponding to the aforementioned distance and rotation angle. To easily distinguish it from the three-dimensional voxel features obtained in the aforementioned inference stage, in this embodiment, the feature obtained based on the training samples is referred to as the three-dimensional voxel sample feature.

[0089] Furthermore, based on the three-dimensional voxel sample features, according to the preset voxel grid size, the three-dimensional voxel sample features are projected into a two-dimensional pseudo-sample image through the PillarScatter. Then, the two-dimensional pseudo-sample image is sparsely convolved with ResNet18 as the backbone to obtain two-dimensional image sample features.

[0090] Next, the features of the two-dimensional image sample will be input into the detection head, in this example the Center Heatmap Head, to perform regression detection based on thermal peaks to obtain the prediction results of the corresponding target object, including: target prediction box, target object size, orientation, velocity and other information.

[0091] The predicted result is compared with the annotation information corresponding to the training samples in that frame, and a loss value is calculated based on a preset loss function. Then, backpropagation is performed based on this loss value to adjust the model parameters. The loss function can be implemented by anyone skilled in the art according to actual needs, such as the loss function of a conventional pillar-based 3D object detection model, etc. This application does not impose any limitations on this.

[0092] The above training process uses a single training sample as an example. However, those skilled in the art should understand that all training samples will be input into the 3D object detection model in the above manner for iterative training until the training termination conditions are met, such as reaching a preset number of training iterations or the loss value meeting a preset threshold.

[0093] Once trained, the 3D target detection model can be tested and / or used for inference. Optionally, it can also be integrated with downstream business processes to achieve corresponding downstream tasks.

[0094] Step S208: Perform 3D object detection using the trained 3D object detection model.

[0095] This includes: acquiring a 3D point cloud of the environment where the autonomous driving device is located; extracting 3D voxel features based on the 3D point cloud to obtain the corresponding 3D voxel features; projecting the 3D voxel features into a 2D pseudo-image and performing sparse convolution on the 2D pseudo-image to obtain the corresponding 2D image features; calculating thermal peaks based on the 2D image features; and performing 3D target detection based on the calculation results and the 2D image features.

[0096] The specific implementation of the above process can be found in the relevant description in the aforementioned Embodiment 1, and will not be repeated here.

[0097] This embodiment achieves several advantages. First, by augmenting the training samples, the tolerance for the collected training samples becomes very relaxed. Furthermore, new training samples can be generated by combining similar training samples with a certain probability, and new training samples can be derived from existing training samples, thus maximizing sample diversity with minimal acquisition cost. Second, by adding three-dimensional voxel features, including distance and rotation angle features, the description of the environment in which the autonomous driving device operates becomes richer. The combination of 2D sparse convolution and ResNet18 better captures feature descriptions that fit the sparse characteristics of point clouds. Finally, the Center HeatmapHead allows for more targeted regression of bounding boxes for various object categories, improving bounding box detection accuracy and significantly saving hardware computing power.

[0098] Example 3

[0099] Reference Figure 4 The diagram shows a structural block diagram of a three-dimensional target detection device according to Embodiment 3 of this application.

[0100] The three-dimensional target detection device in this embodiment includes: a first acquisition module 302, used to acquire a three-dimensional point cloud of the environment where the autonomous driving device is located; a second acquisition module 304, used to extract three-dimensional voxel features based on the three-dimensional point cloud to acquire the corresponding three-dimensional voxel features; a third acquisition module 306, used to project the three-dimensional voxel features into a two-dimensional pseudo-image and perform sparse convolution on the two-dimensional pseudo-image to obtain the corresponding two-dimensional image features; and a detection module 308, used to calculate thermal peaks based on the two-dimensional image features and to perform three-dimensional target detection based on the calculation results and the two-dimensional image features.

[0101] Optionally, the second acquisition module 304 is used to voxelize the three-dimensional point cloud and calculate the distance of the three-dimensional point cloud from the horizontal plane in the cylindrical coordinate system and the rotation angle of the three-dimensional point cloud around the vertical axis in the cylindrical coordinate system; based on the voxelization result, the distance and the rotation angle, three-dimensional voxel features are extracted from the three-dimensional point cloud to obtain three-dimensional voxel features including the distance feature of the three-dimensional point cloud from the horizontal plane in the cylindrical coordinate system and the rotation angle feature of the three-dimensional point cloud around the vertical axis in the cylindrical coordinate system.

[0102] Optionally, the detection module 308 is used to determine the corresponding heat map based on the feature map corresponding to the two-dimensional image features; perform peak calculation on the heat map to obtain at least one detection center point; perform regression calculation based on at least one detection center point and the feature map to determine the detection box corresponding to each of the at least one detection center point; and determine the three-dimensional detection result based on the detection box.

[0103] Optionally, the 3D target detection device in this embodiment performs 3D target detection through a pillar-based 3D target detection model, which includes: a 3D voxel feature extraction part, a 2D image feature extraction part, and a detection head part; wherein: the 3D voxel feature extraction part is used to extract 3D voxel features based on 3D point clouds to obtain corresponding 3D voxel features; the 2D image feature extraction part is used to project the 3D voxel features into a 2D pseudo-image and perform sparse convolution on the 2D pseudo-image to obtain corresponding 2D image features; the detection head part is used to calculate thermal peaks based on the 2D image features and perform 3D target detection based on the calculation results and the 2D image features.

[0104] Optionally, the above-mentioned three-dimensional voxel feature extraction part includes: a voxelization part and a three-dimensional voxel feature encoder part; wherein: the voxelization part is used to voxelize the three-dimensional point cloud, and calculate the distance of the three-dimensional point cloud from the horizontal plane in the cylindrical coordinate system, and the rotation angle of the three-dimensional point cloud around the vertical axis in the cylindrical coordinate system; the three-dimensional voxel feature encoder part is used to extract three-dimensional voxel features from the three-dimensional point cloud based on the voxelization result, the distance and the rotation angle, to obtain the corresponding three-dimensional voxel features.

[0105] Optionally, the above-mentioned two-dimensional image feature extraction part includes a projection part and a two-dimensional encoder part; wherein: the projection part is used to project the three-dimensional voxel features into a two-dimensional pseudo-image; the two-dimensional encoder part is used to perform sparse convolution on the two-dimensional pseudo-image to obtain the corresponding two-dimensional image features, which are sparse features.

[0106] Optionally, the two-dimensional encoder part includes multiple layers of two-dimensional sparse convolutional layers; the two-dimensional encoder part performs sparse convolution on the two-dimensional pseudo-image through multiple layers of two-dimensional sparse convolutional layers to obtain the corresponding two-dimensional image features.

[0107] Optionally, the multi-layer two-dimensional sparse convolutional layers are multiple sparse convolutional layers in a ResNet18 model that employs sparse convolution.

[0108] Optionally, the detection head is divided into a central thermal mapping detection head.

[0109] Optionally, the three-dimensional target detection device in this embodiment further includes a training module 310, used to train the three-dimensional target detection model using three-dimensional point cloud training samples that have undergone data augmentation processing.

[0110] Optionally, the above data augmentation process includes at least one of the following processes: performing point cloud deletion processing on the object faces of the target objects in the 3D point cloud training samples; performing point cloud swapping processing on target objects of the same type according to the types of multiple target objects in the 3D point cloud training samples; and performing sparsification processing on the point cloud of at least one target object in the 3D point cloud training samples.

[0111] The three-dimensional target detection device of this embodiment is used to implement the corresponding three-dimensional target detection methods in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each module in the three-dimensional target detection device of this embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0112] Example 4

[0113] Reference Figure 5 The diagram shows a structural schematic of an autonomous driving device according to Embodiment 4 of this application.

[0114] The autonomous driving device in this embodiment includes a three-dimensional point cloud acquisition device 402 and a processor 404.

[0115] The 3D point cloud acquisition device 402 is used to acquire 3D point clouds of the environment in which the autonomous driving equipment is located. For example, the 3D point cloud acquisition device 402 can be implemented as a LiDAR (Light Detection and Ranging) system.

[0116] Processor 404 is used for 3D object detection based on 3D point clouds. Optionally, processor 404 can perform 3D object detection based on 3D point clouds and a pre-trained 3D object detection model. The specific implementation of the 3D object detection, and the 3D object detection performed using the 3D object detection model, can be implemented with reference to the relevant descriptions in Embodiments 1 or 2 above, and will not be repeated here. For example, processor 404 can be implemented as an automotive embedded chip.

[0117] Optionally, the processor 404 can also make autonomous driving decisions based on the detection results of the 3D target detection; and drive the autonomous driving equipment to perform driving task processing based on the decision results.

[0118] Generally, autonomous driving devices also include a drive unit 406 and a control unit 408. After the processor 404 determines the corresponding autonomous driving decision, it sends a drive signal to the drive unit 406. The drive unit 406 uses this drive signal to control the control unit 408 (such as brake, accelerator, steering wheel, etc.) in the autonomous driving device to perform corresponding control operations on the autonomous driving settings, thereby realizing the autonomous driving of the autonomous driving device.

[0119] For ease of description and brevity, the specific working process of the autonomous driving device described above can be referred to the corresponding process description in the foregoing method embodiments, and will not be repeated here.

[0120] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0121] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0122] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0123] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A three-dimensional target detection method, characterized in that, include: Obtain the 3D point cloud of the environment in which the autonomous driving device is located; Based on the three-dimensional point cloud, three-dimensional voxel features are extracted to obtain the corresponding three-dimensional voxel features; The three-dimensional voxel features are projected into a two-dimensional pseudo-image, and the two-dimensional pseudo-image is sparsely convolved to obtain the corresponding two-dimensional image features. The process involves calculating thermal peak values ​​based on the two-dimensional image features, and then performing three-dimensional target detection based on the calculation results and the two-dimensional image features. This includes: determining a corresponding thermal map based on the feature map corresponding to the two-dimensional image features; calculating peak values ​​on the thermal map to obtain at least one detection center point; performing regression calculations based on the at least one detection center point and the feature map to determine the detection boxes corresponding to the at least one detection center point; and determining the three-dimensional detection result based on the detection boxes.

2. The method according to claim 1, characterized in that, The step of extracting three-dimensional voxel features based on the three-dimensional point cloud to obtain the corresponding three-dimensional voxel features includes: The three-dimensional point cloud is voxelized, and the distance of the three-dimensional point cloud from the horizontal plane in the cylindrical coordinate system and the rotation angle of the three-dimensional point cloud around the vertical axis in the cylindrical coordinate system are calculated. Based on the voxelization result, the distance, and the rotation angle, three-dimensional voxel features are extracted from the three-dimensional point cloud to obtain three-dimensional voxel features including the distance feature of the three-dimensional point cloud in the horizontal direction in the cylindrical coordinate system and the rotation angle feature of the three-dimensional point cloud around the vertical axis in the cylindrical coordinate system.

3. The method according to claim 1, characterized in that, The three-dimensional target detection method is executed through a pillar-based three-dimensional target detection model, which includes: a three-dimensional voxel feature extraction part, a two-dimensional image feature extraction part, and a detection head part; in: The three-dimensional voxel feature extraction part is used to extract three-dimensional voxel features based on the three-dimensional point cloud and obtain the corresponding three-dimensional voxel features. The two-dimensional image feature extraction part is used to project the three-dimensional voxel features into a two-dimensional pseudo-image, and perform sparse convolution on the two-dimensional pseudo-image to obtain the corresponding two-dimensional image features. The detection head is used to calculate thermal peaks based on the two-dimensional image features, and to perform three-dimensional target detection based on the calculation results and the two-dimensional image features.

4. The method according to claim 3, characterized in that, The three-dimensional voxel feature extraction part includes: a voxelization part and a three-dimensional voxel feature encoder part; in: The voxelization part is used to voxelize the three-dimensional point cloud, and to calculate the distance of the three-dimensional point cloud from the horizontal plane in the cylindrical coordinate system, and the rotation angle of the three-dimensional point cloud around the vertical axis in the cylindrical coordinate system. The three-dimensional voxel feature encoder is used to extract three-dimensional voxel features from the three-dimensional point cloud based on the voxelization result, the distance, and the rotation angle, to obtain three-dimensional voxel features including the distance feature of the three-dimensional point cloud in the horizontal direction in the cylindrical coordinate system and the rotation angle feature of the three-dimensional point cloud around the vertical axis in the cylindrical coordinate system.

5. The method according to claim 3, characterized in that, The two-dimensional image feature extraction part includes a projection part and a two-dimensional encoder part; in: The projection portion is used to project the three-dimensional voxel features into a two-dimensional pseudo-image; The two-dimensional encoder part is used to perform sparse convolution on the two-dimensional pseudo-image to obtain the corresponding two-dimensional image features.

6. The method according to claim 5, characterized in that, The two-dimensional encoder portion includes multiple two-dimensional sparse convolutional layers; The two-dimensional encoder part performs sparse convolution on the two-dimensional pseudo-image through the multi-layer two-dimensional sparse convolutional layer to obtain the corresponding two-dimensional image features.

7. The method according to claim 6, characterized in that, The multi-layer two-dimensional sparse convolutional layer refers to multiple sparse convolutional layers in the ResNet18 model that employs sparse convolution.

8. The method according to claim 3, characterized in that, The detection head is divided into a central thermal mapping detection head.

9. The method according to any one of claims 3-8, characterized in that, The method further includes: The 3D target detection model is trained using data-augmented 3D point cloud training samples.

10. The method according to claim 9, characterized in that, The data augmentation process includes at least one of the following processes: The target object in the three-dimensional point cloud training sample is subjected to point cloud deletion processing of the object surface; Based on the types of multiple target objects in the 3D point cloud training samples, point cloud swapping is performed on target objects of the same type. The point cloud of at least one target object in the three-dimensional point cloud training sample is subjected to sparsification processing.

11. A three-dimensional target detection device, characterized in that, include: The first acquisition module is used to acquire the three-dimensional point cloud of the environment where the autonomous driving device is located; The second acquisition module is used to extract three-dimensional voxel features based on the three-dimensional point cloud and acquire the corresponding three-dimensional voxel features. The third acquisition module is used to project the three-dimensional voxel features into a two-dimensional pseudo-image and perform sparse convolution on the two-dimensional pseudo-image to obtain the corresponding two-dimensional image features. The detection module is used to calculate thermal peaks based on the two-dimensional image features, and to perform three-dimensional target detection based on the calculation results and the two-dimensional image features, including: determining the corresponding thermal map based on the feature map corresponding to the two-dimensional image features; calculating the peaks on the thermal map to obtain at least one detection center point; performing regression calculation based on the at least one detection center point and the feature map to determine the detection boxes corresponding to the at least one detection center point; and determining the three-dimensional detection result based on the detection boxes.

12. An autonomous driving device, characterized in that, include: 3D point cloud acquisition device and processor; in: The three-dimensional point cloud acquisition device is used to acquire three-dimensional point clouds of the environment where the autonomous driving equipment is located. The processor is configured to perform three-dimensional target detection based on the three-dimensional point cloud, according to any one of claims 1-11.

13. The autonomous driving device according to claim 12, characterized in that, The processor is also used to make autonomous driving decisions based on the detection results of the three-dimensional target detection; and, based on the decision results, drive the autonomous driving device to perform driving task processing.

14. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-10.

Images