Point cloud data labeling method, model training method, electronic device, and storage medium

Abstract

The application discloses a point cloud data labeling method, a model training method, an electronic device and a storage medium. The point cloud data labeling method comprises the following steps: performing grid processing on current frame point cloud data to obtain a backbone feature of the current frame point cloud data; obtaining at least one ground truth box obtained by labeling adjacent two frame point cloud data; processing the at least one ground truth box to obtain a ground truth feature of the labeled adjacent two frame point cloud data; and performing splicing and fusion processing on the backbone feature of the current frame point cloud data and the ground truth feature of the labeled adjacent two frame point cloud data to obtain a fusion feature, so that a detection box of the current frame point cloud data is obtained according to the fusion feature, and labeling is realized. According to the scheme, the ground truth feature of the labeled adjacent two frame point cloud data is integrated into the backbone feature of the point cloud data to be labeled, better results are output with less resources, and manpower and material resources are saved.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a point cloud data annotation method, a model training method, an electronic device, and a storage medium. Background Technology

[0002] In the process of training a high-precision point cloud 3D target detection model, a large amount of point cloud labeled data is required. In some existing methods, all the data to be labeled is labeled manually to obtain point cloud labeled data, which consumes a lot of manpower and resources. Summary of the Invention

[0003] This application provides at least one point cloud data annotation method, model training method, electronic device, and storage medium to solve the above-mentioned problems.

[0004] The first aspect of this application provides a point cloud data annotation method, the method comprising: acquiring current frame point cloud data to be annotated, wherein the current frame point cloud data is located between two adjacent frame point cloud data that have been annotated in a series of consecutive frame point cloud data;

[0005] The current frame point cloud data is rasterized to obtain the backbone features of the current frame point cloud data;

[0006] At least one truth box is obtained by annotating each frame in the two adjacent point cloud data frames, and the at least one truth box is processed to obtain the truth features of each frame in the annotated two adjacent point cloud data frames.

[0007] The backbone features of the current frame point cloud data and the ground truth features of the two adjacent frame point cloud data that have been labeled are spliced ​​and fused to obtain fused features. The detection box of the current frame point cloud data is then obtained based on the fused features to achieve labeling.

[0008] Among them, the continuous multi-frame point cloud data refers to the point cloud data in which each pair of adjacent frame point cloud data is separated by the same number of frames.

[0009] The core features are represented by a first three-dimensional tensor, and the truth features are represented by a second three-dimensional tensor. The size of the two-dimensional tensor of the second three-dimensional tensor is equal to the size of the two-dimensional tensor of the first three-dimensional tensor.

[0010] The process of splicing and fusing the backbone features of the current frame point cloud data and the ground truth features of the two adjacent frame point cloud data includes: splicing and fusing the backbone features of the current frame point cloud data and the ground truth features of the two adjacent frame point cloud data along the two-dimensional tensor of equal size.

[0011] Wherein, the at least one truth box corresponds to at least one detection category; the at least one truth box is processed to obtain the truth features of the labeled two adjacent frames of point cloud data, including: obtaining the size ratio between the preset interest region in the three-dimensional space where the at least one truth box is located and the two-dimensional tensor of the second three-dimensional tensor;

[0012] Based on the size ratio, the initial coordinates of the center point of each of the at least one truth box in the preset region of interest, and the detection category corresponding to each of the at least one truth box, the mapping coordinates of the center point of each of the at least one truth box in the second three-dimensional tensor are determined to obtain at least one of the mapping coordinates;

[0013] The truth feature is obtained by assigning a value to each of at least one of the mapped coordinates.

[0014] The initial coordinates include the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the center point; assigning values ​​to each of at least one of the mapped coordinates includes: assigning the truth box length, width, height, orientation, z-axis coordinate of the center point, and the time interval between the labeled adjacent frame point cloud data corresponding to the truth box and the current frame point cloud data.

[0015] The annotation of each frame in the two adjacent point cloud data includes: inputting each frame in the two adjacent point cloud data into a pre-annotation model to output at least one annotation box.

[0016] The at least one annotation box is modified to obtain the at least one truth box.

[0017] The method of obtaining the detection box of the current frame point cloud data based on the fusion features further includes: correcting the detection box.

[0018] The second aspect of this application provides a model training method, including:

[0019] Obtain the detection box of the point cloud data annotation; wherein the detection box of the point cloud data annotation is obtained using the point cloud data annotation method in the first aspect above;

[0020] The model is trained using the detection boxes labeled with the point cloud data to obtain the trained target model.

[0021] A third aspect of this application provides an electronic device including a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the point cloud data annotation method of the first aspect or the model training method of the second aspect.

[0022] The fourth aspect of this application provides a non-volatile computer-readable storage medium for storing program instructions, which, when executed by a processor, are used to implement the point cloud data annotation method in the first aspect or the model training method in the second aspect.

[0023] The above scheme involves the current frame point cloud data to be labeled located between two adjacent labeled point cloud data frames in a continuous series of point cloud data. The current frame point cloud data is rasterized to obtain its backbone features. At least one ground truth bounding box is obtained from the labeled points in the two adjacent frames. This ground truth bounding box is processed to obtain the ground truth features of the labeled points in the two adjacent frames. The backbone features of the current frame point cloud data are then concatenated and fused with the ground truth features of the labeled points in the two adjacent frames to obtain a fused feature. Based on this fused feature, the detection box of the current frame point cloud data is obtained, thus achieving labeling. The scheme in this application, by incorporating the ground truth features of the labeled points in the two adjacent frames into the backbone features of the point cloud data to be labeled, outputs better results with fewer resources, saving manpower and material resources.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0026] Figure 1 This is a flowchart illustrating the point cloud data annotation method in the embodiments of this application;

[0027] Figure 2 This is a first distribution diagram of point cloud data in an embodiment of this application;

[0028] Figure 3 This is a schematic diagram of the workflow of the point cloud data annotation method in the embodiments of this application;

[0029] Figure 4 This is a schematic diagram of the second distribution of point cloud data in an embodiment of this application;

[0030] Figure 5(a) is a schematic diagram of the shape of the main features in an embodiment of this application;

[0031] Figure 5(b) is a schematic diagram of the shape of the truth feature in an embodiment of this application;

[0032] Figure 5(c) is a schematic diagram of the shape of the fusion feature in an embodiment of this application;

[0033] Figure 6 This is a schematic diagram of the mapping relationship in the embodiments of this application;

[0034] Figure 7 This is a schematic diagram of the assignment scenario in the embodiments of this application;

[0035] Figure 8 This is a flowchart illustrating the model training method in an embodiment of this application;

[0036] Figure 9 This is a schematic diagram of the structure of the electronic device in the embodiments of this application;

[0037] Figure 10 This is a schematic diagram of the structure of a non-volatile computer-readable storage medium in an embodiment of this application. Detailed Implementation

[0038] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the application. Similarly, the following embodiments are only some, not all, embodiments of the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0039] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0040] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, "many" in this document means two or more. Additionally, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. Furthermore, the terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0041] As mentioned above, in some existing methods, all the data to be labeled is labeled manually to obtain point cloud labeled data, which consumes a lot of manpower and resources.

[0042] To address these issues, this application provides a point cloud data annotation method, a model training method, an electronic device, and a storage medium.

[0043] Please see Figure 1 , Figure 1 This is a flowchart illustrating the point cloud data annotation method in an embodiment of this application. It should be noted that if substantially the same result is obtained, the method of this application is not necessarily identical. Figure 1 The illustrated process sequence is limited. This method can be applied to electronic devices with computational functions. These devices can execute this method by receiving information collected by sensor devices, such as millimeter-wave radar, lidar, or cameras equipped on the model training vehicle. During the model training vehicle's operation, the sensor devices perceive the dynamic real-world scene surrounding the vehicle, including vehicles, pedestrians, and buildings on the road. Figure 1 As shown, the point cloud data annotation method includes the following steps:

[0044] S11. Obtain the current frame point cloud data to be labeled, wherein the current frame point cloud data is located between two adjacent frame point cloud data that have been labeled in a series of consecutive frame point cloud data.

[0045] Point cloud data can be collected using radar sensors, for example, by mounting the radar sensor on a mobile device. This mobile device can be an automated mobile device, such as a robot or a model training vehicle.

[0046] In some embodiments, the radar sensor may be a lidar sensor, such as a mechanical lidar, semi-solid-state lidar, or solid-state lidar. In one embodiment, the radar sensor may be any radar device that provides point cloud data and is used for model training to meet perception accuracy requirements.

[0047] In one application scenario, an autonomous vehicle is driving on a road. The radar sensors installed on the autonomous vehicle acquire point cloud data around the vehicle to obtain point cloud data for each frame within a preset time period.

[0048] The process involves acquiring the current frame's point cloud data to be labeled. This current frame's point cloud data is located between two adjacent labeled frames within a continuous series of point cloud data. This is understandable. Figure 2 This is a first distribution diagram of point cloud data in an embodiment of this application, such as... Figure 2As shown, the preset time period includes multiple frames of point cloud data. The solid boxes represent labeled point cloud data (a1, a2, a3, a4, a5), and the dashed boxes represent point cloud data to be labeled (b1, b2, b3, b4, b5, b6, b7, b8). The labeled point cloud data (a1), the point cloud data to be labeled (b1, b2), the labeled point cloud data (a2), the point cloud data to be labeled (b3, b4, b5), the labeled point cloud data (a3), the point cloud data to be labeled (b6, b7), the labeled point cloud data (a4), the point cloud data to be labeled (b8), and the labeled point cloud data (a5) constitute a continuous series of multiple frames of point cloud data. When it is necessary to label any frame of the point cloud data to be labeled, that frame of point cloud data is the current frame of point cloud data to be labeled.

[0049] For example, when it is necessary to annotate the point cloud data (b1) to be labeled, it can be determined that the current frame point cloud data (b1) to be labeled is located between the two adjacent labeled frame point cloud data (a1, a2), that is, the current frame point cloud data (b1) is located between the adjacent labeled frame point cloud data (a1) and the adjacent labeled frame point cloud data (a2); as another example, when it is necessary to annotate the point cloud data (b6) to be labeled, it can be determined that the current frame point cloud data (b6) to be labeled is located between the two adjacent labeled frame point cloud data (a3, a4), that is, the current frame point cloud data (b6) is located between the adjacent labeled frame point cloud data (a3) ​​and the adjacent labeled frame point cloud data (a4).

[0050] S12. Rasterize the current frame point cloud data to obtain the backbone features of the current frame point cloud data.

[0051] The current frame point cloud data is rasterized, for example, by processing it through a grid and then using a SparseConv3d backbone to obtain the backbone features of the current frame point cloud data. The specific rasterization method can be set according to actual usage requirements and is not specifically limited. (Understandably, continuing with the above...) Figure 2 Taking the current frame point cloud data (b3) as an example, we perform rasterization processing on the current frame point cloud data (b3) to obtain the backbone features of the current frame point cloud data (b3).

[0052] S13. Obtain at least one truth box obtained by annotating each frame in two adjacent point cloud data frames, and process the at least one truth box to obtain the truth features of each frame in the annotated two adjacent point cloud data frames.

[0053] Obtaining at least one truth bounding box from the annotation of each frame in two adjacent frames of point cloud data can be understood as follows: annotating any frame of point cloud data in a series of consecutive frames will yield at least one truth bounding box, ensuring that each frame of annotated point cloud data corresponds to at least one truth bounding box. Processing at least one truth bounding box allows us to obtain the truth features of the corresponding annotated frame of point cloud data, thereby obtaining the truth features of each frame in two adjacent frames of annotated point cloud data.

[0054] For example, continuing with the above Figure 2 Taking this as an example, when it is necessary to annotate the current frame point cloud data (b3), the two adjacent frame point cloud data can be point cloud data (a2) and point cloud data (a3), respectively. Annotating the point cloud data (a2) yields at least one truth box. Processing the at least one truth box yields the truth features of the point cloud data (a2). Annotating the point cloud data (a3) ​​yields at least one truth box. Processing the at least one truth box yields the truth features of the point cloud data (a3). That is, the truth features of each frame in the two adjacent frame point cloud data (a2, a3) that have been annotated are obtained.

[0055] S14. The backbone features of the current frame point cloud data and the ground truth features of the two adjacent labeled point cloud data are spliced ​​and fused to obtain fused features. The detection box of the current frame point cloud data is obtained based on the fused features to achieve annotation.

[0056] Understandably, concatenating and fusing the backbone features of the current frame's point cloud data with the ground truth features of the two adjacent labeled point cloud data frames yields corresponding fused features. The specific fusion method can be set according to actual usage requirements and is not specifically limited. Based on the obtained fused features, detection boxes for the current frame's point cloud data can be obtained, enabling annotation. For example, the fused features can be sequentially passed through the Neck and Head parts of a deep learning network to output corresponding detection boxes, which can be 3D detection boxes.

[0057] For example, continuing with the above Figure 2 Let's take an example to illustrate. Figure 3 This is a schematic diagram illustrating the workflow of the point cloud data annotation method in this application embodiment, as follows: Figure 3As shown, when it is necessary to annotate the current frame point cloud data (b3), the current frame point cloud data (b3) is subjected to fine rasterization processing to obtain the backbone features of the current frame point cloud data (b3). The two adjacent frame point cloud data that have been annotated are point cloud data (a2) and point cloud data (a3), respectively. Each frame in the two adjacent frame point cloud data (a2, a3) is annotated to obtain at least one ground truth box. At least one ground truth box corresponding to each frame is obtained, and at least one ground truth box is processed to obtain the ground truth features of each frame in the two adjacent frame point cloud data (a2, a3) that have been annotated. The backbone features are concatenated and fused with the corresponding two ground truth features to obtain the fused features corresponding to the current frame point cloud data (b3). Then, based on the fused features, the detection box of the current frame point cloud data (b3) can be obtained, thereby realizing the annotation.

[0058] The above scheme involves the current frame point cloud data to be labeled located between two adjacent labeled point cloud data frames in a continuous series of point cloud data. The current frame point cloud data is rasterized to obtain its backbone features. At least one ground truth bounding box is obtained from the labeled points in the two adjacent frames. This ground truth bounding box is processed to obtain the ground truth features of the labeled points in the two adjacent frames. The backbone features of the current frame point cloud data are then concatenated and fused with the ground truth features of the labeled points in the two adjacent frames to obtain a fused feature. Based on this fused feature, the detection box of the current frame point cloud data is obtained, thus achieving labeling. The scheme in this application, by incorporating the ground truth features of the labeled points in the two adjacent frames into the backbone features of the point cloud data to be labeled, outputs better results with fewer resources, saving manpower and material resources.

[0059] As described above, the current frame point cloud data is located between two adjacent labeled frame point cloud data in a series of consecutive frame point cloud data. In one embodiment of this application, each pair of adjacent labeled frame point cloud data in a series of consecutive frame point cloud data is spaced out by the same number of frames.

[0060] In continuous multi-frame point cloud data, labeled point cloud data refers to point cloud data where every two adjacent labeled frames are spaced out by the same number of frames. This means that the continuous multi-frame point cloud data includes labeled point cloud data and point cloud data to be labeled. The point cloud data to be labeled is located between two adjacent labeled frames, and every two adjacent labeled frames are spaced out by the same number of frames of unlabeled point cloud data. The number of frames between adjacent labeled frames can be set according to actual usage requirements, such as 3, 5, or 10 frames, or other achievable values; there is no specific limitation. In other words, the point cloud data in continuous multi-frame point cloud data is labeled at the same interval to obtain labeled point cloud data, thus ensuring that every two adjacent labeled frames are spaced out by the same number of frames.

[0061] For example, Figure 4 This is a schematic diagram of the second distribution of point cloud data in an embodiment of this application, as shown below. Figure 4 As shown, the continuous multi-frame point cloud data includes point cloud data (a1, b1, b2, b3, a2, b4, b5, b6, a3, b7, b8, b9, a4). The continuous multi-frame point cloud data is labeled at intervals of 3 frames, resulting in labeled point cloud data (a1, a2, a3, a4) and unlabeled point cloud data (b1, b2, b3, b4, b5, b6, b7, b8, b9). This ensures that the intervals between adjacent labeled point cloud data frames (a1, a2), adjacent labeled point cloud data frames (a2, a3), and adjacent labeled point cloud data frames (a3, a4) are all the same, which is 3 frames of point cloud data.

[0062] As described above, the backbone features of the current frame point cloud data are obtained by rasterization, and the truth features of the labeled point cloud data are obtained by processing at least one truth box. In one embodiment of this application, the backbone features are represented by a first three-dimensional tensor, and the truth features are represented by a second three-dimensional tensor. The size of the two-dimensional tensor of the second three-dimensional tensor is equal to the size of the two-dimensional tensor of the first three-dimensional tensor.

[0063] The core features are represented by a first three-dimensional tensor, which can be understood as having a three-dimensional matrix shape. The truth features are represented by a second three-dimensional tensor, which can also be understood as having a three-dimensional matrix shape. The size of the second three-dimensional tensor's two-dimensional tensor is equal to the size of the first three-dimensional tensor's two-dimensional tensor. This can be understood as treating the first three-dimensional tensor as a first cuboid and the second three-dimensional tensor as a second cuboid, then the size of one surface of the first cuboid is equal to the size of one surface of the second cuboid.

[0064] For example, Figure 5(a) is a schematic diagram of the shape of the backbone feature in an embodiment of this application, and Figure 5(b) is a schematic diagram of the shape of the truth feature in an embodiment of this application. As shown in Figures 5(a) and 5(b), the backbone feature is represented by a first three-dimensional tensor with a size of C×W×H, and the truth feature is represented by a second three-dimensional tensor with a size of D×W×H. The size of the two-dimensional tensor (W×H) in the first three-dimensional tensor is equal to the size of the two-dimensional tensor (W×H) in the second three-dimensional tensor.

[0065] The process involves splicing and fusing the backbone features of the current frame point cloud data with the ground truth features of the two adjacent frame point cloud data along a two-dimensional tensor of equal size.

[0066] Along a two-dimensional tensor of equal size, the backbone features of the current frame point cloud data and the ground truth features of the two adjacent labeled frame point cloud data are spliced ​​and fused. To understand further, we will continue to use the backbone features and ground truth features shown in Figures 5(a) and 5(b) as examples. The two adjacent labeled frame point cloud data can obtain two corresponding ground truth features. The backbone features are represented by a first three-dimensional tensor (C×W×H), and both ground truth features are represented by a second three-dimensional tensor (D×W×H). Figure 5(c) is a schematic diagram of the shape of the fused feature in this embodiment. As shown in Figure 5(c), along a two-dimensional tensor (W×H) of equal size, the backbone features and two ground truth features are spliced ​​and fused. That is, the first three-dimensional tensor (C×W×H) is spliced ​​and fused with two second three-dimensional tensors (D×W×H) to obtain the fused feature [(C+D+D)×W×H].

[0067] As described above, labeling any frame in a series of consecutive point cloud data can yield at least one truth box. In one embodiment of this application, at least one truth box corresponds to at least one detection category.

[0068] Understandably, during operation, autonomous vehicles perceive their surroundings in real time through sensors mounted on them, obtaining continuous multi-frame point cloud data. Each frame contains point clouds of at least one detection category, such as vehicles, pedestrians, etc., which can be set according to actual usage requirements without specific limitations. Annotating any frame of point cloud data yields at least one ground truth bounding box, and each ground truth bounding box corresponds to at least one detection category.

[0069] Processing at least one truth bounding box to obtain truth features of two adjacent labeled point cloud data includes: obtaining the size ratio between a preset region of interest in the three-dimensional space where the at least one truth bounding box is located and a two-dimensional tensor of a second three-dimensional tensor; determining the mapping coordinates of the center point of each of the at least one truth bounding box in the second three-dimensional tensor based on the size ratio, the initial coordinates of the center point of each of the at least one truth bounding box in the preset region of interest, and the detection category corresponding to each of the at least one truth bounding box, to obtain at least one mapping coordinate; assigning a value to each of the at least one mapping coordinate to obtain the truth features.

[0070] The method involves obtaining the dimensional ratio between a preset region of interest (ROI) in the 3D space containing at least one truth bounding box and the 2D tensor of the second 3D tensor. This means that when labeling any frame in a series of consecutive point cloud data frames, at least one truth bounding box can be obtained in the 3D space, where each truth bounding box corresponds to at least one detection category. The preset ROI can be set according to actual usage requirements, and its specific size is not limited. For example, centered on the location of the autonomous vehicle, with a radius of 100 meters in front and behind the vehicle and 50 meters to the left and right, the preset ROI can be defined as a rectangular area with a length H' = 200 meters and a width W' = 100 meters. Taking the 2D tensor (W×H) of the second 3D tensor (D×W×H) as an example, the specific values ​​of the length (H) and width (W) of the 2D tensor (W×H) can be set according to actual usage requirements and are not specifically limited. Therefore, once the size of the preset region of interest (W'×H') and the size of the two-dimensional tensor (W×H) are determined, the size ratio between the preset region of interest (W'×H') and the two-dimensional tensor (W×H) can be further determined.

[0071] Based on the size ratio, the initial coordinates of the center point of each of the at least one truth bounding boxes within a preset region of interest (ROI), and the corresponding detection category of each of the at least one truth bounding boxes, the mapped coordinates of the center point of each of the at least one truth bounding boxes within a second 3D tensor are determined to obtain at least one mapped coordinate. It can be understood that after setting the length and width of the preset ROI, a preset ROI coordinate system is obtained by taking any vertex of the preset ROI as the origin and the two sides intersecting that vertex as the horizontal and vertical axes, respectively. This allows the coordinates of the truth bounding boxes within the preset ROI coordinate system, i.e., the initial coordinates of the truth bounding boxes within the preset ROI. The detection categories are set according to actual usage requirements, ensuring that each frame of point cloud data includes point clouds of at least one detection category. Therefore, the at least one truth bounding box annotated for any frame of point cloud data corresponds to at least one detection category. The mapped coordinates of the center point of each of the at least one truth bounding boxes within a second 3D tensor are determined to obtain at least one mapped coordinate. A value is assigned to each of the at least one mapped coordinate to obtain the truth feature.

[0072] For example, Figure 6 This is a schematic diagram of the mapping relationship in the embodiments of this application, such as... Figure 6As shown, multiple ground truth boxes are obtained by labeling any frame of point cloud data. Assuming the preset region of interest has a length H' = 200 and a width W' = 100, there are four ground truth boxes (d11, d21, d31, d32) within the preset region of interest. These four ground truth boxes (d11, d21, d31, d32) correspond to three detection categories (D1, D2, D3). Specifically, the initial coordinates of the center point of ground truth box (d11) are (20, 80, 0.5), corresponding to detection category (D1); the initial coordinates of the center point of ground truth box (d21) are (80, 30, 1), corresponding to detection category (D2); the initial coordinates of the center point of ground truth box (d31) are (120, 20, 0.6); and the initial coordinates of the center point of ground truth box (d32) are (122, 20, 0.65). Both ground truth boxes (d31 and d32) correspond to detection category (D3).

[0073] The second 3D tensor (D×W×H) has a length H=100 and a width W=50, thus determining that the size ratio between the preset region of interest (W'×H') and the 2D tensor (W×H) is 2:1. The three detection categories (D1, D2, D3) are superimposed to form the tensor (D) in the second 3D tensor. Based on the size ratio, the initial coordinates of the center points of the four truth boxes, and the corresponding detection categories, the mapping coordinates of each truth box are determined. Specifically, if the truth box (d11) corresponds to the detection category (D1), then the center point of the truth box (d11) is mapped to the detection category (D1), and the mapped point (d11') obtained by mapping the truth box (d11) to the detection category (D1) has coordinates of (10, 40). If the truth box (d21) corresponds to the detection category (D2), then the center point of the truth box (d21) is mapped to the detection category (D2), and the mapped point (d21') obtained by mapping the truth box (d21) to the detection category (D2) has coordinates of (40, 15). If the truth box (d31) corresponds to the detection category (D3), then the center point of the truth box (d31) is mapped to the detection category (D3), and the mapped point (d31') obtained by mapping the truth box (d31) to the detection category (D3) has the coordinates (60, 10). If the truth box (d32) corresponds to the detection category (D3), then the center point of the truth box (d32) is mapped to the detection category (D3), and the mapped point (d32') obtained by mapping the truth box (d32) to the detection category (D3) has the coordinates (61, 10). The obtained mapped coordinates (10, 40), (40, 15), (60, 10), and (61, 10) are assigned values ​​respectively to obtain the truth features of the point cloud data of this frame.

[0074] As described above, a value is assigned to each of at least one mapped coordinate to obtain the true value feature. In one embodiment of this application, the initial coordinates include the x-axis coordinate of the center point, the y-axis coordinate of the center point, and the z-axis coordinate of the center point.

[0075] It is understandable that the center point of the truth box in three-dimensional space has three-dimensional coordinates, that is, the initial coordinates of the center point include the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the center point.

[0076] Assign values ​​to each of at least one mapped coordinate, including: assigning the truth box length, truth box width, truth box height, truth box orientation, center point z-axis coordinate, and the time interval between the labeled adjacent frame point cloud data and the current frame point cloud data.

[0077] It is understandable that after labeling any frame of point cloud data, at least one truth box can be obtained, and at the same time, the truth box length, truth box width, truth box height and truth box orientation of each truth box in at least one truth box can be obtained.

[0078] For example, Figure 7 This is a schematic diagram of the assignment scenario in the embodiments of this application, such as... Figure 7 As shown, each detection category (D1, D2, D3) corresponds to 6 elements: truth box length, truth box width, truth box height, truth box orientation, center point z-axis coordinate, and time interval. Continuing with the above... Figure 4 and Figure 6 The corresponding embodiment is used as an example for illustration. Suppose that the current frame point cloud data (b1) needs to be labeled. The current frame point cloud data (b1) is located between the labeled adjacent frame point cloud data (a1) and the labeled adjacent frame point cloud data (a2). The time interval between the labeled adjacent frame point cloud data (a1) and the current frame point cloud data (b1) is 1, and the time interval between the labeled adjacent frame point cloud data (a2) and the current frame point cloud data (b1) is -3.

[0079] Then, the four truth boxes (d11, d21, d31, d32) corresponding to the labeled adjacent frame point cloud data (a1) are mapped to obtain the mapped coordinates (10, 40) in the detection category (D1), (40, 15) in the detection category (D2), (60, 10) in the detection category (D3), and (61, 10) in the detection category (D3), respectively. Accordingly, the truth box length (2), width (1), height (1), orientation (0.1), center point z-axis coordinate (0.5), and time interval (1) of the truth box (d11) are assigned to the mapped coordinates (10, 40) within the detection category (D1). Similarly, the truth box length, width, height, orientation, center point z-axis coordinate, and time interval (1) of the other three truth boxes (d21, d31, d32) are assigned to the corresponding mapped coordinates, thereby obtaining the truth features of the labeled adjacent frame point cloud data (a1). Similarly, the truth features of the labeled adjacent frame point cloud data (a2) can be obtained.

[0080] In one embodiment of this application, labeling each frame in two adjacent point cloud data includes: inputting each frame in two adjacent point cloud data into a pre-labeling model to output at least one label box; and correcting the at least one label box to obtain at least one truth box.

[0081] Understandably, any frame of point cloud data is input into a pre-labeled model, which then annotates the point cloud data to output at least one bounding box. The pre-labeled model can be configured according to actual usage requirements and is not specifically limited. At least one bounding box is then corrected to obtain at least one ground truth box. For example, this can be done manually to ensure the accuracy of the output ground truth box, or other feasible correction methods are acceptable and are not specifically limited.

[0082] In one embodiment of this application, obtaining the detection box of the current frame point cloud data based on the fusion features further includes: correcting the detection box.

[0083] Understandably, after obtaining the detection bounding boxes of the current frame point cloud data, the detection bounding boxes can be corrected. For example, they can be corrected manually to ensure the accuracy of the output ground truth box, or other possible correction methods are acceptable and not specifically limited.

[0084] Please see Figure 8 , Figure 8 This is a flowchart illustrating the model training method in an embodiment of this application. The model training method includes the following steps:

[0085] S21. Obtain the detection box of the point cloud data annotation; wherein the detection box of the point cloud data annotation is obtained using the point cloud data annotation method in the above embodiment.

[0086] The detection boxes for point cloud data annotation are obtained through the point cloud data annotation method described in the above embodiments. For example, the current frame point cloud data to be annotated is obtained, wherein the current frame point cloud data is located between two adjacent annotated point cloud data in a series of consecutive point cloud data; the current frame point cloud data is rasterized to obtain the backbone features of the current frame point cloud data; at least one ground truth box obtained by annotating each frame in the two adjacent frame point cloud data is obtained, and the at least one ground truth box is processed to obtain the ground truth features of each frame in the two adjacent frame point cloud data; the backbone features of the current frame point cloud data and the ground truth features of the two adjacent frame point cloud data are concatenated and fused to obtain fused features, and the detection boxes of the current frame point cloud data are obtained based on the fused features.

[0087] S22. The model is trained using the detection boxes labeled with point cloud data to obtain the trained target model.

[0088] The model is trained using the detection boxes labeled with the obtained point cloud data, thus obtaining the trained target model.

[0089] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0090] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. The electronic device 900 includes a memory 901 and a processor 902 coupled to each other. The processor 902 is used to execute program instructions stored in the memory 901 to implement the steps in the above-described point cloud data annotation method embodiment or the steps in the above-described model training method embodiment. In a specific implementation scenario, the electronic device 900 may include, but is not limited to, a microcomputer or a server.

[0091] Specifically, processor 902 controls itself and memory 901 to implement the steps in the above-described point cloud data annotation method embodiments or the above-described model training method embodiments. Processor 902 can also be called a CPU (Central Processing Unit), and may be an integrated circuit chip with signal processing capabilities. Processor 902 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 902 can be implemented using integrated circuit chips.

[0092] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a non-volatile computer-readable storage medium in an embodiment of this application. The computer-readable storage medium 1000 is used to store program instructions 1001. When executed by the processor 902, the program instructions 1001 are used to implement the steps in the above-described point cloud data annotation method embodiment or the steps in the above-described model training method embodiment.

[0093] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed methods and related devices can be implemented in other ways. For example, the related device implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication disconnection shown or discussed may be indirect coupling or communication disconnection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A point cloud data annotation method, characterized in that, include: Obtain the current frame point cloud data to be labeled, wherein the current frame point cloud data is located between two adjacent labeled frame point cloud data in a series of consecutive frame point cloud data; The current frame point cloud data is rasterized to obtain the backbone features of the current frame point cloud data; At least one truth box is obtained by annotating each frame in the two adjacent frames of point cloud data, and the at least one truth box is processed to obtain the truth features of each frame in the annotated two adjacent frames of point cloud data, wherein the backbone features are represented by a first three-dimensional tensor, the truth features are represented by a second three-dimensional tensor, and the size of the two-dimensional tensor of the second three-dimensional tensor is equal to the size of the two-dimensional tensor of the first three-dimensional tensor. The backbone features of the current frame point cloud data and the ground truth features of the two adjacent frame point cloud data that have been labeled are spliced ​​and fused to obtain fused features. The detection box of the current frame point cloud data is then obtained based on the fused features to achieve labeling. The process of concatenating and fusing the backbone features of the current frame point cloud data and the ground truth features of the two adjacent frame point cloud data includes: Along the two-dimensional tensor of equal size, the backbone features of the current frame point cloud data and the ground truth features of the labeled two adjacent frame point cloud data are spliced ​​and fused.

2. The method according to claim 1, characterized in that, The continuous multi-frame point cloud data refers to the point cloud data in which each pair of adjacent frame point cloud data is separated by the same number of frames.

3. The method according to claim 1, characterized in that, The at least one truth box corresponds to at least one detection category; Processing the at least one truth box to obtain the truth features of the labeled point cloud data of the two adjacent frames includes: Obtain the size ratio between a preset region of interest in the three-dimensional space where the at least one truth box is located and the two-dimensional tensor of the second three-dimensional tensor; Based on the size ratio, the initial coordinates of the center point of each of the at least one truth box in the preset region of interest, and the detection category corresponding to each of the at least one truth box, the mapping coordinates of the center point of each of the at least one truth box in the second three-dimensional tensor are determined to obtain at least one of the mapping coordinates; The truth feature is obtained by assigning a value to each of at least one of the mapped coordinates.

4. The method according to claim 3, characterized in that, The initial coordinates include the x-axis coordinates, y-axis coordinates, and z-axis coordinates of the center point; Assigning a value to each of at least one of the mapped coordinates includes: Assign values ​​to the truth box's length, width, height, orientation, center point z-axis coordinate, and the time interval between the labeled adjacent frame point cloud data and the current frame point cloud data.

5. The method according to claim 2, characterized in that, Labeling is performed on each frame of the two adjacent point cloud data frames, including: Each frame of the two adjacent point cloud data is input into the pre-labeled model to output at least one label box; The at least one annotation box is modified to obtain the at least one truth box.

6. The method according to claim 1, characterized in that, The detection bounding box of the current frame point cloud data obtained based on the fusion features further includes: The detection frame is then modified.

7. A model training method, characterized in that, include: Obtain detection boxes for point cloud data annotation, wherein the detection boxes for data annotation are obtained using the point cloud data annotation method according to any one of claims 1-6; The model is trained using the detection boxes labeled with the point cloud data to obtain the trained target model.

8. An electronic device, characterized in that, It includes a memory and a processor that are coupled to each other, the processor being used to execute program instructions stored in the memory to implement the point cloud data annotation method as described in any one of claims 1-6 or the model training method as described in claim 7.

9. A non-volatile computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions, which, when executed by a processor, are used to implement the point cloud data annotation method as described in any one of claims 1-6 or the model training method as described in claim 7.

Images