Point cloud data labeling method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a point cloud data labeling method and device, electronic equipment and a storage medium, and relates to the technical field of automatic driving. The method comprises the following steps: acquiring a to-be-processed point cloud sequence; inputting the to-be-processed point cloud sequence into a labeling model to obtain an initial labeling result of the to-be-processed point cloud sequence output by the labeling model; correcting the initial labeling result to obtain a target labeling result corresponding to the to-be-processed point cloud sequence; wherein the labeling model is trained based on the following mode: obtaining an object point cloud, a wind sand point cloud and a plurality of first sample point clouds; for each first sample point cloud, adding the object point cloud and / or the sandstorm point cloud into the first sample point cloud to obtain a second sample point cloud; and training the initial labeling model based on the plurality of second sample point clouds, and taking the trained initial labeling model as a labeling model. According to the embodiment of the invention, the processing capability of the labeling model for the wind and sand scene is improved, and the labeling quality is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology. Specifically, the present disclosure relates to a point cloud data annotation method, device, electronic device and storage medium. Background Art

[0002] At present, perception systems usually rely on deep learning networks for environmental understanding and target recognition. The training of deep learning models usually requires a large amount of high-quality labeled data to ensure that the model can accurately perceive in complex and dynamic environments. This dependence places higher requirements on the scale and quality of the dataset for model training. The existing manual data labeling method requires a lot of human resources and time, which is inefficient and costly. Summary of the invention

[0003] The embodiments of the present disclosure provide a point cloud data annotation method, device, electronic device and storage medium, which can solve the problems of low efficiency and high cost of existing manual data annotation methods. The technical solutions provided by the present disclosure are as follows: According to one aspect of an embodiment of the present disclosure, a point cloud data annotation method is provided, the method comprising: Acquire a point cloud sequence to be processed; the point cloud sequence to be processed includes at least two consecutive frames of point clouds; Inputting the to-be-processed point cloud sequence into the annotation model, obtaining an initial annotation result of the to-be-processed point cloud sequence output by the annotation model; the initial annotation result includes initial category labels corresponding to each point in each frame of the point cloud; Correcting the initial annotation result to obtain a target annotation result corresponding to the point cloud sequence to be processed; The annotation model is trained based on the following method: Obtaining an object point cloud, a wind and sand point cloud, and a plurality of first sample point clouds; For each first sample point cloud, adding the object point cloud and / or the wind and sand point cloud to the first sample point cloud to obtain a second sample point cloud; The initial annotation model is trained based on the multiple second sample point clouds, and the trained initial annotation model is used as the annotation model.

[0004] Optionally, adding the object point cloud to the first sample point cloud comprises: Determining a candidate object position in the first sample point cloud based on a three-dimensional detection box corresponding to the object point cloud; If the candidate object position includes a road surface point, the target object position is determined based on the ground height; otherwise, the candidate object position is used as the target object position; If the candidate object position does not include a soil wall point, and the detection frame corresponding to the candidate object position does not collide with other detection frames in the first sample point cloud, the candidate object position is used as the target object position; Based on the target object position, the object point cloud is added to the first sample point cloud.

[0005] Optionally, adding the wind and sand point cloud to the first sample point cloud includes: Determine a candidate wind sand position in the first sample point cloud based on a two-dimensional detection frame corresponding to the wind sand point cloud; If the number of earth wall points included in the candidate wind and sand positions is not greater than the first number, and the number of vehicle points is not greater than the second number, determining the target wind and sand position corresponding to the wind and sand point cloud based on the candidate wind and sand positions; Based on the target wind sand position, the wind sand point cloud is added to the first sample point cloud.

[0006] Optionally, the adding the object point cloud and the wind and sand point cloud to the first sample point cloud to obtain a second sample point cloud further includes: Acquire at least one three-dimensional detection box in the second sample point cloud; For each three-dimensional detection frame in the second sample point cloud, the sandstorm points in the three-dimensional detection frame are deleted.

[0007] Optionally, obtaining the object point cloud includes: Acquire a first initial point cloud sequence; the first initial point cloud sequence includes at least two consecutive frames of first initial point clouds; extracting at least one dynamic obstacle from the first initial point cloud sequence; For each dynamic obstacle, if the distance between the dynamic obstacle and its corresponding associated wind-blown sand group is less than a preset distance threshold, the point cloud of the dynamic obstacle and the point cloud of the associated wind-blown sand group are taken as a first object point cloud; otherwise, the point cloud of the dynamic obstacle is taken as a second object point cloud; The associated sand group is the sand group closest to the dynamic obstacle.

[0008] Optionally, obtaining a sandstorm point cloud includes: Acquire a second initial point cloud sequence; the second initial point cloud sequence includes at least two consecutive frames of second initial point clouds; extracting at least one wind-blown sand mass from the second initial point cloud sequence; For each sand group, if the distance between the sand group and the vehicle is less than a preset distance threshold, the sand group is regarded as a sand group close to the vehicle; otherwise, the sand group is regarded as a sand group far from the vehicle; wherein the vehicle is the vehicle that obtains the second initial point cloud sequence; The wind-sand point cloud is determined based on the wind-sand group close to the vehicle and the wind-sand group far from the vehicle.

[0009] Optionally, the modifying the initial annotation result includes: Splicing multiple frames of point cloud in the to-be-processed point cloud sequence to obtain a spliced ​​point cloud sequence; Selecting at least one isolated point from the stitched point cloud sequence; For each isolated point, a neighborhood range corresponding to the isolated point is determined; based on a plurality of neighborhood points in the neighborhood range, the category of the isolated point is corrected, and the corrected category is used as a target category label corresponding to the isolated point.

[0010] Optionally, the method further comprises: For each frame of point cloud, determining at least one road surface point, at least one non-ignored category point, and at least one ignored category point from the point cloud based on the target annotation result; voxelize the at least one road surface point to obtain at least one first voxel grid; for each first voxel grid, if the standard deviation of the height coordinates corresponding to each road surface point in the first voxel grid is less than a preset threshold, use the voxel grid as a road surface grid; determine the target road surface points corresponding to the at least one road surface grid; voxelize the at least one non-ignored category point to obtain a grid center point corresponding to at least one second voxel grid; For each grid center point, searching for a plurality of adjacent road surface points corresponding to the grid center point from the at least one target road surface point, performing road surface fitting on the plurality of adjacent road surface points, and obtaining road surface parameters corresponding to the fitted road surface; Based on at least one road surface parameter, the heights corresponding to the respective points in the point cloud are calculated.

[0011] According to another aspect of an embodiment of the present disclosure, a point cloud data annotation device is provided, the device comprising: A data acquisition module, used to acquire a point cloud sequence to be processed; the point cloud sequence to be processed includes at least two consecutive frames of point clouds; A pre-annotation module is used to input the to-be-processed point cloud sequence into an annotation model to obtain an initial annotation result of the to-be-processed point cloud sequence output by the annotation model; the initial annotation result includes initial category labels corresponding to each point in each frame of the point cloud; A correction module, used to correct the initial annotation result to obtain a target annotation result corresponding to the point cloud sequence to be processed; The annotation model is trained based on the following method: Obtaining an object point cloud, a wind and sand point cloud, and a plurality of first sample point clouds; For each first sample point cloud, adding the object point cloud and / or the wind and sand point cloud to the first sample point cloud to obtain a second sample point cloud; The initial annotation model is trained based on the multiple second sample point clouds, and the trained initial annotation model is used as the annotation model.

[0012] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of any one of the above-mentioned point cloud data labeling methods when executing the program.

[0013] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned point cloud data labeling methods are implemented.

[0014] According to one aspect of an embodiment of the present disclosure, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned point cloud data labeling methods are implemented.

[0015] The technical solution provided by the embodiments of the present disclosure brings the following beneficial effects: The target annotation results corresponding to the point cloud sequence to be processed are obtained through the annotation model, which reduces the cost of manual annotation, improves the annotation efficiency, avoids the influence of the subjectivity of manual annotation on the annotation quality, and improves the annotation quality; based on the object point cloud and the wind and sand point cloud, the first sample point cloud after data enhancement is used as the second sample point cloud, so that the second sample point cloud includes rich information on object categories and wind and sand categories; based on the second sample point cloud, the recognition ability of object categories and wind and sand categories in the initial annotation model is trained in a targeted manner, which improves the accuracy of the trained annotation model in identifying object categories and wind and sand categories, improves the processing ability of the annotation model for wind and sand scenes, and further improves the annotation quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for describing the embodiments of the present disclosure are briefly introduced below.

[0017] Figure 1A schematic diagram of a process of a point cloud data annotation method provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of a process of adding an object point cloud provided in an embodiment of the present disclosure; Figure 3 A schematic diagram of adding a wind and sand point cloud provided in an embodiment of the present disclosure; Figure 4 A schematic diagram of a process for extracting object point cloud provided by an embodiment of the present disclosure; Figure 5 A schematic diagram of a process for extracting a sand point cloud provided by an embodiment of the present disclosure; Figure 6 A schematic diagram of a process for calculating height provided in an embodiment of the present disclosure; Figure 7 A schematic diagram of a process flow of another point cloud data annotation method provided in an embodiment of the present disclosure; Figure 8 A schematic diagram of the structure of a point cloud data annotation device provided in an embodiment of the present disclosure; Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] The embodiments of the present disclosure are described below in conjunction with the drawings in the present disclosure. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions of the embodiments of the present disclosure.

[0019] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present disclosure refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" or "A, B" indicates that it is implemented as "A", or implemented as "B", or implemented as "A and B".

[0020] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0021] In recent years, autonomous driving technology has gradually become a research hotspot in the automotive field. As the top priority in the field of autonomous driving, perception technology can help vehicles understand and make decisions about the surrounding environment based on the environmental information around the vehicle obtained by sensors and combined with preset algorithms, and complete the path planning and vehicle behavior control of the vehicle's autonomous driving. The BEV (Bird's Eye View) perception model based on the bird's-eye view fusion of multi-camera image features around the vehicle body is the mainstream of current vehicle perception technology.

[0022] In open-pit mines, due to extreme weather conditions such as dust, rain and fog, existing perception systems usually rely on deep learning networks for environmental understanding and target recognition. The training of deep learning models usually requires a large amount of high-quality annotated data to ensure that the model can accurately perceive in complex and dynamic environments. This dependence places higher requirements on the scale and quality of the dataset for model training.

[0023] However, the traditional manual data labeling method requires a lot of human resources and time, is inefficient and costly, especially in open-pit mines, which are vast and have ever-changing environments. In addition, the subjective judgment differences between different labelers may lead to consistency issues in the labeling results. This subjectivity not only increases the variability of the labeling results, but may also directly affect the accuracy and reliability of the model, thereby reducing the effectiveness of the perception system in practical applications.

[0024] The point cloud data annotation method, device, electronic device and storage medium provided by the present invention are intended to solve the above technical problems in the prior art.

[0025] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present disclosure and the technical effects produced by the technical solutions of the present disclosure. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0026] Figure 1 A schematic diagram of a point cloud data annotation method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the method includes: Step S110, obtaining a point cloud sequence to be processed; the point cloud sequence to be processed includes at least two consecutive frames of point clouds.

[0027] Specifically, the point cloud sequence to be processed may be a point cloud sequence that needs to be labeled. After being labeled, the point cloud sequence to be processed may be used to train the BEV perception model. The point cloud sequence to be processed may include at least two consecutive frames of point clouds.

[0028] The point cloud sequence to be processed can be acquired by laser radar. For example, the laser radar can be set on a vehicle, and the point cloud sequence to be processed can be acquired by the laser radar carried by the vehicle while the vehicle is driving.

[0029] Step S120, inputting the to-be-processed point cloud sequence into the annotation model, and obtaining the initial annotation result of the to-be-processed point cloud sequence output by the annotation model; the initial annotation result includes initial category labels corresponding to each point in each frame of the point cloud; Among them, the annotation model is trained based on the following method: Obtaining an object point cloud, a wind and sand point cloud, and a plurality of first sample point clouds; For each first sample point cloud, adding the object point cloud and / or the wind and sand point cloud to the first sample point cloud to obtain a second sample point cloud; The initial annotation model is trained based on the multiple second sample point clouds, and the trained initial annotation model is used as the annotation model.

[0030] Specifically, after obtaining the point cloud sequence to be processed, the point cloud sequence to be processed can be input into a trained annotation model, and the annotation model is used to predict the categories corresponding to each point in the point cloud sequence to be processed, so as to obtain the initial annotation results corresponding to the point cloud to be processed.

[0031] Among them, the initial annotation results may include the initial category labels corresponding to each point in the point cloud sequence to be processed. For the open-pit mine scene, the initial category labels may include road surface points, earth wall points, vehicle points, and wind and sand points, etc.; for the scene of severe weather in the city, the initial category labels may include road surface points, building points, vehicle points, pedestrian points, and wind and sand points, etc., which are not limited to the embodiments of the present disclosure.

[0032] Before executing step S120, the annotation model may be trained in advance. The training process of the annotation model may include: First, multiple first sample point clouds may be obtained. One first sample point cloud may be a frame of point cloud. For example, in an open-pit mine scene, multiple first sample point clouds may be point cloud data collected by observing the open-pit mine scene. Multiple first sample point clouds may include manually annotated category labels.

[0033] Object point clouds and wind and sand point clouds can be obtained. The object point cloud can be point cloud data of the dynamic obstacle category. The dynamic obstacles can include vehicles, pedestrians, etc. The wind and sand point cloud can be point cloud data of the wind and sand category.

[0034] The object point cloud and the wind and sand point cloud can be obtained from an existing laser point cloud database, or extracted from a collected point cloud sequence. When extracting the object point cloud and the wind and sand point cloud from the collected point cloud sequence, they can be directly extracted from multiple first sample point clouds, or they can be extracted from other point cloud sequences. The point cloud sequence of the object point cloud and the wind and sand point cloud to be extracted and the multiple first sample point clouds can belong to the same application scenario, or they can belong to different application scenarios. The disclosed embodiments do not limit the specific method of obtaining the object point cloud and the wind and sand point cloud.

[0035] On this basis, for each first sample point cloud, the object point cloud and / or the wind and sand point cloud can be added to the first sample laser point cloud to obtain a second sample laser point cloud.

[0036] Optionally, for any first sample point cloud, at least one object point cloud can be added to the first sample point cloud, at least one wind and sand point cloud can be added to the first sample point cloud, or at least one object point cloud and at least one wind and sand point cloud can be added to the first sample point cloud at the same time, which is not limited to the embodiments of the present disclosure.

[0037] After obtaining a plurality of second sample laser point clouds, the initial annotation model can be trained based on the plurality of second sample point clouds, and the trained initial annotation model can be used as the annotation model. The initial annotation model can be constructed based on a point prompt training (PPT) network.

[0038] Optionally, multiple second sample point clouds and a large number of publicly available sample point clouds can be input into the initial annotation model, and the categories corresponding to each point in each frame of each sample point cloud are predicted through the initial annotation model. Based on the predicted categories and manually labeled categories corresponding to each point in each frame of each sample point cloud, a loss function is determined, and the network parameters in the initial annotation model are optimized based on the loss function until the training end conditions are met, and the initial annotation model that meets the training end conditions is used as the trained annotation model.

[0039] Among them, the training end condition may be the convergence of the loss function, for example, the loss function is less than a set value or the loss function calculated for a set number of consecutive times is less than the set value; the training end condition may also be that the number of training times reaches a preset number, which is not limited in the embodiments of the present disclosure.

[0040] Step S130, correcting the initial annotation result to obtain the target annotation result corresponding to the point cloud sequence to be processed.

[0041] Specifically, after obtaining the initial annotation result corresponding to the point cloud sequence to be processed output by the annotation model, the initial category label corresponding to at least one point in the initial annotation result can be corrected, and the category labels corresponding to each corrected point can be used as the target category labels, thereby obtaining the target annotation result corresponding to the point cloud sequence to be processed.

[0042] In the disclosed embodiment, a target annotation result corresponding to a point cloud sequence to be processed is obtained through an annotation model, thereby reducing the cost of manual annotation, improving annotation efficiency, avoiding the influence of the subjectivity of manual annotation on the annotation quality, and improving the annotation quality; data enhancement is performed on the first sample point cloud based on the object point cloud and the wind and sand point cloud, and the first sample point cloud after data enhancement is used as the second sample point cloud, so that the second sample point cloud includes rich information on object categories and wind and sand categories; based on the second sample point cloud, the recognition ability of the object category and the wind and sand category in the initial annotation model is trained in a targeted manner, thereby improving the accuracy of the trained annotation model in identifying the object category and the wind and sand category, improving the processing ability of the annotation model for wind and sand scenes, and further improving the annotation quality.

[0043] As an optional embodiment, adding the object point cloud to the first sample point cloud includes: Determine the position of the candidate object in the first sample point cloud based on the three-dimensional detection frame corresponding to the object point cloud; If the candidate object position includes a road surface point, the target object position is determined based on the ground height; otherwise, the candidate object position is used as the target object position; If the candidate object position does not include the earth wall point, and the detection frame corresponding to the candidate object position does not collide with other detection frames in the first sample point cloud, the candidate object position is used as the target object position; Based on the target object position, the object point cloud is added to the first sample point cloud.

[0044] Specifically, Figure 2 A schematic diagram of a process of adding an object point cloud provided in an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, in order to add the object point cloud to a reasonable position in the first sample point cloud, the object point cloud to be added and the first sample point cloud to be enhanced can be determined, and the three-dimensional detection box corresponding to the object point cloud can be determined, and the corresponding position of the three-dimensional detection box of the object point cloud in the first sample point cloud is used as the candidate object position corresponding to the object point cloud.

[0045] Among them, the three-dimensional detection box of the object point cloud can be expressed as gt_box3d (Ground Truth 3D BoundingBox).

[0046] It should be noted that the position representations in the object point cloud to be added and the first sample point cloud to be enhanced are located in the same coordinate system. The categories of each point in the first sample point cloud are manually marked.

[0047] To ensure the rationality of the candidate object position, the candidate object position needs to be further screened. It can be determined whether the area of ​​the candidate object position in the first sample point cloud includes road points. If no road points are included, the candidate object position can be used as the target object position, and the object point cloud can be pasted on the target object position. If the road points are included, the height of the object from the ground can be calculated, and based on the ground height, the lowest point of the candidate object position can be moved to the ground, thereby avoiding the object from being suspended in the air.

[0048] Optionally, the height of the object from the ground may be determined based on the difference between the lowest point of the candidate object position (ie, the point with the smallest height coordinate Z) and the height coordinate Z of the road surface point.

[0049] It is also possible to determine whether the region of the candidate object position in the first sample point cloud includes earth wall points. If the region includes earth wall points, but the object usually does not appear on the earth wall, the object point cloud is not added to the first sample point cloud.

[0050] If the earth wall point is not included, a collision detection can be performed based on the detection frame corresponding to the candidate object position and other known detection frames in the first sample point cloud. If the detection frame corresponding to the candidate object position does not collide with other detection frames, the candidate object position is used as the target object position and the object point cloud is pasted on the target object position. If it is detected that the detection frame corresponding to the candidate object position collides with other detection frames, the object point cloud is not added to the first sample point cloud to avoid pasting the object point cloud on the position of the original object in the first sample point cloud.

[0051] Among them, the detection frame for collision detection can be a two-dimensional detection frame, and the detection frame corresponding to the candidate object position can transform the three-dimensional detection frame of the object point cloud, for example, adjusting the height coordinates of each point in the three-dimensional detection frame (such as height coordinate * 100), thereby simulating the corresponding two-dimensional detection frame.

[0052] Among them, the earth wall point in the mining scene can represent a wall or retaining structure made of soil, gravel, rock or similar materials, and the earth wall point in the urban scene can represent a building in the city.

[0053] It should be noted that, when multiple object point clouds are added to the first sample point cloud, the above steps may be performed for each object point cloud to ensure that the multiple object point clouds are pasted to reasonable positions in the first sample point cloud.

[0054] After adding at least one object point cloud to the first sample point cloud, the original point clouds in the areas of the target object positions corresponding to the object point clouds can be deleted, and then the object point clouds can be pasted to the corresponding target object positions to prevent interference with the original point clouds in the first sample point cloud.

[0055] In the disclosed embodiment, the target object position is determined by screening the candidate object positions in the first sample point cloud, and then the object point cloud is pasted at a reasonable position in the first sample point cloud, so that the enhanced data is more in line with the actual situation of the application scenario, ensuring the quality of the training samples and further ensuring the training effect of the annotation model.

[0056] As an optional embodiment, adding the wind and sand point cloud to the first sample point cloud includes: Determine the candidate wind sand position in the first sample point cloud based on the two-dimensional detection frame corresponding to the wind sand point cloud; If the number of earth wall points included in the candidate wind and sand positions is not greater than the first number, and the number of vehicle points is not greater than the second number, determining the target wind and sand position corresponding to the wind and sand point cloud based on the candidate wind and sand positions; Based on the target wind sand position, the wind sand point cloud is added to the first sample point cloud.

[0057] Specifically, Figure 3 A schematic diagram of adding a sand point cloud provided by an embodiment of the present disclosure, such as Figure 3 As shown, in order to determine the position where the wind and sand point cloud is added to the first sample point cloud, the wind and sand point cloud to be added and the first sample point cloud to be enhanced can be determined, and the two-dimensional detection frame corresponding to the wind and sand point cloud can be determined, and the corresponding position of the two-dimensional detection frame of the wind and sand point cloud in the first sample point cloud can be used as the candidate wind and sand position corresponding to the wind and sand point cloud.

[0058] Optionally, the minimum bounding rectangle of each sand point in the sand point cloud may be determined, and the minimum bounding rectangle may be used as the corresponding two-dimensional detection frame.

[0059] After obtaining the candidate wind and sand position, the points in the candidate wind and sand position in the first sample point cloud can be extracted. If it is detected that the number of earth wall points included in the area of ​​the candidate wind and sand position in the first sample point cloud is greater than the first number, or the number of vehicle points included is greater than the second number, it means that the area of ​​the candidate wind and sand position is likely to be the area where the earth wall or the vehicle is located, then the wind and sand point cloud will not be added to the first sample point cloud.

[0060] The first number and the second number can be specifically set according to the actual application scenario, and the first number and the second number can have the same value or different values, which is not limited in the embodiment of the present disclosure. For example, the first number can be set to 100 and the second number can also be 100.

[0061] If it is detected that the number of earth wall points included in the area of ​​the candidate wind and sand position in the first sample point cloud is not greater than the first number, and the number of vehicle points included is not greater than the second number, then the target wind and sand position corresponding to the wind and sand point cloud can be determined based on the candidate wind and sand position, and the wind and sand point cloud can be added to the target wind and sand position.

[0062] Optionally, after obtaining the candidate wind and sand position, it is also possible to determine whether the area of ​​the candidate wind and sand position in the first sample point cloud includes a road surface point. If it includes a road surface point, the ground height of the wind and sand group corresponding to the wind and sand point cloud from the ground can be calculated, and based on the ground height, the wind and sand point cloud can be moved to a preset height range from the ground (for example, 0-0.5 meters) to simulate the actual wind and sand situation.

[0063] If the road surface point is not included, the candidate wind sand position may be used as the target wind sand position, or when processing the near-ground wind sand situation, the wind sand point cloud may not be added to the first sample point cloud.

[0064] It should be noted that, for any first sample point cloud, at least one object point cloud and / or at least one wind and sand point cloud can be added to the first sample point cloud; for different first sample point clouds, the number and categories of added object point clouds or wind and sand point clouds can be the same or different, so as to improve the randomness of data enhancement, which can not only expand the number of training samples, but also improve the richness of training samples, which is conducive to improving the training effect of the annotation model.

[0065] In the disclosed embodiment, the target wind and sand position is determined by screening the candidate wind and sand positions in the first sample point cloud, and then the wind and sand point cloud is pasted at a reasonable position in the first sample point cloud to simulate the actual wind and sand situation, so that the enhanced data is more in line with the actual situation of the application scenario, ensuring the quality of the training samples and further ensuring the training effect of the annotation model.

[0066] As an optional embodiment, the object point cloud and the wind and sand point cloud are added to the first sample point cloud to obtain a second sample point cloud, and then the method further includes: Acquire at least one three-dimensional detection frame in the second sample point cloud; For each 3D detection frame in the second sample point cloud, the sand points in the 3D detection frame are deleted.

[0067] Specifically, after performing data enhancement on the first sample point cloud based on the object point cloud and the wind and sand point cloud to obtain the second sample point cloud, at least one three-dimensional detection frame in the second sample point cloud can be obtained. The three-dimensional detection frame is usually for an object. Therefore, for each three-dimensional detection frame in the second sample point cloud, the wind and sand points in the three-dimensional detection frame can be deleted to prevent mutual influence between the wind and sand points and the object points, thereby further improving the quality of the training samples.

[0068] As an optional embodiment, obtaining an object point cloud includes: Acquire a first initial point cloud sequence; the first initial point cloud sequence includes at least two consecutive frames of first initial point clouds; extracting at least one dynamic obstacle from the first initial point cloud sequence; For each dynamic obstacle, if the distance between the dynamic obstacle and its corresponding associated wind-blown sand group is less than a preset distance threshold, the point cloud of the dynamic obstacle and the point cloud of the associated wind-blown sand group are taken as the first object point cloud; otherwise, the point cloud of the dynamic obstacle is taken as the second object point cloud; Among them, the associated wind and sand group is the wind and sand group closest to the dynamic obstacle.

[0069] Specifically, to obtain the object point cloud, a first initial point cloud sequence can be obtained. The first initial point cloud sequence can be point cloud data used to extract the object point cloud. The first initial point cloud sequence can include at least two consecutive frames of the first initial point cloud. The first initial point cloud sequence includes category labels corresponding to each point.

[0070] After the first initial point cloud sequence is obtained, the first initial point cloud sequence may be processed to extract at least one dynamic obstacle in the first initial point cloud sequence.

[0071] It should be noted that different vehicles or different pedestrians correspond to different types of dynamic obstacles.

[0072] On this basis, at least one wind sand group in the first initial point cloud of each frame in the first initial point cloud sequence can be extracted. For each dynamic obstacle, the wind sand group closest to the dynamic obstacle is taken as an associated wind sand group. If the distance between the associated wind sand group and the dynamic obstacle is less than a preset distance threshold (for example, 10 meters), it means that the associated wind sand group is likely to be generated by the movement of the dynamic obstacle, such as the wind and sand raised around the vehicle when the vehicle is driving. The point cloud of the dynamic obstacle and the point cloud of the associated wind sand group corresponding to it are taken as the first object point cloud. That is to say, the first object point cloud includes a point cloud of a dynamic obstacle and a point cloud of wind and sand around it.

[0073] If the distance between the associated wind-blown sand group and the dynamic obstacle is not less than a preset distance threshold, the point cloud of the dynamic obstacle is used as the second object point cloud.

[0074] After obtaining at least one first object point cloud and at least one second object point cloud, data enhancement is performed on the first sample point cloud by performing data enhancement on the at least one first object point cloud and / or the at least one second object point cloud.

[0075] Figure 4A schematic diagram of a process for extracting object point cloud provided by an embodiment of the present disclosure, such as Figure 4 As shown, the object identifier corresponding to each point in the first initial point cloud of each frame in the first initial point cloud sequence can be obtained, and all the object identifiers that appear and are not repeated can be obtained. One object has an object identifier corresponding to it; for each object identifier, the object detection frame corresponding to the object identifier in each frame of the point cloud is obtained, and multiple object points in the object detection frame in each frame of the point cloud are extracted. For each object identifier, if it is detected that the object detection frame of the object identifier is included in the first initial point cloud of a continuous preset number of frames (for example, 5 consecutive frames), and the average number of object points in the first initial point cloud of the preset number of frames is greater than the preset number of object points (for example, 100), then the object corresponding to the object identifier is identified as a dynamic obstacle, and the point cloud corresponding to the object identifier is extracted to obtain the point cloud corresponding to the dynamic obstacle.

[0076] Optionally, for each dynamic obstacle, the point cloud of the dynamic obstacle may be determined based on point clouds in each object detection frame of the dynamic obstacle in a first initial point cloud of a preset number of consecutive frames.

[0077] like Figure 4 As shown, on the basis of executing the above steps, the sand points in the first initial point cloud sequence can be clustered to obtain multiple candidate sand clusters, and the candidate sand cluster with a point number greater than a preset sand point number (for example, 1000) can be extracted as a valid sand cluster.

[0078] In the disclosed embodiment, by acquiring a first initial point cloud sequence, at least one dynamic obstacle in the first initial point cloud sequence is extracted, and by clustering the wind and sand around the dynamic obstacles, a first object point cloud including the dynamic obstacles and their corresponding associated wind and sand groups, and a second object point cloud including only the dynamic obstacles are obtained. By distinguishing different object point clouds, the point cloud situation around the moving object can be better simulated when data enhancement is performed based on the object point cloud, thereby further improving the quality of the training samples and facilitating improving the training effect of the annotation model.

[0079] As an optional embodiment, obtaining a sandstorm point cloud includes: Acquire a second initial point cloud sequence; the second initial point cloud sequence includes at least two consecutive frames of second initial point clouds; extracting at least one wind-blown sand cluster from the second initial point cloud sequence; For each sand group, if the distance between the sand group and the ego vehicle is less than a preset distance threshold, the sand group is regarded as a sand group close to the ego vehicle; otherwise, the sand group is regarded as a sand group far from the ego vehicle; wherein the ego vehicle is the vehicle that obtains the second initial point cloud sequence; The sand point cloud is determined based on the sand clusters near the vehicle and the sand clusters far from the vehicle.

[0080] Specifically, Figure 5 A schematic diagram of a process for extracting a sandstorm point cloud provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, in order to obtain a wind-blown sand point cloud, a second initial point cloud sequence can be obtained. The second initial point cloud sequence can be point cloud data used to extract the wind-blown sand point cloud. The second initial point cloud sequence can include at least two consecutive frames of second initial point clouds. The second initial point cloud sequence includes category labels corresponding to each point.

[0081] It should be noted that the first initial point cloud sequence and the second initial point cloud sequence may be the same data or different data, which is not limited in the embodiment of the present disclosure.

[0082] After obtaining the second initial point cloud sequence, for each frame of the second initial point cloud, multiple sand points in the second initial point cloud can be extracted, and at least one initial sand group can be obtained by clustering the multiple sand points, for example, using the DBSCAN algorithm (Density-Based Spatial Clustering of Applications with Noise, density-based noise-resistant clustering method). Each initial sand group in each frame of the second initial point cloud is processed (for example, multiple initial sand groups at the same position in each frame of the second initial point cloud are averaged) to obtain at least one candidate sand group. For each candidate sand group, if the number of points included in the candidate sand group is greater than the preset number of sand points (for example, greater than 1000), the candidate sand group is regarded as a valid sand group; otherwise, the candidate sand group is not used to generate the sand point cloud.

[0083] After obtaining at least one sand group, for each sand group, the distance between the sand group and the vehicle can be calculated. If the distance between the sand group and the vehicle is less than a preset distance threshold (for example, less than 10 meters), it means that the sand group is likely to be generated by the vehicle during its movement, and the sand group is regarded as a sand group near the vehicle; otherwise, the sand group is regarded as a sand group far from the vehicle. The vehicle can be understood as the vehicle that collects the second initial point cloud sequence, that is, the second initial point cloud sequence does not include the vehicle point corresponding to the vehicle.

[0084] After classifying at least one wind sand group, the wind sand group near the vehicle and the wind sand group far from the vehicle can be combined to generate a wind sand point cloud. For example, the wind sand group near the vehicle and the wind sand group far from the vehicle can be fused according to a certain preset ratio, and the fused wind sand group is used as the wind sand point cloud.

[0085] In the disclosed embodiment, at least one sand cluster is extracted from the second initial point cloud sequence, and divided into sand clusters near the vehicle and sand clusters far from the vehicle according to the distance between each sand cluster and the vehicle. The sand point cloud is generated by combining the sand clusters near the vehicle and the sand clusters far from the vehicle, thereby increasing the richness of the sand point cloud and improving the randomness of data enhancement, which is beneficial to improving the quality of training samples and the training effect of the annotation model.

[0086] As an optional embodiment, the initial annotation result is corrected, including: Splicing multiple frames of point cloud in the point cloud sequence to be processed to obtain a spliced ​​point cloud sequence; Select at least one isolated point from the spliced ​​point cloud sequence; For each isolated point, the neighborhood range corresponding to the isolated point is determined; the category of the isolated point is corrected based on multiple neighboring points in the neighborhood range, and the corrected category is used as the target category label corresponding to the isolated point.

[0087] Specifically, the initial annotation results include the annotation results corresponding to each frame of point cloud output by the annotation model. For a single frame of point cloud, when there are isolated points (i.e., outliers) in the single frame of point cloud, the annotation model may not accurately predict the category of the isolated points. In addition, there may be a situation where the same point has inconsistent categories in two consecutive frames of point cloud. Therefore, in order to ensure the accuracy of the annotation results, the initial annotation results can be further corrected.

[0088] Multiple frames of point clouds in the point cloud sequence to be processed can be spliced ​​together to obtain an overall spliced ​​point cloud sequence.

[0089] Optionally, for each frame of point cloud, the pose information of each frame of point cloud can be used to transform it from the local coordinate system to the global (world) coordinate system. Through matrix transformation, the coordinates of each point are applied to the corresponding pose matrix so that each frame of point cloud is in the same coordinate system. All the converted point cloud data are spliced ​​together, and the spliced ​​point cloud data set is used as a spliced ​​point cloud sequence. The spliced ​​point cloud sequence is a more complete and denser point cloud data set, which helps to improve the quality of the overall point cloud and reduce the number of isolated points.

[0090] After obtaining the spliced ​​point cloud sequence, at least one isolated point can be extracted from the spliced ​​point cloud sequence. For example, a neighborhood range can be set, and if the number of points contained in the neighborhood range of the point is less than a preset number of points, the point is regarded as an isolated point.

[0091] For each isolated point, a neighborhood range can be set, and all neighborhood points in the neighborhood range corresponding to the isolated point can be obtained. According to the categories corresponding to each neighborhood point, the category of the isolated point is corrected, and the corrected category is used as the target category label of the isolated point.

[0092] Optionally, the category of the isolated point can be corrected based on a voting mechanism. For example, the category that appears most frequently among all the neighboring points can be used as the corrected category of the isolated point. Alternatively, corresponding weights can be set based on the distance between each neighboring point and the isolated point. For example, the closer the distance, the greater the weight. Then, the corrected category of the isolated point is determined based on the category of each neighboring point and its corresponding weight.

[0093] In the disclosed embodiment, a spliced ​​point cloud sequence is obtained by splicing multiple frames of point clouds in a point cloud sequence to be processed, and the category of each isolated point in the spliced ​​point cloud sequence is corrected, thereby eliminating the influence of the labeling model's poor reasoning effect on isolated points due to insufficient single-frame information, solving the problem of inconsistent categories between multiple-frame point clouds, improving the accuracy of the category labels of isolated points, and improving the accuracy and reliability of the labeled point clouds, which is conducive to improving the training effect of the BEV perception model trained based on these point clouds.

[0094] As an optional embodiment, the method further includes: For each frame of point cloud, determining at least one road surface point, at least one non-ignored category point, and at least one ignored category point from the point cloud based on the target annotation result; voxelize the at least one road surface point to obtain at least one first voxel grid; for each first voxel grid, if the standard deviation of the height coordinates corresponding to each road surface point in the first voxel grid is less than a preset threshold, use the voxel grid as a road surface grid; determine the target road surface points corresponding to the at least one road surface grid; voxelize the at least one non-ignored category point to obtain a grid center point corresponding to at least one second voxel grid; For each grid center point, searching for a plurality of adjacent road surface points corresponding to the grid center point from the at least one target road surface point, performing road surface fitting on the plurality of adjacent road surface points, and obtaining road surface parameters corresponding to the fitted road surface; Based on at least one road surface parameter, the heights corresponding to the respective points in the point cloud are calculated.

[0095] Specifically, after obtaining the target annotation result of the point cloud sequence to be processed, the height information of each point in the point cloud sequence to be processed can be further calculated.

[0096] For each frame of point cloud in the point cloud sequence to be processed, based on the target category labels of each point in the frame of point cloud, at least one point of the road surface point with the target category label in the frame of point cloud can be used as at least one road surface point for subsequent road surface fitting. At the same time, based on the target category labels of each point in the frame of point cloud, each point can be divided into at least one non-ignored category point and at least one ignored category point.

[0097] Among them, ignored category points may include isolated points. For example, when splicing the processed point cloud sequence, there are still some points in the spliced ​​point cloud sequence with fewer points around the points. It may be because the number of neighborhood points is small that the categories of these discrete points cannot be corrected, so these points are still isolated points. In this case, these points can be classified as ignored category points. All other points in the frame point cloud except the ignored category points are classified as non-ignored category points, that is, non-ignored category points also include road surface points.

[0098] After obtaining at least one road surface point, the at least one road surface point may be voxelized to obtain at least one first voxel grid.

[0099] The specific process of voxelization can include: Assuming a frame of point cloud is given, represented as p=[x, y, z, label] T , where x, y, z are the coordinates of the point in the LiDAR coordinate system, and label represents the category of the point. First, get the minimum coordinate min_xy of the point cloud of the frame to determine the minimum boundary of the point cloud. Then, according to the minimum coordinate and the specified voxel size, convert the coordinates of each point into a voxel index, thereby mapping the two-dimensional coordinates of the point cloud to the voxel grid. Subsequently, find all unique voxel centers and return the voxel center index corresponding to each point, so that the point cloud data is effectively divided by voxels.

[0100] For each first voxel grid, the standard deviation of the height coordinate can be calculated based on the height coordinate Z of each road surface point in the first voxel grid. If the standard deviation is less than the preset threshold, the first voxel grid is regarded as a valid road surface grid; if the standard deviation is not less than the preset threshold, it means that the thickness of the corresponding road surface is thick, and all road surface points in the first voxel grid are not used for road surface fitting, so as to avoid the inability to accurately fit the correct road surface parameters due to an overly thick road surface, and effectively screen out reasonable road surface columns.

[0101] After at least one road surface grid is screened out, for each road surface grid, all road surface points in the road surface grid may be downsampled to obtain a target road surface for representing the road surface grid.

[0102] At the same time, at least one non-ignored category point may be voxelized to obtain at least one second voxel grid and a grid center point corresponding to each second voxel grid, wherein the second voxel grid is a non-empty voxel grid.

[0103] For each grid center point of the second voxel grid, multiple adjacent road surface points corresponding to the grid center point can be searched from multiple target road surface points. For example, a K-nearest neighbor search can be used to search for target road surface points within 5 meters from the grid center point in the xy plane as multiple adjacent road surface points. By performing road surface fitting on multiple adjacent road surface points, road surface parameters corresponding to the fitted road surface are obtained.

[0104] Optionally, the road surface parameters corresponding to the fitted road surface may include the center point of the road surface and the normal vector of the road surface.

[0105] For example, the mean of the input point cloud data can be calculated by mean=np.mean(points, axis=0), where np.mean() is a function in the NumPy library (an open source library) that is used to calculate the mean of an array, points represents the point cloud data set to be calculated, and axis=0 represents averaging along the column direction. When points is set to multiple road surface points in the fitted road surface, the average position of all points can be calculated, which is the center point of the road surface.

[0106] For another example, for multiple road surface points of the fitted road surface, the covariance matrix cov of the point cloud is calculated, and then the matrix is ​​subjected to singular value decomposition (SVD) to obtain a normal vector matrix. Finally, the normal vector is normalized to make its length 1, that is, the normal vector of the road surface is obtained.

[0107] After obtaining the road surface parameters corresponding to each fitted road surface, for each non-ignored category point in the point cloud, the second voxel grid to which the non-ignored category point belongs is determined, and based on the road surface parameters corresponding to the second voxel grid and the coordinates of the non-ignored category point, the height information between the non-ignored category point and the ground can be calculated.

[0108] The height calculation process may specifically include: Assuming that the non-ignored category point is point, calculate the vector difference between it and the center point of the road surface (mean): diff=np.array(point)[:,:3]-mean, where np.array(point) means converting point into a NumPy array for array operations, [:,:3] means all rows (:), but only the first 3 columns (:3) are selected, which means only the first three dimensions of each point are taken (for example, the x, y, z coordinates in three-dimensional space). By multiplying the vector difference diff and the road surface normal vector (normal), the height between the point and the road surface is obtained as dist=(np.array(point)[:,:3]-mean)@normal, where "@" is the matrix dot multiplication symbol in NumPy.

[0109] For each ignored category point, the voxel grid where the ignored category point is located can be calculated. If the voxel grid has corresponding road surface parameters, the height information of the ignored category point can be calculated based on the road surface parameters corresponding to the voxel grid and the coordinates corresponding to the ignored category point. The process of height calculation is described above and will not be repeated here; otherwise, the height information of the ignored category point is set to a preset height (for example, -1).

[0110] Figure 6 A schematic diagram of a process for calculating height provided in an embodiment of the present disclosure, such as Figure 6 As shown, on the basis of executing the above steps, for each grid center point corresponding to the second voxel grid, a grid center point can be used as a key, and the road surface parameters of the road surface corresponding to the grid center point can be used as a value to construct a dictionary recording the road surface parameters. When calculating the height information of each ignored category point, the voxel grid where each ignored category point is located can be first determined, and it can be determined whether the voxel grid is in the dictionary containing the road surface parameters. If not, the height information of the point is set to a preset value; if so, the height calculation is performed.

[0111] In the disclosed embodiment, by voxelizing the road surface points, a plurality of first voxel grids are obtained, and road surface grids with appropriate road surface thickness are screened out; by voxelizing the non-ignored category points, a plurality of second voxel grids are obtained, and by obtaining a plurality of adjacent road surface points corresponding to the grid center of each second voxel grid, the road surface is fitted for the plurality of adjacent road surface points to obtain the road surface parameters of the fitted road surface, and then the height information of each point relative to the nearest ground is calculated, thereby improving the accuracy of the height calculation of the point cloud data, effectively eliminating the influence of ups and downslopes on the height measurement, and making the generated height information more reasonable and consistent, so as to more accurately capture the real terrain characteristics of the point cloud, thereby improving the uniformity of the height concept during model training. Standardization of height information not only helps to improve the performance of the model, but also ensures adaptability under different terrain conditions, thereby enhancing the model's ability to understand complex environments.

[0112] In addition, the calculated height information of each point can also be used to assist data labeling. For example, in an open-pit mining scenario, it is difficult to accurately mark the boundary line between the earth wall and the road surface during manual labeling. If the detected category is an earth wall point, but the corresponding height information value is small (for example, less than 0.2 meters), in actual situations, vehicles can travel on these lower earth walls. It is not appropriate to label these points as earth wall points. The category of these points can be corrected to road surface points, so that the labeling results are more in line with the actual situation of the application scenario, further improving the accuracy of the labeling results.

[0113] Figure 7 A schematic diagram of a point cloud data annotation method provided by an embodiment of the present disclosure is shown in FIG. Figure 7 As shown, the point cloud data annotation method provided by the embodiment of the present disclosure includes pre-annotation through the Point-Prompt-Training annotation model, and the annotation model is obtained based on the training samples after data enhancement for the wind and sand scene. The training process of the annotation model can be found in the description above and will not be repeated here; after obtaining the initial annotation result output by the annotation model, the multi-frame point cloud is spliced ​​and combined with the voting mechanism to correct the category of the isolated points to obtain a refined target annotation result; finally, the height information of the point cloud sequence is further calculated according to the target annotation result to obtain a point cloud sequence with accurate target category labels and height information, so that the annotated point cloud sequence can be used to train the BEV perception model.

[0114] The point cloud data annotation method provided by the disclosed embodiment adopts the Point-Prompt-Training point-level prediction model to classify dynamic and static obstacles, and trains through a large number of public data sets to achieve category semantic associations between different data sets, thereby expanding the diversity of training data and improving the generalization ability of the model; through the road surface fitting method and columnar processing, the road surface and non-road surface categories are distinguished, and the height difference of each point relative to the nearest ground is calculated to ensure the accuracy of the height information and provide a reliable basis for subsequent annotation; combined with the voting mechanism, BEV segmentation and automatic annotation of height information are realized. This mechanism not only improves the accuracy and efficiency of annotation, but also significantly reduces the cost of manual annotation, providing rich real data support for the training and evaluation of perception models.

[0115] Figure 8 A schematic diagram of a point cloud data annotation device provided in an embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the device of this embodiment may include: The data acquisition module 210 is used to acquire a point cloud sequence to be processed; the point cloud sequence to be processed includes at least two consecutive frames of point clouds; The pre-labeling module 220 is used to input the to-be-processed point cloud sequence into the labeling model to obtain the initial labeling result of the to-be-processed point cloud sequence output by the labeling model; the initial labeling result includes initial category labels corresponding to each point in each frame of the point cloud; A correction module 230 is used to correct the initial annotation result to obtain a target annotation result corresponding to the point cloud sequence to be processed; The annotation model is trained based on the following method: Obtaining an object point cloud, a wind and sand point cloud, and a plurality of first sample point clouds; For each first sample point cloud, adding the object point cloud and / or the wind and sand point cloud to the first sample point cloud to obtain a second sample point cloud; The initial annotation model is trained based on the multiple second sample point clouds, and the trained initial annotation model is used as the annotation model.

[0116] As an optional embodiment, when the pre-labeling module adds the object point cloud to the first sample point cloud, it is specifically used to: Determining a candidate object position in the first sample point cloud based on a three-dimensional detection box corresponding to the object point cloud; If the candidate object position includes a road surface point, the target object position is determined based on the ground height; otherwise, the candidate object position is used as the target object position; If the candidate object position does not include a soil wall point, and the detection frame corresponding to the candidate object position does not collide with other detection frames in the first sample point cloud, the candidate object position is used as the target object position; Based on the target object position, the object point cloud is added to the first sample point cloud.

[0117] As an optional embodiment, when the pre-labeling module adds the wind and sand point cloud to the first sample point cloud, it is specifically used to: Determine a candidate wind sand position in the first sample point cloud based on a two-dimensional detection frame corresponding to the wind sand point cloud; If the number of earth wall points included in the candidate wind and sand positions is not greater than the first number, and the number of vehicle points is not greater than the second number, determining the target wind and sand position corresponding to the wind and sand point cloud based on the candidate wind and sand positions; Based on the target wind sand position, the wind sand point cloud is added to the first sample point cloud.

[0118] As an optional embodiment, the pre-labeling module adds the object point cloud and the wind and sand point cloud to the first sample point cloud to obtain the second sample point cloud, and is further used for: Acquire at least one three-dimensional detection box in the second sample point cloud; For each three-dimensional detection frame in the second sample point cloud, the sandstorm points in the three-dimensional detection frame are deleted.

[0119] As an optional embodiment, when acquiring the object point cloud, the pre-labeling module is specifically used to: Acquire a first initial point cloud sequence; the first initial point cloud sequence includes at least two consecutive frames of first initial point clouds; extracting at least one dynamic obstacle from the first initial point cloud sequence; For each dynamic obstacle, if the distance between the dynamic obstacle and its corresponding associated wind-blown sand group is less than a preset distance threshold, the point cloud of the dynamic obstacle and the point cloud of the associated wind-blown sand group are taken as a first object point cloud; otherwise, the point cloud of the dynamic obstacle is taken as a second object point cloud; The associated sand group is the sand group closest to the dynamic obstacle.

[0120] As an optional embodiment, when acquiring the wind and sand point cloud, the pre-labeling module is specifically used to: Acquire a second initial point cloud sequence; the second initial point cloud sequence includes at least two consecutive frames of second initial point clouds; extracting at least one wind-blown sand mass from the second initial point cloud sequence; For each sand group, if the distance between the sand group and the vehicle is less than a preset distance threshold, the sand group is regarded as a sand group close to the vehicle; otherwise, the sand group is regarded as a sand group far from the vehicle; wherein the vehicle is the vehicle that obtains the second initial point cloud sequence; The wind-sand point cloud is determined based on the wind-sand group close to the vehicle and the wind-sand group far from the vehicle.

[0121] As an optional embodiment, when the correction module corrects the initial annotation result, it is specifically used to: Splicing multiple frames of point cloud in the to-be-processed point cloud sequence to obtain a spliced ​​point cloud sequence; Selecting at least one isolated point from the stitched point cloud sequence; For each isolated point, a neighborhood range corresponding to the isolated point is determined; based on a plurality of neighborhood points in the neighborhood range, the category of the isolated point is corrected, and the corrected category is used as a target category label corresponding to the isolated point.

[0122] As an optional embodiment, the device further includes a height calculation module, which is used to: For each frame of point cloud, determining at least one road surface point, at least one non-ignored category point, and at least one ignored category point from the point cloud based on the target annotation result; voxelize the at least one road surface point to obtain at least one first voxel grid; for each first voxel grid, if the standard deviation of the height coordinates corresponding to each road surface point in the first voxel grid is less than a preset threshold, use the voxel grid as a road surface grid; determine the target road surface points corresponding to the at least one road surface grid; voxelize the at least one non-ignored category point to obtain a grid center point corresponding to at least one second voxel grid; For each grid center point, searching for a plurality of adjacent road surface points corresponding to the grid center point from the at least one target road surface point, performing road surface fitting on the plurality of adjacent road surface points, and obtaining road surface parameters corresponding to the fitted road surface; Based on at least one road surface parameter, the heights corresponding to the respective points in the point cloud are calculated.

[0123] The device of the embodiment of the present disclosure can execute the method provided by the embodiment of the present disclosure, and its implementation principle is similar and has corresponding technical effects. The actions performed by each module in the device of each embodiment of the present disclosure correspond to the steps in the method of each embodiment of the present disclosure. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0124] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0125] In an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, and the processor executes the above computer program to implement the steps of the method provided by any optional embodiment of the present disclosure. Compared with the prior art, it can be achieved that: the target annotation result corresponding to the point cloud sequence to be processed is obtained through the annotation model, the cost of manual annotation is reduced, the annotation efficiency is improved, the influence of the subjectivity of manual annotation on the annotation quality is avoided, and the annotation quality is improved; based on the object point cloud and the wind and sand point cloud, the first sample point cloud after data enhancement is performed on the data as the second sample point cloud, so that the second sample point cloud includes rich information on object categories and wind and sand categories; based on the second sample point cloud, the recognition ability of the object category and the wind and sand category in the initial annotation model is trained in a targeted manner, the accuracy of the trained annotation model in identifying the object category and the wind and sand category is improved, the processing ability of the annotation model for the wind and sand scene is improved, and the annotation quality is further improved.

[0126] In an alternative embodiment, an electronic device is provided, such as Fig. 9 As shown, Fig. 9The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present disclosure.

[0127] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0128] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0129] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0130] The memory 4003 is used to store the computer program for executing the embodiment of the present disclosure, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0131] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0132] An embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0133] The embodiments of the present disclosure also provide a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiments when executed by a processor.

[0134] It should be understood that, although the flowchart of the embodiment of the present disclosure indicates each operation step by arrows, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present disclosure, the implementation steps in each flowchart can be executed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times. In scenarios with different execution times, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present disclosure does not limit this.

[0135] The above is only an optional implementation method for some implementation scenarios of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the scheme of the present disclosure, other similar implementation methods based on the technical ideas of the present disclosure are also within the protection scope of the embodiments of the present disclosure.

Claims

1. A point cloud data annotation method, characterized in that: include: Acquire a point cloud sequence to be processed; the point cloud sequence to be processed includes at least two consecutive frames of point clouds; Inputting the to-be-processed point cloud sequence into a labeling model to obtain an initial labeling result of the to-be-processed point cloud sequence output by the labeling model; The initial annotation result includes initial category labels corresponding to each point in each frame of the point cloud; Correcting the initial annotation result to obtain a target annotation result corresponding to the point cloud sequence to be processed; The annotation model is trained based on the following method: Obtaining an object point cloud, a wind and sand point cloud, and a plurality of first sample point clouds; For each first sample point cloud, adding the object point cloud and / or the wind and sand point cloud to the first sample point cloud to obtain a second sample point cloud; The initial annotation model is trained based on the multiple second sample point clouds, and the trained initial annotation model is used as the annotation model.

2. The method according to claim 1, characterized in that The step of adding the object point cloud to the first sample point cloud comprises: Determining a candidate object position in the first sample point cloud based on a three-dimensional detection box corresponding to the object point cloud; If the candidate object position includes a road surface point, the target object position is determined based on the ground height; otherwise, the candidate object position is used as the target object position; If the candidate object position does not include a soil wall point, and the detection frame corresponding to the candidate object position does not collide with other detection frames in the first sample point cloud, the candidate object position is used as the target object position; Based on the target object position, the object point cloud is added to the first sample point cloud.

3. The method according to claim 1, characterized in that The step of adding the wind and sand point cloud to the first sample point cloud comprises: Determine a candidate wind sand position in the first sample point cloud based on a two-dimensional detection frame corresponding to the wind sand point cloud; If the number of earth wall points included in the candidate wind and sand positions is not greater than the first number, and the number of vehicle points is not greater than the second number, determining the target wind and sand position corresponding to the wind and sand point cloud based on the candidate wind and sand positions; Based on the target wind sand position, the wind sand point cloud is added to the first sample point cloud.

4. The method according to claim 1, characterized in that: The object point cloud and the wind and sand point cloud are added to the first sample point cloud to obtain a second sample point cloud, and then the method further includes: Acquire at least one three-dimensional detection box in the second sample point cloud; For each three-dimensional detection frame in the second sample point cloud, the sandstorm points in the three-dimensional detection frame are deleted.

5. The method according to claim 1, characterized in that The step of obtaining the object point cloud comprises: Acquire a first initial point cloud sequence; the first initial point cloud sequence includes at least two consecutive frames of first initial point clouds; extracting at least one dynamic obstacle from the first initial point cloud sequence; For each dynamic obstacle, if the distance between the dynamic obstacle and its corresponding associated wind-blown sand group is less than a preset distance threshold, the point cloud of the dynamic obstacle and the point cloud of the associated wind-blown sand group are taken as a first object point cloud; otherwise, the point cloud of the dynamic obstacle is taken as a second object point cloud; The associated sand group is the sand group closest to the dynamic obstacle.

6. The method according to claim 1, characterized in that The step of obtaining the wind and sand point cloud comprises: Acquire a second initial point cloud sequence; the second initial point cloud sequence includes at least two consecutive frames of second initial point clouds; extracting at least one wind-blown sand mass from the second initial point cloud sequence; For each sand group, if the distance between the sand group and the vehicle is less than a preset distance threshold, the sand group is regarded as a sand group close to the vehicle; otherwise, the sand group is regarded as a sand group far from the vehicle; wherein the vehicle is the vehicle that obtains the second initial point cloud sequence; The wind-sand point cloud is determined based on the wind-sand group close to the vehicle and the wind-sand group far from the vehicle.

7. The method according to any one of claims 1 to 6, characterized in that The correcting the initial annotation result includes: Splicing multiple frames of point cloud in the to-be-processed point cloud sequence to obtain a spliced ​​point cloud sequence; Selecting at least one isolated point from the stitched point cloud sequence; For each isolated point, a neighborhood range corresponding to the isolated point is determined; based on a plurality of neighborhood points in the neighborhood range, the category of the isolated point is corrected, and the corrected category is used as a target category label corresponding to the isolated point.

8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: For each frame of point cloud, determining at least one road surface point, at least one non-ignored category point, and at least one ignored category point from the point cloud based on the target annotation result; voxelize the at least one road surface point to obtain at least one first voxel grid; for each first voxel grid, if the standard deviation of the height coordinates corresponding to each road surface point in the first voxel grid is less than a preset threshold, use the voxel grid as a road surface grid; determine the target road surface points corresponding to the at least one road surface grid; voxelize the at least one non-ignored category point to obtain a grid center point corresponding to at least one second voxel grid; For each grid center point, searching for a plurality of adjacent road surface points corresponding to the grid center point from the at least one target road surface point, performing road surface fitting on the plurality of adjacent road surface points, and obtaining road surface parameters corresponding to the fitted road surface; Based on at least one road surface parameter, the heights corresponding to the respective points in the point cloud are calculated.

9. A point cloud data annotation device, characterized in that: include: A data acquisition module, used to acquire a point cloud sequence to be processed; the point cloud sequence to be processed includes at least two consecutive frames of point clouds; A pre-annotation module, used for inputting the to-be-processed point cloud sequence into an annotation model, and obtaining an initial annotation result of the to-be-processed point cloud sequence output by the annotation model; The initial annotation result includes initial category labels corresponding to each point in each frame of the point cloud; A correction module, used to correct the initial annotation result to obtain a target annotation result corresponding to the point cloud sequence to be processed; The annotation model is trained based on the following method: Obtaining an object point cloud, a wind and sand point cloud, and a plurality of first sample point clouds; For each first sample point cloud, adding the object point cloud and / or the wind and sand point cloud to the first sample point cloud to obtain a second sample point cloud; The initial annotation model is trained based on the multiple second sample point clouds, and the trained initial annotation model is used as the annotation model.

10. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images