A method, device, equipment and medium for auxiliary annotation of point cloud data

Abstract

The present invention discloses an auxiliary annotation method, device, equipment and medium for point cloud data, relating to the field of data processing technology. The method comprises: obtaining original point cloud data, determining the initial point of the current frame from the original point cloud data, and extracting the target data point of the current frame from the original point cloud data based on the initial point; generating an initial target detection frame based on the target data point; adjusting the size of the initial target detection frame, and projecting the original point cloud data within the frame onto various two-dimensional planes, performing size regression on all projected images and aggregating the regressed projected images to obtain a single-frame target detection frame for the current frame; determining the motion trajectory of the original point cloud data, and generating a single-frame target detection frame for each frame of consecutive frames after the current frame based on the motion trajectory. The present invention can generate detection frames for consecutive frames, which can greatly reduce the workload of manual annotation and improve the accuracy of the frame center point and size information.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and medium for auxiliary annotation of point cloud data. Background Art

[0002] With the rapid development of fields such as autonomous driving, robotic vision, and geographic information systems, the demand for annotated 3D point cloud data is increasing. To improve the performance of data in these areas, more accurate 3D object detection and annotation are essential. However, the current 3D point cloud annotation process still relies heavily on human intervention, especially when annotating large datasets. Traditional annotation methods are not only time-consuming and labor-intensive, but also prone to significant data errors.

[0003] Although there are some deep learning-based 3D point cloud target detection methods (such as PV-RCNN, CenterPoint, DSVT, etc.) that can perform target detection, most of these algorithms only consider single-frame target recognition and do not fully combine the generation of target detection boxes with continuous frame target tracking.

[0004] In order to solve these problems, it is necessary to provide a new 3D point cloud object detection method. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method, apparatus, device, and medium for auxiliary annotation of point cloud data, so as to solve the problem of inaccurate annotation of three-dimensional point cloud data.

[0006] According to a first aspect, an embodiment of the present invention provides an auxiliary annotation method for point cloud data, the method comprising:

[0007] Obtaining raw point cloud data, determining the initial point of the current frame from the raw point cloud data, and extracting target data points of the current frame from the raw point cloud data based on the initial point; the distance between each target data point and the initial point does not exceed a preset search radius;

[0008] Generate an initial target detection frame based on the target data points;

[0009] Adjust the size of the initial target detection frame and project the original point cloud data within the adjusted initial target detection frame onto each two-dimensional plane to obtain the projection image of each two-dimensional plane. Perform size regression on all the projection images and aggregate the regressed projection images to obtain the single-frame target detection frame of the current frame.

[0010] Obtain motion information of the original point cloud data in the frame after the current frame, determine the motion trajectory of the original point cloud data based on the motion information, generate a single-frame target detection frame for each frame of the consecutive frames after the current frame based on the motion trajectory, and confirm whether the frames corresponding to the consecutive frames meet the consistency and preset annotation requirements. If it is determined that the consistency or the preset annotation requirements are not met, adjust the single-frame target detection frame that does not meet the requirements; the consistency is that the single-frame target detection frames of adjacent frames belong to one frame, and the size deviation, orientation, and overlap ratio between the single-frame target detection frames of adjacent frames meet the preset requirements.

[0011] In combination with the first aspect, in a first implementation of the first aspect, obtaining the original point cloud data, determining the initial point of the current frame from the original point cloud data, and extracting the target data point of the current frame from the original point cloud data based on the initial point specifically includes:

[0012] Obtain original point cloud data and determine the initial point of the current frame from the original point cloud data;

[0013] With the initial point as the center point and the preset search radius as the interception radius, a search area is delineated from the original point cloud data;

[0014] The original point cloud data within the search area is used as the target data point, and the target data point of the current frame is extracted.

[0015] In combination with the first aspect, in the second embodiment of the first aspect, the initial target detection frame is generated by a trained target detection model, and the target detection model is trained based on historical point cloud data and label information extracted from the historical point cloud data.

[0016] In combination with the second implementation of the first aspect, in the third implementation of the first aspect, the target detection model is trained by the following steps:

[0017] Acquiring the historical point cloud data, and determining the tag information from the historical point cloud data;

[0018] The historical point cloud data is used as input data for training, and the label information corresponding to the historical point cloud data is used as the label for training. Deep learning is used for training to obtain the target detection model for generating the initial target detection frame of the target data point.

[0019] In combination with the first aspect, in a fourth implementation of the first aspect, adjusting the size of the initial target detection frame, projecting the original point cloud data within the adjusted initial target detection frame onto each two-dimensional plane to obtain a projection image of each two-dimensional plane, performing size regression on all projection images and aggregating the regressed projection images to obtain a single-frame target detection frame for the current frame, specifically includes:

[0020] Keep the center point of the initial target detection frame unchanged and enlarge the size of the initial target detection frame by a preset multiple;

[0021] Extract the original point cloud data within the adjusted initial target detection frame, and project the original point cloud data onto each two-dimensional plane to obtain the projection image of each two-dimensional plane;

[0022] The projection center points of all projection planes and the regression detection frames are aggregated to obtain the single-frame target detection frame of the current frame.

[0023] In combination with the first aspect, in a fifth implementation of the first aspect, before the steps of adjusting the size of the initial target detection frame, projecting the original point cloud data within the adjusted initial target detection frame onto each two-dimensional plane to obtain a projection image of each two-dimensional plane, performing size regression on all the projection images, and aggregating the regressed projection images to obtain a single-frame target detection frame for the current frame, the method further includes:

[0024] Determine the preset labeling requirements, verify the single-frame target frame according to the preset labeling requirements, and if it is determined that the verification fails, perform a frame adjustment operation on the single-frame target detection frame.

[0025] In combination with the first aspect, in a sixth implementation of the first aspect, a single-frame object detection frame of each frame in the continuous frames is obtained by the following steps:

[0026] voxelize the original point cloud data of consecutive frames to obtain voxelized data;

[0027] Extract point cloud feature information from voxelized data and use the self-attention mechanism to obtain contextual information from continuous frame point cloud data;

[0028] Perform feature fusion on point cloud feature information and context information to obtain continuous frame feature information;

[0029] Extract local spatial information of feature information of consecutive frames, and obtain motion information of the original point cloud data in the frame after the current frame based on the local spatial information;

[0030] The motion trajectory of the original point cloud data is determined according to the motion information, and a single-frame target detection frame is generated for each frame of the consecutive frames after the current frame according to the motion trajectory.

[0031] According to a second aspect, an embodiment of the present invention further provides an auxiliary annotation device for point cloud data, the device comprising:

[0032] A data extraction module is used to obtain raw point cloud data, determine the initial point of the current frame from the raw point cloud data, and extract the target data points of the current frame from the raw point cloud data based on the initial point; the distance between each target data point and the initial point does not exceed a preset search radius;

[0033] The initial generation module is used to generate the initial target detection box based on the target data points;

[0034] The size optimization module is used to adjust the size of the initial target detection frame and project the original point cloud data within the adjusted initial target detection frame onto various two-dimensional planes to obtain the projection images of each two-dimensional plane. The size of all the projected images is regressed and the regressed projection images are aggregated to obtain the single-frame target detection frame of the current frame.

[0035] The continuous frame generation module is used to obtain the motion information of the original point cloud data in the frame after the current frame, determine the motion trajectory of the original point cloud data based on the motion information, generate a single-frame target detection frame for each frame of the continuous frames after the current frame based on the motion trajectory, and confirm whether the frame corresponding to the continuous frames meets the consistency and preset labeling requirements. If it is determined that the consistency or the preset labeling requirements are not met, the unsatisfactory single-frame target detection frame is adjusted; the consistency is that the single-frame target detection frames of adjacent frames belong to one frame, and the size deviation, orientation, and overlap ratio between the single-frame target detection frames of adjacent frames meet the preset requirements.

[0036] According to the third aspect, an embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the steps of the auxiliary annotation method for point cloud data as described in any one of the above are implemented.

[0037] According to a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described auxiliary annotation methods for point cloud data.

[0038] The point cloud data auxiliary annotation method, apparatus, device, and medium of the present invention select an initial point and determine a search area based on the position of the initial point. Target data points sufficiently close to the initial point are extracted from the original point cloud data based on the search area. An initial target detection frame is then automatically generated based on the target data points. The size of the initial target detection frame is then optimized to obtain a single-frame target detection frame for the current frame. Motion information of frames subsequent to the current frame is used to determine a motion trajectory of the original point cloud data based on the motion information. Single-frame target detection frames are then generated for each of the consecutive frames subsequent to the current frame based on the motion trajectory. Generating detection frames for consecutive frames in this manner significantly reduces the workload of manual annotation and improves the accuracy of frame center point and size information. Multi-directional projection optimization technology is used in the size optimization process of the initial target detection frame to accurately adjust the size and position of the target frame, ensuring high-precision annotation. By continuously tracking the target during the generation of single-frame target detection frames for consecutive frames based on the single-frame target detection frame of the current frame, the method can stably process targets in dynamic scenes. The method is suitable for complex dynamic environments, such as autonomous driving, robotic vision, geographic information systems, and other fields, and can process different types of three-dimensional point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0040] Figure 1 A schematic diagram showing the flow of the auxiliary annotation method for point cloud data provided by the present invention is shown;

[0041] Figure 2 A schematic structural diagram of the auxiliary annotation device for point cloud data provided by the present invention is shown;

[0042] Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0044] With the rapid development of fields such as autonomous driving, robotic vision, and geographic information systems, the demand for annotated 3D point cloud data is increasing. To improve the performance of data in these areas, more accurate 3D object detection and annotation are essential. However, the current 3D point cloud annotation process still relies heavily on human intervention, especially when annotating large datasets. Traditional annotation methods are not only time-consuming and labor-intensive, but also prone to significant data errors.

[0045] Although there are some deep learning-based 3D point cloud target detection methods (such as PV-RCNN, CenterPoint, DSVT, etc.) that can perform target detection, most of these algorithms only consider single-frame target recognition and do not fully combine the generation of target detection boxes with continuous frame target tracking.

[0046] In summary, it is necessary to provide a new 3D point cloud object detection method.

[0047] In order to solve the above problems, this embodiment provides an auxiliary annotation method for point cloud data, which aims to generate detection frames for continuous frames, greatly reduce the workload of manual annotation, and improve the accuracy of the frame center point and size information. The auxiliary annotation method for point cloud data of the embodiment of the present invention can be used in electronic devices, including but not limited to computers, mobile terminals, etc. Figure 1 FIG. 1 is a flow chart of an auxiliary annotation method for point cloud data according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0048] S10, acquiring original point cloud data, determining an initial point of a current frame from the original point cloud data, and extracting a target data point of the current frame from the original point cloud data based on the initial point.

[0049] The original point cloud data is obtained as a continuous sequence of point clouds. The original point cloud data contains M consecutive frames of point cloud data, that is, continuous M frames of point cloud data at the moment.

[0050] In this embodiment, the target data points include the initial point, and the distance between each target data point and the initial point does not exceed the preset search radius, that is, the target data point is a point cloud subset extracted from the original point cloud data that is close enough to the initial point.

[0051] For example, taking the initial point as the center point, it is possible to determine the target data point with a distance that meets the requirements from the original point cloud data based on the initial point of the current frame.

[0052] The original point cloud data may be pre-stored in the electronic device, or may be acquired by the electronic device from the outside, for example, acquired by the electronic device from an external acquisition device, or extracted from an external video capture device.

[0053] There is no restriction on the specific acquisition method of the original point cloud data. It only needs to ensure that the electronic device can obtain the original point cloud data.

[0054] S20: Generate an initial target detection frame based on the target data points.

[0055] During the initial object detection frame generation phase, the core task is to infer the target's bounding box based on the user-selected initial point (i.e., coordinate prompt) and the target data points. This allows the user to provide precise point information and then assist in generating the target's bounding box based on this information. After the user provides the initial point in the previous step, the point cloud data of the target area has been initially extracted.

[0056] In this step, the trained target detection model is used to generate the initial target detection frame. Specifically, the target data point is input into the trained target detection model, and the target detection model outputs the initial target detection frame based on the target data point. The initial target detection frame includes the center point of the target. , frame information and category information, Represents the x-axis coordinate of the center point of the initial target detection frame, Indicates the y-axis coordinate of the center point of the initial target detection frame, Represents the z-axis coordinate of the center point of the initial target detection frame, Indicates the length of the initial target detection box, Indicates the width of the initial target detection box, Indicates the height of the initial object detection box.

[0057] The target detection model can be trained by historical point cloud data and label information extracted from the historical point cloud data. For example, the historical point cloud data is first obtained, and the label information is determined from the historical point cloud data. Then, the historical point cloud data is used as input data for training, and the label information corresponding to the historical point cloud data is used as the label for training. Then, deep learning is used for training to obtain the target detection model for generating the initial target detection frame of the target data point. There is no restriction on the specific training method of the target detection model.

[0058] Preferably, you can also use existing 3D object detection models (such as PointNet++ or PV-RCNN) to generate an initial object detection box, input the extracted point cloud data, and perform object detection. These models will output an initial object detection box, which represents the bounding box of the target, and the bounding box also includes the center point of the target / object. , frame information and category information.

[0059] S30. Adjust the size of the initial target detection frame, and project the original point cloud data within the adjusted initial target detection frame to each two-dimensional plane to obtain the projection image of each two-dimensional plane. Perform size regression on all projection images and aggregate the regressed projection images to obtain a single-frame target detection frame for the current frame.

[0060] In this embodiment, this step optimizes the generated initial target detection frame to expand the original point cloud data contained in the detection frame, and then projects these original data point clouds to each two-dimensional plane. For three-dimensional point cloud data, there are three two-dimensional planes that can be projected, namely the xy, xz and yz planes, and the projection results obtained on each two-dimensional plane are the projection images. Next, the size of the initial target detection frame in the two-dimensional plane can be obtained by performing size regression on the projected image. Finally, by aggregating the results of the size regression of each two-dimensional plane, the center coordinates and size of the final optimized target frame can be obtained, that is, the single-frame target detection frame of the current frame. It can be understood that the obtained single-frame target detection frame contains the center point and size information of the frame.

[0061] By first expanding the initial target detection frame to expand the original point cloud data contained in the frame, and then performing a two-dimensional plane projection on the original point cloud data, the center coordinates of these original point cloud data on the two-dimensional plane and the two-dimensional frame information formed based on these original point cloud data are obtained. After that, size regression processing and merging processing are performed. The frame information of the single-frame target detection frame of the current frame is finally obtained more accurate, so that the single-frame target detection frame of the frame meets the accuracy requirements, and lays the foundation for the subsequent calculation of the single-frame target detection frame of each frame of the subsequent consecutive frames based on the single-frame target detection frame of the frame that meets the accuracy requirements.

[0062] S40. Obtain motion information of the original point cloud data in the frame after the current frame, determine the motion trajectory of the original point cloud data based on the motion information, generate a single-frame target detection frame for each frame of the consecutive frames after the current frame based on the motion trajectory, and confirm whether the frames corresponding to the consecutive frames meet the consistency and preset labeling requirements. If it is determined that the consistency or preset labeling requirements are not met, adjust the single-frame target detection frame that does not meet the requirements.

[0063] In this embodiment, the consistency of the frames primarily refers to the fact that these single-frame object detection frames belong to a single frame, and that the changes in the single-frame object detection frames between adjacent frames are relatively natural, such as maintaining consistent orientation. That is, the size deviation, orientation, and overlap ratio between the single-frame object detection frames of adjacent frames all meet preset requirements. If inconsistency is determined or the preset annotation requirements are not met, the user can choose to fine-tune the frame if the deviation is small; if the deviation is large, the user can choose to delete the frame generation result and re-execute steps S10 to S40 for annotation to generate single-frame object detection frames at other times.

[0064] Specifically: the original point cloud data of continuous frames is voxelized to obtain voxelized data, which contains the voxel coordinates of the original point cloud data. Then, the point cloud feature information of the voxelized data is extracted and the context information of the continuous frame point cloud data is obtained using the self-attention mechanism. Next, the point cloud feature information and the context information are fused to obtain the continuous frame feature information. Then, the local spatial information of the continuous frame feature information is extracted, so that features containing spatial information can be obtained. Finally, the motion information of the original point cloud data in the frame after the current frame is obtained based on the local spatial information, the motion trajectory of the original point cloud data is determined based on the motion information, and a single-frame target detection frame is generated for each frame of the continuous frames after the current frame based on the motion trajectory.

[0065] Assume that in the current In the frame, the target detection box is , and in the previous frame In the frame, the target detection box is , the target's motion in consecutive frames can be estimated by the tracking algorithm:

[0066]

[0067] in, The motion trajectory representing the length of the target detection box between adjacent frames; The motion trajectory representing the width of the target detection box between adjacent frames; Represents the motion trajectory of the target detection box height between adjacent frames.

[0068] Using these motion trajectories and based on the registration and motion of the point clouds of the previous and next frames, the position change of the target in the subsequent frames is estimated, and a new single-frame target detection box is generated for each frame.

[0069] In order to generate a single-frame target detection frame for each frame in the continuous frames, in this embodiment, a common point cloud single target tracking algorithm (such as M2Track, MBPTrack, etc.) can also be used to track the motion trajectory of the point cloud, and generate a single-frame target detection frame for the continuous frames based on the motion trajectory.

[0070] The auxiliary annotation method for point cloud data of the present invention selects an initial point and determines a search area based on the position of the initial point. Target data points sufficiently close to the initial point are extracted from the original point cloud data based on the search area. An initial target detection frame is then automatically generated based on the target data points. The size of the initial target detection frame is then optimized to obtain a single-frame target detection frame for the current frame. Motion information of frames subsequent to the current frame is used to determine a motion trajectory of the original point cloud data based on the motion information. Single-frame target detection frames are then generated for each of the consecutive frames subsequent to the current frame based on the motion trajectory. Generating detection frames for consecutive frames in this manner significantly reduces the workload of manual annotation and improves the accuracy of frame center points and size information. Multi-directional projection optimization technology is used in the size optimization process of the initial target detection frame to accurately adjust the size and position of the target frame, ensuring high-precision annotation. By continuously tracking the target during the generation of single-frame target detection frames for consecutive frames based on the single-frame target detection frame of the current frame, the method can stably process targets in dynamic scenes. The method is suitable for complex dynamic environments, such as autonomous driving, robotic vision, geographic information systems, and other fields, and can process different types of three-dimensional point cloud data.

[0071] In this embodiment, step S10 specifically includes:

[0072] S11. Obtain original point cloud data, and determine the initial point of the current frame from the original point cloud data.

[0073] Preferably, the user, such as the annotator, clicks on the approximate location of the target object to select the initial point of the current frame. Assume that the coordinates of the three-dimensional point cloud selected by the user are ,in, Indicates the x-axis coordinate of the initial point; Indicates the y-axis coordinate of the initial point; Represents the z-axis coordinate of the initial point, so that a search area can be determined based on the position of the initial point, and based on the search area, a point cloud subset that is close enough to the initial point can be extracted from the original point cloud data to obtain the target data point.

[0074] S12. Delineating a search area from the original point cloud data with the initial point as the center point and a preset search radius as the interception radius.

[0075] Specifically, set a preset search radius , used to determine the size of the intercepted search area. In this embodiment, the search range of the search area is a spherical area or a cubic area.

[0076] S13. Use the original point cloud data within the search area as target data points and extract the target data points of the current frame.

[0077] In particular, for the search area of the spherical area, the search range of the search area is:

[0078]

[0079] in, Represents the point cloud subset contained in the search area, that is, the set of target data points; represents the target data point; Indicates the The x-axis coordinate of the target data point; Indicates the The y-axis coordinate of the target data point; Indicates the The z-axis coordinate of the target data point.

[0080] In particular, for the search area of the cube area, the search range of the search area is:

[0081]

[0082] In this embodiment, step S30 specifically includes:

[0083] S31. Keep the center point of the initial target detection frame unchanged, and enlarge the size of the initial target detection frame by a preset multiple.

[0084] In this embodiment, the size of the initial target detection frame is enlarged by a preset multiple during optimization. , and keep the center point unchanged during the zoom process to ensure that the box contains more original point cloud data.

[0085] The adjusted box size is therefore:

[0086]

[0087] in, Indicates the The length of the adjusted initial target detection box corresponding to the frame; Indicates the The width of the adjusted initial target detection box corresponding to the frame; Indicates the The height of the adjusted initial object detection box corresponding to the frame.

[0088] S32: extracting original point cloud data within the adjusted initial target detection frame, and projecting the original point cloud data onto each two-dimensional plane to obtain a projection image of each two-dimensional plane.

[0089] Next, the original point cloud data within the magnified frame is extracted. It can be understood that these point cloud data contain the target data points obtained previously.

[0090] The projection process specifically projects these point cloud data in three directions:

[0091] Front and rear projection: Select - Project the plane to obtain the front and back projection images .

[0092] Top and Bottom Projection: Select - Project the plane to get the up and down projection image .

[0093] Left and right projection: Select - Project the plane to get the left and right projection images .

[0094] S33 , respectively determining the projection center point of each projection center, and performing size regression processing on the projection image according to the projection center point to obtain a regression detection frame of each projection center.

[0095] Preferably, for each projection image, a convolutional neural network can be used to train a key point detection model and a size regression model, and the trained key point detection model and size regression model can be used to determine the projection center point and the regression size, respectively.

[0096] S34: Aggregate the projection center points of all projection planes and the regression detection frames to obtain a single-frame target detection frame of the current frame.

[0097] For the projected image, its key point is the projection center point. Assume that the projection center point of each projected image is determined as , the regression detection frame obtained after size regression processing based on the projection center point is ,in, , then the single-frame target detection frame of the current frame obtained by aggregating the projection results is:

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] in, Represents the x-axis coordinate of the center point of the single-frame target detection frame after aggregation; Indicates the y-axis coordinate of the center point of the single-frame target detection frame after aggregation; Indicates the length of the aggregated single-frame target detection frame; Indicates the width of the aggregated single-frame target detection frame; Indicates the height of the aggregated single-frame target detection box.

[0105] Through this aggregation method, the center coordinates and size information corresponding to the single-frame target detection frame can be obtained, and more refined frame information can be obtained.

[0106] In this embodiment, the method further includes the following steps after step S30:

[0107] Determine the preset labeling requirements, verify the single-frame target frame according to the preset labeling requirements, and if it is determined that the verification fails, perform a frame adjustment operation on the single-frame target detection frame.

[0108] After optimizing the initial target detection frame and obtaining the single-frame target detection frame, the user can also verify the optimized frame and confirm whether it meets the preset annotation requirements such as fit, size, orientation, and clear outline. If the frame target detection frame verification fails, the frame adjustment operation is performed on the single-frame target detection frame that fails the verification and does not meet the requirements (for example, the frame size has a large deviation). In this embodiment, the frame adjustment operation includes but is not limited to dragging, rotating, scaling, etc. on the single-frame target detection frame. For example, the user can use the frame fine-tuning tool to manually adjust the size and position of the frame. Assume that the fine-tuning operation is to adjust the boundary of the frame by arrive , the frame after fine-tuning is:

[0109]

[0110] in, Represents the box after fine-tuning.

[0111] During the labeling process, the labeler only needs to fine-tune a few boxes that do not meet the requirements, and most detection boxes can be generated through the above-mentioned automated process.

[0112] The following describes an auxiliary annotation device for point cloud data provided by an embodiment of the present invention. The auxiliary annotation device for point cloud data described below and the auxiliary annotation method for point cloud data described above can correspond to each other.

[0113] In order to solve the above problems, an auxiliary annotation device for point cloud data is provided in this embodiment, which aims to generate detection frames for continuous frames, greatly reduce the workload of manual annotation, and improve the accuracy of the frame center point and size information. Figure 2 FIG. 1 is a schematic diagram of a structure of an auxiliary annotation method for point cloud data according to an embodiment of the present invention. Figure 2 As shown, the device may include:

[0114] The data extraction module 10 is used to obtain original point cloud data, determine the initial point of the current frame from the original point cloud data, and extract the target data point of the current frame from the original point cloud data based on the initial point.

[0115] The original point cloud data is obtained as a continuous sequence of point clouds. The original point cloud data contains M consecutive frames of point cloud data, that is, continuous M frames of point cloud data at the moment.

[0116] In this embodiment, the target data points include the initial point, and the distance between each target data point and the initial point does not exceed the preset search radius, that is, the target data point is a point cloud subset extracted from the original point cloud data that is close enough to the initial point.

[0117] For example, taking the initial point as the center point, it is possible to determine the target data point with a distance that meets the requirements from the original point cloud data based on the initial point of the current frame.

[0118] The original point cloud data may be pre-stored in the electronic device, or may be acquired by the electronic device from the outside, for example, acquired by the electronic device from an external acquisition device, or extracted from an external video capture device.

[0119] There is no restriction on the specific acquisition method of the original point cloud data. It only needs to ensure that the electronic device can obtain the original point cloud data.

[0120] The initial generation module 20 is used to generate an initial target detection frame according to the target data points.

[0121] In the initial target detection frame generation stage, the core task is to infer the target bounding box based on the initial point selected by the user (i.e., coordinate prompt) and the target data points. After the user provides the initial point in the previous step, the point cloud data of the target area has been preliminarily extracted.

[0122] In this step, the trained target detection model is used to generate the initial target detection frame. Specifically, the target data point is input into the trained target detection model, and the target detection model outputs the initial target detection frame based on the target data point. The initial target detection frame includes the center point of the target. , frame information and category information, Indicates the x-axis coordinate of the center point, Indicates the y-axis coordinate of the center point, Indicates the z-axis coordinate of the center point, Indicates the length of the initial target detection box, Indicates the width of the initial target detection box, Indicates the height of the initial object detection box.

[0123] The target detection model can be trained using historical point cloud data and label information determined from the historical point cloud data. There is no restriction on the specific training method of the target detection model.

[0124] Preferably, you can also use existing 3D object detection models (such as PointNet++ or PV-RCNN) to generate an initial object detection box, input the extracted point cloud data, and perform object detection. These models will output an initial object detection box, which represents the bounding box of the target, and the bounding box also includes the center point of the target / object. , frame information and category information.

[0125] The size optimization module 30 is used to adjust the size of the initial target detection frame and project the original point cloud data within the adjusted initial target detection frame to each two-dimensional plane to obtain the projection image of each two-dimensional plane. The size of all the projected images is regressed and the regressed projection images are aggregated to obtain the single-frame target detection frame of the current frame.

[0126] In this embodiment, this step optimizes the generated initial target detection frame to expand the original point cloud data contained in the detection frame, and then projects these original data point clouds to each two-dimensional plane. For three-dimensional point cloud data, there are three two-dimensional planes that can be projected, namely the xy, xz and yz planes, and the projection results obtained on each two-dimensional plane are the projection images. Next, the size of the initial target detection frame in the two-dimensional plane can be obtained by performing size regression on the projected image. Finally, by aggregating the results of the size regression of each two-dimensional plane, the center coordinates and size of the final optimized target frame can be obtained, that is, the single-frame target detection frame of the current frame. It can be understood that the obtained single-frame target detection frame contains the center point and size information of the frame.

[0127] The continuous frame generation module 40 is used to obtain the motion information of the original point cloud data in the frame after the current frame, determine the motion trajectory of the original point cloud data based on the motion information, generate a single-frame target detection frame for each frame of the continuous frames after the current frame based on the motion trajectory, and confirm whether the frame corresponding to the continuous frames meets the consistency and preset labeling requirements. If it is determined that the consistency or preset labeling requirements are not met, the single-frame target detection frame that does not meet the requirements is adjusted.

[0128] In this embodiment, the consistency of the frames primarily refers to the fact that these single-frame object detection frames belong to a single frame, and that the changes in the single-frame object detection frames between adjacent frames are relatively natural, such as maintaining consistent orientation. That is, the size deviation, orientation, and overlap ratio between the single-frame object detection frames of adjacent frames all meet preset requirements. If inconsistency is determined or the preset annotation requirements are not met, the user can choose to fine-tune the frame if the deviation is small, or delete the frame generation result if the deviation is large, and start annotation from scratch to generate the single-frame object detection frames for another time.

[0129] Assume that at present In the frame, the target detection box is , and in the previous frame In the frame, the target detection box is , the target's motion in consecutive frames can be estimated by the tracking algorithm:

[0130]

[0131] in, Indicates the The length of the adjusted initial target detection box corresponding to the frame; Indicates the The width of the adjusted initial target detection box corresponding to the frame; Indicates the The height of the adjusted initial object detection box corresponding to the frame.

[0132] Using these motion trajectories and based on the registration and motion of the point clouds of the previous and next frames, the position change of the target in the subsequent frames is estimated, and a new single-frame target detection box is generated for each frame.

[0133] The auxiliary annotation device for point cloud data of the present invention selects an initial point and determines a search area based on the position of the initial point. Target data points sufficiently close to the initial point are extracted from the original point cloud data based on the search area. An initial target detection frame is then automatically generated based on the target data points. The size of the initial target detection frame is then optimized to obtain a single-frame target detection frame for the current frame. Motion information of frames subsequent to the current frame is used to determine a motion trajectory of the original point cloud data based on the motion information. Single-frame target detection frames are then generated for each of the consecutive frames subsequent to the current frame based on the motion trajectory. Generating detection frames for consecutive frames in this manner significantly reduces the workload of manual annotation and improves the accuracy of frame center point and size information. Multi-directional projection optimization technology is used in the size optimization process of the initial target detection frame to accurately adjust the size and position of the target frame, ensuring high-precision annotation. By continuously tracking the target during the generation of single-frame target detection frames for consecutive frames based on the single-frame target detection frame of the current frame, the device can stably process targets in dynamic scenes. The device is suitable for complex dynamic environments, such as autonomous driving, robotic vision, geographic information systems, and other fields, and can process different types of three-dimensional point cloud data.

[0134] Figure 3 An example of a physical structure diagram of an electronic device is shown below. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call the logic commands in the memory 330 to execute the auxiliary annotation method of point cloud data, which includes:

[0135] Obtaining raw point cloud data, determining the initial point of the current frame from the raw point cloud data, and extracting target data points of the current frame from the raw point cloud data based on the initial point; the distance between each target data point and the initial point does not exceed a preset search radius;

[0136] Generate an initial target detection frame based on the target data points;

[0137] Adjust the size of the initial target detection frame and project the original point cloud data within the adjusted initial target detection frame onto each two-dimensional plane to obtain the projection image of each two-dimensional plane. Perform size regression on all the projection images and aggregate the regressed projection images to obtain the single-frame target detection frame of the current frame.

[0138] Obtain motion information of the original point cloud data in the frame after the current frame, determine the motion trajectory of the original point cloud data based on the motion information, generate a single-frame target detection frame for each frame of the consecutive frames after the current frame based on the motion trajectory, and confirm whether the frames corresponding to the consecutive frames meet the consistency and preset annotation requirements. If it is determined that the consistency or the preset annotation requirements are not met, adjust the single-frame target detection frame that does not meet the requirements; the consistency is that the single-frame target detection frames of adjacent frames belong to one frame, and the size deviation, orientation, and overlap ratio between the single-frame target detection frames of adjacent frames meet the preset requirements.

[0139] In addition, the logic instructions in the aforementioned memory 330 can be implemented in the form of a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0140] On the other hand, the present invention further provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions. When the program instructions are executed by a computer, the computer is capable of performing the auxiliary annotation method for point cloud data provided by the above methods, the method comprising:

[0141] Obtaining raw point cloud data, determining the initial point of the current frame from the raw point cloud data, and extracting target data points of the current frame from the raw point cloud data based on the initial point; the distance between each target data point and the initial point does not exceed a preset search radius;

[0142] Generate an initial target detection frame based on the target data points;

[0143] Adjust the size of the initial target detection frame and project the original point cloud data within the adjusted initial target detection frame onto each two-dimensional plane to obtain the projection image of each two-dimensional plane. Perform size regression on all the projection images and aggregate the regressed projection images to obtain the single-frame target detection frame of the current frame.

[0144] Obtain motion information of the original point cloud data in the frame after the current frame, determine the motion trajectory of the original point cloud data based on the motion information, generate a single-frame target detection frame for each frame of the consecutive frames after the current frame based on the motion trajectory, and confirm whether the frames corresponding to the consecutive frames meet the consistency and preset annotation requirements. If it is determined that the consistency or the preset annotation requirements are not met, adjust the single-frame target detection frame that does not meet the requirements; the consistency is that the single-frame target detection frames of adjacent frames belong to one frame, and the size deviation, orientation, and overlap ratio between the single-frame target detection frames of adjacent frames meet the preset requirements.

[0145] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the above-mentioned method for performing auxiliary annotation of point cloud data, the method comprising:

[0146] Obtaining raw point cloud data, determining the initial point of the current frame from the raw point cloud data, and extracting target data points of the current frame from the raw point cloud data based on the initial point; the distance between each target data point and the initial point does not exceed a preset search radius;

[0147] Generate an initial target detection frame based on the target data points;

[0148] Adjust the size of the initial target detection frame and project the original point cloud data within the adjusted initial target detection frame onto each two-dimensional plane to obtain the projection image of each two-dimensional plane. Perform size regression on all the projection images and aggregate the regressed projection images to obtain the single-frame target detection frame of the current frame.

[0149] Obtain motion information of the original point cloud data in the frame after the current frame, determine the motion trajectory of the original point cloud data based on the motion information, generate a single-frame target detection frame for each frame of the consecutive frames after the current frame based on the motion trajectory, and confirm whether the frames corresponding to the consecutive frames meet the consistency and preset annotation requirements. If it is determined that the consistency or the preset annotation requirements are not met, adjust the single-frame target detection frame that does not meet the requirements; the consistency is that the single-frame target detection frames of adjacent frames belong to one frame, and the size deviation, orientation, and overlap ratio between the single-frame target detection frames of adjacent frames meet the preset requirements.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An auxiliary annotation method for point cloud data, characterized in that: The method comprises: Obtaining raw point cloud data, determining the initial point of the current frame from the raw point cloud data, and extracting target data points of the current frame from the raw point cloud data based on the initial point; the distance between each target data point and the initial point does not exceed a preset search radius; Generate an initial target detection frame based on the target data points; Adjust the size of the initial target detection frame and project the original point cloud data within the adjusted initial target detection frame onto each two-dimensional plane to obtain the projection image of each two-dimensional plane. Perform size regression on all the projection images and aggregate the regressed projection images to obtain the single-frame target detection frame of the current frame. Obtain motion information of the original point cloud data in the frame after the current frame, determine the motion trajectory of the original point cloud data based on the motion information, generate single-frame target detection frames for each frame of the consecutive frames after the current frame based on the motion trajectory, and confirm whether the frames corresponding to the consecutive frames meet the consistency and preset annotation requirements. If it is determined that the consistency or the preset annotation requirements are not met, adjust the single-frame target detection frames that do not meet the requirements; the consistency means that the single-frame target detection frames of adjacent frames belong to the same frame, and the size deviation, orientation, and overlap ratio between the single-frame target detection frames of adjacent frames meet the preset requirements; The single-frame object detection box of each frame in the continuous frames is obtained by the following steps: voxelize the original point cloud data of consecutive frames to obtain voxelized data; Extract point cloud feature information from voxelized data and use the self-attention mechanism to obtain contextual information from continuous frame point cloud data; Perform feature fusion on point cloud feature information and context information to obtain continuous frame feature information; Extract local spatial information of feature information of consecutive frames, and obtain motion information of the original point cloud data in the frame after the current frame based on the local spatial information; The motion trajectory of the original point cloud data is determined according to the motion information, and a single-frame target detection frame is generated for each frame of the consecutive frames after the current frame according to the motion trajectory.

2. The auxiliary annotation method of point cloud data according to claim 1, characterized in that: The obtaining of the original point cloud data, determining the initial point of the current frame from the original point cloud data, and extracting the target data point of the current frame from the original point cloud data based on the initial point, specifically includes: Obtain original point cloud data and determine the initial point of the current frame from the original point cloud data; With the initial point as the center point and the preset search radius as the interception radius, a search area is delineated from the original point cloud data; The original point cloud data within the search area is used as the target data point, and the target data point of the current frame is extracted.

3. The auxiliary annotation method of point cloud data according to claim 1, characterized in that: The initial target detection frame is generated by using a trained target detection model, where the target detection model is trained based on historical point cloud data and label information extracted from the historical point cloud data.

4. The auxiliary annotation method of point cloud data according to claim 3, characterized in that: The target detection model is trained by the following steps: Acquiring the historical point cloud data, and determining the tag information from the historical point cloud data; The historical point cloud data is used as input data for training, and the label information corresponding to the historical point cloud data is used as the label for training. Deep learning is used for training to obtain the target detection model for generating the initial target detection frame of the target data point.

5. The auxiliary annotation method of point cloud data according to claim 1, characterized in that: The method of adjusting the size of the initial target detection frame and projecting the original point cloud data within the adjusted initial target detection frame onto each two-dimensional plane to obtain projection images of each two-dimensional plane, performing size regression on all projection images and aggregating the regressed projection images to obtain a single-frame target detection frame of the current frame specifically includes: Keep the center point of the initial target detection frame unchanged and enlarge the size of the initial target detection frame by a preset multiple; Extract the original point cloud data within the adjusted initial target detection frame, and project the original point cloud data onto each two-dimensional plane to obtain the projection image of each two-dimensional plane; The projection center points of all projection planes and the regression detection frames are aggregated to obtain the single-frame target detection frame of the current frame.

6. The auxiliary annotation method of point cloud data according to claim 1, characterized in that: Before the steps of adjusting the size of the initial target detection frame, projecting the original point cloud data within the adjusted initial target detection frame onto each two-dimensional plane to obtain projection images of each two-dimensional plane, performing size regression on all projection images and aggregating the regressed projection images to obtain a single-frame target detection frame of the current frame, the method further includes: Determine the preset labeling requirements, verify the single-frame target frame according to the preset labeling requirements, and if it is determined that the verification fails, perform a frame adjustment operation on the single-frame target detection frame.

7. An auxiliary annotation device for point cloud data, characterized in that: The device comprises: A data extraction module is used to obtain raw point cloud data, determine the initial point of the current frame from the raw point cloud data, and extract the target data points of the current frame from the raw point cloud data based on the initial point; the distance between each target data point and the initial point does not exceed a preset search radius; The initial generation module is used to generate the initial target detection box based on the target data points; The size optimization module is used to adjust the size of the initial target detection frame and project the original point cloud data within the adjusted initial target detection frame onto various two-dimensional planes to obtain the projection images of each two-dimensional plane. The size of all the projected images is regressed and the regressed projection images are aggregated to obtain the single-frame target detection frame of the current frame. The continuous frame generation module is used to obtain the motion information of the original point cloud data in the frame after the current frame, determine the motion trajectory of the original point cloud data based on the motion information, generate a single-frame target detection frame for each frame of the continuous frames after the current frame based on the motion trajectory, and confirm whether the frames corresponding to the continuous frames meet the consistency and preset labeling requirements. If it is determined that the consistency or the preset labeling requirements are not met, adjust the unsatisfactory single-frame target detection frame; the consistency is that the single-frame target detection frames of adjacent frames belong to the same frame, and the size deviation, orientation, and overlap ratio between the single-frame target detection frames of adjacent frames meet the preset requirements; The single-frame object detection box of each frame in the continuous frames is obtained by the following steps: voxelize the original point cloud data of consecutive frames to obtain voxelized data; Extract point cloud feature information from voxelized data and use the self-attention mechanism to obtain contextual information from continuous frame point cloud data; Perform feature fusion on point cloud feature information and context information to obtain continuous frame feature information; Extract local spatial information of feature information of consecutive frames, and obtain motion information of the original point cloud data in the frame after the current frame based on the local spatial information; The motion trajectory of the original point cloud data is determined according to the motion information, and a single-frame target detection frame is generated for each frame of the consecutive frames after the current frame according to the motion trajectory.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the auxiliary annotation method for point cloud data according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the auxiliary annotation method for point cloud data as described in any one of claims 1 to 6 are implemented.

Images