Auxiliary marking method and device for point cloud data, equipment and medium

Abstract

The invention discloses an auxiliary marking method and device for point cloud data, equipment and a medium, and relates to the technical field of data processing, and the method comprises the steps: obtaining original point cloud data, determining an initial point of a current frame from the original point cloud data, and extracting a target data point of the current frame from the original point cloud data based on the initial point; generating an initial target detection frame according to the target data point; adjusting the size of the initial target detection frame, projecting original point cloud data in the frame to each two-dimensional plane, performing size regression on all projection images, and aggregating the regressed projection images to obtain a single-frame target detection frame of the current frame; and determining a motion trail of the original point cloud data, and generating a single-frame target detection frame of each frame of continuous frames after the current frame according to the motion trail. According to the method, the detection frame of continuous frames can be generated, the workload of manual labeling can be greatly reduced, and the accuracy of the frame body center point and the size information is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an auxiliary annotation method, device, equipment and medium for point cloud data. Background Art

[0002] With the rapid development of fields such as autonomous driving, robot vision, and geographic information systems, the demand for annotating three-dimensional point cloud data is also increasing day by day. In order to improve the performance of data in the above fields, more accurate three-dimensional object detection and annotation have become essential foundations. However, currently in the process of three-dimensional point cloud annotation, a large amount of manual intervention is still relied on. Especially in the annotation of large-scale data sets, the traditional annotation method not only takes a lot of time and effort, but also easily generates large data errors.

[0003] Although there are already some three-dimensional point cloud object detection methods based on deep learning (such as PV-RCNN, CenterPoint, DSVT, etc.) that can perform object detection, most of these algorithms only consider single-frame object recognition and do not fully combine the generation of object detection boxes with continuous-frame object tracking.

[0004] To solve these problems, it is necessary to provide a new three-dimensional point cloud object detection method. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an auxiliary annotation method, device, equipment and medium for point cloud data to solve the problem that the annotation of three-dimensional point cloud data is not accurate enough.

[0006] According to a first aspect, an embodiment of the present invention provides an auxiliary annotation method for point cloud data, the method comprising: Obtain original point cloud data, determine an initial point of the current frame from the original point cloud data, and extract target data points of the current frame from the original point cloud data based on the initial point; the distance between each target data point and the initial point does not exceed a preset search radius; Generate an initial object detection box according to the target data points; Adjust the size of the initial object detection box, project the original point cloud data located within the adjusted initial object detection box onto each two-dimensional plane to obtain projection images of each two-dimensional plane, perform size regression on all projection images and aggregate the regression-projected images to obtain a single-frame object detection box of the current frame; Obtain the motion information of the original point cloud data in the frames after the current frame, determine the motion trajectory of the original point cloud data according to the motion information, generate a single-frame target detection box for each frame of the consecutive frames after the current frame according to the motion trajectory, and confirm whether the boxes corresponding to the consecutive frames meet the consistency and the preset annotation requirements. In the case of determining that the consistency or the preset annotation requirements are not met, adjust the single-frame target detection boxes that do not meet the requirements; the consistency means that the single-frame target detection boxes of adjacent frames belong to one box, and the size deviation, orientation, and coincidence ratio between the single-frame target detection boxes of adjacent frames all meet the preset requirements.

[0007] Combined with the first aspect, in the first embodiment of the first aspect, the obtaining of the original point cloud data, determining the initial point of the current frame from the original point cloud data, and extracting the target data points of the current frame based on the initial point specifically include: Obtain the original point cloud data and determine the initial point of the current frame from the original point cloud data; Taking the initial point as the center point and the preset search radius as the intercept radius, delimit a search area from the original point cloud data; Take the original point cloud data located in the search area as the target data points and extract the target data points of the current frame.

[0008] Combined with the first aspect, in the second embodiment of the first aspect, the initial target detection box is generated by the trained target detection model, and the target detection model is trained based on the historical point cloud data and the label information extracted from the historical point cloud data.

[0009] Combined with the second embodiment of the first aspect, in the third embodiment of the first aspect, the target detection model is trained through the following steps: Obtain the historical point cloud data and determine the label information from the historical point cloud data; Take the historical point cloud data as the input data for training, take the label information corresponding to the historical point cloud data as the label for training, and perform training in a deep learning manner to obtain the target detection model for generating the initial target detection box of the target data points.

[0010] Combined with the first aspect, in the fourth embodiment of the first aspect, adjusting the size of the initial target detection box, projecting the original point cloud data located within the adjusted initial target detection box onto each two-dimensional plane to obtain the projection images of each two-dimensional plane, performing size regression on all the projection images and aggregating the projection images after regression to obtain the single-frame target detection box of the current frame specifically includes: Keep the center point of the initial target detection box unchanged and enlarge the size of the initial target detection box by a preset multiple; Extract the original point cloud data located within the adjusted initial target detection box, and project the original point cloud data onto each two-dimensional plane to obtain the projection images of each two-dimensional plane; Aggregate the projection center points of all projection planes and the regression detection boxes to obtain the single-frame target detection box for the current frame.

[0011] Combined with the first aspect, in the fifth implementation manner of the first aspect, before the steps of adjusting the size of the initial target detection box, projecting the original point cloud data located within the adjusted initial target detection box onto each two-dimensional plane to obtain the projection images of each two-dimensional plane, performing size regression on all projection images, and aggregating the regression-projected images to obtain the single-frame target detection box for the current frame, the method further includes: Determine the preset annotation requirements, verify the single-frame target box according to the preset annotation requirements, and perform a frame adjustment operation on the single-frame target detection box when it is determined that the verification fails.

[0012] Combined with the first aspect, in the sixth implementation manner of the first aspect, the single-frame target detection box for each frame in a continuous frame is obtained through the following steps: Perform voxelization processing on the original point cloud data of the continuous frame to obtain voxelized data; Extract the point cloud feature information of the voxelized data and obtain the context information of the continuous frame point cloud data by using the self-attention mechanism; Perform feature fusion on the point cloud feature information and the context information to obtain continuous frame feature information; Extract the local spatial information of the continuous frame feature information, and obtain the motion information of the original point cloud data in the frame after the current frame according to the local spatial information; Determine the motion trajectory of the original point cloud data according to the motion information, and generate the single-frame target detection box for each frame in the continuous frames after the current frame according to the motion trajectory.

[0013] According to the second aspect, an auxiliary annotation device for point cloud data is further provided in an embodiment of the present invention. The device includes: A data extraction module, configured to obtain the original point cloud data, determine the initial points of the current frame from the original point cloud data, and extract the target data points of the current frame from the original point cloud data based on the initial points; the distance between each target data point and the initial point does not exceed a preset search radius; An initial generation module, configured to generate an initial target detection box according to the target data points; A size optimization module, configured to adjust the size of the initial target detection box, project the original point cloud data located within the adjusted initial target detection box onto each two-dimensional plane to obtain the projection images of each two-dimensional plane, perform size regression on all projection images, and aggregate the regression-projected images to obtain the single-frame target detection box for the current frame; A continuous frame generation module is configured to obtain the motion information of the original point cloud data in the frames after the current frame, determine the motion trajectory of the original point cloud data according to the motion information, generate a single-frame target detection box for each frame of the continuous frames after the current frame according to the motion trajectory, and confirm whether the boxes corresponding to the continuous frames meet the consistency and the preset annotation requirements. In the case where it is determined that the consistency or the preset annotation requirements are not met, the unsatisfied single-frame target detection boxes are adjusted; the consistency means that the single-frame target detection boxes of adjacent frames belong to one box, and the size deviation, orientation, and coincidence ratio between the single-frame target detection boxes of adjacent frames all meet the preset requirements.

[0014] According to a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the method for assisting in annotating point cloud data as described in any one of the above are implemented.

[0015] According to a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for assisting in annotating point cloud data as described in any one of the above are implemented.

[0016] The method, device, equipment, and medium for assisting in annotating point cloud data of the present invention select an initial point, determine a search area based on the position of the initial point, extract target data points that are close enough to the initial point from the original point cloud data based on the search area, then automatically generate an initial target detection box according to the target data points, and then optimize the size of the initial target detection box to obtain the single-frame target detection box of the current frame. Using the motion information of the frames after the current frame, the motion trajectory of the original point cloud data is determined according to the motion information, and a single-frame target detection box for each frame of the continuous frames after the current frame is generated according to the motion trajectory. By generating the detection boxes of the continuous frames in this way, the workload of manual annotation can be greatly reduced and the accuracy of the center point and size information of the box can be improved. In the process of optimizing the size of the initial target detection box, a multi-directional projection optimization technology is adopted, which can accurately adjust the size and position of the target box to ensure high-precision annotation. In the process of generating the single-frame target detection boxes of the continuous frames based on the single-frame target detection box of the current frame, the target is continuously tracked, and the target in the dynamic scene can be stably processed, which is applicable to complex dynamic environments, such as multiple fields including autonomous driving, robot vision, and geographic information systems, and can process different types of three-dimensional point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings. The drawings are schematic and should not be construed as limiting the present invention in any way. In the drawings: Figure 1 The flowchart of the auxiliary annotation method for point cloud data provided by the present invention is shown; Figure 2 The structural schematic diagram of the auxiliary annotation device for point cloud data provided by the present invention is shown; Figure 3 It is the hardware structural schematic diagram of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0018] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0019] With the rapid development of fields such as autonomous driving, robot vision, and geographic information systems, the demand for annotating three-dimensional point cloud data is also increasing day by day. To improve the performance of data in the above fields, more accurate three-dimensional object detection and annotation have become essential foundations. However, currently in the process of three-dimensional point cloud annotation, a large amount of manual intervention is still relied on. Especially in the annotation of large-scale data sets, the traditional annotation method is not only time-consuming and laborious, but also prone to large data errors.

[0020] Although there are already some three-dimensional point cloud object detection methods based on deep learning (such as PV-RCNN, CenterPoint, DSVT, etc.) that can perform object detection, most of these algorithms only consider the object recognition of a single frame and do not fully combine the generation of object detection boxes and continuous frame object tracking.

[0021] In summary, it is necessary to provide a new three-dimensional point cloud object detection method.

[0022] To solve the above problems, in this embodiment, an auxiliary annotation method for point cloud data is provided, aiming to generate detection boxes for continuous frames, greatly reducing the workload of manual annotation and improving the accuracy of the center point and size information of the boxes. The auxiliary annotation method for point cloud data in the embodiments of the present invention can be used in an electronic device, and the electronic device includes but is not limited to a computer, a mobile terminal, etc. Figure 1 It is the flowchart of the auxiliary annotation method for point cloud data according to the embodiments of the present invention. As Figure 1 shown, the method may include the following steps: S10. Obtain the original point cloud data, determine the initial points of the current frame from the original point cloud data, and extract the target data points of the current frame from the original point cloud data based on the initial points.

[0023] The obtained original point cloud data is a continuous sequence of point clouds. The original point cloud data contains the point cloud data of continuous M frames, that is, the point cloud data of M frames at consecutive moments.

[0024] In this embodiment, the target data points include initial points, and the distance between each target data point and the initial point does not exceed a preset search radius. That is, the target data points are a subset of point clouds extracted from the original point cloud data that are close enough to the initial point.

[0025] For example, with the initial point as the center point, it is possible to determine target data points that meet the requirements from the original point cloud data based on the initial point of the current frame.

[0026] The original point cloud data can be stored in the electronic device in advance, or can be obtained by the electronic device from the outside world. For example, the electronic device obtains it from an external acquisition device, or extracts video image frames from an external acquisition video, etc.

[0027] There is no limitation on the specific acquisition form of the original point cloud data here, as long as it is ensured that the electronic device can obtain the original point cloud data.

[0028] S20. Generate an initial target detection box according to the target data points.

[0029] In the generation stage of the initial target detection box, the core task is to calculate the bounding box of the target based on the initial point (i.e., coordinate hint) selected by the user and the target data points. In this way, it is possible to first obtain an accurate point position information provided by the user, and based on this point position information, assist the user in generating the bounding box of the target. After the user provides the initial point in the previous step, the point cloud data of the target area has been preliminarily extracted.

[0030] In this step, an initial target detection box is generated through a trained target detection model. Specifically, the target data points are input into the trained target detection model, and the initial target detection box is output by the target detection model based on the target data points. Among them, the initial target detection box includes the center point of the target , the box information and the category information. represents the x-axis coordinate of the center point of the initial target detection box, represents the y-axis coordinate of the center point of the initial target detection box, represents the z-axis coordinate of the center point of the initial target detection box, represents the length of the initial target detection box, represents the width of the initial target detection box, represents the height of the initial target detection box.

[0031] The target detection model can be trained using historical point cloud data and label information extracted from the historical point cloud data. For example, first obtain the historical point cloud data, determine the label information from the historical point cloud data, then use the historical point cloud data as the input data for training, use the label information corresponding to the historical point cloud data as the label for training, and then use deep learning methods for training to obtain the target detection model for generating the initial target detection box of the target data points. Here, the specific training method of the target detection model is not limited.

[0032] Preferably, an existing 3D target detection model (such as PointNet++ or PV-RCNN) can also be used to generate the initial target detection box, input the extracted point cloud data, and perform target detection. These models will output an initial target detection box representing the bounding box of the target, and this bounding box also includes the center point of the target / object. , the bounding box information and the category information.

[0033] S30. Adjust the size of the initial target detection box, project the original point cloud data located within the adjusted initial target detection box onto each two-dimensional plane to obtain the projection images of each two-dimensional plane, perform size regression on all the projection images, and aggregate the regression-projected images to obtain the single-frame target detection box of the current frame.

[0034] In this embodiment, this step will optimize the generated initial target detection box to expand the original point cloud data contained within the detection box, and then project these original data point clouds onto each two-dimensional plane. For three-dimensional point cloud data, there are a total of three two-dimensional planes onto which it can be projected, namely the x-y, x-z, and y-z planes, and the projection results obtained on each two-dimensional plane are the projection images. Next, performing size regression on the projection images can obtain the size of the initial target detection box in the two-dimensional plane. Finally, by aggregating the results of size regression of each two-dimensional plane, the center coordinates and size of the finally optimized target box, that is, the single-frame target detection box of the current frame, can be obtained. It can be understood that the obtained single-frame target detection box contains the center point and size information of the box body.

[0035] By first expanding the initial target detection box to include more original point cloud data within the box, and then performing a two-dimensional plane projection on the original point cloud data, the central coordinates of these original point cloud data on the two-dimensional plane and the two-dimensional dimension box information formed based on these original point cloud data are obtained. After that, size regression processing and association processing are carried out. Finally, the box information of the single-frame target detection box of the current frame obtained is more accurate, enabling the single-frame target detection box of this frame to meet the accuracy requirements and laying a foundation for calculating the single-frame target detection box of each subsequent continuous frame based on the single-frame target detection box of this frame that meets the accuracy requirements.

[0036] S40. Obtain the motion information of the original point cloud data in the frame after the current frame, determine the motion trajectory of the original point cloud data according to the motion information, generate the single-frame target detection box of each frame in the subsequent continuous frames after the current frame according to the motion trajectory, and confirm whether the boxes corresponding to the continuous frames meet the consistency and preset annotation requirements. In the case where it is determined that the consistency or preset annotation requirements are not met, adjust the single-frame target detection box that does not meet the requirements.

[0037] In this embodiment, the consistency of the boxes mainly means that these single-frame target detection boxes belong to one box, and the changes in the single-frame target detection boxes between adjacent frames are relatively natural. For example, the orientations are kept consistent, that is, the size deviation, orientation, and coincidence ratio between the single-frame target detection boxes of adjacent frames all meet the preset requirements. In the case where it is determined that they are inconsistent or do not meet the preset annotation requirements, the user can, according to the situation: if the deviation of the box is relatively small, fine-tuning can be selected; if the deviation of the box is relatively large, the box generation result can be selected to be deleted, and steps S10 to S40 are re-executed for annotation, so as to generate the single-frame target detection boxes at other times.

[0038] Specifically: perform voxelization processing on the original point cloud data of the continuous frames to obtain voxelized data, and the voxelized data contains the voxel coordinates of the original original point cloud data. After that, extract the point cloud feature information of the voxelized data and use the self-attention mechanism to obtain the context information of the continuous frame point cloud data. Next, perform feature fusion on the point cloud feature information and the context information to obtain the continuous frame feature information. Then, extract the local spatial information of the continuous frame feature information, so that the feature containing spatial information can be obtained. Finally, obtain the motion information of the original point cloud data in the frame after the current frame according to the local spatial information, determine the motion trajectory of the original point cloud data according to the motion information, and generate the single-frame target detection box of each frame in the subsequent continuous frames after the current frame according to the motion trajectory.

[0039] Suppose in the current frame, the target detection box is while in the previous frame, the target detection box is , the movement of the target in consecutive frames can be estimated by a tracking algorithm:

[0040] Among them, represents the motion trajectory of the length of the target detection box between adjacent frames; represents the motion trajectory of the width of the target detection box between adjacent frames; represents the motion trajectory of the height of the target detection box between adjacent frames.

[0041] Using these motion trajectories, and based on the registration and motion of the point clouds of the previous and subsequent frames, the position change of the target in the subsequent frames is estimated, and a new single-frame target detection box for each frame is generated.

[0042] In order to generate a single-frame target detection box for each frame in consecutive frames, in this embodiment, common single-object tracking algorithms for point clouds (such as M2Track, MBPTrack, etc.) can also be used to track the motion trajectory of the point cloud, and a single-frame target detection box for consecutive frames is generated based on the motion trajectory.

[0043] The method for assisting in annotating point cloud data of the present invention selects an initial point, determines a search area based on the position of the initial point, extracts target data points that are close enough to the initial point from the original point cloud data, then automatically generates an initial target detection box according to the target data points, and then optimizes the size of the initial target detection box to obtain the single-frame target detection box of the current frame. Using the motion information of the frames after the current frame, the motion trajectory of the original point cloud data is determined according to the motion information, and a single-frame target detection box for each frame of the consecutive frames after the current frame is generated. By generating the detection boxes of consecutive frames in this way, the workload of manual annotation can be greatly reduced and the accuracy of the center point and size information of the box body can be improved. In the process of optimizing the size of the initial target detection box, a multi-directional projection optimization technique is adopted, which can accurately adjust the size and position of the target box to ensure high-precision annotation. In the process of generating a single-frame target detection box for consecutive frames based on the single-frame target detection box of the current frame, by continuously tracking the target, it can stably process the targets in dynamic scenes and is applicable to complex dynamic environments, such as multiple fields including autonomous driving, robot vision, and geographic information systems, and can process different types of three-dimensional point cloud data.

[0044] In this embodiment, step S10 specifically includes: S11. Obtain the original point cloud data and determine the initial point of the current frame from the original point cloud data.

[0045] Preferably, the user, such as an annotator, clicks on the approximate position of the target object to select the initial point of the current frame. Assume that the coordinates of the three-dimensional point cloud selected by the user's click in the coordinate system are , where represents the x-axis coordinate of the initial point; represents the y-axis coordinate of the initial point; represents the z-axis coordinate of the initial point. In this way, a search area can be determined based on the position of the initial point, and a subset of point clouds that is close enough to the initial point can be extracted from the original point cloud data to obtain the target data points.

[0046] S12: With the initial point as the center point and a preset search radius as the intercept radius, delimit a search area from the original point cloud data.

[0047] Specifically, set a preset search radius to determine the size of the intercepted search area. In this embodiment, the search range of the search area is a spherical area or a cubic area.

[0048] S13: Take the original point cloud data located within the search area as the target data points, and extract the target data points of the current frame.

[0049] Particularly, for the search area of the spherical area, the search range of the search area is:

[0050] where represents the subset of point clouds included in the search area, that is, the set of target data points; represents the target data point; represents the x-axis coordinate of the th target data point; represents the th y-axis coordinate of the target data point;

[0051] Particularly, for the search area of the cubic area, the search range of the search area is:

[0052] In this embodiment, step S30 specifically includes: S31: Keep the center point of the initial target detection frame unchanged, and enlarge the size of the initial target detection frame by a preset multiple.

[0053] In this embodiment, when optimizing, the size of the initial target detection frame is enlarged by a preset multiple , and the center point remains unchanged during the enlargement process to ensure that more original point cloud data is included in the frame.

[0054] Therefore, the adjusted frame size is:

[0055] Among them, represents the length of the adjusted initial target detection box corresponding to the th frame; represents the width of the adjusted initial target detection box corresponding to the th frame; represents the height of the adjusted initial target detection box corresponding to the th frame.

[0056] S32. Extract the original point cloud data located within the adjusted initial target detection box, and project the original point cloud data onto each two-dimensional plane to obtain the projected images of each two-dimensional plane.

[0057] Next, extract the original point cloud data within the enlarged box body. It can be understood that this point cloud data contains the target data points obtained previously.

[0058] The projection process specifically projects these point cloud data in three directions: Front-back projection: Select the - plane for projection to obtain the front-back direction projected image .

[0059] Up-down projection: Select the - plane for projection to obtain the up-down direction projected image .

[0060] Left-right projection: Select the - plane for projection to obtain the left-right direction projected image .

[0061] S33. Respectively determine the projection center points of each projection center, and perform size regression processing on the projected images according to the projection center points to obtain the regression detection boxes of each projection center.

[0062] Preferably, for each projected image, a convolutional neural network can be used to train the key point detection model and the size regression model respectively, and the trained key point detection model and size regression model are used to determine the projection center point and the regression size respectively.

[0063] S34. Aggregate the projection center points and regression detection boxes of all projection planes to obtain the single-frame target detection box of the current frame.

[0064] For the projected image, its key point is the projection center point. Assume that the projection center point of each projected image is determined as , the regression detection box obtained after size regression processing based on the projection center point is , where , then the single-frame object detection box of the current frame obtained by aggregating the projection results is:

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] where represents the x-axis coordinate of the center point after aggregation of the single-frame object detection box; represents the y-axis coordinate of the center point after aggregation of the single-frame object detection box; represents the length after aggregation of the single-frame object detection box; represents the width after aggregation of the single-frame object detection box; represents the height after aggregation of the single-frame object detection box.

[0071] Through this aggregation method, the center coordinates and size information corresponding to the single-frame object detection box can be obtained, and more refined box information can be obtained.

[0072] In this embodiment, after step S30, the method further includes the following steps: Determine the preset annotation requirements, verify the single-frame object box according to the preset annotation requirements, and perform a box adjustment operation on the single-frame object detection box when it is determined that the verification fails.

[0073] After optimizing the initial object detection box and obtaining the single-frame object detection box, the user can also verify the optimized box and confirm whether it meets the preset annotation requirements such as fitting degree, size, orientation, and contour clarity. If the frame object detection box verification fails, for the single-frame object detection box that fails the verification, that is, does not meet the requirements (for example, there is a large deviation in the box size), a box adjustment operation is performed. In this embodiment, the box adjustment operation includes but is not limited to operations such as dragging, rotating, and scaling the single-frame object detection box. Specifically, for example, the user can use the block diagram fine-tuning tool to manually adjust the size and position of the box. Assume that the fine-tuning operation is to adjust the boundary of the box from to , the fine-tuned box is:

[0074] Among them, represents the box after fine-tuning.

[0075] During the annotation process, the annotator only needs to fine-tune a small number of boxes that do not meet the requirements, and most of the detection boxes can be generated through the above-mentioned automated process.

[0076] Next, the auxiliary annotation device for point cloud data provided by the embodiments of the present invention will be described. The auxiliary annotation device for point cloud data described below can be correspondingly referred to the auxiliary annotation method for point cloud data described above.

[0077] To solve the above problems, in this embodiment, an auxiliary annotation device for point cloud data is provided, aiming to generate detection boxes for consecutive frames, greatly reducing the workload of manual annotation and improving the accuracy of the center point and size information of the boxes. Figure 2 is a schematic structural diagram of the auxiliary annotation method for point cloud data according to the embodiments of the present invention, as Figure 2 shown, this device may include: A data extraction module 10, configured to obtain original point cloud data, determine an initial point of the current frame from the original point cloud data, and extract target data points of the current frame from the original point cloud data based on the initial point.

[0078] The obtained original point cloud data is a point cloud continuous sequence, and the original point cloud data includes point cloud data of consecutive M frames, that is, consecutive M frames of point cloud data at time.

[0079] In this embodiment, the target data points include the initial point, and the distance between each target data point and the initial point does not exceed a preset search radius, that is, the target data points are a subset of point clouds extracted from the original point cloud data that are close enough to the initial point.

[0080] For example, taking the initial point as the center point, the target data points that meet the requirements can be determined from the original point cloud data based on the initial point of the current frame.

[0081] The original point cloud data can be stored in the electronic device in advance, or can be obtained by the electronic device from the outside. For example, the electronic device obtains it from an external acquisition device, or extracts video image frames from an external acquisition video, etc.

[0082] Here, no specific limitation is imposed on the specific acquisition form of the original point cloud data, as long as it is ensured that the electronic device can obtain the original point cloud data.

[0083] An initial generation module 20, configured to generate an initial target detection box according to the target data points.

[0084] In the initial stage of generating the target detection box, the core task is to calculate the bounding box of the target based on the initial points (i.e., coordinate hints) selected by the user and the target data points. After the user provided the initial points in the previous step, the point cloud data of the target area has been preliminarily extracted.

[0085] In this step, the trained target detection model is used to generate the initial target detection box. Specifically, the target data points are input into the trained target detection model, and the initial target detection box is output by the target detection model based on the target data points. Among them, the initial target detection box includes the center point of the target , the box body information and the category information. represents the x-axis coordinate of the center point, represents the y-axis coordinate of the center point, represents the z-axis coordinate of the center point, represents the length of the initial target detection box, represents the width of the initial target detection box, represents the height of the initial target detection box.

[0086] The target detection model can be trained by historical point cloud data and label information determined from the historical point cloud data. The specific training method of the target detection model is not limited here.

[0087] Preferably, an existing 3D target detection model (such as PointNet++ or PV-RCNN) can also be used to generate the initial target detection box. The extracted point cloud data is input for target detection. These models will output an initial target detection box representing the bounding box of the target, and this bounding box also includes the center point of the target / object , the box body information and the category information.

[0088] The size optimization module 30 is used to adjust the size of the initial target detection box, project the original point cloud data located within the adjusted initial target detection box onto each two-dimensional plane to obtain the projection images of each two-dimensional plane, perform size regression on all the projection images, and aggregate the regression-projected images to obtain the single-frame target detection box of the current frame.

[0089] In this embodiment, this step will optimize the generated initial target detection box to expand the original point cloud data contained within the detection box, and then project these original data point clouds onto each two-dimensional plane. For three-dimensional point cloud data, there are a total of three two-dimensional planes onto which it can be projected, namely the x-y, x-z, and y-z planes, and the projection results obtained on each two-dimensional plane are the projection images. Next, by performing size regression on the projection images, the size of the initial target detection box on the two-dimensional plane can be obtained. Finally, by aggregating the results of the size regression of each two-dimensional plane, the center coordinates and size of the finally optimized target box can be obtained, which is the single-frame target detection box of the current frame. It can be understood that the obtained single-frame target detection box contains the center point and size information of the box body.

[0090] The continuous frame generation module 40 is configured to obtain the motion information of the original point cloud data in the frames after the current frame, determine the motion trajectory of the original point cloud data according to the motion information, generate the single-frame target detection box for each frame of the continuous frames after the current frame according to the motion trajectory, and confirm whether the box bodies corresponding to the continuous frames meet the consistency and preset annotation requirements. In the case where it is determined that they do not meet the consistency or preset annotation requirements, the non-conforming single-frame target detection boxes are adjusted.

[0091] In this embodiment, the consistency of the box bodies mainly means that these single-frame target detection boxes belong to one box body, and the changes in the single-frame target detection boxes between adjacent frames are relatively natural. For example, the orientations are kept consistent, that is, the size deviation, orientation, and coincidence ratio between the single-frame target detection boxes of adjacent frames all meet the preset requirements. In the case where it is determined that they are inconsistent or do not meet the preset annotation requirements, the user can, according to the situation: if the deviation of the box is relatively small, fine-tuning can be selected; if the deviation of the box is relatively large, the generated result of the box body can be selected to be deleted, and annotation can be started from the beginning to generate the single-frame target detection boxes at other times.

[0092] Assume that in the current frame, the target detection box is , and in the frame of the previous frame, the target detection box is , and the motion of the target in the continuous frames can be estimated through a tracking algorithm:

[0093] Among them, represents the length of the adjusted initial target detection box corresponding to the th frame; represents the width of the adjusted initial target detection box corresponding to the th frame; represents the height of the adjusted initial target detection box corresponding to the th frame.

[0094] Use these motion trajectories, and estimate the position change of the target in subsequent frames based on the registration and motion of the point clouds in the front and back frames, and generate a new single-frame target detection box for each frame.

[0095] The auxiliary annotation device for point cloud data of the present invention selects an initial point, determines a search area based on the position of the initial point, extracts target data points that are close enough to the initial point from the original point cloud data, then automatically generates an initial target detection box according to the target data points, and optimizes the size of the initial target detection box to obtain the single-frame target detection box of the current frame. Using the motion information of the frames after the current frame, the motion trajectory of the original point cloud data is determined according to the motion information, and the single-frame target detection box of each frame of the continuous frames after the current frame is generated. By generating the detection boxes of the continuous frames in this way, the workload of manual annotation can be greatly reduced and the accuracy of the center point and size information of the box can be improved. In the process of optimizing the size of the initial target detection box, a multi-directional projection optimization technique is adopted, which can accurately adjust the size and position of the target box to ensure high-precision annotation. In the process of generating the single-frame target detection box of the continuous frames based on the single-frame target detection box of the current frame, the target is continuously tracked, and the target in the dynamic scene can be stably processed, which is applicable to complex dynamic environments, such as multiple fields of autonomous driving, robot vision, geographic information system, etc., and can process different types of three-dimensional point cloud data.

[0096] Figure 3 An example of the physical structure diagram of an electronic device is shown as Figure 3 shown. The electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340. Among them, the processor 310, the communication interface 320, and the memory 330 complete mutual communication through the communication bus 340. The processor 310 can call the logical commands in the memory 330 to execute the auxiliary annotation method for point cloud data, and the method includes: Obtain the original point cloud data, determine the initial point of the current frame from the original point cloud data, and extract the target data points of the current frame from the original point cloud data based on the initial point; the distance between each target data point and the initial point does not exceed a preset search radius; Generate an initial target detection box according to the target data points; Adjust the size of the initial target detection box, project the original point cloud data located within the adjusted initial target detection box onto each two-dimensional plane to obtain the projection images of each two-dimensional plane, perform size regression on all projection images and aggregate the regression-projected images to obtain the single-frame target detection box of the current frame; Obtain the motion information of the original point cloud data in the frames after the current frame, determine the motion trajectory of the original point cloud data according to the motion information, generate a single-frame target detection box for each frame of the consecutive frames after the current frame according to the motion trajectory, and confirm whether the boxes corresponding to the consecutive frames meet the consistency and the preset annotation requirements. In the case of determining that the consistency or the preset annotation requirements are not met, adjust the single-frame target detection boxes that do not meet the requirements; the consistency means that the single-frame target detection boxes of adjacent frames belong to one box, and the size deviation, orientation, and coincidence ratio between the single-frame target detection boxes of adjacent frames all meet the preset requirements.

[0097] In addition, when the logic instructions in the above-mentioned memory 330 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memor), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0098] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the method for assisting in annotating point cloud data provided by the above-mentioned various methods. The method includes: Obtain the original point cloud data, determine the initial point of the current frame from the original point cloud data, and extract the target data points of the current frame from the original point cloud data based on the initial point; the distance between each target data point and the initial point does not exceed the preset search radius. Generate an initial target detection box according to the target data points. Adjust the size of the initial target detection box, project the original point cloud data located within the adjusted initial target detection box onto each two-dimensional plane to obtain the projection images of each two-dimensional plane, perform size regression on all the projection images and aggregate the regression-projected images to obtain the single-frame target detection box of the current frame. Obtain the motion information of the original point cloud data in the frames after the current frame, determine the motion trajectory of the original point cloud data according to the motion information, generate a single-frame target detection box for each frame of the consecutive frames after the current frame according to the motion trajectory, and confirm whether the boxes corresponding to the consecutive frames meet the consistency and the preset annotation requirements. In the case of determining that the consistency or the preset annotation requirements are not met, adjust the single-frame target detection boxes that do not meet the requirements; the consistency means that the single-frame target detection boxes of adjacent frames belong to one box, and the size deviation, orientation, and coincidence ratio between the single-frame target detection boxes of adjacent frames all meet the preset requirements.

[0099] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is used to execute the auxiliary annotation method of point cloud data provided above. The method includes: Obtain the original point cloud data, determine the initial points of the current frame from the original point cloud data, and extract the target data points of the current frame from the original point cloud data based on the initial points; the distance between each target data point and the initial point does not exceed the preset search radius. Generate an initial target detection box according to the target data points. Adjust the size of the initial target detection box, project the original point cloud data located inside the adjusted initial target detection box onto each two-dimensional plane to obtain the projection images of each two-dimensional plane, perform size regression on all the projection images and aggregate the regression-projected images to obtain the single-frame target detection box of the current frame. Obtain the motion information of the original point cloud data in the frames after the current frame, determine the motion trajectory of the original point cloud data according to the motion information, generate a single-frame target detection box for each frame of the consecutive frames after the current frame according to the motion trajectory, and confirm whether the boxes corresponding to the consecutive frames meet the consistency and the preset annotation requirements. In the case of determining that the consistency or the preset annotation requirements are not met, adjust the single-frame target detection boxes that do not meet the requirements; the consistency means that the single-frame target detection boxes of adjacent frames belong to one box, and the size deviation, orientation, and coincidence ratio between the single-frame target detection boxes of adjacent frames all meet the preset requirements.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An auxiliary annotation method for point cloud data, characterized in that The method includes: Obtain the original point cloud data, determine the initial points of the current frame from the original point cloud data, and extract the target data points of the current frame from the original point cloud data based on the initial points; the distance between each target data point and the initial point does not exceed a preset search radius. Generate an initial target detection box according to the target data points. Adjust the size of the initial target detection box, project the original point cloud data located within the adjusted initial target detection box onto each two-dimensional plane to obtain projection images of each two-dimensional plane, perform size regression on all the projection images, and aggregate the regression-projected images to obtain the single-frame target detection box of the current frame. Obtain the motion information of the original point cloud data in the frames after the current frame, determine the motion trajectory of the original point cloud data according to the motion information, generate the single-frame target detection box of each frame in the consecutive frames after the current frame according to the motion trajectory, and confirm whether the boxes corresponding to the consecutive frames meet the consistency and preset annotation requirements. In the case of determining that the consistency or the preset annotation requirements are not met, adjust the non-conforming single-frame target detection boxes; the consistency means that the single-frame target detection boxes of adjacent frames belong to one box, and the size deviation, orientation, and coincidence ratio between the single-frame target detection boxes of adjacent frames all meet the preset requirements.

2. The auxiliary annotation method for point cloud data according to claim 1, wherein The obtaining of the original point cloud data, determining the initial points of the current frame from the original point cloud data, and extracting the target data points of the current frame from the original point cloud data based on the initial points specifically includes: Obtain the original point cloud data and determine the initial points of the current frame from the original point cloud data. With the initial points as the center points and the preset search radius as the intercepting radius, delimit a search area from the original point cloud data. Use the original point cloud data located within the search area as the target data points and extract the target data points of the current frame.

3. The auxiliary annotation method for point cloud data according to claim 1, wherein Generate the initial target detection box through the trained target detection model. The target detection model is trained based on the historical point cloud data and the label information extracted from the historical point cloud data.

4. The auxiliary annotation method for point cloud data according to claim 3, wherein, The target detection model is trained through the following steps: Obtain the historical point cloud data and determine the label information from the historical point cloud data. Use the historical point cloud data as the input data for training, use the label information corresponding to the historical point cloud data as the label for training, and perform training in a deep learning manner to obtain the target detection model for generating the initial target detection box of the target data points.

5. The auxiliary annotation method for point cloud data according to claim 1, wherein The adjusting of the size of the initial target detection box, projecting the original point cloud data located within the adjusted initial target detection box onto each two-dimensional plane to obtain projection images of each two-dimensional plane, performing size regression on all the projection images, and aggregating the regression-projected images to obtain the single-frame target detection box of the current frame specifically includes: Keep the center point of the initial target detection box unchanged and enlarge the size of the initial target detection box by a preset multiple. Extract the original point cloud data located within the adjusted initial target detection box and project the original point cloud data onto each two-dimensional plane to obtain projection images of each two-dimensional plane. Aggregate the projection center points of all projection planes and the regression detection boxes to obtain the single-frame object detection box for the current frame.

6. The auxiliary annotation method for point cloud data according to claim 1, wherein Before the steps of adjusting the size of the initial object detection box, projecting the original point cloud data located within the adjusted initial object detection box onto each two-dimensional plane to obtain the projection images of each two-dimensional plane, performing size regression on all projection images, and aggregating the regression-projected images to obtain the single-frame object detection box for the current frame, the method further includes: Determine the preset annotation requirements, verify the single-frame object box according to the preset annotation requirements, and perform a box adjustment operation on the single-frame object detection box when it is determined that the verification fails.

7. The auxiliary annotation method for point cloud data according to claim 1, wherein The single-frame object detection box for each frame in a continuous frame is obtained through the following steps: Perform voxelization processing on the original point cloud data of the continuous frame to obtain voxelized data; Extract the point cloud feature information of the voxelized data and obtain the context information of the continuous frame point cloud data using the self-attention mechanism; Perform feature fusion on the point cloud feature information and the context information to obtain the continuous frame feature information; Extract the local spatial information of the continuous frame feature information, and obtain the motion information of the original point cloud data in the frame after the current frame according to the local spatial information; Determine the motion trajectory of the original point cloud data according to the motion information, and generate the single-frame object detection box for each frame of the continuous frames after the current frame according to the motion trajectory.

8. An auxiliary annotation device for point cloud data, characterized in that The device includes: A data extraction module for obtaining the original point cloud data, determining the initial points of the current frame from the original point cloud data, and extracting the target data points of the current frame from the original point cloud data based on the initial points; the distance between each target data point and the initial point does not exceed a preset search radius; An initial generation module for generating an initial object detection box according to the target data points; A size optimization module for adjusting the size of the initial object detection box, projecting the original point cloud data located within the adjusted initial object detection box onto each two-dimensional plane to obtain the projection images of each two-dimensional plane, performing size regression on all projection images, and aggregating the regression-projected images to obtain the single-frame object detection box for the current frame; A continuous frame generation module for obtaining the motion information of the original point cloud data in the frame after the current frame, determining the motion trajectory of the original point cloud data according to the motion information, generating the single-frame object detection box for each frame of the continuous frames after the current frame according to the motion trajectory, and confirming whether the boxes corresponding to the continuous frames meet the consistency and preset annotation requirements, and adjusting the non-conforming single-frame object detection boxes when it is determined that the consistency or the preset annotation requirements are not met; the consistency means that the single-frame object detection boxes of adjacent frames belong to one box, and the size deviation, orientation, and coincidence ratio between the single-frame object detection boxes of adjacent frames all meet the preset requirements.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method for assisting in annotating point cloud data according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for assisting in annotating point cloud data according to any one of claims 1 to 7.

Images