Point cloud data labeling method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention discloses a point cloud data labeling method and device, electronic equipment and a computer readable medium. A specific embodiment of the method comprises the following steps: in response to a received point cloud data sequence, for each frame of point cloud image in the point cloud data sequence, determining attribute information of a target object in the frame of point cloud image; associating a target object in each frame of point cloud image of the point cloud data sequence to generate a moving point cloud image sequence of the target object; according to the attribute information of the target object in the moving point cloud image sequence, correcting the attribute information of the target object in the preset frame point cloud image; and marking the target object in the preset frame point cloud image based on the correction result. The implementation mode is related to a point cloud data processing technology, multiple frames of point cloud data are adopted for processing, and dynamic changes of an object are obtained through fusion and correlation analysis. Therefore, the time sequence information can be fully utilized, so that the accuracy of object labeling is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of point cloud data processing, and in particular to a point cloud data annotation method, device, electronic device, and computer-readable medium. Background Art

[0002] Data pre-flashing (pre-labeling) for point cloud detection tasks is one of the key technologies for reducing costs and increasing efficiency in autonomous driving. Point cloud data is usually represented by a large number of three-dimensional points. The labeling of point cloud data generally involves drawing bounding boxes or marking the positions of points in three-dimensional space and assigning labels to each object category.

[0003] However, the inventors found that related annotation methods are usually based on the detection results of a single frame of point cloud images as an annotation reference. Since each frame of point cloud image data is independent, it is impossible to accurately predict the motion state and behavior of the object, which will affect the accuracy of the pre-annotation results and the annotation effect.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content section of this disclosure is used to introduce concepts in a brief form, and these concepts will be described in detail in the specific implementation section below. The content section of this disclosure is not intended to identify the key features or necessary features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection. Some embodiments of the present disclosure propose a point cloud data annotation method, a point cloud data annotation device, an electronic device, a computer-readable medium, and a computer program product to solve one or more of the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a point cloud data annotation method, including: in response to receiving a point cloud data sequence, determining, for each frame of point cloud image in the point cloud data sequence, attribute information of a target object in the frame of point cloud image; associating the target objects in each frame of point cloud image in the point cloud data sequence to generate a moving point cloud image sequence of the target object; correcting the attribute information of the target object in a preset frame of point cloud image based on the attribute information of the target object in the moving point cloud image sequence; and annotating the target object in the preset frame of point cloud image based on the correction result.

[0007] In some embodiments, based on the attribute information of the target object in the moving point cloud image sequence, the attribute information of the target object in the preset frame point cloud image is corrected, including: selecting a first preset number of frames of candidate moving point cloud images from the moving point cloud image sequence of the target object; selecting a second preset number of reference points from each candidate moving point cloud image, wherein the reference points are related to the target object; based on the attribute information of the reference points in each candidate moving point cloud image, the attribute information of the target object in the preset frame point cloud image is corrected.

[0008] In some embodiments, attribute information of a target object in a preset frame point cloud image is corrected based on attribute information of a reference point in each candidate moving point cloud image, including: determining position feature data of a reference point in each frame of a candidate moving point cloud image, wherein the reference point is used to characterize shape information of the target object; selecting a frame of candidate moving point cloud images from each candidate moving point cloud image as a preset frame point cloud image, and performing similarity calculation on the position feature data of the reference point in the preset frame point cloud image and the position feature data in other frames of candidate moving point cloud images; and determining a corrected shape size of the target object in the preset frame point cloud image based on the calculation result.

[0009] In some embodiments, a frame of candidate moving point cloud images is selected from each candidate moving point cloud image as a preset frame point cloud image, including: selecting each candidate moving point cloud image as a preset frame point cloud image; and based on the calculation results, determining the corrected shape size of the target object in the preset frame point cloud image, including: determining the corrected shape size of the target object in each preset frame point cloud image according to the calculation results, and obtaining a first preset number of corrected shape sizes; and determining the shape size of the target object according to the first preset number of corrected shape sizes.

[0010] In some embodiments, based on the attribute information of the reference point in each candidate moving point cloud image, the attribute information of the target object in the preset frame point cloud image is corrected, and it also includes: selecting a frame of candidate moving point cloud image from each candidate moving point cloud image, and determining the position of the selected candidate moving point cloud image in the trajectory represented by the point cloud data sequence as the origin of the local coordinate system; converting the position information of the reference point in other candidate moving point cloud images to the local coordinate system to obtain local coordinate position feature data; and determining the corrected position information of the target object in the preset frame point cloud image based on the local coordinate position feature data of the reference point in each candidate moving point cloud image.

[0011] In some embodiments, the corrected position information of the target object in the preset frame point cloud image is determined based on the local coordinate position feature data of the reference point in each candidate mobile point cloud image, including: selecting each candidate mobile point cloud image as a preset frame point cloud image; calculating the similarity between the local coordinate position feature data of the reference point in the current preset frame point cloud image and the local coordinate position feature data in other frame candidate mobile point cloud images; based on the calculation results, determining the three-dimensional coordinate offset of the target object in the current preset frame point cloud image and the target heading angle offset in the local coordinate system.

[0012] In some embodiments, the target object in each frame of the point cloud image of the point cloud data sequence is associated to generate a moving point cloud image sequence of the target object, including: determining an identifier of the target object, and according to the identifier, associating the target object in each frame of the point cloud image in the point cloud data sequence to obtain an object image sequence; from each object image in the object image sequence, point cloud data in a specified area is selected as the moving point cloud image of the target object to obtain a moving point cloud image sequence of the target object, wherein the specified area at least includes point cloud data representing the target object.

[0013] In a second aspect, some embodiments of the present disclosure provide a point cloud data annotation device, comprising: an object attribute determination unit, configured to determine, in response to receiving a point cloud data sequence, attribute information of a target object in each frame of point cloud image in the point cloud data sequence; an object association unit, configured to associate the target objects in each frame of point cloud image in the point cloud data sequence to generate a moving point cloud image sequence of the target object; an attribute correction unit, configured to correct the attribute information of the target object in a preset frame of point cloud image according to the attribute information of the target object in the moving point cloud image sequence; and an object annotation unit, configured to annotate the target object in the preset frame of point cloud image based on the correction result.

[0014] In some embodiments, the attribute correction unit is further configured to select a first preset number of frames of candidate moving point cloud images from the moving point cloud image sequence of the target object; select a second preset number of reference points from each candidate moving point cloud image, wherein the reference points are related to the target object; and correct the attribute information of the target object in the preset frame point cloud image based on the attribute information of the reference points in each candidate moving point cloud image.

[0015] In some embodiments, the attribute correction unit includes a first correction subunit, which is configured to determine the position feature data of the reference point in each frame of the candidate moving point cloud image, wherein the reference point is used to characterize the shape information of the target object; select a frame of candidate moving point cloud image from each candidate moving point cloud image as a preset frame point cloud image, and perform similarity calculation on the position feature data of the reference point in the preset frame point cloud image and the position feature data in other frames of candidate moving point cloud images; based on the calculation result, determine the corrected shape size of the target object in the preset frame point cloud image.

[0016] In some embodiments, the first correction subunit is further configured to select each candidate mobile point cloud image as a preset frame point cloud image; and determine the corrected shape size of the target object in each preset frame point cloud image based on the calculation results, to obtain a first preset number of corrected shape sizes; and determine the shape size of the target object based on the first preset number of corrected shape sizes.

[0017] In some embodiments, the attribute correction unit also includes a second correction subunit, which is configured to select a frame of candidate moving point cloud images from each candidate moving point cloud image, and determine the position of the selected candidate moving point cloud image in the trajectory represented by the point cloud data sequence as the origin of the local coordinate system; convert the position information of the reference point in other candidate moving point cloud images to the local coordinate system to obtain local coordinate position feature data; determine the corrected position information of the target object in the preset frame point cloud image based on the local coordinate position feature data of the reference point in each candidate moving point cloud image.

[0018] In some embodiments, the second correction subunit is further configured to select each candidate mobile point cloud image as a preset frame point cloud image; perform similarity calculation on the local coordinate position feature data of the reference point in the current preset frame point cloud image and the local coordinate position feature data in other frame candidate mobile point cloud images; based on the calculation results, determine the three-dimensional coordinate offset of the target object in the current preset frame point cloud image and the target heading angle offset in the local coordinate system.

[0019] In some embodiments, the object association unit is further configured to determine the identification of the target object, and based on the identification, associate the target object in each frame of the point cloud image in the point cloud data sequence to obtain an object image sequence; from each object image in the object image sequence, select the point cloud data within a specified area as a moving point cloud image of the target object to obtain a moving point cloud image sequence of the target object, wherein the specified area includes at least the point cloud data representing the target object.

[0020] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation method in the above-mentioned first aspect.

[0021] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any one of the implementations in the first aspect above is implemented.

[0022] In a fifth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which, when executed by a processor, implements the method described in any implementation manner in the above-mentioned first aspect.

[0023] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the point cloud data annotation method of some embodiments of the present disclosure can improve the accuracy of the annotation results. Specifically, the traditional manual annotation of point cloud data is a time-consuming and labor-intensive task. Because point cloud data may contain a lot of details and noise, this process requires a high degree of expertise and accuracy. The annotator must carefully check each point and ensure that the object is correctly labeled for subsequent tasks. To this end, it is further proposed to use a large model to predict the detection results of point cloud data, realize the pre-annotation of data, and provide a preliminary annotation reference for the annotator, thereby effectively reducing the annotation cost. Data pre-annotation technology not only saves time and human resources, but also accelerates the development and optimization process of the autonomous driving system, providing important support for achieving the goal of reducing costs and increasing efficiency of autonomous driving.

[0024] However, this pre-labeling technology usually performs pre-detection on single-frame point cloud image data. Since each frame of data is independent, it is impossible to accurately predict the motion state and behavior of the object. In other words, key information such as the object's motion trajectory, speed, and acceleration is often difficult to determine, which results in less than ideal pre-brush results.

[0025] Based on this, the point cloud data annotation method disclosed in the present invention can associate the target objects in each frame of the point cloud image of the point cloud data sequence to generate a moving point cloud image sequence of the target object. Then, according to the attribute information of the target object in the moving point cloud image sequence, the attribute information of the target object in the preset frame point cloud image can be corrected, thereby realizing the annotation of the target object. In other words, it is necessary to use multiple frames of point cloud data for processing, and obtain the dynamic changes of the object through fusion and association analysis. In this way, the time series information can be fully utilized to improve the annotation effect, thereby improving the accuracy and robustness of the annotation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0027] Figure 1 is a flow chart of some embodiments of the point cloud data annotation method disclosed in the present invention;

[0028] Figure 2 is a flow chart of some embodiments of shape size correction in the point cloud data annotation method disclosed in the present invention;

[0029] Figure 3 is a flow chart of some embodiments of position information correction in the point cloud data annotation method disclosed in the present invention;

[0030] Figure 4 It is a schematic diagram of the structure of some embodiments of the point cloud data annotation device disclosed in the present invention;

[0031] Figure 5 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0033] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0034] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0035] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0036] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0037] Figure 1 The process 100 of some embodiments of the point cloud data annotation method according to the present disclosure is shown. The method comprises the following steps:

[0038] Step 101 , in response to receiving a point cloud data sequence, for each frame of point cloud image in the point cloud data sequence, determining attribute information of a target object in the frame of point cloud image.

[0039] In some embodiments, the execution entity of the point cloud data annotation method of some embodiments of the present disclosure (such as a point cloud data processing server) can be connected to other electronic devices (such as point cloud data acquisition equipment) through a wired connection or a wireless connection. Here, point cloud data acquisition equipment such as (vehicle-mounted radar, etc.) can usually collect data in real time. Among them, point cloud data generally refers to a set of vectors in a three-dimensional coordinate system. Scanning data is generally recorded in the form of points, each point contains three-dimensional coordinates, and some may contain color information (RGB, red, green, blue) or reflection intensity information (Intensity).

[0040] In some embodiments, the point cloud data adopting device may send the collected point cloud data to the execution subject. When the execution subject receives the point cloud data sequence, it may perform object detection on each frame of the point cloud image in the point cloud data sequence. And the attribute information of the target object in each frame of the point cloud image may be determined. The target object here may be any object that needs to be analyzed and labeled, such as a vehicle, a road, etc. Among them, the attribute information may be relevant information characterizing the attributes of the target object, such as the shape of the external contour or the bounding box, the category, etc.

[0041] In some embodiments, the execution subject can perform object detection based on features. For example, features in point cloud data can be extracted, and the features can be used to classify and locate objects. Then the attribute information of the target object in each frame of the point cloud image is determined. For example, the object represented by the point cloud data can be distinguished based on feature data such as the color or reflection intensity of the point cloud data. And the classification of the object can be determined based on the shape surrounded by similar point cloud data. Optionally, the execution subject can use deep learning technology to detect objects. For example, a deep learning model can be trained using sample point cloud images of the target object. After the training is completed, the model can be used to detect the target object in each frame of the point cloud image of the point cloud data sequence, thereby obtaining the attribute information of the target object.

[0042] Step 102 , associating the target object in each frame of the point cloud image of the point cloud data sequence to generate a moving point cloud image sequence of the target object.

[0043] In some embodiments, when the target object in each frame of the point cloud image of the point cloud data sequence is detected, the execution subject can associate the target object in each frame of the point cloud image to generate a moving point cloud image sequence of the target object. For example, each frame of the point cloud image in which the target object is detected can be used as the moving point cloud image of the target object. And these moving point cloud images can be associated in time series to obtain a moving point cloud image sequence of the target object.

[0044] Optionally, the execution subject may determine the identifier of the target object. The identifier is used to uniquely characterize the target object and may be at least one character such as text, numbers, letters, symbols, etc. The execution subject may associate the target objects in each frame of the point cloud image in the point cloud data sequence according to the identifier to obtain an object image sequence. As an example, the execution subject may assign identifiers to detected objects, such as determining object numbers according to the detection order. For another example, the execution subject may determine the identifier of the target object by referring to the object identifier relationship table. The object identifier relationship table may represent the correspondence between objects and identifiers.

[0045] Furthermore, the execution subject can select point cloud data in a specified area from each object image in the object image sequence as a moving point cloud image of the target object, thereby obtaining a moving point cloud image sequence of the target object. The specified area includes at least point cloud data representing the target object. As an example, the area where the outer contour shape or bounding box of the target object is located can be selected as the specified area. For another example, in order to compensate for the contextual information between each frame of point cloud images and enrich the data content, an area slightly larger than the outer contour shape or bounding box of the target object, such as a certain distance enlarged in the three-dimensional direction, can be selected as the specified area.

[0046] It is understandable that a designated area is selected from each frame of the point cloud image as the moving point cloud image of the target object. Further analysis is then performed based on the moving point cloud image sequence of the target object. This not only preserves key data information and ensures the accuracy of subsequent analysis, but also reduces the amount of point cloud data to be processed and improves data processing efficiency.

[0047] Step 103 , correcting the attribute information of the target object in the preset frame point cloud image according to the attribute information of the target object in the moving point cloud image sequence.

[0048] In some embodiments, the execution subject may correct the attribute information of the target object in the preset frame point cloud image according to the attribute information of the target object in the moving point cloud image sequence. As an example, for each frame of the moving point cloud image in the moving point cloud image sequence, the execution subject may select a preset number of reference points therefrom. Among them, the reference point is usually related to the target object. The selection method here is also not limited, for example, a part may be selected from the external contour shape representing the target object, and a part may be selected from the inside of the contour shape. Furthermore, the attribute information of these reference points in each frame of the moving point cloud image may be used as feature data to analyze and predict the moving trajectory of the target object. In this way, the attribute information of the target object at the moment (i.e., the preset frame point cloud image) corresponding to the preset frame point cloud image on the moving trajectory may be predicted and determined as a correction reference. It can be understood that the above-mentioned preset frame point cloud image may be a point cloud image that has been acquired, or a point cloud image that will be acquired in the future.

[0049] In some embodiments, first, the execution subject may select a first preset number of candidate moving point cloud images from the moving point cloud image sequence of the target object. Then, a second preset number of reference points may be selected from each candidate moving point cloud image. After that, the attribute information of the target object in the preset frame point cloud image may be corrected according to the attribute information of the reference points in each candidate moving point cloud image. The first preset number and the second preset number here may also be set according to actual needs.

[0050] As an example, first, the execution entity can determine the position feature data of the reference point in each frame of the candidate moving point cloud image. The reference point is used to characterize the shape information of the target object. Then, a frame of candidate moving point cloud images can be selected from each candidate moving point cloud image as a preset frame point cloud image, and the position feature data of the reference point in the preset frame point cloud image is similarly calculated with the position feature data in other frames of candidate moving point cloud images. The similarity calculation method is also not limited. Afterwards, the corrected shape size of the target object in the preset frame point cloud image can be determined based on the calculation results.

[0051] For example, similarity calculation can be used to determine other frame point cloud images that are valuable to or related to the preset frame point cloud image, such as point cloud images whose similarity of position feature data is greater than a threshold. In this way, the corrected shape size of the target object in the preset frame point cloud image can be determined through these valuable or related point cloud images. For example, the average value of the shape size of the target object in these (preset frame + associated frame) point cloud images is used as the corrected shape size. Alternatively, the shape size of the target object in the preset frame point cloud image and the average value of the shape size in the associated frame point cloud image are weighted and summed to serve as the corrected shape size.

[0052] Furthermore, in order to improve the accuracy of the attribute information correction result, the execution subject can also select each candidate moving point cloud image as a preset frame point cloud image, so as to calculate the similarity with other frame candidate moving point cloud images in turn. Then, based on the calculation results, the corrected shape size of the target object in each preset frame point cloud image can be determined to obtain a first preset number of corrected shape sizes. Then, the shape size of the target object can be determined based on the first preset number of corrected shape sizes. For example, the average value of the first preset number of corrected shape sizes is determined as the final shape size of the target object.

[0053] Optionally, the execution subject can not only modify the shape and size of the target object (such as Figure 2 As shown), you can also modify its location information (as shown Figure 3 As shown). Specifically, first, the execution subject can select a frame of candidate moving point cloud images from each candidate moving point cloud image, and determine the position of the selected candidate moving point cloud image in the trajectory represented by the point cloud data sequence as the origin of the local coordinate system. Then, the position information of the reference point in other candidate moving point cloud images can be converted to the local coordinate system to obtain local coordinate position feature data. After that, the corrected position information of the target object in the preset frame point cloud image can be determined based on the local coordinate position feature data of the reference point in each candidate moving point cloud image.

[0054] Here, the execution entity can also select a frame of candidate mobile point cloud images as the preset frame point cloud image to correct the position information. In some embodiments, in order to further improve the accuracy of the correction result, the execution entity can select each candidate mobile point cloud image as the preset frame point cloud image. Then, the local coordinate position feature data of the reference point in the current preset frame point cloud image is similar to the local coordinate position feature data in other frames of candidate mobile point cloud images. Afterwards, based on the calculation results, the three-dimensional coordinate offset of the target object in the current preset frame point cloud image and the target heading angle offset can be determined in the local coordinate system. In other words, for the annotated point cloud image, the annotation results can also be corrected and referenced according to the method disclosed in the present invention, thereby improving the accuracy of the annotation results.

[0055] Step 104 : annotating the target object in the preset frame point cloud image based on the correction result.

[0056] In some embodiments, based on the correction result of step 103, the execution entity may use the correction result as a labeling reference to label the target object in the preset frame point cloud image.

[0057] The point cloud data annotation method of some embodiments of the present disclosure can associate the target objects in each frame of the point cloud image of the point cloud data sequence to generate a moving point cloud image sequence of the target object. Then, the attribute information of the target object in the preset frame point cloud image can be corrected according to the attribute information of the target object in the moving point cloud image sequence, thereby realizing the annotation of the target object. In other words, it is necessary to use multiple frames of point cloud data for processing, and obtain the dynamic changes of the object through fusion and association analysis. In this way, the time series information can be fully utilized to improve the annotation effect, thereby improving the accuracy and robustness of the annotation results.

[0058] In some application scenarios, using the point cloud data annotation method disclosed in the present invention, the result of the pre-annotated box can be outputted using time series information. Specifically:

[0059] First, multi-frame 3D target detection: using a standard point cloud detection network, input continuous point cloud frame data and output the bounding box and category information of the 3D object in each frame.

[0060] Next, offline multi-target tracking: associate the objects detected in each frame to form an object sequence and assign a unique object ID;

[0061] Second, correct data preparation: Given a sequence, we first slightly enlarge the RoI (Return-on-investment) region of the tracking box along the 3D parameters [x, y, z] direction to compensate for the large amount of context information. Then, the points enclosed by these enlarged RoI regions are taken out.

[0062] Afterwards, the process is corrected based on the shape and position attributes of the sequence results containing timing information:

[0063] 1. Shape information correction:

[0064] like Figure 2As shown in the figure, by obtaining the time-series detection results of the target, complementary information about the target appearance and shape can be obtained. First, t samples are selected from the sequence, and 256 points are selected for each sample to generate a geometry query (GeometryQuery). Then, the features are extracted through the point cloud image encoding (Query Encoder) module as the query input of the network (Transformer). After that, a local coordinate transformation operation is applied to align the target points at different times to the local box coordinates at different positions, and the points from different frames are merged to generate a geometry key-value pair (Geometry Key&Value). Among them, the key represents each frame of the point cloud image, and the value represents the position of each point in the point cloud image. The features are extracted through the position encoding (Value Encoder) module and then used as the key value input of the network. After generating the query (the preset frame point cloud image at the current moment) and key (the frame point cloud image at other moments), value (the position feature data of the point in the point cloud image) information, the self-attention mechanism (self-attention) and cross-attention mechanism (cross-attention) are performed to calculate the size (shape size, such as length, width and height) corresponding to each query. Then the average of t sizes is taken as the final size.

[0065] 2. Location information correction:

[0066] like Figure 3 As shown, first, 256 points are selected for each sample in the sequence to generate a position query (PositionQuery). Then, features are extracted through the point cloud image encoding (Query Encoder) module as the Query input of the network (Transformer). After that, one is randomly selected from the trajectory as the new local coordinate system. Then, all points in the boxes of other trajectories are converted to this coordinate system. Point Net is used to extract the entire target trajectory, and the Key (point cloud image of each frame) and Value (position in the local coordinate system) are obtained based on the position correction. After generating the Query and key value information, self-attention and cross-attention are performed to calculate the position offset (Refined Position) of each sample in the sequence corresponding to the Query output local coordinate system, such as the offsets of dx, dy, and dz, and the heading angle offset of theta (the angle direction of the front of the vehicle relative to the X-axis of the local coordinate system).

[0067] Finally, the correction information is integrated and corrected frame by frame: the size of each box and the offset of x, y, x, and heading angle are output as the result of the integrated correction. The final result of the pre-labeled box obtained using the timing information is output.

[0068] The present invention makes full use of timing information to perform time alignment and registration on multi-frame point cloud data, match their corresponding objects in space, and form a series of object trajectories. Such trajectory information can reveal key attributes such as the object's motion mode, speed, and acceleration. Afterwards, the multi-frame point cloud data on the same trajectory are fused to provide richer geometric information, eliminate noise and occlusion problems in a single-frame point cloud, and obtain a more complete and accurate object point cloud representation, thereby improving the effect of pre-detection.

[0069] Further references Figure 4 , as a response to the above Figures 1 to 3 The present disclosure provides some embodiments of point cloud data annotation devices, and these device embodiments are similar to Figures 1 to 3 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0070] like Figure 4 As shown, some embodiments of the point cloud data annotation device 400 may include: an object attribute determination unit 401, configured to determine the attribute information of the target object in each frame of point cloud image in the point cloud data sequence in response to receiving the point cloud data sequence; an object association unit 402, configured to associate the target objects in each frame of point cloud image in the point cloud data sequence to generate a moving point cloud image sequence of the target object; an attribute correction unit 403, configured to correct the attribute information of the target object in a preset frame of point cloud image according to the attribute information of the target object in the moving point cloud image sequence; and an object annotation unit 404, configured to annotate the target object in the preset frame of point cloud image based on the correction result.

[0071] In some embodiments, the attribute correction unit 403 can be further configured to select a first preset number of frames of candidate moving point cloud images from the moving point cloud image sequence of the target object; select a second preset number of reference points from each candidate moving point cloud image, wherein the reference points are related to the target object; and correct the attribute information of the target object in the preset frame point cloud image based on the attribute information of the reference points in each candidate moving point cloud image.

[0072] In some embodiments, the attribute correction unit 403 may include a first correction subunit (not shown in the figure), which is configured to determine the position feature data of the reference point in each frame of the candidate moving point cloud image, wherein the reference point is used to characterize the shape information of the target object; select a frame of candidate moving point cloud image from each candidate moving point cloud image as a preset frame point cloud image, and perform similarity calculation on the position feature data of the reference point in the preset frame point cloud image and the position feature data in other frames of candidate moving point cloud images; based on the calculation result, determine the corrected shape size of the target object in the preset frame point cloud image.

[0073] In some embodiments, the first correction subunit can be further configured to select each candidate mobile point cloud image as a preset frame point cloud image; and determine the corrected shape size of the target object in each preset frame point cloud image based on the calculation results, to obtain a first preset number of corrected shape sizes; and determine the shape size of the target object based on the first preset number of corrected shape sizes.

[0074] In some embodiments, the attribute correction unit 403 may also include a second correction subunit (not shown in the figure), which is configured to select a frame of candidate moving point cloud images from each candidate moving point cloud image, and determine the position of the selected candidate moving point cloud image in the trajectory represented by the point cloud data sequence as the origin of the local coordinate system; convert the position information of the reference point in other candidate moving point cloud images to the local coordinate system to obtain local coordinate position feature data; determine the corrected position information of the target object in the preset frame point cloud image according to the local coordinate position feature data of the reference point in each candidate moving point cloud image.

[0075] In some embodiments, the second correction subunit can be further configured to select each candidate mobile point cloud image as a preset frame point cloud image; perform similarity calculation on the local coordinate position feature data of the reference point in the current preset frame point cloud image and the local coordinate position feature data in other frame candidate mobile point cloud images; based on the calculation results, determine the three-dimensional coordinate offset of the target object in the current preset frame point cloud image and the target heading angle offset in the local coordinate system.

[0076] In some embodiments, the object association unit 402 can be further configured to determine the identification of the target object, and based on the identification, associate the target object in each frame of the point cloud image in the point cloud data sequence to obtain an object image sequence; from each object image in the object image sequence, select the point cloud data in a specified area as the moving point cloud image of the target object to obtain a moving point cloud image sequence of the target object, wherein the specified area at least includes the point cloud data representing the target object.

[0077] It can be understood that the units recorded in the point cloud data annotation device 400 are similar to the reference Figures 1 to 3 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the point cloud data annotation device 400 and the units included therein, and will not be described in detail here.

[0078] Reference below Figure 5 , which shows a structural schematic diagram of an electronic device 500 suitable for implementing some embodiments of the present disclosure. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0079] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0080] Typically, the following devices may be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, physical buttons, etc.; output devices 507 including, for example, a speaker, a vibrator, etc.; storage devices 508 including, for example, a hard disk, a magnetic disk, etc.; and communication devices 509. The communication devices 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0081] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0082] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0083] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0084] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: in response to receiving a point cloud data sequence, for each frame of point cloud image in the point cloud data sequence, determines the attribute information of the target object in the frame of point cloud image; associates the target objects in each frame of point cloud image in the point cloud data sequence to generate a moving point cloud image sequence of the target object; according to the attribute information of the target object in the moving point cloud image sequence, corrects the attribute information of the target object in the preset frame of point cloud image; and based on the correction result, labels the target object in the preset frame of point cloud image.

[0085] In addition, computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0086] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0087] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The described units may also be set in a processor, for example, it may be described as: a processor includes an object attribute determination unit, an object association unit, an attribute correction unit and an object labeling unit. The names of these units do not constitute limitations on the units themselves in some cases. For example, the object attribute determination unit may also be described as "a unit for determining the attribute information of the target object in the point cloud image of the frame".

[0088] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0089] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned point cloud data labeling methods.

[0090] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A point cloud data annotation method, include: In response to receiving a point cloud data sequence, for each frame of point cloud image in the point cloud data sequence, determining attribute information of a target object in the frame of point cloud image; Associating the target object in each frame of the point cloud image of the point cloud data sequence to generate a moving point cloud image sequence of the target object; According to the attribute information of the target object in the moving point cloud image sequence, correcting the attribute information of the target object in the preset frame point cloud image; Based on the correction result, the target object in the preset frame point cloud image is marked.

2. The point cloud data annotation method according to claim 1, in, The step of correcting the attribute information of the target object in the preset frame point cloud image according to the attribute information of the target object in the moving point cloud image sequence includes: Selecting a first preset number of frames of candidate moving point cloud images from the moving point cloud image sequence of the target object; Selecting a second preset number of reference points from each of the candidate moving point cloud images, wherein the reference points are related to the target object; According to the attribute information of the reference point in each of the candidate moving point cloud images, the attribute information of the target object in the preset frame point cloud image is corrected.

3. The point cloud data annotation method according to claim 2, in, The step of correcting the attribute information of the target object in the preset frame point cloud image according to the attribute information of the reference point in each of the candidate moving point cloud images comprises: Determine position feature data of the reference point in each frame of the candidate moving point cloud image, wherein the reference point is used to characterize shape information of the target object; Selecting one frame of the candidate moving point cloud image from each of the candidate moving point cloud images as a preset frame point cloud image, and performing similarity calculation on the position feature data of the reference point in the preset frame point cloud image and the position feature data of the reference point in other frames of the candidate moving point cloud images; Based on the calculation result, a corrected shape and size of the target object in the preset frame point cloud image is determined.

4. The point cloud data annotation method according to claim 3, in, The step of selecting a frame of the candidate moving point cloud image from each of the candidate moving point cloud images as a preset frame of point cloud image includes: Selecting each of the candidate moving point cloud images as a preset frame point cloud image; and The step of determining the corrected shape and size of the target object in the point cloud image of the preset frame based on the calculation result includes: Determine the corrected shape size of the target object in each preset frame point cloud image according to the calculation result, and obtain the first preset number of corrected shape sizes; and The shape and size of the target object are determined according to the first preset number of corrected shape and sizes.

5. The point cloud data annotation method according to claim 2, in, The step of correcting the attribute information of the target object in the preset frame point cloud image according to the attribute information of the reference point in each of the candidate moving point cloud images further includes: Selecting a frame of the candidate moving point cloud image from each of the candidate moving point cloud images, and determining the position of the selected candidate moving point cloud image in the trajectory represented by the point cloud data sequence as the origin of the local coordinate system; Convert the position information of the reference point in the other candidate moving point cloud images to the local coordinate system to obtain local coordinate position feature data; According to the local coordinate position feature data of the reference point in each of the candidate moving point cloud images, the corrected position information of the target object in the preset frame point cloud image is determined.

6. The point cloud data annotation method according to claim 5, in, The step of determining the corrected position information of the target object in the preset frame point cloud image according to the local coordinate position feature data of the reference point in each of the candidate moving point cloud images comprises: Selecting each of the candidate moving point cloud images as a preset frame point cloud image; Calculate the similarity between the local coordinate position feature data of the reference point in the point cloud image of the current preset frame and the local coordinate position feature data of the candidate moving point cloud images in other frames; Based on the calculation results, the three-dimensional coordinate offset of the target object in the point cloud image of the current preset frame and the target heading angle offset are determined in the local coordinate system.

7. The point cloud data annotation method according to any one of claims 1 to 6, in, The step of associating the target object in each frame of the point cloud image of the point cloud data sequence to generate a moving point cloud image sequence of the target object includes: Determine the identifier of the target object, and according to the identifier, associate the target object in each frame of the point cloud image in the point cloud data sequence to obtain an object image sequence; From each object image in the object image sequence, point cloud data in a specified area is selected as a moving point cloud image of the target object to obtain a moving point cloud image sequence of the target object, wherein the specified area at least includes point cloud data representing the target object.

8. A point cloud data annotation device, include: an object attribute determination unit, configured to determine, in response to receiving a point cloud data sequence, attribute information of a target object in each frame of point cloud image in the point cloud data sequence; An object associating unit is configured to associate the target object in each frame of the point cloud image of the point cloud data sequence to generate a moving point cloud image sequence of the target object; an attribute correction unit, configured to correct the attribute information of the target object in the preset frame point cloud image according to the attribute information of the target object in the moving point cloud image sequence; The object labeling unit is configured to label the target object in the preset frame point cloud image based on the correction result.

9. An electronic device, include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, in, When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 7.

Images