Method, device and storage medium for object tracking and annotation in multi-frame 3D point cloud data

Abstract

The present application discloses a method and device for tracking and labeling objects in frame 3D point cloud data, so as to solve the problem that the prior art cannot label objects in 3D point cloud data quickly and efficiently. The method includes: a labeling device displays the Nth frame of 3D point cloud data, receives input labeling data of a 3D labeling box that labels an object; records the labeling data of the Nth frame of the object; in the process of displaying the subsequent N+ith frame of 3D point cloud data, generates and displays a 3D labeling box that matches the recorded labeling data in the 3D coordinate system space of the displayed 3D point cloud data; receives input update data of the 3D labeling box, updates the generated 3D labeling box according to the update data, and obtains the updated labeling data; records the updated labeling data as the labeling data of the N+ith frame of the object; wherein N and i are both natural numbers greater than or equal to 1.

Description

Technical Field

[0001] The present application relates to the field of data annotation, and in particular to a method and device for tracking and annotating objects in multi-frame 3D point cloud data, and a storage medium. Background Art

[0002] In the related art, annotators annotate 3D point cloud data displayed on a front end such as a browser. Usually, when tracking and annotating objects in multiple frames of 3D point cloud data, annotators need to observe and identify the object in one frame of point cloud data and mark the object with a 3D annotation box, and observe and identify the same object in the next frame of 3D point cloud data and mark the object with a 3D annotation box. Summary of the invention

[0003] The present application provides a tracking and labeling solution for objects in multi-frame 3D point cloud data, so as to solve the problem in the prior art that objects in 3D point cloud data cannot be labeled quickly and efficiently.

[0004] On the one hand, an embodiment of the present application provides a method for tracking and annotating objects in multi-frame 3D point cloud data, comprising:

[0005] The annotation device displays the Nth frame of 3D point cloud data, receives input annotation data of a 3D annotation frame annotating an object, and records the annotation data of the Nth frame of the object; wherein the annotation data includes at least one or more of the following: position data of the 3D annotation frame, size data of the 3D annotation frame, and category data of the object;

[0006] In the process of displaying the subsequent N+i-th frame of 3D point cloud data, in the 3D coordinate system space displaying the 3D point cloud data, a 3D annotation box that is consistent with the recorded annotation data is generated and displayed; update data of the input 3D annotation box is received, the generated 3D annotation box is updated according to the update data, and the updated annotation data is obtained; the updated annotation data is recorded as the annotation data of the N+i-th frame of the object; wherein N and i are both natural numbers greater than or equal to 1.

[0007] On the other hand, an embodiment of the present application provides a device for tracking and labeling objects in multi-frame 3D point cloud data, including: a processor and at least one memory, at least one memory storing at least one machine executable instruction, and the processor executes at least one machine executable instruction to execute the method for tracking objects in multi-frame 3D point cloud data as described above.

[0008] On the other hand, an embodiment of the present application provides a non-volatile storage medium storing at least one machine executable instruction, which can execute the method for tracking objects in multi-frame 3D point cloud data as described above after being executed by a processor.

[0009] In the technical solution provided in the embodiment of the present application, in the process of displaying multiple frames of 3D point cloud data, after an object in one of the frames is first annotated, the annotation device records the annotation data of the 3D annotation box annotating the object, and in the process of displaying subsequent frames, generates and displays a 3D annotation box consistent with the recorded annotation data, receives input update data for the 3D annotation box, updates the 3D annotation box according to the update data, obtains the updated annotation data, and records the marking data of the object in the frame. Therefore, the annotation device does not need to receive the annotations of the same object in each frame of 3D point cloud data from the annotator one by one, and only needs to automatically generate the corresponding 3D annotation box in the subsequent frame after an object in the 3D point cloud data is first annotated, and adjust the 3D annotation box according to the input update data, and automatically record the annotation data of the updated 3D annotation box, so that objects in multiple frames of 3D point cloud data can be tracked and annotated quickly and efficiently, and compared with the annotation methods in the prior art, the annotation speed and efficiency can be significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to provide further understanding of the present application and constitute a part of the specification. They are used to explain the present application together with the embodiments of the present application and do not constitute a limitation of the present application.

[0011] Figure 1 A schematic diagram of the structure of a device for tracking and annotating objects in multi-frame 3D point cloud data provided by an embodiment of the present application;

[0012] Figure 2 A schematic diagram of the architecture for tracking and labeling objects in multi-frame 3D point cloud data provided by an embodiment of the present application;

[0013] Figure 3 A processing flow chart of a method for tracking and annotating objects in multi-frame 3D point cloud data provided in an embodiment of the present application;

[0014] Figure 4 Provided in the embodiments of this application Figure 3 An implementation flow chart of step 301;

[0015] Figure 5 Provided in the embodiments of this application Figure 3 Another implementation flow chart of step 301;

[0016] Figure 6 Provided in the embodiments of this application Figure 3 Another implementation flow chart of step 301;

[0017] Figure 7 Provided in the embodiments of this application Figure 3 Another implementation flow chart of step 301;

[0018] Figure 8 Provided in the embodiments of this application Figure 6 or Figure 7 Processing flow chart of update processing in

[0019] Fig. 9 Another processing flow chart of tracking and labeling objects in multi-frame 3D point cloud data provided by an embodiment of the present application;

[0020] Figures 10a to 10h For implementation Fig. 9 An example of the method used to track and label intermittently appearing objects. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0022] In related technologies, when annotators track and annotate objects in multiple frames of 3D point cloud data, they need to observe and identify the same object in each frame of 3D point cloud data and annotate the objects one by one. This annotation method obviously has the problems of slow annotation speed and low efficiency. Especially when there are many frames of multiple 3D point cloud data and many objects to be annotated in each frame of data, the annotation speed and efficiency will drop significantly.

[0023] The embodiment of the present application provides a tracking and annotation scheme for objects in multi-frame 3D point cloud data, which can effectively improve the speed and efficiency of object tracking and annotation. In the embodiment of the present application, in the process of displaying multi-frame 3D point cloud data, after an object in one frame is annotated for the first time, the annotation device records the annotation data of the 3D annotation box annotating the object, and in the process of displaying subsequent frame data, generates and displays a 3D annotation box consistent with the recorded annotation data, receives input update data for the 3D annotation box, updates the 3D annotation box according to the update data, obtains the updated annotation data, and records the marking data of the object in the frame. Therefore, the annotation device does not need to receive the annotation of the object in each frame of data one by one by the annotator, only needs to automatically generate a 3D annotation box in the subsequent frame after the object in the 3D point cloud data is annotated for the first time, and adjusts the 3D annotation box according to the input update data, and automatically records the annotation data of the updated 3D annotation box, which can quickly and efficiently track and annotate objects in multi-frame 3D point cloud data, and can significantly improve the annotation speed and annotation efficiency compared with the annotation method in the prior art.

[0024] Some embodiments of the present application provide a tracking and annotation solution for objects in multi-frame 3D point cloud data. Figure 1 The structure of the marking device provided in an embodiment of the present application is shown. The device 1 includes a processor 11 and at least one memory 12.

[0025] In some embodiments, at least one memory 12 may be a storage device in various forms, such as a transient or non-transient storage medium. At least one machine executable instruction may be stored in the memory 12, and at least one machine executable instruction may be executed by the processor 11 to implement the tracking and annotation processing of objects in the multi-frame 3D point cloud data provided in the embodiment of the present application.

[0026] In some embodiments, the marking device 1 may be located on a server side. In other embodiments, the marking device 1 may also be located on a cloud server. In other embodiments, the marking device 1 may also be located on a client side.

[0027] like Figure 2 As shown, the tracking and annotation processing of objects in multi-frame 3D point cloud data provided by the embodiment of the present application may include front-end processing 12 and back-end processing 14. The front-end processing 12 displays relevant 3D point cloud data or other data, and receives relevant data or information input by the annotator. For example, the front-end processing 12 may be a processing implemented through a web page, or a processing implemented through a separate application interface. The back-end processing 14 performs corresponding annotation processing based on the relevant data and information received by the front-end processing 12. After the annotation processing is completed, the annotation device 1 can further provide the annotation results to other processing or applications on the client, server, or cloud server.

[0028] The following describes the tracking and annotation process of objects in multiple frames of 3D point cloud data implemented by the annotation device 1 executing at least one machine executable instruction.

[0029] Figure 3 The processing flow of tracking and labeling objects in multi-frame 3D point cloud data provided by the embodiment of the present application is shown in FIG. 1 , that is, the processing of object tracking and labeling by the labeling device includes:

[0030] Step 301: The annotation device displays the Nth frame of 3D point cloud data, receives input annotation data of a 3D annotation frame that annotates an object, and records the annotation data of the Nth frame of the object; wherein N is a natural number greater than or equal to 1, and the annotation data includes at least one or more of the following: position data of the 3D annotation frame, size data of the 3D annotation frame, and category data of the object;

[0031] Step 302: During display of the subsequent N+i-th frame of 3D point cloud data, the annotation device generates and displays a 3D annotation box that matches the recorded annotation data in the 3D coordinate system space displaying the 3D point cloud data; receives update data of the input 3D annotation box, updates the generated 3D annotation box according to the update data, and obtains updated annotation data; and records the updated annotation data as the annotation data of the N+i-th frame of the object; wherein i is a natural number greater than or equal to 1.

[0032] according to Figure 3 According to the method shown, the annotation device does not need to receive the annotations of the same object in each frame of 3D point cloud data from the annotator one by one, but only needs to perform the first annotation of an object in the 3D point cloud data according to the received annotation data, automatically generate the corresponding 3D annotation frame in the subsequent frame, adjust the 3D annotation frame according to the received update data, and automatically record the annotation data of the updated 3D annotation frame. It can quickly and efficiently track and annotate objects in multiple frames of 3D point cloud data, and can significantly improve the annotation speed and efficiency compared with the annotation method in the prior art.

[0033] Figure 3 The processing shown can be implemented through multiple implementations and alternative implementations, and the implementation provided in the examples of the present application is described below.

[0034] In the above steps 301 and 302, when the annotation device displays the 3D point cloud data, the 3D point cloud data may be displayed according to a specified display direction. The specified display direction may be a preset display direction or a display direction input by the annotator.

[0035] For example, in some embodiments, after the annotation device reads a frame of 3D point cloud data, the frame of 3D point cloud data can be displayed according to a preset display direction. For another example, in some embodiments, when the annotator needs to carefully observe the scene or object expressed by the 3D point cloud data, the desired display direction can be selected and input, and the annotation device displays the 3D point cloud data according to the received display direction, so as to facilitate the annotator to observe and identify.

[0036] In some embodiments of the present application, multi-frame 3D point cloud data may be continuous multi-frame 3D point cloud data, for example, multi-frame continuous 3D point cloud data within 1 second. In some application scenarios, the time interval for acquiring 3D point cloud data is very short, and the scene difference between frames is not large. In order to save processing power, it is not necessary to annotate objects in each frame of 3D point cloud data, and only 3D point cloud data with a certain time interval may be annotated, so that in some embodiments, multi-frame 3D point cloud data may be multi-frame 3D point cloud data with the same time interval. In other scenarios, it is only necessary to annotate 3D point cloud data with obvious changes in the scene, and some data may be selected from multi-frame 3D point cloud data for annotation, so that in some embodiments, multi-frame 3D point cloud data may be multi-frame data with different time intervals.

[0037] In the embodiment of the present application, multiple objects may appear in the multi-frame 3D point cloud data, and the first appearance of an object may be in any frame in the multi-frame data. In step 301, when the annotator tracks and annotates the objects in the multi-frame 3D point cloud data, it is necessary to manually annotate the objects first recognized in one of the frames. That is, the annotator can manually annotate the objects that first appear in any frame in the multi-frame 3D point cloud, and input the annotation data of the 3D annotation box annotating the object into the annotation device.

[0038] In some application scenarios, the annotator can input the annotation data of the 3D annotation box through the human-machine interface provided by the annotation device in a variety of ways, for example, directly enter a specific parameter value in the data input box in the human-machine interface, click a preset button or key on the human-machine interface, the button or key has a corresponding preset instruction or data, or select a corresponding option in the drop-down menu provided by the human-machine interface, the drop-down menu may include one or more levels of submenus, each submenu may include one or more options, and the annotation device receives the annotation data input by the annotator through the human-machine interface.

[0039] The annotation device performs the first annotation operation according to the received annotation data, including generating a corresponding 3D annotation frame and recording the received annotation data. In some embodiments, after the annotator inputs the annotation data of the 3D annotation frame of an object, the annotation device can also generate and display the 3D annotation frame to facilitate the annotator to observe whether the annotation operation is accurate. Figure 4 A processing flow of the first marking operation performed by the marking device is shown, that is, an implementation of the above step 301, including:

[0040] Step 3011a: The labeling device displays the Nth frame of 3D point cloud data and receives input labeling data of a 3D labeling box that labels an object;

[0041] Step 3012a: the annotation device generates and displays a 3D annotation frame that matches the annotation data in the 3D coordinate system space displaying the 3D point cloud data;

[0042] Step 3013a: The labeling device records the labeling data of the object.

[0043] In the embodiment of the present application, the 3D annotation box can be a cuboid, for example. In different application scenarios, 3D annotation boxes of different shapes can also be set according to the needs of the application scenarios.

[0044] In some embodiments, when the labeling device needs to label multiple objects that appear for the first time in a frame of data, it can also establish an identifier for each object and a correspondence between the identifier and the labeling data to achieve accurate labeling and recording of multiple objects. Figure 5 The processing flow of another implementation of the above step 301 is shown:

[0045] Step 3011b: the labeling device displays the Nth frame of 3D point cloud data and receives input labeling data of a 3D labeling box that labels an object;

[0046] Step 3012b, establishing the identification of the object;

[0047] Step 3013b: the annotation device generates and displays a 3D annotation frame that matches the annotation data in the 3D coordinate system space displaying the 3D point cloud data;

[0048] Step 3014b: the labeling device records the labeling data of the object;

[0049] Step 3015b: The labeling device establishes an association relationship between the identifier of the object and the labeling data of the object.

[0050] In some embodiments, after the labeling device generates a 3D labeling box based on the labeling data input by the labeler, the labeler can further adjust the generated 3D labeling box to achieve accurate labeling of the object, that is, the first labeling operation performed by the labeling device, and can further receive the updated data of the 3D labeling box input by the labeler to adjust the 3D labeling box and record the updated labeling data of the 3D labeling box. Figure 6 The processing flow of another implementation of step 301 is shown, including:

[0051] Step 3011c: the labeling device displays the Nth frame of 3D point cloud data and receives input labeling data of a 3D labeling box that labels an object;

[0052] Step 3012c: the annotation device generates and displays a 3D annotation frame that matches the annotation data in the 3D coordinate system space displaying the 3D point cloud data;

[0053] Step 3013c: receiving the input update data of the 3D annotation frame, updating the generated 3D annotation frame accordingly according to the update data, and obtaining the updated annotation data;

[0054] Step 3014c: record the updated annotation data of the object in the Nth frame.

[0055] above Figures 4 to 6 The embodiments shown in the figure can also be combined with each other to obtain new implementation modes. For example, the following Figure 7 The embodiment shown includes:

[0056] Step 3011d: the labeling device displays the Nth frame of 3D point cloud data and receives input labeling data of a 3D labeling box that labels an object;

[0057] Step 3012d, establishing the identification of the object;

[0058] Step 3013d: the annotation device generates and displays a 3D annotation frame that matches the annotation data in the 3D coordinate system space displaying the 3D point cloud data;

[0059] Step 3014d: receiving the input update data of the 3D annotation frame, updating the generated 3D annotation frame accordingly according to the update data, and obtaining the updated annotation data;

[0060] Step 3015d, recording the updated annotation data of the object in the Nth frame;

[0061] Step 3016d: The labeling device establishes an association relationship between the identifier of the object and the updated labeling data of the object.

[0062] Although the embodiments of the present application list some implementation methods of the marking device performing the first marking operation, in specific application scenarios, other equivalent implementation methods or alternative implementation methods may also be included.

[0063] In the above Figure 3 to Figure 7 In the processing of the illustrated embodiment and in the processing of the following embodiments, the operation of generating a 3D annotation box that matches the annotation data according to different data included in the annotation data may include the following multiple implementations.

[0064] Generation method 1: In some scenarios, when tracking and annotating objects in multiple frames of 3D point cloud data, the objects must be accurately annotated, and the position of the objects is a factor to be considered, that is, the annotation data may include the position data of the objects, and the annotation device generates a 3D annotation frame based on the received position data.

[0065] In some embodiments, when the annotation data includes position data of a 3D annotation box, the operation of generating a 3D annotation box consistent with the annotation data may include: generating a 3D annotation box at a position indicated by the position data included in the annotation data.

[0066] For example, after the annotator identifies an object in the displayed 3D point cloud data, he clicks to select a position in the 3D coordinate system space displaying the data, and the annotation device identifies the three-dimensional coordinate value of the position, and the position data includes the three-dimensional coordinate value (x, y, z) of the position; or when the annotator can identify the three-dimensional coordinate value of the selected position, he can also directly enter the position data, including the three-dimensional coordinate value (x, y, z) of the position in the numerical input box of the human-computer interface. In this way, the annotation device can generate a 3D annotation box at the position indicated by the received three-dimensional coordinate value. When generating, a rectangular 3D annotation box can be generated with the position indicated by the three-dimensional coordinate value as the center point of the cube, or a cube 3D annotation box can be generated with the position indicated by the three-dimensional coordinate value as a predetermined corner point, and a cube 3D annotation box can also be generated with the position indicated by the three-dimensional coordinate value as the center point of a face.

[0067] Furthermore, the generated 3D annotation box may have a predetermined size. For example, in some scenarios, the objects to be annotated are relatively simple and have similar or identical sizes. In this case, the size of the 3D annotation box may be preset, and the annotation device may generate a 3D annotation box with a predetermined size based on the received position data.

[0068] Generation method 2: In some scenarios, the labeling work has high requirements on the accuracy of the size of the labeled object, so the labeling device may first receive the size data of the 3D labeling box input by the labeler.

[0069] In some embodiments, when the annotation data includes size data of the 3D annotation box, the operation of generating a 3D annotation box that matches the annotation data may include: generating a 3D annotation box whose size is consistent with the size data included in the annotation data. The size data may be set according to the proportional relationship between the size of the data point in the 3D point cloud data and the size of the actual object, and the value of the size is the value s*t*r in the 3D space coordinate system.

[0070] Furthermore, the process of generating a 3D annotation frame may also be to generate a 3D annotation frame consistent with the dimension data at a predetermined position, and the predetermined position may be, for example, a position approximately in the center of a display window in a human-machine interface, or a preset position in a 3D coordinate system space. If the position data of the 3D annotation frame input by the annotator is subsequently received, a 3D annotation frame consistent with the dimension data may be further regenerated at the position indicated by the position data according to the position data.

[0071] Generation method three: In some scenarios, the labeling work has high requirements on the accuracy of object positions, and the differences between the multiple objects to be labeled are large. The same size of 3D labeling box cannot be used to label multiple different objects. After entering the position data, the labeler also needs to enter the size data of the 3D labeling box.

[0072] In some embodiments, when the annotation data includes position data and size data of a 3D annotation box, the operation of generating a 3D annotation box consistent with the annotation data may include: generating a 3D annotation box of the size indicated by the size data at the position indicated by the position data included in the annotation data.

[0073] Generation method 4: In some scenarios, the category of the object to be annotated is known, and the size of the 3D annotation box corresponding to the object category can be pre-set, and the annotation device pre-stores this information. When the annotation device receives the object category data, it can directly generate a 3D annotation box of a preset size corresponding to the object category, which can improve the annotation efficiency.

[0074] In some embodiments, when the annotation data includes category data of the object, the operation of generating a 3D annotation box that matches the annotation data may include: generating a 3D annotation box of a preset size corresponding to the object category.

[0075] Furthermore, the process of generating a 3D annotation frame may also be to generate a 3D annotation frame of a preset size corresponding to the object category at a predetermined position, and the predetermined position may be, for example, a position approximately in the center of a display window in a human-machine interface, or a preset position in a 3D coordinate system space. Subsequently, based on the position data input by the annotator, a 3D annotation frame of a size corresponding to the object category may be further regenerated at the position indicated by the position data.

[0076] Generation method five: In some scenarios, the category of the object to be labeled is known, the category of the object and the size of the 3D labeling box corresponding to the object category can be preset. The labeler also needs to enter the object category data after entering the position data. The labeling device generates a 3D labeling box with a size corresponding to the object category based on the received position data.

[0077] In some embodiments, when the annotation data includes position data of a 3D annotation box and category data of an object, the operation of generating a 3D annotation box that matches the annotation data may include: generating a 3D annotation box of a preset size corresponding to the object category at the position indicated by the position data included in the annotation data.

[0078] Although the above describes how to generate corresponding 3D annotation boxes according to different data when the annotation data includes different data, the above embodiments can also be combined to obtain various other implementations or modified implementations.

[0079] Generation method six: In some embodiments, the annotation device pre-stores the size of a universal 3D annotation box and the size data of 3D annotation boxes corresponding to different object categories, and the annotator inputs the different data included in the annotation data multiple times; when the annotation data first received by the annotation device includes position data, the annotation device can first generate a 3D annotation box with a universal size according to the position data, and after the annotation device further receives the object type data, the generated 3D annotation box is regenerated, and a 3D annotation box with a size consistent with the size data corresponding to the object type is generated at the position indicated by the position data.

[0080] Generation method seven, in some embodiments, the annotation device pre-stores the size of a general 3D annotation box and the size data of 3D annotation boxes corresponding to different object categories, and the annotator inputs different parameters in the annotation data multiple times; when the annotation device first receives the category data included in the annotation data, a 3D annotation box with a size corresponding to the object category is generated at a predetermined position in the 3D coordinate system space. For example, the predetermined position can be the center of the display interface in the human-computer interface. After the annotation device further receives the position data, a 3D annotation box with a size corresponding to the object category is regenerated at the position indicated by the position data.

[0081] The above examples exemplarily illustrate how to generate a 3D annotation box when the annotator inputs multiple different data included in the annotation data in multiple times. Similarly, the annotator can also input different data in other different orders. The annotation device performs corresponding processing according to the specific data included in the received annotation data. The specific processing process can be a combination of one or more of the above-mentioned generation methods. Those skilled in the art can obtain a variety of equivalent, interchangeable, or deformed processing methods based on the records and disclosures of this application, which are not listed one by one in this application.

[0082] The above describes the process of the labeling device labeling an object for the first time based on the labeling data input by the labeler.

[0083] In such Figure 6 or Figure 7 In the processing process shown in FIG. 1 , the annotation device updates the 3D annotation frame according to the received update data, such as Figure 8 As shown, the following processing may be included:

[0084] Step 801: The annotation device receives input update data of the 3D annotation frame, and updates the generated 3D annotation frame accordingly according to the update data to obtain updated annotation data;

[0085] Step 802: Record the updated annotation data of the object.

[0086] according to Figure 8 The processing process shown can adjust the 3D annotation frame according to the updated data and record the updated annotation data.

[0087] The processing of step 801 above may include the following multiple implementations according to different data included in the update data.

[0088] Update method 1: In some embodiments, Fig. 9 As shown, the received update data of the 3D annotation box includes the position update data of the 3D annotation box; the annotation device determines the updated position of the 3D annotation box according to the position update data of the 3D annotation box, and displays the 3D annotation box at the updated position; it is determined that the updated annotation data includes the updated position data of the 3D annotation box.

[0089] In some embodiments, when the position update data of the 3D annotation box includes the updated position data of the 3D annotation box, the position indicated by the updated position data of the 3D annotation box is determined as the updated position of the 3D annotation box. The position update data may include the updated three-dimensional space coordinate value (x', y', z') of the 3D annotation box. The annotation device regenerates the 3D annotation box at the position indicated by the updated three-dimensional space coordinate value (x', y', z').

[0090] In some embodiments, when the position update data of the 3D annotation box includes the position update direction and position offset of the 3D annotation box, the annotation device determines the updated position of the 3D annotation box according to the position of the generated 3D annotation box and the update direction and position offset of the 3D annotation box included in the position update data. For example, the position coordinates of the generated 3D annotation box before the update are (x, y, z), and the position update data includes an update direction of the x-axis direction and an offset of a, then the updated position is (x', y, z), x'=x+a. The annotation device regenerates the 3D annotation box at the position indicated by the updated three-dimensional space coordinate value (x', y, z).

[0091] Update method two: In some embodiments, the received update data of the 3D annotation box includes size update data of the 3D annotation box; the annotation device determines the updated size of the 3D annotation box based on the size update data, and displays the 3D annotation box using the updated size; it is determined that the updated annotation data includes the updated size data of the 3D annotation box.

[0092] In some embodiments, when the size update data of the 3D annotation box includes the updated size data of the 3D annotation box, the annotation device determines the size indicated by the updated size data of the 3D annotation box as the updated size of the 3D annotation box. The size update data may include the updated size data s'*r'*t'. The annotation device regenerates a 3D annotation box with a size of s'*r'*t'.

[0093] In some embodiments, when the size update data of the 3D annotation box includes the adjustment data of the 3D annotation box, the annotation device determines the updated size of the 3D annotation box according to the size of the generated 3D annotation box and the adjustment data of the 3D annotation box included in the update data. For example, the size data before the update includes s*r*t, the adjustment data includes (+i, -j, +k), and the updated size data is s'*r'*t', s'=s+i, r'=rj, t'=t+k. The annotation device regenerates a 3D annotation box with a size of s'*r'*t'.

[0094] Update method three, in some embodiments, the received update data of the 3D annotation box includes size update data and position update data of the 3D annotation box, and the annotation device can implement the update of the 3D annotation box according to the above-mentioned update method one and update method two.

[0095] According to the above embodiment, after the annotator inputs annotation data for an object that first appears in multiple frames of 3D point cloud data, and the annotation device completes the first annotation of the object and records the annotation data of the object, when the annotation device displays data of subsequent frames, it can automatically generate and display a 3D annotation box based on the recorded annotation data. The annotator adjusts the 3D annotation box based on the observation results and inputs update data for the 3D annotation box. The annotation device updates the 3D annotation box accordingly based on the update data, and generates and records the updated annotation data, thereby completing the tracking and annotation processing of the same object in subsequent frames of 3D point cloud data.

[0096] After the labeling device has finished recording the labeling data of the objects in one frame of data, the next frame of 3D point cloud data can be displayed to perform labeling operations on the objects in the subsequent frames of data.

[0097] Various implementations of step 302 are described below.

[0098] In step 302, the annotation device needs to refer to the recorded annotation data to generate a 3D annotation box that matches the recorded annotation data. The above generation methods 1 to 7 illustrate the process of generating a 3D annotation box that matches the annotation data. In step 302, the annotation device can combine any one of the above generation methods 1 to 7 and any one of the following generation methods 8 to 10 to perform the process of generating a 3D annotation box that matches the recorded annotation data.

[0099] Generation method 8. In some embodiments, the annotation device can generate a 3D annotation box that matches the annotation data of the recorded Nth frame; that is, the annotation device generates a 3D annotation box based on the annotation data of the object recorded for the first time. For example, in some scenarios, the annotator needs to observe and identify the 3D annotation box of an object recorded for the first time as a reference, so the annotation device can generate a 3D annotation box based on the annotation data recorded for the first time when displaying the subsequent frame 3D point cloud data.

[0100] Generation method 9. In some embodiments, the annotation device generates a 3D annotation box that matches the annotation data of the previous frame. For example, in some scenarios, the annotator needs to observe and identify the 3D annotation box of an object recorded in the previous frame as a reference, so the annotation device can generate a 3D annotation box based on the annotation data recorded in the previous frame during the display of the subsequent frame 3D point cloud data.

[0101] Generation method 10: In some embodiments, the annotation device generates a 3D annotation box that matches the annotation data of any previous frame recorded. For example, in some scenes, the annotator can observe and identify the object to be annotated well without the annotation data of a specific frame as a reference. In this case, the annotation device can generate a 3D annotation box based on the annotation data recorded in any previous frame during the display of the subsequent frame 3D point cloud data.

[0102] In step 302, after the annotation device generates and displays a 3D annotation frame that matches the recorded annotation data, the annotator can adjust the 3D annotation frame to update the 3D annotation frame that accurately annotates the object. The annotation device updates the generated 3D annotation frame according to the updated data and obtains the updated annotation data, which can refer to one of the above-mentioned update methods 1 to 3.

[0103] In step 302, the processing of the updated annotation data is recorded. When the annotation device has established an association between the object identifier and the annotation data of the object, the annotation data of the object's N+ith frame can be associated with the object identifier and recorded, thereby achieving accurate recording and tracking of the annotation data of the object in multiple frames of 3D point cloud data.

[0104] Through the processing of step 302, the labeling device can automatically generate a 3D labeling box to label the object in the subsequent frame of 3D point cloud data after initially labeling an object in the Nth frame of 3D point cloud data, and update the generated 3D labeling box accordingly according to the updated data of the 3D labeling box input by the labeler, and record the updated labeling data.

[0105] Through the processing of the above embodiments, after the labeling device initially labels an object according to the received labeling data, it can automatically track and label the object in subsequent multiple frames of 3D point cloud data according to the updated data of the received 3D labeling box, thereby significantly improving the labeling speed and efficiency.

[0106] In some scenes, some objects do not appear continuously in multiple frames of 3D point cloud data. After the first appearance, they may disappear in a frame of data and reappear in another subsequent frame of data. The technical solutions provided by some embodiments of the present application can effectively annotate objects that appear intermittently in multiple frames of 3D point cloud data.

[0107] like Fig. 9 As shown, in Figure 3 After step 302, the process further includes:

[0108] Step 303: when displaying the N+xth frame of 3D point cloud data, if the labeling device receives the first instruction for the object, cancel the display of the 3D labeling frame for labeling the object and stop the operation of labeling data for the object; wherein x is a natural number greater than or equal to 1;

[0109] Step 304: when displaying the N+kth frame of 3D point cloud data, skip the labeling operation of the object in the frame data, that is, skip the processing of the above step 302; wherein k is a natural number greater than or equal to 1, and x≤k;

[0110] Step 305: When displaying the N+yth frame of 3D point cloud data, upon receiving a second instruction for the object, a 3D annotation box that matches the recorded annotation data is regenerated for the object in the frame data, and operations on the annotation data of the object are performed, that is, the processing of the above step 302 is re-executed; wherein y is a natural number greater than or equal to 1, and k≤y.

[0111] In the above step 303, the first instruction may be an instruction in various forms, such as a preset button, a preset command, etc. The meaning of the first instruction may include that a continuously tracked and annotated object disappears in the current N+xth frame of 3D point cloud data. After the annotation device receives the first instruction for an object, it may stop the annotation operation of the object accordingly, that is, cancel the currently displayed 3D annotation box of the object, and cancel the operation of the annotation data of the object.

[0112] Furthermore, in the N+kth frame of 3D point cloud data, if the second instruction is not received, the labeling operation on the object is skipped in the frame data, including not displaying the 3D labeling box of the object and not operating the labeling data of the object.

[0113] If a second instruction for an object is received in the N+yth frame of 3D point cloud data, the instruction may mean that an object that has been marked and once disappeared reappears in the current N+yth frame of 3D point cloud data, then the marking device should re-execute the marking operation on the object, that is, regenerate a 3D marking box that is consistent with the recorded marking data, and perform operations on the 3D marking box and the marking data of the object.

[0114] Figures 10a to 10h An example of tracking and labeling objects that appear intermittently is shown in FIG. Figures 10a to 10h The figure shows a multi-frame 3D point cloud data of a road scene, in which object 2 is a vehicle equipped with sensors to obtain environmental information, and object 1 is a vehicle appearing in an adjacent lane. For the sake of simplicity and clarity, these multi-frame point cloud data are selected from continuous frame data. For a clearer explanation, Figures 10a to 10h In the example, the labeling device automatically generates a 3D labeling frame in the current frame data according to the labeling data of the previous frame of the recorded object.

[0115] Fig.10a FIG. 3 shows a 3D annotation frame annotated by the annotation device for two objects 1 and 2 appearing in the first frame of 3D point cloud data according to the annotation data input by the annotator.

[0116] exist Fig.10b FIG. 3 shows the 3D annotation frames of object 1 and object 2 generated according to the annotation data of the first frame after the annotation device displays the second frame of 3D point cloud data. Since object 2 is a vehicle for data collection, object 2 is always at a relatively fixed position in the 3D point cloud data. Figure 10b~10h In the second frame data, the position of the 3D annotation box of object 2 generated by the annotation device does not change, and object 2 can be accurately annotated. The position of object 1 in the second frame data has changed relative to the position of object 1 in the first frame, and the annotation box of object 1 generated by the annotation device based on the annotation data of the first frame cannot correctly annotate object 1.

[0117] exist Fig.10c , it is shown that the labeling device updates the 3D labeling frame of the object 1 to the accurate position of the object 1 according to the position update data of the object 1 input by the labeler.

[0118] exist Fig.10d FIG. 3 shows a 3D annotation frame generated by the annotation device based on the annotation data of the second frame after the annotation device displays the third frame of 3D point cloud data. It can be seen that in this frame of data, object 1 has disappeared in the data, and the 3D annotation frame of object 1 generated by the annotation device based on the annotation data of the second frame cannot correctly identify object 1.

[0119] exist Fig.10e In the example, after the annotation device receives the first instruction for object 1 input by the annotator, it cancels the 3D annotation frame of object 1. That is, after the annotation device receives the first instruction for object 1, it does not perform annotation processing on the third frame point cloud data of object 1.

[0120] exist Fig.10f FIG. 3 shows a 3D annotation box generated by the annotation device according to the annotation data of the third frame after the annotation device displays the fourth frame of 3D point cloud data. Since the annotation device does not record the annotation data of object 1 in the third frame of point cloud data, the annotation device only generates a 3D annotation box for object 2 in the fourth frame of point cloud data. However, it can be seen that object 1 reappears in the fourth frame of point cloud data.

[0121] exist Figure 10g In the example, after the labeling device receives the second instruction for object 1, it regenerates the 3D labeling box of object 1 according to the most recently recorded labeling data of object 1. After the labeler recognizes that object 1 reappears in the fourth frame of point cloud data, he inputs the second instruction for object 1. After the labeling device receives the second instruction for object 1, it generates the 3D labeling box for object 1 according to the most recently recorded labeling data of object 1, that is, the labeling data of object 1 in the second frame of point cloud data. However, it can be seen that the generated 3D labeling box of object 1 cannot correctly label object 1.

[0122] exist Fig.10h In the process, the labeling device updates the 3D labeling frame of object 1 to the accurate position of object 1 according to the received position update data of object 1. After the labeler identifies the position difference between object 1 and the 3D labeling frame of object 1, he inputs the position update data, and the labeling device updates the position of the 3D labeling frame of object 1 according to the position update data, so that the 3D labeling frame can correctly label object 1.

[0123] pass Fig. 9 The processing shown can track and label objects that appear intermittently in multiple frames of 3D point cloud data.

[0124] The following is a processing flow chart for tracking object 1 and object 2 in six consecutive frames of 3D point cloud data using the method provided in the above embodiment of the present application in an application scenario, including:

[0125] Step 1001: The labeling device displays the first frame of 3D point cloud data on the human-machine interface;

[0126] Step 1002: The labeling device receives the labeling data of object 1 and object 2 input by the labeler on the human-machine interface, wherein the labeling data includes position data and size data;

[0127] Step 1003: the labeling device creates labels for object 1 and object 2 respectively;

[0128] Step 1004: the labeling device generates a 3D labeling frame of object 1 according to the labeling data of object 1, and generates a 3D labeling frame of object 2 according to the labeling data of object 2, wherein the position and size of the generated 3D labeling frame are consistent with the position data and size data included in the labeling data;

[0129] Step 1005: the labeling device displays 3D labeling frames of object 1 and object 2 on the human-machine interface;

[0130] Step 1006: the labeling device receives update data of the 3D labeling frame of the object 1 input by the labeler on the human-machine interface, wherein the update data includes size update data of the 3D labeling frame;

[0131] Step 1007: the labeling device updates the size of the 3D labeling frame of the object 1 according to the update data of the 3D labeling frame of the object 1, and regenerates the 3D labeling frame after the size is updated;

[0132] Step 1008: the labeling device records the labeling data of the first frame of object 1, the labeling data including the updated size data and the received position data; and records the labeling data of the first frame of object 2, the standard data including the received position data and size data;

[0133] Step 1009: the labeling device establishes a corresponding relationship between the identifier of object 1 and the first frame of labeling data of object 1, and establishes a corresponding relationship between the identifier of object 2 and the first frame of labeling data of object 2;

[0134] Step 1010: The labeling device displays the second frame of 3D point cloud data on the human-machine interface;

[0135] Step 1011: the labeling device reads the labeling data of the first frame of object 1 according to the identification of object 1, generates a 3D labeling frame according to the read labeling data, and displays the 3D labeling frame on the human-machine interface; reads the labeling data of the first frame of object 2 according to the identification of object 2, generates a 3D labeling frame according to the read labeling data, and displays the 3D labeling frame on the human-machine interface;

[0136] Step 1012: the labeling device receives update data of the 3D labeling frame of the object 1 input by the labeler through the human-machine interface, wherein the update data includes position update data;

[0137] Step 1013: The labeling device updates the position of the 3D labeling frame of the object 1 according to the position update data to obtain updated position data;

[0138] Step 1014: the labeling device determines that the labeling data of the second frame of the object 1 includes the read size data and the updated position data, and associates and records the identifier of the object 1 and the labeling data of the second frame of the object 1;

[0139] Step 1015: the labeling device receives update data of the 3D labeling frame of the object 2 input by the labeler through the human-machine interface, wherein the update data includes position update data;

[0140] Step 1016: The labeling device updates the position of the 3D labeling frame of the object 2 according to the position update data to obtain updated position data;

[0141] Step 1017: The labeling device determines that the labeling data of the second frame of the object 2 includes the read size data and the updated position data, and associates and records the identifier of the object 2 and the labeling data of the second frame of the object 2;

[0142] Step 1018: The annotation device displays the third frame of 3D point cloud data on the human-machine interface;

[0143] Step 1019: the labeling device reads the labeling data of the second frame of object 1 according to the identification of object 1, generates a 3D labeling frame according to the read labeling data, and displays the 3D labeling frame on the human-machine interface; reads the labeling data of the second frame of object 2 according to the identification of object 2, generates a 3D labeling frame according to the read labeling data, and displays the 3D labeling frame on the human-machine interface;

[0144] Step 1020: the labeling device receives update data of the 3D labeling frame of the object 1 input by the labeler through the human-machine interface, wherein the update data includes position update data and size update data;

[0145] Step 1021: The labeling device updates the position of the 3D labeling frame of the object 1 according to the position update data to obtain updated position data, and updates the size of the 3D frame of the object 1 according to the size update data to obtain updated size data;

[0146] Step 1022: The labeling device determines that the labeling data of the third frame of the object 1 includes the updated position data and the updated size data, and associates and records the identifier of the object 1 and the labeling data of the third frame of the object 1;

[0147] Step 1023: the labeling device receives update data of the 3D labeling frame of the object 2 input by the labeler through the human-machine interface, wherein the update data includes position update data and size update data;

[0148] Step 1024: the labeling device updates the position of the 3D labeling frame of the object 2 according to the position update data to obtain updated position data, and updates the size of the 3D frame of the object 2 according to the size update data to obtain updated size data;

[0149] Step 1025: The labeling device determines that the labeling data of the third frame of the object 2 includes the updated position data and the updated size data, and associates and records the identifier of the object 2 and the labeling data of the third frame of the object 2;

[0150] Step 1026: The annotation device displays the fourth frame of 3D point cloud data through the human-machine interface;

[0151] Step 1027: the labeling device reads the labeling data of the third frame of object 1 according to the identification of object 1, generates a 3D labeling frame according to the read labeling data, and displays the 3D labeling frame on the human-machine interface; reads the labeling data of the third frame of object 2 according to the identification of object 2, generates a 3D labeling frame according to the read labeling data, and displays the 3D labeling frame on the human-machine interface;

[0152] Step 1028: The labeling device receives the first instruction for the object 1, cancels the 3D labeling frame of the object 1 displayed on the human-machine interface, and stops the operation on the 3D labeling frame of the object 1 and the labeling data;

[0153] Step 1029: the labeling device receives update data of the 3D labeling frame of the object 2 input by the labeler through the human-machine interface, wherein the update data includes position update data and size update data;

[0154] Step 1030: The labeling device updates the position of the 3D labeling box of the object 2 according to the position update data to obtain updated position data, and updates the size of the 3D box of the object 2 according to the size update data to obtain updated size data;

[0155] Step 1031: The labeling device determines that the labeling data of the fourth frame of the object 2 includes the updated position data and the updated size data, and associates and records the identifier of the object 2 and the labeling data of the fourth frame of the object 2;

[0156] Step 1032: the annotation device displays the fifth frame of 3D point cloud data through the human-machine interface;

[0157] Step 1033: the labeling device reads the labeling data of the fourth frame of the object 2 according to the identifier of the object 2, generates a 3D labeling frame according to the read labeling data, and displays the 3D labeling frame on the human-machine interface;

[0158] Step 1034: the labeling device receives update data of the 3D labeling frame of the object 2 input by the labeler through the human-machine interface, wherein the update data includes position update data;

[0159] Step 1035: The labeling device updates the position of the 3D labeling frame of the object 2 according to the position update data to obtain updated position data;

[0160] Step 1036: The labeling device determines that the labeling data of the fifth frame of the object 2 includes the updated position data and the read size data, and associates and records the identifier of the object 2 and the labeling data of the fifth frame of the object 2;

[0161] Step 1037: the annotation device displays the sixth frame of 3D point cloud data through the human-machine interface;

[0162] Step 1038: The labeling device receives the second instruction for object 1, reads the labeling data of the third frame of object 1 according to the identifier of object 1, generates a 3D labeling frame according to the read labeling data, and displays the 3D labeling frame on the human-machine interface; reads the labeling data of the fifth frame of object 2 according to the identifier of object 2, generates a 3D labeling frame according to the read labeling data, and displays the 3D labeling frame on the human-machine interface;

[0163] Step 1039: the marking device receives the first instruction for the object 2, cancels the display of the 3D marking frame of the object 2 on the human-machine interface, and stops the operation on the 3D marking frame of the object 2 and the marking data;

[0164] Step 1040: the labeling device receives update data of the 3D labeling frame of the object 1 input by the labeler through the human-machine interface, wherein the update data includes position update data;

[0165] Step 1041: The labeling device updates the position of the 3D labeling frame of the object 1 according to the position update data to obtain updated position data;

[0166] Step 1042 : The labeling device determines that the labeling data of the sixth frame of object 1 includes the updated position data and the read size data, and associates and records the identifier of object 1 and the labeling data of the sixth frame of object 1 .

[0167] The above exemplary embodiment provides a tracking and annotation process of objects in multiple frames of 3D point cloud data in an application scenario. According to multiple embodiments provided in the embodiments of the present application, more implementation situations can be combined in multiple application scenarios.

[0168] The embodiment of the present application also provides a non-volatile storage medium, in which at least one machine executable instruction is stored, and the processor executes the at least one machine executable instruction to perform the above-mentioned Figures 3 to 9 Any of the methods in .

[0169] The implementation of the substantive content and functional operations described in this application can be implemented by a variety of systems, digital electronic circuits, or computer software, firmware or hardware, and these implementations include the structures disclosed in the specification and equivalent structures, or a combination of these structures. The implementation of the substantive content described in the specification can be implemented as one or more computer program products, for example, one or more modules of computer program instructions, which are encoded and stored in a tangible and non-volatile computer-readable medium, and the computer program instructions can be executed by a data processing device, or used to control the operation of a data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device, a composition that can affect a machine-readable propagation signal, or a combination of these substances. The term "data processing unit" or "data processing device" includes all devices, equipment and machines for processing data, exemplarily including a programmable processor, a computer, or a multi-processor, a multi-computer. In addition to hardware, these devices may include code to establish an executable environment for the computer program in question, for example, code constituting a processor firewall, a protocol stack, a database management system, an operating system, or a combination of these codes.

[0170] A computer program (also referred to as a program, software, software application, script, or code) can be written in any programming language, including compiled or translated speech; and can be deployed in any form, including a stand-alone program or a module, component, sub-use case, or other unit suitable for a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as part of a file that also stores other programs or data (for example, one or more scripts stored in a markup language document), or the program can be stored in a separate file specifically for the program in question, or in a mutually coordinated file (for example, multiple files storing one or more modules, subroutines, or code portions). A computer program can be deployed to be executed by one or more computers located at one address, or distributed at multiple addresses, and interconnected by a communication network.

[0171] The processes or logic diagrams described in this specification can be performed by one or more programmable processors to execute one or more computer programs and generate output results based on input data. The processes or logic diagrams can be performed by dedicated logic circuits, and various devices can also be implemented as dedicated logic circuits, such as field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs).

[0172] Processors for executing computer programs include, for example, general-purpose microprocessors and special-purpose microprocessors, and any one or more processors in any type of digital computer. Typically, a processor receives instructions and data from a read-only memory and / or a random access memory. The basic unit of a computer includes a processor and one or more storage devices, the processor is used to execute instructions, and the storage device is used to store instructions and data. Typically, a computer also includes or is operatively coupled to one or more large storage devices to receive data and / or send data, and the large storage device includes a magnetic disk, a magneto-optical disk, or an optical disk. However, a computer does not necessarily include these devices. Computer-readable media for storing instructions and data include all forms of non-volatile memory, media, and storage devices, exemplarily including semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices. The processor and memory can be replaced by dedicated logic circuits or incorporated into dedicated logic circuits.

[0173] Although the present application document includes a variety of embodiments, these embodiments are not intended to be interpreted as limiting the scope of protection of the present application, but are only descriptions of features, which can be implemented in specific embodiments of specific inventions. Some features described in the independent embodiments of the present application can also be combined and implemented in a separate embodiment. A plurality of features described in a separate embodiment can also be implemented in a plurality of embodiments, respectively, or in any suitable combination of a further step. Moreover, although some features are described in a specific combination, one or more features can also be removed from the desired one or more combinations, and the desired combination can be further combined or deformed to further combinations.

[0174] Similarly, although various operations are described in a certain order in the accompanying drawings, it should not be understood that these operations must be performed in such an order to achieve the desired results. Moreover, the splitting of various system components in the embodiments should not be understood as requiring such splitting in all embodiments. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these changes and variations.

Claims

1. A method for tracking and annotating objects in multi-frame 3D point cloud data, characterized in that: include: The annotation device displays the Nth frame of 3D point cloud data and receives input annotation data of a 3D annotation box that annotates an object; Recording the annotation data of the Nth frame of the object; wherein the annotation data includes at least one or more of the following: position data of the 3D annotation box, size data of the 3D annotation box, and category data of the object; In the process of displaying the subsequent N+i-th frame of 3D point cloud data, in the 3D coordinate system space displaying the 3D point cloud data, a 3D annotation frame consistent with the recorded annotation data is generated and displayed; receiving the input update data of the 3D annotation frame, updating the generated 3D annotation frame according to the update data, and obtaining the updated annotation data; recording the updated annotation data as the annotation data of the N+i-th frame of the object; wherein N and i are both natural numbers greater than or equal to 1; When displaying the N+xth frame of 3D point cloud data, in response to receiving a first instruction for the object, canceling the display of the 3D annotation box annotating the object, and stopping the operation of the marking data of the object, wherein the first instruction indicates that the object does not exist in the current N+xth frame of 3D point cloud data; When displaying the N+kth frame of 3D point cloud data, skip the labeling operation of the object in the frame data; When displaying the N+yth frame of 3D point cloud data, in the case of receiving a second instruction for the object, regenerating a 3D annotation frame for the object in the frame data that matches the recorded annotation data, and performing an operation on the annotation data of the object, wherein the second instruction indicates that the object reappears in the current N+yth frame of 3D point cloud data; Wherein, x, k and y are all natural numbers greater than or equal to 1, and x≤k≤y.

2. The method according to claim 1, characterized in that In the process of displaying the Nth frame of 3D point cloud data by the annotation device, after receiving the input annotation data of the 3D annotation frame annotating an object, it also includes: establishing an identifier of the object; After recording the annotation data of one frame of the object, the method further includes: establishing an association relationship between the identifier of the object and the annotation data of the frame of the object.

3. The method according to claim 1, characterized in that Generate a 3D annotation box that matches the recorded annotation data, including one of the following: Generate a 3D annotation box that matches the annotation data of the recorded Nth frame; Generate a 3D annotation box that matches the annotation data of the previous frame recorded; Generate a 3D annotation box that matches the annotation data of any previous frame recorded.

4. The method according to claim 1, characterized in that: In the case where the annotation data includes position data of a 3D annotation box, generating a 3D annotation box that matches the annotation data includes: A 3D annotation box is generated at the position indicated by the position data included in the annotation data.

5. The method according to claim 1, characterized in that In the case where the annotation data includes the size data of the 3D annotation box, a 3D annotation box consistent with the annotation data is generated, including: Generate a 3D annotation box whose dimensions are consistent with the dimension data included in the annotation data.

6. The method according to claim 1, characterized in that When the annotation data includes the category data of the object, a 3D annotation box that matches the annotation data is generated, including: Generate a 3D annotation box of a preset size corresponding to the object category.

7. The method according to claim 1, characterized in that In the process of displaying the Nth frame of 3D point cloud data by the annotation device, after generating a 3D annotation frame that matches the annotation data, the method further includes: Receive the input updated data of the 3D annotation box, update the generated 3D annotation box accordingly according to the updated data, and obtain the updated annotation data; Recording the Nth frame of the annotation data of the object includes: recording the updated annotation data of the Nth frame of the object.

8. The method according to claim 1 or 7, characterized in that: The received update data of the 3D annotation frame includes the position update data of the 3D annotation frame; The generated 3D annotation box is updated accordingly according to the updated data, and the updated annotation data is obtained, including: determining the updated position of the 3D annotation box according to the position update data of the 3D annotation box, and displaying the 3D annotation box at the updated position; determining that the updated annotation data includes the updated position data of the 3D annotation box.

9. The method according to claim 8, characterized in that The position update data of the 3D annotation frame includes updated position data of the 3D annotation frame; Determining the updated position of the 3D annotation frame according to the position update data of the 3D annotation frame includes: determining the position indicated by the updated position data of the 3D annotation frame as the updated position of the 3D annotation frame.

10. The method according to claim 8, characterized in that The position update data of the 3D annotation frame includes the position update direction and position offset of the 3D annotation frame; Determining the updated position of the 3D annotation frame according to the position update data of the 3D annotation frame includes: determining the updated position of the 3D annotation frame according to the position of the generated 3D annotation frame and the update direction and position offset of the 3D annotation frame included in the position update data.

11. The method according to claim 1 or 7, characterized in that: The received update data of the 3D annotation box includes size update data of the 3D annotation box; The generated 3D annotation box is updated accordingly according to the updated data, and the updated annotation data is obtained, including: determining the updated size of the 3D annotation box according to the size update data, and displaying the 3D annotation box using the updated size; and determining that the updated annotation data includes the updated size data of the 3D annotation box.

12. The method according to claim 11, characterized in that The size update data of the 3D annotation frame includes updated size data of the 3D annotation frame; Determining the updated size of the 3D annotation frame according to the size update data includes: determining the size indicated by the updated size data of the 3D annotation frame as the updated size of the 3D annotation frame.

13. The method according to claim 11, characterized in that The size update data of the 3D annotation frame includes adjustment data of the 3D annotation frame; Determining the updated size of the 3D annotation box according to the size update data includes: determining the updated size of the 3D annotation box according to the size of the generated 3D annotation box and the adjustment data of the 3D annotation box included in the update data.

14. The method according to claim 1, characterized in that After the labeling device receives input labeling data of a 3D labeling frame labeling an object, the method further includes: In the 3D coordinate system space displaying the 3D point cloud data, a 3D annotation box that matches the annotation data is generated and displayed.

15. The method according to claim 1, characterized in that Also includes: After recording one frame of annotation data, the next frame of 3D point cloud data is displayed.

16. The method according to claim 1, characterized in that The annotation device includes a human-machine interface; the annotation device displays a frame of 3D point cloud data through the human-machine interface; receives input annotation data and update data through the human-machine interface; or, The marking device receives the marking data and the update data from the client.

17. The method according to claim 1, characterized in that The annotation device displays a frame of 3D point cloud data, including: Display a frame of 3D point cloud data according to a preset display direction; or, An input display direction is received, and a frame of 3D point cloud data is displayed according to the received display direction.

18. The method according to claim 1, characterized in that The multi-frame 3D point cloud data includes one of the following: continuous multi-frame 3D point cloud data, multi-frame 3D point cloud data with the same time interval, and multi-frame 3D point cloud data with different time intervals.

19. A tracking and labeling device for objects in multi-frame 3D point cloud data, characterized in that: include: A processor and at least one memory, wherein at least one machine executable instruction is stored in the at least one memory, and the processor executes the at least one machine executable instruction to perform the method according to any one of claims 1 to 18.

20. A non-volatile storage medium, characterized in that: At least one machine executable instruction is stored, and after the at least one machine executable instruction is executed by a processor, the method according to any one of claims 1 to 18 is executed.

Images