Object labeling method, object labeling device, and computer-readable storage medium

Abstract

The application provides an object labeling method, an object labeling device and a computer readable storage medium. The object labeling method comprises: obtaining an image point cloud pair based on a first timestamp of a laser radar, wherein each image point cloud pair comprises a two-dimensional image and a three-dimensional point cloud; extracting a two-dimensional detection box of a target object in the two-dimensional image and a three-dimensional detection box of the target object in the three-dimensional point cloud; obtaining a projection detection box; matching the two-dimensional detection box of each image point cloud pair with the projection detection box to obtain a to-be-supplemented frame image point cloud pair that fails to match; generating a supplemented frame three-dimensional detection box of the to-be-supplemented frame image point cloud pair by using the three-dimensional detection box in adjacent image point cloud pairs of the to-be-supplemented frame image point cloud pair, wherein the adjacent image point cloud pairs are image point cloud pairs that successfully match; and labeling the target object in the two-dimensional image of the to-be-supplemented frame image point cloud pair by using the supplemented frame three-dimensional detection box. In this way, the accuracy and completeness of the object labeling method can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of visual perception for autonomous driving, and in particular to an object annotation method, an object annotation device, and a computationally readable storage medium. Background Technology

[0002] In the field of autonomous driving visual perception, given an image captured by an onboard camera (i.e., a two-dimensional image), we need to know the three-dimensional (3D) coordinates of target objects in the image, such as obstacles (vehicles, pedestrians, etc.), relative to the vehicle. This information is crucial for subsequent path planning and control. Utilizing deep learning to directly extract the 3D coordinates of obstacles from images end-to-end is currently a popular technology, thus creating a need for 3D image annotation.

[0003] 3D image annotation needs to meet several basic conditions: "completeness of annotation": annotating as many 3D obstacles in the image as possible; "accuracy of annotation": the annotation of 3D obstacles must be accurate; 3D annotations used for deep learning training must cover a large number of scenes. Therefore, improving the completeness and accuracy of 3D annotation is a necessary technical problem to be solved to improve the safety of autonomous driving. Summary of the Invention

[0004] This application provides an object annotation method, an object annotation apparatus, and a computationally readable storage medium to improve the accuracy and completeness of the object annotation method, thereby effectively enhancing the safety of autonomous driving.

[0005] To solve the above-mentioned technical problems, the technical solution adopted in this application is as follows: 1. An object annotation method, the object annotation method comprising: acquiring image point cloud pairs based on a first timestamp of a LiDAR, wherein each image point cloud pair includes a two-dimensional image and a three-dimensional point cloud; extracting a two-dimensional detection box of a target object in the two-dimensional image, and extracting a three-dimensional detection box of the target object in the three-dimensional point cloud; projecting the three-dimensional detection box of each image point cloud pair onto the two-dimensional image to obtain a projected detection box; matching the two-dimensional detection box and the projected detection box of each image point cloud pair to obtain a mismatched image point cloud pair to be framed; generating a framed three-dimensional detection box for the image point cloud pair to be framed using the three-dimensional detection boxes in adjacent image point cloud pairs, wherein adjacent image point cloud pairs are successfully matched image point cloud pairs; and annotating the target object in the two-dimensional image of the image point cloud pair to be framed using the framed three-dimensional detection box.

[0006] To solve the above-mentioned technical problems, the technical solution adopted in this application is: to provide an object annotation device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the object annotation method of any of the above embodiments.

[0007] To solve the above-mentioned technical problems, the technical solution adopted in this application is to provide a computer-readable storage medium for storing program instructions, which, when executed by a processor, are used to implement the object annotation method of any of the above embodiments.

[0008] The beneficial effects of the embodiments of this application are as follows: The object annotation method of this application includes acquiring image point cloud pairs based on the first timestamp of a LiDAR, wherein each image point cloud pair includes a two-dimensional image and a three-dimensional point cloud; extracting a two-dimensional detection box of a target object in the two-dimensional image, and extracting a three-dimensional detection box of the target object in the three-dimensional point cloud; projecting the three-dimensional detection box of each image point cloud pair onto the two-dimensional image to obtain a projected detection box; matching the two-dimensional detection box and the projected detection box of each image point cloud pair to obtain a matching but not matched image point cloud pair to be supplemented; generating a supplemented three-dimensional detection box of the image point cloud pair to be supplemented using the three-dimensional detection boxes in the adjacent image point cloud pairs to be supplemented, wherein the adjacent image point cloud pairs are successfully matched image point cloud pairs; and annotating the target object in the two-dimensional image of the image point cloud pair to be supplemented using the supplemented three-dimensional detection box. Acquiring image point cloud pairs based on the first timestamp of a LiDAR ensures that the timestamps of the two-dimensional image and the three-dimensional point cloud are as close as possible, thereby effectively improving the accuracy of target object annotation while also effectively reducing the computational load of the object annotation device. Furthermore, the object annotation method of this application determines the image point cloud pair to be framed by matching the 3D detection box and the 2D detection box in each image point cloud pair, and obtains the target object in the 2D image of the image point cloud pair to be framed by the successfully matched image point cloud pair, thereby completing the 3D annotation information of the target object at each time, and thus effectively improving the completeness of the target object annotation. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the method steps in the first embodiment of the object annotation method of this application;

[0010] Figure 2 This is a schematic diagram of the steps in step S100 of the object annotation method of this application;

[0011] Figure 3 This is a schematic diagram of the method steps for extracting the 3D detection box of the target object in the 3D point cloud in step S200 of the object annotation method of this application;

[0012] Figure 4 This is a schematic diagram of an embodiment of matching a two-dimensional detection frame and a three-dimensional detection frame according to this application;

[0013] Figure 5 This is a schematic diagram of the method steps of step S500 of the object annotation method of this application.

[0014] Figure 6 This is a schematic diagram of the method steps of the second embodiment of the object annotation method of this application;

[0015] Figure 7 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0017] The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise expressly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0018] This application provides an object annotation method. In the embodiments of this application, the object annotation method is mainly applied to the field of autonomous driving. At the same time, the object annotation method of this application is also applicable to other corresponding technical fields. This application mainly describes the object annotation method in detail with the application of the object annotation method in the field of autonomous driving. The execution subject of the object annotation method of this application is an object annotation device installed on a vehicle or other equipment.

[0019] Among them, such as Figure 1 As shown, Figure 1 This is a schematic diagram of the method steps of the first embodiment of the object annotation method of this application. The object annotation method can be found in [reference needed]. Figure 1 The method steps shown are implemented, specifically including steps S100 to S600, as follows:

[0020] Step S100: Obtain image point cloud pairs based on the first timestamp of the LiDAR, wherein each image point cloud pair includes a two-dimensional image and a three-dimensional point cloud.

[0021] Specifically, before acquiring image point cloud pairs, it is necessary to prepare the input data for the annotation system. For example, this involves preparing the vehicle for data collection and ensuring that the time bases of all sensors are consistent. This includes ensuring the time bases of the LiDAR, camera, and IMU (Inertial Measurement Unit) are consistent, guaranteeing the time base consistency of LiDAR data, camera data, and IMU data. LiDAR data includes 3D point clouds acquired by the LiDAR, camera data includes 2D images acquired by the camera, and IMU data includes the angular velocity and linear acceleration data of the vehicle's motion. Furthermore, it is necessary to calibrate the intrinsic and extrinsic parameters of the sensors. This includes calibrating the intrinsic parameters of the camera and the extrinsic parameters between the LiDAR and camera. Camera intrinsic parameter calibration refers to calibrating the inherent parameters of the camera, such as the focal length and pixel pitch of the camera's internal optical elements. Extrinsic parameter calibration between the LiDAR and camera refers to the positional relationship and orientation between the LiDAR and camera, as well as their positional relationship in the 3D environment. In this embodiment, to ensure a large overlap in the fields of view between the camera and the LiDAR, a camera with a large field of view is used, and the intrinsic parameter model of the camera is Scaramuzha. In other embodiments, a pinhole camera model or the fisheye camera model built into OpenCV can also be used. The extrinsic parameters between the LiDAR and the camera are calibrated using a common calibration board method.

[0022] The object annotation device completes the preliminary preparations based on the above method. Further, the object annotation device acquires image point cloud pairs containing two-dimensional images and three-dimensional point clouds based on the first timestamp of the LiDAR. In this embodiment, the first timestamp is the time corresponding to each frame of three-dimensional point cloud acquired by the LiDAR. That is, the time corresponding to each frame of three-dimensional point cloud is called the first timestamp. The image point cloud pairs are acquired based on the first timestamp of the LiDAR to improve the accuracy of the object annotation method by ensuring that the timestamp of the three-dimensional point cloud in the image point cloud pair is as close as possible to the timestamp of the two-dimensional image.

[0023] like Figure 2 As shown, Figure 2 This is a schematic diagram of step S100 of the object annotation method of this application; wherein, the acquisition of image point cloud pairs based on the first timestamp of the LiDAR can be specifically achieved through... Figure 2 The method steps shown are implemented, specifically including steps S110 to S120.

[0024] Step S110: Use a lidar to acquire several frames of 3D point cloud data.

[0025] Step S120: Traverse all the first timestamps of the three-dimensional point clouds, and form image-point cloud pairs with the two-dimensional image whose second timestamp is closest to the first timestamp and the three-dimensional point cloud.

[0026] The parameters for steps S110 and S120 are standardized.

[0027] In this embodiment, the object annotation device uses a lidar to collect several frames of 3D point clouds at different times and a camera to collect several frames of 2D images at different times. The time corresponding to each frame of 2D image is called the second timestamp. Furthermore, the object annotation device traverses the first timestamps of all 3D point clouds and forms an image point cloud pair with the 2D image whose second timestamp is closest to the first timestamp and the 3D point cloud. This effectively ensures that the 3D point cloud and the 2D image in the image point cloud pair are data collected by the lidar and the camera, respectively, and are synchronized as much as possible.

[0028] For example, in practical applications, the frequency of LiDAR is generally 10Hz, and the acquisition frequency of camera is generally 30Hz. Therefore, the data volume of the two is generally in a one-to-three ratio. Thus, the object labeling device can effectively reduce the amount of computation by acquiring image point cloud pairs based on the first timestamp of LiDAR. That is, when selecting two-dimensional images, the object labeling device only needs to select the two-dimensional image whose second timestamp is closest to the first timestamp of each frame of three-dimensional point cloud.

[0029] Step S200: Extract the two-dimensional detection bounding box of the target object in the two-dimensional image, and extract the three-dimensional detection bounding box of the target object in the three-dimensional point cloud.

[0030] The object annotation device acquires image point cloud pairs based on the method described in step S100 above. Further, the object annotation device extracts two-dimensional detection boxes of target objects from the two-dimensional image and three-dimensional detection boxes of target objects from the three-dimensional point cloud. The object annotation device uses a deep learning model to identify target objects in the two-dimensional image, acquires the two-dimensional detection boxes of the target objects, and uses an algorithm to perform two-dimensional tracking of the target objects. The deep learning model includes You Only LookOnce (YOLO, an object detection algorithm model), and the algorithm for two-dimensional tracking of target objects includes a multi-object tracking algorithm (DeepSORT). The object annotation device acquires the two-dimensional detection boxes of the target image in the two-dimensional image according to the above-described algorithm and tracks the two-dimensional detection boxes based on the above-described tracking algorithm to assign an ID corresponding to the two-dimensional detection box of each target object, thereby distinguishing different target objects.

[0031] Furthermore, such as Figure 3 As shown, Figure 3This is a schematic diagram illustrating the method steps in step S200 of the object annotation method of this application for extracting the 3D detection bounding box of the target object in the 3D point cloud. The extraction of the 3D detection bounding box of the target object in the 3D point cloud can be achieved through… Figure 3 The method steps shown are implemented, specifically including steps S210 to S230.

[0032] Step S210: Obtain the LiDAR pose corresponding to each 3D point cloud and extract the 3D detection box of each 3D point cloud.

[0033] Step S220: Convert the 3D detection box to the global coordinate system according to the LiDAR pose of all corresponding points in the 3D point cloud.

[0034] Step S230: Track the 3D detection box of the target object in the global coordinate system to obtain the motion trajectory of the target object.

[0035] Steps S210 to S230 will be described in a unified manner.

[0036] The object annotation device acquires the 3D bounding box of the target object in each 3D point cloud, and also acquires the LiDAR pose corresponding to each 3D point cloud. Specifically, the object annotation device acquires 3D point clouds based on IMU data and LiDAR data, and uses SLAM algorithms such as Fast-LIO to obtain the LiDAR pose when acquiring each frame of the 3D point cloud, i.e., acquires the LiDAR pose. Further, the object annotation device transforms the 3D bounding boxes to the global coordinate system according to the LiDAR poses of all corresponding points in the 3D point cloud, so as to track the 3D bounding boxes in the global coordinate system and acquire the motion trajectory of the target object. Based on the above method, the object annotation device can more accurately track the 3D bounding boxes and acquire the motion trajectory of the target object by transforming the 3D bounding boxes to the global coordinate system based on the LiDAR pose, thereby effectively improving the accuracy of the object annotation method. Specifically, after transforming the 3D bounding boxes to the global coordinate system, the object annotation device uses a Kalman filter to perform 3D tracking of the target object, thereby obtaining attributes such as the target object's motion trajectory and velocity.

[0037] Step S300: Project the 3D detection box of each image point cloud pair onto the 2D image to obtain the projected detection box.

[0038] Step S400: Match the two-dimensional detection box and the projected detection box of each image point cloud pair to obtain the image point cloud pairs that failed to match.

[0039] Steps S300 to S400 will be described in a unified manner.

[0040] In this embodiment, after obtaining the 3D and 2D detection boxes in step S200, the object annotation device uses Hungarian matching to match the 3D and 2D detection boxes. Specifically, the 3D detection box of each image point cloud pair is projected onto the 2D image to obtain a projected detection box. The projected detection box is the 2D projected detection box formed after the 3D detection box is projected onto the 2D image. Further, the 2D detection box of each image point cloud pair is Hungarian matched with the projected detection box using the Intersection Over Union (IOU). For image point cloud pairs where the 2D detection box and the projected detection box successfully match, the corresponding 3D detection box is used to annotate the target object in the successfully matched 2D image, thereby achieving 3D annotation of the target object in the successfully matched image point cloud pair. Furthermore, the two-dimensional detection box and the projected detection box of each image point cloud pair are matched. Based on this, the image point cloud pair to be supplemented can be obtained, that is, the image point cloud pair in which the two-dimensional detection box and the projected detection box fail to match. For example, at a certain moment, the lidar did not collect the three-dimensional point cloud data of the target object, or the collected three-dimensional point cloud data of the target object had a deviation, etc., which resulted in the inability to obtain the three-dimensional detection box corresponding to the two-dimensional detection box at that moment in the three-dimensional point cloud, thus causing the two-dimensional detection box and the projected detection box to fail to match, resulting in incomplete three-dimensional annotation of the target object.

[0041] Step S500: Generate the frame-complementing 3D detection box of the image point cloud pair to be framed using the 3D detection boxes in the neighboring image point cloud pairs of the image point cloud pair to be framed, wherein the neighboring image point cloud pairs are successfully matched image point cloud pairs.

[0042] Step S600: Use the frame-complementing 3D detection box to annotate the target object in the 2D image of the point cloud pair of the image to be framed.

[0043] Steps S500 to S600 will be described in a unified manner.

[0044] For example Figure 4 As shown, Figure 4This is a schematic diagram of an embodiment of the matching of two-dimensional and three-dimensional detection boxes according to this application. The object annotation device obtains two-dimensional and three-dimensional detection boxes from image point cloud pairs at different times along the time axis t0 to t4 through steps S100 to S200. The image point cloud pairs at times t0, t2, and t4 are image point cloud pairs to be framed, and the image point cloud pairs at times t1 and t3 are successfully matched image point cloud pairs. The object annotation device generates a framed three-dimensional detection box corresponding to the framed image point cloud pair using the three-dimensional detection boxes in adjacent image point cloud pairs of the framed image point cloud pair to be framed. This allows the object to be annotated in the two-dimensional image of the framed image point cloud pair using the framed three-dimensional detection boxes, thereby completing the three-dimensional annotation information of the target object at each time point and effectively improving the completeness of the target object annotation. Adjacent image point cloud pairs are those adjacent to and successfully matched with the corresponding framed image point cloud pair to be framed. For example... Figure 4 Image point cloud pairs at time t1 and t3.

[0045] Optionally, see Figure 5 As shown, Figure 5 This is a schematic diagram of a method step S500 of the object annotation method of this application. Step S500 can be achieved through... Figure 5 The method shown is implemented in the manner described, specifically including steps S510 to S520.

[0046] Step S510: Obtain the kinematic properties of the target object using the 3D detection bounding boxes in adjacent image point cloud pairs, and obtain the adjacent orientation angles of adjacent image point cloud pairs; wherein, the kinematic properties include position and velocity.

[0047] Step S520: Generate a frame-complemented 3D detection box using kinematic properties, and generate a frame-complemented orientation angle using adjacent orientation angles.

[0048] Step S530: Generate a frame-patterned 3D detection box using kinematic properties and frame-patterning orientation angle.

[0049] Steps S510 and S530 will be described in a unified manner.

[0050] Specifically, the 3D detection box is the 3D data of the target object in the global coordinate system actually acquired by the LiDAR at a corresponding moment, while the 3D detection boxes in adjacent image point cloud pairs are the 3D data actually detected by the LiDAR. Therefore, the object annotation device uses the 3D detection boxes in adjacent image point cloud pairs to obtain the corresponding kinematic properties and orientation angles of the target object at the corresponding moment, and then uses these kinematic properties and orientation angles to calculate the frame-complemented 3D detection box. Among them, the adjacent orientation angle is also called the orientation angle of the target object in the adjacent image point cloud pair, and the frame-complemented orientation angle is also called the orientation angle of the target object in the point cloud pair to be frame-complemented.

[0051] For example, see Figure 4 Let P be the position of the target object in three-dimensional space at time t. t The velocity of the target object in three-dimensional space at time t is V. t And the orientation angle of the target object in three-dimensional space at time t is θ. t In this context, the image point cloud pairs at times t0, t2, and t4 are the image point cloud pairs to be framed, and the image point cloud pairs at times t1 and t3 are the adjacent image point cloud pairs of the image point cloud pairs to be framed, i.e., the successfully matched image point cloud pairs. The three-dimensional data of the image point cloud pairs to be framed at times t0, t2, and t4 can be calculated based on the following formulas ① to ⑥, thereby generating the corresponding framed three-dimensional detection boxes based on the three-dimensional data. The specific calculation formulas ① to ⑥ are as follows:

[0052] P0=P1+(t0-t1)*V1 ①;

[0053] θ0=θ1 ②;

[0054] P4=P3+(t4-t3)*V3 ③;

[0055] θ4=θ3 ④;

[0056] P2=P1+(t2-t1)*V1 ⑤;

[0057] θ2=(θ3-θ1) / (t3-t1)*(t2-t1)+θ1 ⑥.

[0058] Unlike existing technologies, the object annotation method of this application includes acquiring image point cloud pairs based on the first timestamp of a LiDAR scanner, wherein each image point cloud pair includes a two-dimensional image and a three-dimensional point cloud; extracting two-dimensional detection boxes of target objects from the two-dimensional image and extracting three-dimensional detection boxes of target objects from the three-dimensional point cloud; projecting the three-dimensional detection boxes of each image point cloud pair onto the two-dimensional image to obtain projected detection boxes; matching the two-dimensional detection boxes and projected detection boxes of each image point cloud pair to obtain image point cloud pairs that fail to match; generating supplementary three-dimensional detection boxes for the supplementary image point cloud pair using the three-dimensional detection boxes in adjacent image point cloud pairs, wherein adjacent image point cloud pairs are successfully matched image point cloud pairs; and annotating the target object in the two-dimensional image of the supplementary image point cloud pair using the supplementary three-dimensional detection boxes. Acquiring image point cloud pairs based on the first timestamp of a LiDAR scanner ensures that the timestamps of the two-dimensional image and the three-dimensional point cloud are as close as possible, thereby effectively improving the accuracy of target object annotation while also effectively reducing the computational load of the object annotation device. Furthermore, the object annotation method of this application determines the image point cloud pair to be framed by matching the 3D detection box and the 2D detection box in each image point cloud pair, and obtains the target object in the 2D image of the image point cloud pair to be framed by the successfully matched image point cloud pair, thereby completing the 3D annotation information of the target object at each time, thus effectively improving the completeness of the target object annotation and thus effectively improving the safety of autonomous driving.

[0059] This application also proposes a second embodiment of the object annotation method, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of the method steps in the second embodiment of the object annotation method of this application. The object annotation method can also be achieved through… Figure 6 The steps shown are as follows:

[0060] Step S700: Acquire several frames of 3D point cloud using a lidar and acquire several frames of 2D images using a camera.

[0061] Step S800: Traverse all the first timestamps of the three-dimensional point clouds, and form image-point cloud pairs with the two-dimensional image whose second timestamp is closest to the first timestamp and the three-dimensional point cloud.

[0062] For specific implementation methods of steps S700 to S800, please refer to steps S110 to S120 of the first embodiment of the object annotation method, which will not be repeated here.

[0063] Step S900: Extract the two-dimensional detection bounding box of the target object in the two-dimensional image.

[0064] For a detailed implementation of step S900, please refer to step S200 of the first embodiment of the object annotation method, which will not be repeated here.

[0065] Step S1000: Obtain the LiDAR pose corresponding to each 3D point cloud and extract the 3D detection box of each 3D point cloud.

[0066] Step S1100: Convert the 3D detection box to the global coordinate system according to the LiDAR pose of all corresponding points in the 3D point cloud.

[0067] Step S1200: Track the 3D detection box of the target object in the global coordinate system to obtain the motion trajectory of the target object.

[0068] The implementation methods of steps S1000 to S1200 can be found in steps S210 to S230, and will not be described in detail here.

[0069] Step S1300: Obtain the first and second timestamps for each image point cloud pair.

[0070] Step S1400: Based on the difference between the first and second timestamps and the motion trajectory of the target object, calculate the compensated 3D detection box of each image point cloud pair at the second timestamp.

[0071] The parameters for steps S1300 and S1400 are standardized.

[0072] Unlike the first embodiment of the object annotation method in this application, in this embodiment, before executing step S1500, the object annotation device further compensates the 3D detection box through steps S1300 and S1400 to make the 2D detection box and the 3D detection box more synchronized in time, thereby further improving the accuracy of the object annotation method. Specifically, after obtaining the motion trajectory of the target object through step S1200, the object annotation device further obtains the first timestamp of the 3D point cloud and the second timestamp of the 2D image in each image point cloud pair, and determines the degree of synchronization between the 3D point cloud and the 2D image based on the difference between the first timestamp and the second timestamp, and further calculates a compensated 3D detection box that is closer to the second timestamp based on the motion trajectory of the target object. The absolute value of the difference between the timestamp of the compensated 3D detection box and the second timestamp is less than the absolute value of the difference between the first timestamp and the second timestamp. Based on this, matching the 2D detection box with the compensated 3D detection box that is more synchronized in time with the 2D detection box can further improve the accuracy of the object annotation method.

[0073] Step S1500: Project the 3D detection box of each image point cloud pair onto the 2D image to obtain the projected detection box.

[0074] Step S1600: Match the two-dimensional detection box and the projection detection box of each image point cloud pair to obtain the image point cloud pairs that failed to match.

[0075] The implementation methods for steps S1500 to S1600 can be found in steps S300 to S400, and will not be described in detail here.

[0076] Step S1700: Generate the frame-complementing 3D detection box of the point cloud pair of the image to be framed using the 3D detection boxes in the neighboring point cloud pairs of the point cloud pair of the image to be framed, wherein the neighboring point cloud pairs are the successfully matched point cloud pairs.

[0077] For a detailed implementation of step S1700, please refer to step S500 of the first embodiment of the object annotation method, which will not be described in detail here.

[0078] Step S1800: Optimize the 3D detection frame for frame interpolation.

[0079] Unlike the first embodiment of the object annotation method, in this embodiment, before executing step S1900, the object annotation device further optimizes the frame-complemented 3D detection box through step S1800. Specifically, the frame-complemented 3D detection box is projected onto a 2D image to obtain the projection center. In other words, the corresponding frame-complemented 3D detection box is first projected onto a 2D image, where the center of the projected detection box in the 2D image is the projection center of the frame-complemented 3D detection box. The deviation between the projection center and the center of the 2D detection box in the 2D image is defined as the projection error between the frame-complemented 3D detection box and the corresponding 2D detection box. Further, based on the projection center of the frame-complemented 3D detection box, a search is performed in 3D space, traversing the projection errors between the searched 3D and 2D detection boxes. In other words, the object annotation device uses the projection center of the frame-complemented 3D detection box as the search center, searches for 3D detection boxes in 3D space, and obtains the projection errors between the searched 3D and 2D detection boxes. The 3D detection box with the smallest projection error is then used as the corrected frame-complemented 3D detection box, i.e., the optimized frame-complemented 3D detection box.

[0080] Furthermore, when the object annotation device searches in three-dimensional space based on the projection center of the frame-complemented 3D detection box, it also obtains the frame-complemented orientation angle (i.e., the orientation angle of the frame-complemented 3D detection box) of the frame-complemented 3D detection box, so as to add the frame-complemented orientation angle into the search space, so that the projection error between the obtained optimized frame-complemented 3D detection box and the 2D detection box is smaller, thereby further improving the accuracy of the object annotation method.

[0081] For example, the object annotation device uses the projection center of the frame-complemented 3D detection box as the search center, searches in 3D space with a side length of 20 meters and a step size of 0.1 meters, and searches in 3D space based on the frame-complemented orientation angle.

[0082] Step S1900: Use the optimized frame-complementing 3D detection box to annotate the target object in the 2D image of the point cloud pair of the image to be framed.

[0083] Unlike the first embodiment of the object annotation method, in this embodiment, the object annotation device uses an optimized frame-complementing 3D detection box to annotate the target object in the 2D image of the point cloud pair of the image to be framed. Based on this, the accuracy and completeness of the object annotation method can be effectively improved.

[0084] The second embodiment of the object annotation method has the advantages of the first embodiment. In this second embodiment, before executing step S1500, the object annotation device compensates the 3D detection box through steps S1300 and S1400. This makes the 2D and 3D detection boxes more synchronized, further improving the accuracy of the object annotation method. Furthermore, before executing step S1900, the object annotation device optimizes the interpolated 3D detection box through step S1800, thereby annotating the target object based on the optimized interpolated 3D detection box, effectively improving the accuracy and completeness of the object annotation method.

[0085] This application also proposes an object annotation apparatus, wherein the object annotation apparatus includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the object annotation method of any of the above embodiments.

[0086] This application relates to a computer-readable storage medium; please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium is used to store program instructions, which, when executed by a processor, are used to implement the object annotation method of any of the above embodiments.

[0087] The computer-readable storage medium 30 of this application embodiment stores program instructions 31, which are executed to implement the above-described object annotation method.

[0088] Specifically, program instructions 31 can be formed into a program file and stored in the aforementioned storage medium in the form of a software product, so that an electronic device (which may be a personal computer, server, or network device, etc.) or processor can execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0089] In this embodiment, the computer-readable storage medium 30 may be, but is not limited to, a USB flash drive, SD card, PD optical drive, portable hard drive, large-capacity floppy drive, flash memory, multimedia memory card, server, etc.

[0090] In one embodiment, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the steps described in the object annotation method embodiment.

[0091] Furthermore, if the aforementioned functions are implemented as software functions and sold or used as independent products, they can be stored in a mobile terminal-readable storage medium. That is, this application also provides a storage device storing program instructions that can be executed to implement the methods of the above embodiments. This storage device can be, for example, a USB flash drive, an optical disc, or a server. In other words, this application can be embodied in the form of a software product, which includes several instructions to cause a smart terminal to execute all or part of the steps of the methods described in the various embodiments.

[0092] In summary, the object annotation method of this application includes acquiring image point cloud pairs based on the first timestamp of a LiDAR scanner, wherein each image point cloud pair includes a two-dimensional image and a three-dimensional point cloud; extracting two-dimensional detection boxes of target objects from the two-dimensional image and extracting three-dimensional detection boxes of target objects from the three-dimensional point cloud; projecting the three-dimensional detection boxes of each image point cloud pair onto the two-dimensional image to obtain projected detection boxes; matching the two-dimensional detection boxes and projected detection boxes of each image point cloud pair to obtain image point cloud pairs that fail to match; generating supplementary three-dimensional detection boxes for the supplementary image point cloud pair using the three-dimensional detection boxes in adjacent image point cloud pairs, wherein adjacent image point cloud pairs are successfully matched image point cloud pairs; and annotating the target object in the two-dimensional image of the supplementary image point cloud pair using the supplementary three-dimensional detection boxes. Acquiring image point cloud pairs based on the first timestamp of a LiDAR scanner ensures that the timestamps of the two-dimensional image and the three-dimensional point cloud are as close as possible, thereby effectively improving the accuracy of target object annotation while also effectively reducing the computational load of the object annotation device. Furthermore, the object annotation method of this application determines the image point cloud pair to be framed by matching the 3D detection box and the 2D detection box in each image point cloud pair, and obtains the target object in the 2D image of the image point cloud pair to be framed by the successfully matched image point cloud pair, thereby completing the 3D annotation information of the target object at each time, thus effectively improving the completeness of the target object annotation and thus effectively improving the safety of autonomous driving.

[0093] It is worth noting that the accompanying drawings are only for illustrating the structural and connection relationships of the product of this invention, and do not limit the specific structural dimensions of the product of this invention.

[0094] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An object labeling method characterized by comprising: The object labeling method comprises: obtaining an image point cloud pair based on a first timestamp of a laser radar, wherein each image point cloud pair comprises a two-dimensional image and a three-dimensional point cloud; extracting a two-dimensional detection box of a target object in the two-dimensional image, and extracting a three-dimensional detection box of the target object in the three-dimensional point cloud; projecting the three-dimensional detection box of each image point cloud pair into the two-dimensional image to obtain a projected detection box; matching the two-dimensional detection box and the projected detection box of each image point cloud pair to obtain a to-be-supplemented frame image point cloud pair that fails to match; generating a supplemented three-dimensional detection box of the to-be-supplemented frame image point cloud pair by using the three-dimensional detection box in a neighboring image point cloud pair of the to-be-supplemented frame image point cloud pair, wherein the neighboring image point cloud pair is a matched image point cloud pair; labeling the target object in the two-dimensional image of the to-be-supplemented frame image point cloud pair by using the supplemented three-dimensional detection box. The method for obtaining an image point cloud pair based on a first timestamp of a laser radar comprises: collecting a plurality of frames of three-dimensional point clouds by using the laser radar; traversing the first timestamps of all the three-dimensional point clouds, and combining the two-dimensional image closest to the second timestamp with the three-dimensional point cloud to form the image point cloud pair.

2. The object labeling method according to claim 1, wherein the extraction of the three-dimensional detection box of the target object in the three-dimensional point cloud comprises: obtaining the laser radar pose corresponding to each three-dimensional point cloud, and extracting the three-dimensional detection box of each three-dimensional point cloud; converting the three-dimensional detection box to a global coordinate system according to the laser radar pose of the corresponding point of all the three-dimensional point clouds; tracking the three-dimensional detection box of the target object in the global coordinate system to obtain the motion trajectory of the target object.

3. The object labeling method according to claim 2, wherein before the projection of the three-dimensional detection box of each image point cloud pair into the two-dimensional image to obtain a projected detection box, the object labeling method further comprises: obtaining the first timestamp and the second timestamp of each image point cloud pair; calculating a compensated three-dimensional detection box of the three-dimensional detection box in the second timestamp of each image point cloud pair according to the difference between the first timestamp and the second timestamp and the motion trajectory of the target object.

4. The object labeling method according to claim 1, wherein the generation of the supplemented three-dimensional detection box of the to-be-supplemented frame image point cloud pair by using the three-dimensional detection box in the neighboring image point cloud pair of the to-be-supplemented frame image point cloud pair comprises: obtaining the kinematic property of the target object by using the three-dimensional detection box in the neighboring image point cloud pair; generating the supplemented three-dimensional detection box by using the kinematic property; wherein the kinematic property comprises position and velocity.

5. The object labeling method according to claim 4, wherein the generation of the supplemented three-dimensional detection box by using the kinematic property comprises: obtaining a neighboring orientation angle of the neighboring image point cloud pair; generating a supplemented orientation angle by using the neighboring orientation angle; generating the supplemented three-dimensional detection box by using the kinematic property and the supplemented orientation angle.

6. The object labeling method according to claim 1, wherein Before the object labeling method labels the target object in the two-dimensional image of the to-be-infilled frame image point cloud pair by using the infill three-dimensional detection box, the object labeling method further comprises: projecting the infill three-dimensional detection box into the two-dimensional image to obtain a projection center; searching in a three-dimensional space based on the projection center, and traversing a projection error of a searched three-dimensional detection box and the two-dimensional detection box; taking the three-dimensional detection box with the minimum projection error as a corrected infill three-dimensional detection box.

7. The object labeling method according to claim 6, characterized in that, the searching in the three-dimensional space based on the projection center comprises: obtaining an infill orientation angle of the infill three-dimensional detection box; searching in the three-dimensional space according to the infill orientation angle and the projection center.

8. An object labeling apparatus characterized by comprising: The object labeling apparatus comprises a memory and a processor coupled with the memory; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the object labeling method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program instructions, which when executed by a processor, are configured to implement the object labeling method according to any one of claims 1 to 7.

Images