3D image marking method and 3D image marking device based on 2D image marking information

Abstract

The present invention discloses a three-dimensional (3D) image marking device and marking method. The 3D image marking device includes a point cloud clustering module, a projection module, an integration module and a point cloud restoration module. The point cloud clustering module is used to cluster multiple points of a 3D unlabeled image according to a 3D unlabeled image and a clustering algorithm to generate a 3D clustered image having one or more first point groups. The projection module is used to generate a first two-dimensional (2D) image having one or more first objects according to the 3D clustered image, wherein each first point group corresponds to one of the first objects. The integration module is used to mark the first object of the first 2D image according to one or more object frames of one or more 2D labeled images to generate a second 2D image. The point cloud restoration module is used to generate a 3D labeled image according to the second 2D image.

Description

Technical Field

[0001] The invention relates to a 3D image marking method and a 3D image marking device based on 2D image marking information. Background Art

[0002] With the booming development of the autonomous vehicle industry, 3D image labeling technology has received attention. In the prior art, the labeling of three-dimensional (3D) images generally remains at the stage of manual labeling, which is difficult to semi-automate. However, compared with two-dimensional (2D) images, the cost of manual labeling of 3D images is much higher than that of 2D images. Therefore, how to label 3D images at a low cost is one of the goals of the industry. Summary of the invention

[0003] One aspect of the present invention discloses a three-dimensional (3D) image marking device. The 3D image marking device includes a point cloud clustering module, a projection module, an integration module and a point cloud restoration module. The point cloud clustering module is used to cluster multiple points of a 3D unlabeled image according to a 3D unlabeled image and a clustering algorithm to generate a 3D clustered image having one or more first point groups. The projection module is used to generate a first two-dimensional (2D) image having one or more first objects according to the 3D clustered image, wherein each first point group corresponds to one of the first objects. The integration module is used to mark the first object of the first 2D image according to one or more object boxes of one or more 2D labeled images to generate a second 2D image. The point cloud restoration module is used to generate a 3D labeled image according to the second 2D image.

[0004] Another aspect of the present invention discloses a three-dimensional (3D) image labeling method. When the 3D image labeling method is executed by a processor, the processor is caused to: execute a point cloud grouping module to group multiple points of the 3D unlabeled image according to a 3D unlabeled image and a grouping algorithm to generate a 3D grouped image having one or more first point groups; execute a projection module to generate a first two-dimensional (2D) image having one or more first objects according to the 3D grouped image, wherein each first point group corresponds to one of the first objects; execute an integration module to label the first object of the first 2D image according to one or more object boxes of one or more 2D labeled images to generate a second 2D image; and execute a point cloud restoration module to generate a 3D labeled image according to the second 2D image.

[0005] In order to better understand the above and other aspects of the present invention, embodiments are given below and described in detail with reference to the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 FIG. 4 is a block diagram of a 3D image marking apparatus based on 2D image marking information according to an embodiment of the present invention.

[0007] Figure 2 A flow chart of a 3D image marking method based on 2D image marking information according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0008] For the sake of brevity, in the description of this article, "three-dimensional" will be referred to as "3D" and "two-dimensional" will be referred to as "2D".

[0009] In the prior art, the 2D image converted from a 3D image by projection cannot be directly automatically or semi-automatically marked using existing marking software or artificial intelligence. On the other hand, the front-to-back relationship between two objects in a 3D image will be lost due to the conversion to a 2D image (for example, the front object covers the rear object). If the object frame of a 2D image marked with the original 2D image is directly applied to mark the 2D image converted from the 3D image, and then converted back to a 3D image, the point clouds corresponding to the front object and the rear object cannot be distinguished and the marking is chaotic. The 3D image marking device and marking method proposed in the present invention can solve the above problems.

[0010] Please refer to Figure 1 , Figure 1 FIG. 1 is a block diagram of a 3D image labeling apparatus based on 2D image labeling information according to an embodiment of the present invention. The 3D image labeling apparatus 10 comprises a point cloud clustering module 102 , a projection module 104 , an integration module 106 and a point cloud restoration module 108 .

[0011] In one embodiment, the point cloud clustering module 102, the projection module 104, the integration module 106, and the point cloud restoration module 108 may be implemented by a combination of computer-readable instructions and stored in a non-transitory memory (not shown) of the 3D image labeling device 10. When the combination of computer instructions for implementing the point cloud clustering module 102, the projection module 104, the integration module 106, and the point cloud restoration module 108 is executed by a processing unit (not shown), the processing unit may be caused to perform operations for implementing the functions of the point cloud clustering module 102, the projection module 104, the integration module 106, and the point cloud restoration module 108.

[0012] The point cloud clustering module 102 is used for receiving the 3D unlabeled image 3D-IMG-unlab, and clustering a plurality of points of the 3D unlabeled image 3D-IMG-unlab into one or more first point groups according to a clustering algorithm to generate a 3D clustered image 3D-IMG-clus.

[0013] The projection module 104 is configured to project a first point group in the 3D cluster image 3D-IMG-clus onto a two-dimensional plane to generate a first 2D image 2D-IMG1 .

[0014] The integration module 106 is configured to receive a first 2D image 2D-IMG1 and one or more 2D labeled images 2D-IMG-lab, and generate a second 2D image 2D-IMG2 according to the first 2D image 2D-IMG1 and the 2D labeled images 2D-IMG-lab.

[0015] The point cloud restoration module 108 is configured to generate a 3D labeled image 3D-IMG-lab according to the second 2D image 2D-IMG2.

[0016] In order to understand the present invention more clearly, please refer to the following Figure 2 FIG. 1 is a flow chart of a 3D image labeling method according to an embodiment of the present invention. The 3D image labeling method can be applied to the 3D image labeling apparatus 10 and executed by a processing unit.

[0017] In S201, a point cloud clustering module is executed to receive a 3D unlabeled image, and a plurality of points of the 3D unlabeled image 3D-IMG-unlab are clustered into one or more first point groups according to a clustering algorithm to generate a 3D clustered image 3D-IMG-clus. In one embodiment, the 3D unlabeled image 3D-IMG-unlab may come from a 3D original image database. In another embodiment, the 3D unlabeled image 3D-IMG-unlab may come from one or more 3D cameras, such as LiDAR or radar. When the 3D unlabeled image 3D-IMG-unlab comes from multiple 3D cameras, the multiple 3D original images captured by these 3D cameras are aligned in time and space to generate a 3D unlabeled image 3D-IMG-unlab corresponding to a specific time and a specific space, and the 3D unlabeled image 3D-IMG-unlab is provided to the point cloud clustering module 102. In one embodiment, each point may include spatial information (e.g., three-dimensional coordinates). In another embodiment, each point may include spatial information and color information (e.g., RGB values). In one embodiment, the clustering algorithm may adopt an algorithm commonly used in the art, such as a clustering algorithm based on the distance between points. In another embodiment, the clustering algorithm is a clustering algorithm based on the density of points. By using a clustering algorithm based on the density of points, points belonging to the front object (the object closer to the camera) and points belonging to the rear object (the object farther from the camera) among a plurality of points clustered together can be effectively separated and assigned to different first point groups. Each first point group may include one or more points. In one embodiment, each first point group may represent an object. In another embodiment, each first point group may represent one of a plurality of fragments (pieces) of an actual object, for example, three first point groups may respectively represent the head, upper body and lower body (three fragments) of a person (actual object), and when the three first point groups are combined together, they may represent an actual object of a "person".

[0018] In S203, the projection module is executed to project the first point group in the 3D grouped image 3D-IMG-clus onto a two-dimensional plane to generate a first 2D image 2D-IMG1. In one embodiment, the projection module 104 may use a near-point spherical projection method to generate the first 2D image 2D-IMG1. In another embodiment, the projection module 104 may use a projection method commonly used in the art to generate the first 2D image 2D-IMG1. The first 2D image 2D-IMG1 includes one or more first objects formed by the projection of the first point group in the 3D grouped image 3D-IMG-clus. In other words, the first 2D image 2D-IMG1 generated by the projection module is a 2D image with 3D information.

[0019] In S205, the integration module is executed to generate a second 2D image 2D-IMG2 according to the first 2D image 2D-IMG1 and one or more 2D labeled images 2D-IMG-lab. The 2D labeled image 2D-IMG-lab and the 3D unlabeled image 3D-IMG-unlab correspond to the same specific time and specific space. In one embodiment, the 2D labeled image 2D-IMG-lab comes from a 2D labeled image database. In another embodiment, the 2D labeled image 2D-IMG-lab comes from one or more 2D cameras. When the 2D labeled image 2D-IMG-lab comes from multiple 2D cameras, the 2D images taken by the 2D camera at different angles (e.g., 30 degrees, 60 degrees, and 120 degrees) are corrected and labeled to generate the 2D labeled image 2D-IMG-lab, and provided to the integration module 106. Each 2D labeled image 2D-IMG-lab may include one or more object frames. Each object frame has an object category, such as "person", "car", or "motorcycle". The integration module 106 marks the first 2D image 2D-IMG1 according to the object frame of the 2D labeled image 2D-IMG-lab to generate the second 2D image 2D-IMG2. In one embodiment, for each first object in the first 2D image 2D-IMG1, the integration module 106 determines whether the first object falls within the range corresponding to one of the object frames of the labeled image 2D-IMG-lab. If so, the first object is marked with the object category of the corresponding object frame; if not, it is not marked. By the above method, an object category can be determined for each first object in the object frame of the first 2D image 2D-IMG1 corresponding to the labeled image 2D-IMG-lab. That is, in the second 2D image 2D-IMG2, each first object corresponding to the object frame of the labeled image 2D-IMG-lab will have the same object category as the object frame of the corresponding labeled image 2D-IMG-lab, while the first object that does not correspond to the object frame of the labeled image 2D-IMG-lab remains in a state of not having an object category.

[0020] In S207, the point cloud restoration module is executed to generate a 3D labeled image 3D-IMG-lab according to the second 2D image 2D-IMG2. In one embodiment, for each first object in the second 2D image 2D-IMG2, the point cloud restoration module 108 may mark all points included in the first point group of the 3D clustered image 3D-IMG-clus corresponding to the first object according to the object category of the first object (if any) to generate the 3D labeled image 3D-IMG-lab.

[0021] In one embodiment, a manual auxiliary marking step may be further included between steps S205 and S207. The manual auxiliary marking step may mark first objects in the first 2D image 2D-IMG1 that do not correspond to object frames of the marked image 2D-IMG-lab, thereby reducing the number of unmarked first objects in the second 2D image 2D-IMG.

[0022] The 3D image marking device and 3D image marking method proposed in the present invention can group the points in the 3D unlabeled image by a point cloud grouping module before converting the 3D unlabeled image into a first 2D image through projection to generate a 3D grouped image. Each first point group in the 3D grouped image corresponds to a first object in the first 2D image, and the positional relationship between the first point groups is retained by the correspondence between the first point group and the first object. Then, the first object of the first 2D image is marked with the object frame of the 2D marked image to generate a second 2D image. The first object in this second 2D image has an object category because it has been marked. Then, the object category of the first object in the second 2D image is applied to all points in the corresponding first point group in the 3D grouped image. In this way, the generated marked 3D image will not be affected by the loss of the front and back relationship of the 3D points after projection.

[0023] In one practical application, the 3D image marking device and the 3D image marking method proposed in the present invention can be used to establish a 3D marked image database based on a 2D marked image database and a 3D unmarked image database. In another practical application, the 3D image marking device and the 3D image marking method proposed in the present invention can be configured on an autonomous driving vehicle, and can mark the 3D image provided by the 3D camera in real time based on the 2D image provided by the 2D camera that has been marked in real time.

[0024] In summary, although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the appended claims.

[0025]

Explanation of symbols

[0026] 10: 3D image marking device.

[0027] 102: Point cloud clustering module.

[0028] 104: Projection module.

[0029] 106: Integration module.

[0030] 108: Point cloud restoration module.

[0031] 3D-IMG-unlab: 3D unlabeled images.

[0032] 3D-IMG-clus: 3D clustered image. 3D-IMG-lab: 3D labeled image. 2D-IMG-lab: 2D labeled image. 2D-IMG1: first 2D image.

[0033] 2D-IMG2: second 2D image. S201 to S207: steps.

Claims

1. A 3D image marking device, comprising: A point cloud clustering module, configured to cluster a plurality of points of the 3D unlabeled image according to the 3D unlabeled image and the clustering algorithm, so as to generate a 3D clustered image having one or more first point groups; a projection module, configured to generate a first 2D image having one or more first objects according to the 3D grouped image, wherein each first point group corresponds to one of the one or more first objects; an integration module, configured to mark the one or more first objects of the first 2D image according to one or more object frames of one or more 2D marked images to generate a second 2D image; as well as The point cloud restoration module is used to generate a 3D marked image according to the second 2D image. 2 . The 3D image marking device as claimed in claim 1 , wherein the clustering algorithm is a clustering algorithm based on point density of the 3D image.

3. The 3D image marking device as claimed in claim 1, wherein for each first object in the first 2D image, the integration module determines whether the first object falls within a range corresponding to one of the one or more object frames of the one or more marked images, and if so, marks the first object with the object category of the corresponding object frame, and if not, does not mark the first object, thereby generating the second 2D image.

4. The 3D image marking device as described in claim 3, wherein for each first object having the object category in the second 2D image, the point cloud restoration module marks all points contained in the first point group corresponding to the first object in the 3D clustered image according to the object category of the first object. The 3D image marking apparatus of claim 1 , wherein each first point group corresponds to a fragment of an actual object.

6. A 3D image marking method, when executed by a processor, causes the processor to: executing a point cloud clustering module to cluster a plurality of points of the 3D unlabeled image according to the 3D unlabeled image and the clustering algorithm to generate a 3D clustered image having one or more first point groups; executing a projection module to generate a first 2D image having one or more first objects according to the 3D grouped image, wherein each first point group corresponds to one of the one or more first objects; executing an integration module to mark the one or more first objects of the first 2D image according to the one or more object boxes of the one or more 2D marked images to generate a second 2D image; as well as A point cloud restoration module is executed to generate a 3D labeled image according to the second 2D image. 7 . The 3D image labeling method as claimed in claim 6 , wherein the clustering algorithm is a clustering algorithm based on point density of the 3D image.

8. The 3D image marking method as claimed in claim 6, wherein for each first object in the first 2D image, the integration module determines whether the first object falls within a range corresponding to one of the one or more object frames of the one or more marked images, and if so, marks the first object with the object category of the corresponding object frame, and if not, does not mark the first object, thereby generating the second 2D image.

9. The 3D image labeling method as described in claim 8, wherein for each first object having the object category in the second 2D image, the point cloud restoration module labels all points contained in the first point group corresponding to the first object in the 3D clustered image according to the object category of the first object.

10. The 3D image labeling method of claim 6, wherein each first point group corresponds to a fragment of a real object.

Images