3D Object Annotation Method Based on PnP and Device for Implementing 3D Object Annotation Method

Abstract

The present invention belongs to the technical field of computer vision detection, and in particular relates to a 3D object labeling method based on PnP and a device for implementing the 3D object labeling method. The method includes the following steps: S1. Setting a three-dimensional space frame that surrounds the object to be labeled; S2. Setting a world coordinate system; S3. Using a 3D depth perception camera, respectively extracting the RGB image and depth image of the three-dimensional space frame at the same time or position, and extracting the significant corner points at the corresponding positions of the stereo frames in the two images, as well as the world coordinates and pixel coordinates of the corner points; S4. Solving the 3D object pose by introducing a PnP algorithm; S5. Based on the acquired pose, correcting the point cloud data converted from the depth map of all 3D objects to be labeled to be parallel to the horizon. This application converts the 3D object pose solving problem into a camera relative pose solving problem, and converts irregular 3D objects into regular 3D objects according to the three-dimensional space frame for pose solving.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision detection, and in particular relates to a 3D object labeling method based on PnP and a device for implementing the 3D object labeling method. Background Art

[0002] 3D object detection is a 3D analysis method that combines geometric and semantic features of images. It primarily involves two tasks: localization and classification. By processing and analyzing sensor data, it is possible to directly and efficiently obtain the 3D bounding box of a specific target object, as well as information such as the target object's category, center point location, size, and orientation. Compared to 2D object detection, 3D object detection considers the depth information of the target object, making it more suitable for practical applications such as robot navigation and obstacle avoidance. As a data-driven technology, datasets with rich sample sizes will help improve the detection accuracy and generalization capabilities of 3D object detection networks.

[0003] At present, most existing 3D object labeling methods and labeling devices still need to be labeled by building a network, such as the patent application number CN202010822668.X "A three-dimensional point cloud target detection method and labeling device based on weak labeling", so the operation is complicated and not universal. In addition, most 3D object labeling methods and labeling devices are mainly aimed at point cloud data processing based on lidar such as autonomous driving, and the labeling of a few objects such as outdoor cars and pedestrians, such as the patent application number CN201810973255.4 "A method and device for labeling objects in 3D point cloud data" and the patent application number 202010757787.1 "A method and device for labeling objects in 3D point cloud data, storage medium", etc., while 3D object labeling methods and devices based on visual data processing for complex indoor environments are still limited. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a 3D object labeling method based on PnP and a device for implementing the 3D object labeling method. The specific technical solutions are as follows:

[0005] A 3D object annotation method based on PnP includes the following steps:

[0006] S1. According to the actual scale of the object to be labeled, a three-dimensional space frame is set to surround the object to be labeled, and the orientation of the three-dimensional space frame is the same as that of the object to be labeled;

[0007] S2. Set the world coordinate system. Use a corner point of the 3D space frame as the origin of the world coordinate system, and use the sides of the 3D space frame starting from the origin as the three coordinate axes to create the world coordinate system based on the right-hand coordinate system principle.

[0008] S3, using a 3D depth perception camera to extract the RGB image and depth image of the three-dimensional space frame at the same time or position, and extract the significant corner points at the corresponding positions of the stereo frame in the two images, as well as the world coordinates and pixel coordinates of the corner points;

[0009] S4. Solve the 3D object pose by introducing the PnP algorithm. The world coordinates and pixel coordinates of the 3D space frame are used as inputs to the PnP algorithm to solve the pose, obtain the camera pose at that moment, and then convert it into the pose of the 3D space frame in the camera coordinate system. The coordinates of its 3D center point in the camera coordinate system are obtained according to the actual scale of the object.

[0010] S5. Based on the acquired pose, the point cloud data of all 3D objects to be annotated converted from the depth map are corrected to be parallel to the horizon.

[0011] Specifically, the three-dimensional space frame is a stereoscopic frame having the same scale and orientation as the object to be annotated.

[0012] Specifically, in step S3, the corner points of the three-dimensional space frame are marked with special colors.

[0013] Specifically, in step S3, no less than three corner points of the three-dimensional space frame are selected.

[0014] Specifically, the 3D depth perception camera includes an RGB-D camera and a binocular stereo camera that can simultaneously capture RGB images and depth images, or a combination of multiple sensors.

[0015] Specifically, the pose of the three-dimensional space frame in the camera coordinate system in step S4 is R=[(α, β, γ), T]; wherein (α, β, γ) and T are the rotation and translation of the three-dimensional space frame of the object to be annotated in the camera coordinate system respectively; α, β, and γ are the pitch angle, yaw angle, and roll angle respectively.

[0016] The specific operation of step S5 is: let β = 0, and change the pose of the point cloud data to * R, that is, to obtain the corrected data.

[0017] A device for implementing the above-mentioned PnP-based 3D object labeling method includes:

[0018] Input port, the input data includes the internal parameters of the 3D depth perception camera, the actual scale information of the 3D space box, the selected corner points, and the semantic attributes of the target object; the actual scale information of the 3D space box is the scale information of the length, width, and height of the object to be labeled;

[0019] Output port: The output port outputs the pose of the object to be labeled in the camera coordinate system and the point cloud data of the object to be labeled converted from the depth map.

[0020] Specifically, it also includes a memory for storing data sets in the required format.

[0021] Specifically, the semantic attributes of the target object include category, color, operation attribute, dynamic and static attribute, and texture attribute.

[0022] The advantages of the present invention are:

[0023] (1) This application represents the scale and posture of an object in a three-dimensional space frame, converts irregular 3D objects into regular three-dimensional objects, and solves the 3D posture.

[0024] (2) By introducing the PnP algorithm, the 3D object pose problem is converted into a camera relative pose problem to obtain accurate 3D pose.

[0025] (3) This application protects a device for implementing the above method, which makes the labeling results simpler and more accurate by limiting each input. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flowchart of a 3D object annotation method based on PnP.

[0027] Figure 2 A case diagram of the PnP-based 3D object annotation method.

[0028] Figure 3 Build a case diagram for the three-dimensional frame of the object to be labeled.

[0029] Figure 4 Label the result map of the object to be labeled. DETAILED DESCRIPTION

[0030] like Figure 1-2 As shown, a 3D object annotation method based on PnP includes the following steps:

[0031] S1. According to the actual scale of the object to be labeled, set a three-dimensional space frame that surrounds the object to be labeled. In this solution, taking a stool as an example, for the convenience of calculation, the three-dimensional space frame is a cubic frame, and the orientation of the cubic frame is the same as the orientation of the object to be labeled; optionally, the three-dimensional frame is for objects with a regular cubic structure. Objects with other structures can be replaced by frames of other structures, such as other polyhedron frames.

[0032] S2. Set a world coordinate system. Use a corner point of the cube as the origin of the world coordinate system, and the sides of the cube starting from the origin as the three coordinate axes. Create the world coordinate system based on the right-hand coordinate system principle. Specifically, the corner points are points on the cube that have distinct features and are easy to extract, such as the eight corner points of the cube. These can also be marked with a special color to increase differentiation.

[0033] S3, using a 3D depth perception camera to extract the RGB image and depth image of the stereo frame at the same time or position, and extract the significant corner points of the corresponding positions of the stereo frame in the two images, as well as the world coordinates and pixel coordinates of the corner points; specifically, select no less than 3 from the eight corner points. In this solution, if Figure 3 In the example, corner point 3 is the origin, the Z axis is from corner point 3 to corner point 2, the X axis is from corner point 3 to corner point 4, and the Y axis is from corner point 3 to corner point 7. To better extract the corner points, the eight corner points can be marked with special colors, such as red, when designing the cube frame. The 3D depth perception camera can include an RGB-D camera that can simultaneously capture RGB and depth images, a binocular stereo camera, or a combination of multiple sensors.

[0034] S4. Solve the 3D object pose by introducing the PnP algorithm; use the world coordinates and pixel coordinates of the 3D space frame as the input of the PnP algorithm to solve the pose, obtain the camera pose at that moment, and then convert it into the pose R=[(α, β, γ), T] of the 3D space frame in the camera coordinate system (where (α, β, γ) and T are the rotation and translation of the 3D space frame of the object to be annotated in the camera coordinate system; α, β, and γ are the pitch angle, yaw angle, and roll angle, respectively.), and obtain the coordinates of its 3D center point in the camera coordinate system based on the actual scale of the object;

[0035] S5. Based on the acquired posture, the point cloud data of all 3D objects to be annotated converted from the depth map are corrected to be parallel to the horizon. The specific steps are to set β = 0, and to increase the posture of the point cloud data by R, so as to obtain the corrected data.

[0036] Optionally, the 3D depth perception camera includes an RGB-D camera and a binocular stereo camera that can simultaneously capture RGB images and depth maps, or is implemented by a combination of multiple sensors.

[0037] Alternatively, the PnP (Perspective-n-Points) solution, also known as perspective n-point projection, utilizes the relationship between 3D and 2D corresponding points to solve the camera pose problem. The number of corner points can be selected as needed, with n being greater than or equal to 3.

[0038] A device for implementing the above-mentioned 3D object labeling method includes:

[0039] Input port, the input data includes the internal parameters of the 3D depth perception camera, the actual scale information of the stereo frame, the selected corner points, and the semantic attributes of the target object. The actual scale information of the stereo frame includes the actual scale of the length, width, and height of the stereo frame; the semantic attributes of the target object include category, color, operation attributes, dynamic and static attributes, and texture attributes, such as Figure 4 Some semantic and geometric attributes of the displayed annotated objects. The attributes are not limited to the above.

[0040] Output port: The output port outputs the pose of the object to be labeled in the camera coordinate system and the point cloud data of the object to be labeled converted from the depth map.

[0041] Memory for storing data sets in required formats, including SUN-RGBD format, etc.

[0042] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A 3D object annotation method based on PnP, characterized in that: The following steps are involved: S1. According to the actual scale of the object to be labeled, a three-dimensional space frame is set to surround the object to be labeled, and the orientation of the three-dimensional space frame is the same as that of the object to be labeled; S2. Set the world coordinate system; Use a corner point of the 3D space box as the origin of the world coordinate system, and the sides of the 3D space box starting from the origin as the three coordinate axes to create the world coordinate system based on the right-hand coordinate system principle; S3, using a 3D depth perception camera to extract the RGB image and depth image of the three-dimensional space frame at the same time or position, and extract the significant corner points at the corresponding positions of the stereo frame in the two images, as well as the world coordinates and pixel coordinates of the corner points; S4, solve the 3D object pose by introducing the PnP algorithm; The world coordinates and pixel coordinates of the 3D space frame are used as inputs to the PnP algorithm to solve the pose, obtain the camera pose at that moment, and then convert it into the pose of the 3D space frame in the camera coordinate system. The coordinates of its 3D center point in the camera coordinate system are obtained based on the actual scale of the object. S5. Based on the acquired pose, the point cloud data of all 3D objects to be annotated converted from the depth map are corrected to be parallel to the horizon.

2. A 3D object labeling method based on PnP according to claim 1, characterized in that: The three-dimensional space frame is a stereoscopic frame having the same scale and orientation as the object to be annotated.

3. The 3D object annotation method based on PnP according to claim 1, characterized in that: In step S3, the corner points of the three-dimensional space frame are marked with special colors.

4. The 3D object labeling method based on PnP according to claim 1, characterized in that: In step S3, no less than three corner points of the three-dimensional space frame are selected.

5. The 3D object labeling method based on PnP according to claim 1, characterized in that: The 3D depth perception camera includes an RGB-D camera and a binocular stereo camera that can simultaneously capture RGB images and depth maps, or a combination of multiple sensors.

6. The 3D object labeling method based on PnP according to claim 1, characterized in that: In step S4, the pose of the three-dimensional space frame in the camera coordinate system is R=[(α, β, γ), T]; where (α, β, γ) and T are the rotation and translation of the three-dimensional space frame of the object to be annotated in the camera coordinate system; α, β, and γ are the pitch angle, yaw angle, and roll angle respectively; The specific operation of step S5 is: let β = 0, and change the pose of the point cloud data to * R, that is, to obtain the corrected data.

7. A device for implementing the PnP-based 3D object labeling method according to any one of claims 1 to 6, characterized in that: include: Input port: The input data includes the internal parameters of the 3D depth perception camera, the actual scale information of the 3D space box, the selected corner points, and the semantic attributes of the target object; The actual scale information of the three-dimensional space frame is the scale information of the length, width and height of the object to be marked; Output port: The output port outputs the pose of the object to be labeled in the camera coordinate system and the point cloud data of the object to be labeled converted from the depth map.

8. The device according to claim 7, characterized in that Also included is a memory for storing the data set in the required format.

9. The device according to claim 7, characterized in that The semantic attributes of the target object include category, color, operation attribute, dynamic and static attribute, and texture attribute.

Images