Point cloud and camera fusion labeling method, electronic equipment and storage medium

Abstract

The invention provides a point cloud and camera fusion labeling method, electronic equipment and a storage medium. The method comprises the following steps: acquiring point cloud data of a plurality of target objects and constructing a three-dimensional frame for labeling world coordinates corresponding to the point cloud data; each target object has a unique identifier; converting the world coordinates into camera coordinates according to external parameters of a target camera, and projecting the camera coordinates into picture coordinates according to internal parameters of the target camera; performing normalization processing on the picture coordinates to obtain target two-dimensional coordinates, and constructing a two-dimensional frame according to the target two-dimensional coordinates; and storing the three-dimensional frame and the two-dimensional frame of each target object in a preset file format according to the unique identifier of each target object. The point cloud data and the camera data are fused and labeled, so that the labeling efficiency and precision are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data labeling, and in particular to a point cloud and camera fusion labeling method, an electronic device and a storage medium. BACKGROUND

[0002] With the development of automatic driving, intelligent security and intelligent manufacturing industry, the perception system needs to collect data from multiple sensors at the same time, including lidar point cloud data and camera image data; In order to use the subsequent data processing and model training, the data needs to be accurately labeled.

[0003] In the prior art, during the labeling of different data, point cloud labeling and image labeling need to be performed in different labeling tools. Point cloud labeling usually relies on three-dimensional visualization tools, and labeling personnel need to draw a three-dimensional box of the target in a three-dimensional scene. Image labeling relies on two-dimensional image tools, and requires reboxing the same target, resulting in repetitive work. Due to the coordinate system difference and distortion problem between point cloud and image, the labeling results are difficult to quickly align, affecting the overall efficiency and data quality, and the labeling efficiency is low and the consistency between multi-source data is difficult to guarantee. Therefore, there is an urgent need for an efficient labeling scheme that can fuse point cloud and camera data, so that labeling personnel only need to perform one operation to generate corresponding labels in point cloud and image, thereby improving labeling efficiency and accuracy. SUMMARY

[0004] The present application provides a point cloud and camera fusion labeling method, an electronic device and a storage medium, which mainly aims to solve the problem of low labeling efficiency caused by the inability of point cloud and camera data to be fused in the prior art.

[0005] To achieve the above-mentioned purpose, the present application provides a point cloud and camera fusion labeling method, comprising:

[0006] Obtaining point cloud data of a plurality of target objects and constructing a three-dimensional box for labeling world coordinates corresponding to the point cloud data; each target object has a unique identifier;

[0007] According to the extrinsic parameters of the target camera, the world coordinates are transformed into camera coordinates, and according to the intrinsic parameters of the target camera, the camera coordinates are projected into picture coordinates;

[0008] The picture coordinates are normalized to obtain target two-dimensional coordinates, and a two-dimensional box is constructed according to the target two-dimensional coordinates;

[0009] According to the unique identifier of each target object, the three-dimensional box and the two-dimensional box of each target object are stored in a preset file format.

[0010] Optionally, the process of acquiring point cloud data of a plurality of target objects and constructing a three-dimensional box for labeling world coordinates corresponding to the point cloud data comprises:

[0011] The laser radar emits laser to the plurality of target objects, and calculates distance and angle according to the received echo time and packs it into a raw message;

[0012] According to the preset analysis tool, the raw message is read to obtain point cloud data, and a three-dimensional scene is constructed and rendered by a three-dimensional rendering engine tool to obtain the three-dimensional box.

[0013] Optionally, the expression of transforming the world coordinates into camera coordinates according to the extrinsic parameters of the target camera is:

[0014]

[0015] wherein X c , Y c and X w represent the world coordinates, X w , Y w and Z w represent the camera coordinates, r represents the rotation matrix in the extrinsic parameters, and t represents the translation vector in the extrinsic parameters.

[0016] Optionally, the expression of projecting the camera coordinates into picture coordinates according to the intrinsic parameters of the target camera is:

[0017]

[0018] wherein X p , Y p and Z P represent the picture coordinates, X c , Y c and Z c represent the camera coordinates, u and v represent the optical center point of the picture, f x represents the physical focal length of the target camera, and f y represents the pixel value of the target camera.

[0019] Optionally, the method further comprises: when the target camera is a fisheye lens, performing distortion correction on the camera coordinates according to the distortion parameters of the target camera;

[0020] The distortion correction on the camera coordinates comprises:

[0021] According to the camera coordinates, the polar angle θ before distortion is calculated, and the expression is:

[0022] θ = arctan(r);

[0023]

[0024] According to the fisheye distortion model, the polar angle θ after distortion is calculated d , and the expression is:

[0025] θ d = θ + k1θ 3 + k2θ 5 + k3θ 7 + k4θ 9 ;

[0026] Convert the polar angle back to the camera coordinates, and the expression is:

[0027]

[0028] Z c = 1;

[0029] Where X c , Y c and Z c represent the camera coordinates, and k1, k2, k3 and k4 represent the radial distortion coefficients.

[0030] Optionally, the method further comprises:

[0031] Obtaining and storing the extrinsic parameters, intrinsic parameters and distortion parameters of the target camera through a calibration tool; the calibration tool includes OpenCV and calibrateCamer.

[0032] Optionally, the method further comprises:

[0033] Real-time acquisition of the editing record of the three-dimensional frame changed by the three-dimensional rendering engine tool through an event listening tool, and synchronous change of the corresponding two-dimensional frame.

[0034] Optionally, the method further comprises:

[0035] The unique identifiers, the point cloud data, the three-dimensional frame and the two-dimensional frame of a plurality of target objects are stored as logs and saved to a cloud storage platform.

[0036] In order to solve the above problems, the application also provides an electronic device, which comprises:

[0037] At least one processor; and,

[0038] The memory is in communication connection with the at least one processor; wherein,

[0039] The memory has a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the point cloud and camera fusion labeling method described above.

[0040] To solve the above problems, the application further provides a computer readable storage medium, wherein at least one computer program is stored in the computer readable storage medium, and the at least one computer program is executed by a processor in an electronic device to implement the point cloud and camera fusion labeling method described above.

[0041] In the embodiment of the application, the point cloud data provides comprehensive three-dimensional information of a target object, the world coordinates are transformed into camera coordinates according to the external parameters of a target camera, the camera coordinates are projected into picture coordinates according to the internal parameters of the target camera, the dimensionality reduction from three dimensions to two dimensions is realized, and high-precision pictures are provided for analysis and processing based on two-dimensional images; the target two-dimensional coordinates are obtained by normalizing the picture coordinates, and a two-dimensional frame is constructed according to the target two-dimensional coordinates, so as to provide information for subsequent target detection and recognition; the two-dimensional frame and the three-dimensional frame of the same target object are associated by a unique identifier and stored in a preset format, so as to ensure the integrity, accuracy and traceability of the data; the point cloud data and the camera data are fused, so that the point cloud labeling and the image labeling can be performed simultaneously, and the labeling efficiency and accuracy are improved.

[0042] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following figures and detailed description thereof or can be learned by practice of the application. The objects and other advantages of the application can be realized and attained by the structure particularly pointed out in the description and drawings.

[0043] Those skilled in the art will understand that the objects and advantages of the application can be realized and attained by the structure particularly pointed out in the following detailed description and claims, and it will be understood that the application can be carried out by one or more of the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A flowchart of a point cloud and camera fusion labeling method according to an embodiment of the application.

[0045] Figure 2 A flowchart of a three-dimensional frame construction process in a point cloud and camera fusion labeling method according to an embodiment of the application.

[0046] The implementation of the object of the application, the functional features and the advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0047] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application with reference to the embodiments and drawings. Herein, the illustrative embodiments of the present application and the description thereof are used to explain the present application but not to limit the present application.

[0048] It should be noted that, in order to avoid the present application being obscured by unnecessary details, only the structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0049] It should be emphasized that the term "comprising / comprising" as used herein refers to the presence of a feature, element, step or component, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0050] It should be noted that, if not otherwise specified, the term "connection" as used herein can not only refer to direct connection, but also indirect connection with intermediate objects.

[0051] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts or the same or similar steps.

[0052] Referring to Figure 1 As shown in the drawings, the embodiments of the present application provide a point cloud and camera fusion labeling method, which comprises the following steps S101-S104:

[0053] Step S101: acquiring point cloud data of a plurality of target objects and constructing a three-dimensional frame for labeling world coordinates corresponding to the point cloud data; each target object has a unique identifier.

[0054] Step S102: transforming the world coordinates into camera coordinates according to the external parameters of the target camera, and projecting the camera coordinates into picture coordinates according to the internal parameters of the target camera.

[0055] Step S103: normalizing the picture coordinates to obtain target two-dimensional coordinates, and constructing a two-dimensional frame according to the target two-dimensional coordinates.

[0056] Step S104: storing the three-dimensional frame and the two-dimensional frame of each target object into a preset file format according to the unique identifier of each target object.

[0057] In step S101, the target object is an object that needs to be labeled, and the point cloud labeling is a process of labeling and classifying the scene and objects represented by the point cloud data; the image labeling is to draw a bounding box and key points on a two-dimensional image to identify the label and attribute information of the objects in the image. Figure 2A schematic diagram of a three-dimensional frame construction process in a point cloud and camera fusion labeling method according to an embodiment of the present application. In the embodiment of the present application, the process of obtaining point cloud data of a plurality of target objects and constructing a three-dimensional frame for labeling the world coordinates corresponding to the point cloud data includes steps S1011-S1012:

[0058] Step S1011: Emit laser to a plurality of target objects through a laser radar, and calculate distance and angle according to received echo time and pack as original message.

[0059] Step S1012: Obtain point cloud data according to a preset analysis tool reading the original message, and obtain a three-dimensional frame through a three-dimensional rendering engine tool for three-dimensional scene construction and rendering.

[0060] Specifically, the three-dimensional rendering tool adopts Three.js, the analysis tool adopts a PCDLoader extension library, the analysis tool loads the read point cloud data into the three-dimensional rendering tool and obtains a three-dimensional frame. Further, a target object has a unique identifier, and in the conversion process of world coordinates, camera coordinates and picture coordinates, the same identifier is used for the same target in the three-dimensional frame, the two-dimensional frame and all subsequent storage to realize accurate correspondence.

[0061] In step S102, the extrinsic parameters include focal length and stationary point coordinates, the extrinsic parameters include a rotation matrix and a translation vector, and the three-dimensional frame in the world coordinate system is projected into a two-dimensional coordinate system to obtain a two-dimensional frame. In the embodiment of the present application, the expression for transforming the world coordinates into camera coordinates according to the extrinsic parameters of the target camera is:

[0062]

[0063] wherein X c , Y c and X w represent world coordinates, X w , Y w and Z w represent camera coordinates, r represents a rotation matrix in the extrinsic parameters, and t represents a translation vector in the extrinsic parameters. Specifically, r represents a 3*3 camera extrinsic rotation matrix, and t represents a 1*3 camera extrinsic translation vector.

[0064] In the embodiment of the present application, the expression for projecting the camera coordinates into picture coordinates according to the intrinsic parameters of the target camera is:

[0065]

[0066] wherein X p , Y p and Z P represent picture coordinates, X c , Y c and Zc represents the camera coordinates, u and v represent the picture optical center point, f x represents the physical focal length of the target camera, f y represents the pixel value of the target camera.

[0067] Further, in the embodiment of the present application, the method further comprises: acquiring and storing the extrinsic parameters, intrinsic parameters and distortion parameters of the target camera by a calibration tool; the calibration tool comprises OpenCV and calibrateCamera.

[0068] In the embodiment of the present application, the method further comprises: when the target camera is a fisheye lens, performing distortion correction on the camera coordinates according to the distortion parameters of the target camera.

[0069] The distortion correction on the camera coordinates comprises:

[0070] According to the camera coordinates, the polar angle θ before distortion is calculated, and the expression is:

[0071] θ = arctan (r) ;

[0072]

[0073] According to the fisheye distortion model, the polar angle θ after distortion is calculated d , and the expression is:

[0074] θ d = θ + k1θ 3 + k2θ 5 + k3θ 7 + k4θ 9 ;

[0075] The polar angle is converted back to the camera coordinates, and the expression is:

[0076]

[0077] Z c = 1;

[0078] Wherein, X c , Y c and Z c represent the camera coordinates, and k1, k2, k3 and k4 represent the radial distortion coefficients.

[0079] In step S103 and step S104, the picture coordinates (X p , Y p , Z P ) are normalized to obtain the expression of the target two-dimensional coordinates: The target two-dimensional coordinates are drawn and adjusted based on Fabric.js. In the embodiment of the present application, the method further comprises: acquiring, in real time, an editing record of a change of the three-dimensional box by a three-dimensional rendering engine tool through an event listening tool, and synchronously changing the corresponding two-dimensional box. When the three-dimensional box is edited in Three.js, the two-dimensional box in Fabric.js will be synchronously modified.

[0080] Further, before being stored as a preset file format, the projection error of the three-dimensional box and the range of the two-dimensional box are checked to avoid invalid labeling beyond the boundary; the preset file format includes but is not limited to KITTI, nuScenes and COCO.

[0081] In the embodiment of the present application, the method further comprises:

[0082] The unique identifiers, the point cloud data, the three-dimensional boxes and the two-dimensional boxes of the plurality of target objects are stored as logs and saved to a cloud storage platform.

[0083] On the other hand, the embodiment of the present application also provides an electronic device, which comprises at least one processor; and a memory in communication connection with the at least one processor; wherein the memory has a computer program which can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the point cloud and camera fusion labeling method described above.

[0084] On the other hand, the embodiment of the present application also provides a computer readable storage medium, which stores at least one computer program, and the at least one computer program is executed by a processor in an electronic device to implement the point cloud and camera fusion labeling method described above.

[0085] The present application will be described below in conjunction with a specific embodiment:

[0086] The present application proposes a point cloud and camera fusion labeling scheme for a data labeling platform, which comprises the following steps:

[0087] 1. Point cloud loading and three-dimensional stage construction. A point cloud visualization stage is constructed based on Three.js, and point cloud data collected by a laser radar is loaded into a three-dimensional scene. An interactive drawing function is provided to enable a labeling personnel to draw and adjust a three-dimensional box in the three-dimensional stage, for labeling the position and size of a target object.

[0088] 2. Camera parameter modeling. The intrinsic parameters (focal length, principal point coordinates), extrinsic parameters (rotation matrix, translation vector) and distortion parameters of a corresponding camera are read. The three-dimensional box in the point cloud coordinate system is converted into the camera coordinate system.

[0089] 3. In the process of projecting the three-dimensional box to the two-dimensional image, first convert the world coordinates to camera coordinates. If it is a fisheye lens, correct the distortion, then convert the camera coordinates to the picture coordinates and perform normalization processing.

[0090] 4. Two-dimensional label generation and rendering. Use Fabric.js to draw the projected label box on the two-dimensional image stage. Support adjusting and editing the two-dimensional label box to meet the labeling needs in special scenarios.

[0091] 5. Multi-source data labeling result saving. Save the three-dimensional point cloud labeling and two-dimensional image labeling results uniformly, establish a one-to-one correspondence between the point cloud and image data, and lay the foundation for other 2D and 3D linkage functions; the labeling results can be exported to a standardized file format for subsequent data processing and model training.

[0092] In summary, the application provides a point cloud and camera fusion labeling method, an electronic device and a storage medium, the method comprising: obtaining point cloud data of a plurality of target objects and constructing a three-dimensional box for labeling world coordinates corresponding to the point cloud data; each target object has a unique identifier; transforming the world coordinates into camera coordinates according to the external parameters of a target camera, and projecting the camera coordinates into picture coordinates according to the internal parameters of the target camera; performing normalization processing on the picture coordinates to obtain target two-dimensional coordinates, and constructing a two-dimensional box according to the target two-dimensional coordinates; storing the three-dimensional box and the two-dimensional box of each target object in a preset file format according to the unique identifier of each target object.

[0093] The application embodiment further provides an electronic device, which comprises:

[0094] at least one processor; and

[0095] a memory in communication connection with the at least one processor; wherein

[0096] the memory has a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute to realize the point cloud and camera fusion labeling method described above.

[0097] The application embodiment further provides a computer readable storage medium, which stores at least one computer program, and the at least one computer program is executed by a processor in an electronic device to realize the point cloud and camera fusion labeling method described above.

[0098] Any combination of the technical features in the above-described embodiments can be made, and for the sake of brevity, not all possible combinations are described, however, as long as there is no conflict, any combination of the technical features should be considered within the scope of the present disclosure.

[0099] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these are within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A point cloud and camera fusion labeling method, characterized in that, The method comprises: acquiring point cloud data of a plurality of target objects and constructing a three-dimensional box for labeling world coordinates corresponding to the point cloud data; each target object has a unique identifier; transforming the world coordinates into camera coordinates according to extrinsic parameters of a target camera, and projecting the camera coordinates into picture coordinates according to intrinsic parameters of the target camera; normalizing the picture coordinates to obtain target two-dimensional coordinates, and constructing a two-dimensional box according to the target two-dimensional coordinates; storing the three-dimensional box and the two-dimensional box of each target object into a preset file format according to the unique identifier of each target object.

2. The point cloud and camera fusion annotation method of claim 1, wherein, The process of acquiring point cloud data of a plurality of target objects and constructing a three-dimensional box for labeling world coordinates corresponding to the point cloud data comprises: emitting laser to a plurality of target objects through a laser radar, and calculating distance and angle according to received echo time and packing into original message; reading the original message to obtain point cloud data according to a preset analysis tool, and constructing and rendering a three-dimensional scene through a three-dimensional rendering engine tool to obtain the three-dimensional box.

3. The point cloud and camera fusion annotation method of claim 1, wherein, The expression of transforming the world coordinates into camera coordinates according to extrinsic parameters of a target camera is: wherein X c , Y c and X w represent the world coordinates, X w , Y w and Z w represent the camera coordinates, r represents a rotation matrix in the extrinsic parameters, and t represents a translation vector in the extrinsic parameters.

4. The point cloud and camera fusion annotation method of claim 1, wherein, The expression of projecting the camera coordinates into picture coordinates according to intrinsic parameters of the target camera is: where X p , Y p , and Z P represent the picture coordinates, X c , Y c , and Z c represent the camera coordinates, v and v represent the picture optical center points, f x represents the physical focal length of the target camera, f y represents the pixel value of the target camera.

5. The point cloud and camera fusion annotation method of claim 4, wherein, The method further comprises: when the target camera is a fisheye lens, performing distortion correction on the camera coordinates according to distortion parameters of the target camera; The distortion correction on the camera coordinates comprises: calculating the polar angle θ before distortion according to the camera coordinates, the expression is: θ = arctan(r); According to the fisheye distortion model, the polar angle θ after distortion is calculated d , and the expression is: θ d = θ + k1θ 3 + k2θ 5 + k3θ 7 + k4θ 9 ; converting the polar angle back to the camera coordinates, the expression is: Z c ＝1； where X c , Y c , and Z c represent the camera coordinates, and k1, k2, k3, and k4 represent the radial distortion coefficients.

6. The data labeling platform oriented point cloud and camera fusion labeling method according to claim 5, characterized in that, The method further comprises: acquiring and storing the extrinsic parameters, intrinsic parameters and distortion parameters of the target camera through a calibration tool; the calibration tool comprises OpenCV and calibrateCamer.

7. The point cloud and camera fusion annotation method of claim 6, wherein, The method further comprises: real-time acquiring editing records of the three-dimensional rendering engine tool for changing the three-dimensional box through an event listening tool, and synchronously changing the corresponding two-dimensional box.

8. The point cloud and camera fusion annotation method of claim 1, wherein, The method further comprises: storing the unique identifier, the point cloud data, the three-dimensional box and the two-dimensional box of a plurality of target objects into a log and saving to a cloud storage platform.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the point cloud and camera fusion labeling method according to any one of claims 1 to 8.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the point cloud and camera fusion labeling method according to any one of claims 1 to 8.

Images