Three-dimensional target labeling method and device based on point cloud data assisted two-dimensional image

Abstract

The invention provides a three-dimensional target marking method and device based on a point cloud data-assisted two-dimensional image, and the method comprises the steps: fusing precise three-dimensional point cloud information obtained by a laser radar and two-dimensional image data collected by a camera based on a point cloud data and image data cooperation three-dimensional marking method; and a dynamic high-precision three-dimensional target marking system is constructed. Specifically, a space-image relationship is established through three-dimensional point cloud information and a two-dimensional image plane, and three-dimensional information of a target is mapped into a two-dimensional image, so that geometric and semantic information labeling is carried out on the target in the two-dimensional image, information loss of two-dimensional visual data in dimensions such as depth, attitude and space structure is made up, and the accuracy of the three-dimensional information labeling is improved. And meanwhile, when the three-dimensional point cloud is unmeasurable, a three-dimensional target labeling mode is dynamically adjusted in combination with two-dimensional image labeling information. According to the method, through dynamic high-precision three-dimensional target labeling, the obtained image-label can be used for training and reasoning of a three-dimensional target detection network, and the method has a wide application prospect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and data processing, and particularly relates to a three-dimensional target labeling method and device based on point cloud data assisted two-dimensional image. BACKGROUND

[0002] Three-dimensional target labeling of two-dimensional images is a key technology in the field of computer vision, and has important theoretical value and application prospect in many disciplines such as autonomous driving, robot navigation, augmented reality and remote sensing surveying. Traditional two-dimensional labeling methods can only provide planar information and cannot meet the demand of modern intelligent systems for three-dimensional spatial perception.

[0003] In the field of visual perception, the fusion of laser radar and visual sensor has become the mainstream scheme of three-dimensional target detection. However, there are still significant limitations in the existing technology, for example, some technologies have fused the two technologies, but the sparsity of laser radar point cloud is significant in long-distance or small target scene, and it is difficult to accurately regress the three-dimensional bounding box only relying on point cloud clustering, resulting in significant positioning error; while monocular camera has rich texture information, but lacks depth perception ability, and cannot complete spatial positioning independently. In addition, the existing joint labeling method is difficult to realize unified labeling of long-distance targets due to the difference in sensor detection distance, although it divides the target set and processes them respectively to improve the long-distance labeling capability, but it does not solve the data compensation problem when the sensor fails. Some existing technologies use the technical route of fusing point cloud features of multi-view and image segmentation, but rely on fixed projection matrix for explicit alignment, which is not robust enough to handle three-dimensional point cloud data in dynamic scenes; and the existing methods mostly focus on one-way projection from point cloud to image, without establishing a bidirectional correction mechanism.

[0004] There is an urgent need to provide a robust fusion method that can dynamically fuse geometric projection and semantic information, and utilize visual data to realize three-dimensional reconstruction when point cloud is missing. SUMMARY

[0005] The present application proposes a three-dimensional target labeling method and device based on point cloud data assisted two-dimensional image to solve the above technical problems. The method fuses the accurate three-dimensional point cloud information obtained by laser radar (LiDAR) and the two-dimensional image data collected by the camera to construct a high-precision three-dimensional target labeling system. The image-label pair obtained by the method can be used for training and reasoning of monocular three-dimensional target detection network.

[0006] A three-dimensional target labeling method based on point cloud data assisted two-dimensional image, comprising: Step 110, jointly calibrate the laser radar and the optical imaging sensor, obtain the three-dimensional point cloud data and the two-dimensional image of the target and the background, obtain the extrinsic matrix describing the spatial coordinate transformation relationship between the laser radar and the optical imaging sensor, and record the intrinsic matrix of the optical imaging sensor; Step 120, label the point cloud data of the target to obtain the center point coordinates of the target, the target orientation, and the target size; Step 130, convert the world coordinate system in which the point cloud data of the target is located to the image coordinate system; Step 140, label the information of the target in the two-dimensional image, including the category information of the target and the two-dimensional label frame information; Step 150, dynamically switch the labeling mode according to whether the point cloud data is complete, and complete the three-dimensional target labeling; if the point cloud data is complete, directly obtain all the labeling information of the three-dimensional target according to the point cloud data of the target and the information in the two-dimensional image; if the point cloud data is not complete, calculate the target depth using the information of the two-dimensional image, and perform pose solving, and complete the three-dimensional target labeling by reacquiring the target orientation in the world coordinate system.

[0007] On the other hand, the application provides a three-dimensional target labeling device based on point cloud data assisted two-dimensional image, comprising: The first module is used for jointly calibrating the laser radar and the optical imaging sensor, obtaining the three-dimensional point cloud data and the two-dimensional image of the target and the background, obtaining the extrinsic matrix describing the spatial coordinate transformation relationship between the laser radar and the optical imaging sensor, and recording the intrinsic matrix of the optical imaging sensor; The second module is used for labeling the point cloud data of the target to obtain the center point coordinates of the target, the target orientation, and the target size; The third module is used for converting the world coordinate system in which the point cloud data of the target is located to the image coordinate system; The fourth module is used for labeling the information of the target in the two-dimensional image, including the category information of the target and the two-dimensional label frame information; The fifth module is used for dynamically switching the labeling mode according to whether the point cloud data is complete, and completing the three-dimensional target labeling; if the point cloud data is complete, directly obtaining all the labeling information of the three-dimensional target according to the point cloud data of the target and the information in the two-dimensional image; if the point cloud data is not complete, calculating the target depth using the information of the two-dimensional image, and performing pose solving, and completing the three-dimensional target labeling by reacquiring the target orientation in the world coordinate system.

[0008] Compared with the prior art, the technical effects of the application include: (1) The method provided by the application combines two-dimensional images and three-dimensional point cloud data, uses two-dimensional image labeling information to complete three-dimensional target labeling, and improves the spatial accuracy of target detection and recognition.

[0009] (2) The method of the present application, for the case of incomplete point cloud data when the point cloud is sparse or missing, a scheme of dynamic switching of marking mode is designed, based on depth calculation and pose solving, better completing three-dimensional target labeling. The two-dimensional image labeling information is used to realize three-dimensional target space attribute reconstruction without point cloud input, solving the problem of perception continuity when the radar sensor fails. This point cloud data labeling method can provide better training data including 3D labeling for the deep learning model, improving the efficiency of training and learning.

[0010] The three-dimensional target labeling method based on point cloud data assisted two-dimensional image provided by the present application upgrades two-dimensional visual information to three-dimensional space, providing computable spatial prior knowledge for robot navigation, scene reconstruction, human-computer interaction and other tasks, and promoting the evolution of the visual system from passive perception to active cognition. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained from the structures shown in the drawings without creative labor for those skilled in the art.

[0012] Figure 1 The step flow chart of the three-dimensional target labeling method based on point cloud data assisted two-dimensional image in an embodiment of the present application; Figure 2 The point cloud data labeling schematic diagram of the target in an embodiment of the present application; Figure 3 The text example schematic diagram of parameter extraction of the point cloud data of the target in an embodiment of the present application; Figure 4 The three-dimensional point cloud target different position detection image example diagram in an embodiment of the present application, wherein, position 1 is a laser radar detectable position, position 2 is a laser radar undetectable position, Figure 4 (a) is the infrared image of the target at position 1, Figure 4 (b) is the infrared image of the target at position 2; Figure 4 (c) is the point cloud image corresponding to the target at position 1, Figure 4 (d) is the point cloud image corresponding to the target at position 2; Figure 5 The laser radar and infrared camera calibration schematic diagram in the experiment of the present application; Figure 6 The schematic diagram of collecting infrared images for three-dimensional labeling in the outdoor experiment of the experiment of the present application; Figure 7 The schematic diagram of infrared image collected for three-dimensional labeling in the field experiment of the present application is shown in the following figure, Figure 7 (a) is the point cloud data collected for three-dimensional labeling of target A in the field experiment, Figure 7 (b) is the point cloud data collected for three-dimensional labeling of target B in the field experiment, Figure 7 (c) is the point cloud data collected for three-dimensional labeling of target C in the field experiment, Figure 7 (d) is the point cloud data collected for three-dimensional labeling of target D in the field experiment; Figure 8 The schematic diagram of target size of three-dimensional target in the experiment of the present application is shown in the following figure, L is the length of the target ship, W is the width of the target ship, H is the height of the target ship; Figure 9 The schematic diagram of visualization of target three-dimensional detection frame in the experiment of the present application is shown in the following figure, Figure 10 The schematic diagram of visualization of target three-dimensional detection frame and two-dimensional detection frame in the experiment of the present application is shown in the following figure, Figure 11 The schematic diagram of two-dimensional image and point cloud image in the case of lack of point cloud data in the experiment of the present application is shown in the following figure, Figure 11 (a) is the two-dimensional image in the case of lack of point cloud data, Figure 11 (b) is the point cloud image in the case of lack of point cloud data; Figure 12 The schematic diagram of visualization of three-dimensional labeling information in the experiment of the present application is shown in the following figure, Figure 12 (a) is the visualization schematic diagram of three-dimensional labeling of yacht I-class ship, Figure 12 (b) is the visualization schematic diagram of three-dimensional labeling of sand-carrying I-class ship, Figure 12 (c) is the visualization schematic diagram of three-dimensional labeling of yacht II-class ship, Figure 12 (d) is the visualization schematic diagram of three-dimensional labeling of navigation mark-class ship, Figure 12 (e) is the visualization schematic diagram of three-dimensional labeling of sand-carrying II-class ship, Figure 12 (f) is the visualization schematic diagram of three-dimensional labeling of sand-carrying III-class ship. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0014] In one embodiment, as shown in Figure 1 The present application provides a three-dimensional target labeling method based on point cloud data assisted two-dimensional image, comprising: Step 110, jointly calibrate a laser radar (LiDAR) and an optical imaging sensor, the laser radar is used to obtain three-dimensional point cloud data of a target and a background, and the optical imaging sensor is used to obtain a two-dimensional image of the target and the background; solve an extrinsic matrix which describes the spatial coordinate transformation relationship between the laser radar and the optical imaging sensor, including a translation matrix And a rotation matrix .

[0015] Generally, the process of obtaining the translation matrix T And the rotation matrix R includes: (1) Data acquisition Use a checkerboard calibration board to synchronously collect two-dimensional images obtained by the optical imaging sensor (camera) and LiDAR point cloud data under 10 or more different poses (tilt, rotation, and distance combination), and ensure that the calibration board is clearly visible in both sensors.

[0016] (2) Camera intrinsic parameter calibration Load the two-dimensional image through the cameraCalibrator tool of MATLAB, automatically detect the checkerboard corner points, calculate the camera intrinsic parameter matrix K And the distortion coefficient and save.

[0017] (3) Extrinsic calibration (LiDAR→camera) Feature extraction: for each set of data, use detectCheckerboardPoints to extract image corner points, and use extractCheckerboardPoints to extract calibration board plane point cloud from LiDAR point cloud.

[0018] PnP algorithm solving: input 3D-2D point pairs through the lidarCameraCalibrator tool, and use the PnP (Perspective-n-Point) algorithm to optimize and solve the rotation matrix R And the translation matrix , to minimize the reprojection error. The PnP algorithm is a core method for solving camera pose estimation problems in computer vision, which uses a set of known three-dimensional (3D) space points (in the world coordinate system) and their projection points in a two-dimensional (2D) image to solve the rotation matrix R And the translation matrix .

[0019] (4) Result preservation: output the extrinsic matrix through the cameraCalibrator tool of MATLAB R and .

[0020] The optical imaging sensor can be a visible light sensor or an infrared sensor. The process of joint calibration also includes recording the intrinsic matrix of the optical imaging sensor.

[0021] Step 120, label the point cloud data of the target and perform parameter extraction to obtain the center point coordinates of the target , , ), target orientation, and target size. The target orientation is described by the yaw angle of the target in the world coordinate system and is obtained by labeling using the labelCloud open source point cloud labeling project; the parameters describing the target size include length , width , and height . The labeling result of the point cloud data of the target is shown in Figure 2 , and the text example of the point cloud data of the specific target for parameter extraction is shown in Figure 3 .

[0022] Step 130, convert the world coordinate system in which the point cloud data of the target obtained by the laser radar is located to the image coordinate system, and the specific process is as follows: (1) Convert the world coordinate system to the camera coordinate system, and the coordinate transformation is given by the following formula:

[0023] wherein is a 3 3 matrix representing the rotation matrix of the optical imaging sensor, is a 3 1 translation matrix, and both are extrinsic matrices. Among them, is the coordinate of the center point of the target in the camera coordinate system; all corresponding coordinate transformations in the two coordinate systems follow the coordinate change rule of the above formula.

[0024] (2) Convert the camera coordinate system to the image coordinate system, and the coordinate transformation is given by the following formula:

[0025] wherein, is the intrinsic matrix of the optical imaging sensor, is the coordinate of the center point of the target projected to the image coordinate system, represents a three-dimensional all-zero column vector; all corresponding coordinate transformations in the two coordinate systems follow the coordinate change rule of the above formula.

[0026] (3) Image coordinate system conversion to pixel coordinate system: The pixel coordinate system is a discrete representation of the image coordinate system, and each pixel of the actual optical imaging sensor corresponds to a photosensitive point. Assuming that the physical size of a photosensitive point is , the width is . The conversion relationship between the coordinate system established with the upper left corner of the entire image as the origin and the coordinate system established with the center of the imaging plane as the origin is as follows: ; wherein, is the coordinate of the center point of the target in the two-dimensional pixel coordinate system, is the image center pixel point coordinate; The corresponding secondary coordinate matrix form is: ; Combined with the transformation relationships of (1), (2) and (3) above, the total expression of the world coordinate system to the pixel coordinate system transformation can be obtained: ; wherein, is the focal length of the optical imaging sensor in the axis direction, is the focal length of the optical imaging sensor in the axis direction, represents taking the transpose of a vector or matrix.

[0027] Step 140, labeling the information of the target in the two-dimensional image, including the category information (for example, the category information of the target ship) and the two-dimensional labeling box ).

[0028] Step 150, dynamically switching the labeling mode according to whether the point cloud data is complete, and completing the three-dimensional target labeling, including: If the point cloud data is complete, that is, the three-dimensional point cloud target is detectable by the laser radar, then the target size is directly obtained by using the labeled point cloud data of the target; and based on the labeled point cloud data of the target and the information in the two-dimensional image, all labeling information of the three-dimensional target can be obtained, including: ① target category ② two-dimensional labeling box information ) ③ real position of target center point , , ) ④ target length-width-height size ⑤ target orientation ⑥ coordinates of target center point in image ) ⑦ coordinates of eight vertices of target three-dimensional box ) ) ​ () () () () ).

[0029] If the point cloud data is incomplete, meaning the 3D point cloud target is undetectable by the LiDAR, including cases where the point cloud data is partially missing or the target is too far away, the LiDAR cannot obtain complete point cloud data. For example, ... Figure 4 As shown, when the target is Figure 4 When the target is at position 1, it is within the detection range of the lidar, but when the target moves to... Figure 4 At position 2, the target is outside the measurement range of the lidar, making it impossible to obtain information such as target distance and orientation. Figure 4 (a) and Figure 4 (b) shows the infrared images of the target at positions 1 and 2, respectively; Figure 4 (c) and Figure 5 As shown in (d), the point cloud data of the target at positions 1 and 2 are shown respectively. In this case, the steps for annotating the 3D target include: The first step is to calculate the target depth: Using the 2D bounding box information of the 2D image, obtain the height difference of the target in the pixel coordinate system: ; ; in, For the target in pixel coordinates The difference in height in the direction, For the target in pixel coordinates The difference in elevation in the direction.

[0030] Calculate the target depth: , in, To determine the true height of the target.

[0031] The second step is to perform pose calculation, including: Calculate the (2D) position of the target center in the image coordinate system. ): , ; Inverse camera coordinate system: , ; Calculate the target orientation (yaw angle) in the camera coordinate system: Using the yaw angle in the camera coordinate system satisfy: Solving for the problem yields the following: ; Convert to world coordinate system: , ; Extract the relative target orientation in the quadrant rotation matrix : ; Reacquire the target orientation (yaw angle) under the world coordinate system: .

[0032] Thirdly, obtain the three-dimensional detection frame of the three-dimensional target, complete the three-dimensional target labeling, and obtain the labeling information including: the real position of the target center point ( , , ), the coordinates of the eight vertices of the target three-dimensional frame ( )( )( )( )( )( )( )( ).

[0033] Specifically, the real position of the target center ( , , ) is given by the following formula: ; The coordinates of the eight vertices of the target three-dimensional frame in the image coordinate system satisfy:

[0034] Wherein, is the coordinates of the eight vertices of the three-dimensional frame in the world coordinate system, is the rotation matrix of the target around axis.

[0035] Substitute the eight vertices of the three-dimensional frame in the world coordinate system into the mapping from the world coordinate system to the pixel coordinate system (derived in step 130), to obtain ( )( )( )( )( )( )( )( ). According to the total expression of the transformation from the world coordinate system to the pixel coordinate system, we can get:

[0036] Wherein, is the coordinates of the eight vertices of the three-dimensional frame of the target in the world coordinate system, is the coordinates of the eight vertices of the three-dimensional frame of the target in the pixel coordinate system.

[0037] The present application carries out experimental verification of the three-dimensional target labeling method based on point cloud data assisted two-dimensional image of the present application for the above-mentioned embodiments, and the experimental process includes: (I) Experimental equipment: (1) Laser radar (LiDAR), model is Aolun Avia laser radar, and the specific equipment parameters are shown in Table 1 as follows: Table 1: Equipment parameter table of Aolun Avia laser radar

[0038] (2) Infrared camera, model is CCI6903 infrared movement, and the specific equipment parameters are shown in Table 2 as follows: Table 2: Equipment parameter table of CCI6903

[0039] (II) Key data processing steps (1) First, the joint calibration of the laser radar and the infrared camera is carried out by using the lidarCameraCalibrator tool in MATLAB, as shown in Figure 6 , to obtain the projection matrix : ; (2) The data is obtained by shooting in the field experiment, and the three-dimensional point cloud data of the laser radar and the two-dimensional infrared image data set are obtained, as shown in Figure 7 and Figure 6 . Among them, Figure 7 is the two-dimensional infrared image collected in the field experiment for three-dimensional labeling, Figure 7 is the infrared image point cloud data collected in the field experiment for three-dimensional labeling of four different targets A, B, C and D, wherein Figure 7 (a) is the point cloud data collected in the field experiment for three-dimensional labeling of target A, Figure 7 (b) is the point cloud data collected in the field experiment for three-dimensional labeling of target B, Figure 8 (c) is the point cloud data collected in the field experiment for three-dimensional labeling of target C, Figure 9 (d) is the point cloud data collected in the field experiment for three-dimensional labeling of target D.

[0040] (3) The point cloud data is labeled to obtain the three-dimensional label of the ship target, and the length, width and height data of each type of ship are recorded for three-dimensional labeling in the case of lack of point cloud data in the later period, as shown in Figure 10 . Among them: Three-dimensional size label: : 10.7, : 30.7, : 8.8, unit: meter; Target center in world coordinate system coordinates , , ) : , unit: meter; : , unit: radian rad; At the same time, record the length, width and height data of the ship: : 10.7, : 30.7, : 8.8, unit: meter.

[0041] (4) Convert the world coordinates obtained by the laser radar to image coordinates, as shown in Figure 11 .

[0042] (5) Label the two-dimensional image to obtain the two-dimensional label of the target, as shown in Figure 11 .

[0043] (6) For the case of lack of point cloud data, under the condition that the length, width and height of the target ship are known, the three-dimensional label information is calculated according to the two-dimensional label frame.

[0044] As shown in Figure 11 , the target can be clearly seen from the two-dimensional image of Figure 12 (a), but it can be known from the point cloud image of Figure 12 (b) that the laser radar cannot obtain the corresponding point cloud of the target.

[0045] The two-dimensional label corresponding to the ship target in the two-dimensional image is: , unit: pixel; (7) According to step 150 in the method, the three-dimensional data of the labeled ship target is obtained: : 8.8, .

[0046] The label obtained by the three-dimensional target labeling method based on point cloud data assisted two-dimensional image according to the application is visualized in the image, as shown in Figure 12 . Specifically, Figure 12 (a) shows the three-dimensional labeling visualization information of the yacht I class ship, Figure 12 (b) shows the three-dimensional labeling visualization information of the sand transport I class ship, Figure 12 (c) shows the three-dimensional labeling visualization information of the yacht II class ship, Figure 12 (d) shows the three-dimensional labeling visualization information of the navigation mark class ship, ​ (e) is the three-dimensional labeling visualization information of the sand transport II class ship, ​(f) is a three-dimensional annotation visual information of sand-carrying Class III vessel.

[0047] In another embodiment, the present application provides a three-dimensional target annotation device based on point cloud data assisted two-dimensional image, comprising: The first module is configured to jointly calibrate the laser radar and the optical imaging sensor, obtain three-dimensional point cloud data and two-dimensional image of the target and the background, obtain an extrinsic parameter matrix describing the spatial coordinate transformation relationship between the laser radar and the optical imaging sensor, and record an intrinsic parameter matrix of the optical imaging sensor; The second module is configured to annotate the point cloud data of the target to obtain the center point coordinates, the target orientation and the target size of the target; The third module is configured to convert the world coordinate system in which the point cloud data of the target is located to the image coordinate system; The fourth module is configured to annotate the information of the target in the two-dimensional image, including the category information and the two-dimensional annotation box information of the target; The fifth module is configured to dynamically switch the annotation mode according to whether the point cloud data is complete, and complete the three-dimensional target annotation; if the point cloud data is complete, all annotation information of the three-dimensional target is obtained directly according to the point cloud data of the target and the information in the two-dimensional image; if the point cloud data is not complete, the depth of the target is calculated using the information of the two-dimensional image, and the pose is solved, and the three-dimensional target annotation is completed by reacquiring the orientation of the target in the world coordinate system.

[0048] On the other hand, the present application provides a computer device comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the three-dimensional target annotation method based on point cloud data assisted two-dimensional image provided in any of the above embodiments when executing the computer program. The computer device can be a server. The computer device includes a processor, a memory, a network interface and a database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control ability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sample data. The network interface of the computer device is used to communicate with the external terminal through the network connection.

[0049] In addition, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor to realize the steps of the three-dimensional target annotation method based on point cloud data assisted two-dimensional image provided in any of the above embodiments.

[0050] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0051] The details of the present application are as follows.

[0052] The technical features of the above embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0053] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application.

[0054] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for annotating three-dimensional targets in two-dimensional images based on point cloud data, characterized in that, include: Step 110: Perform joint calibration of the lidar and the optical imaging sensor to obtain three-dimensional point cloud data and two-dimensional images of the target and background, obtain the extrinsic parameter matrix describing the spatial coordinate transformation relationship between the lidar and the optical imaging sensor, and record the intrinsic parameter matrix of the optical imaging sensor. Step 120: Label the point cloud data of the target to obtain the center point coordinates, target orientation, and target size; Step 130: Transform the world coordinate system of the target's point cloud data to the image coordinate system; Step 140: Label the target information in the two-dimensional image, including the target category information and the two-dimensional bounding box information; Step 150: Dynamically switch the annotation mode according to whether the point cloud data is complete to complete the 3D target annotation; if the point cloud data is complete, directly obtain all the annotation information of the 3D target based on the point cloud data of the target and the information in the 2D image. If the point cloud data is incomplete, the target depth is calculated using information from the 2D image, and the pose is solved. By re-acquiring the target orientation in the world coordinate system, the 3D target annotation is completed.

2. The method for three-dimensional target annotation based on point cloud data-assisted two-dimensional images according to claim 1, characterized in that, The lidar is used to acquire three-dimensional point cloud data of the target and background; the optical imaging sensor is used to acquire two-dimensional images of the target and background.

3. The method for three-dimensional target annotation based on point cloud data-assisted two-dimensional images according to claim 2, characterized in that, The extrinsic parameter matrix includes a translation matrix. With rotation matrix .

4. The method for three-dimensional target annotation based on point cloud data-assisted two-dimensional images according to claim 3, characterized in that, The parameters that characterize the target size include length. ,width and height .

5. The method for three-dimensional target annotation based on point cloud data-assisted two-dimensional images according to claim 4, characterized in that, Step 130 includes: Step 131: Transform the world coordinate system into the camera coordinate system. The coordinate transformation is given by the following formula: in, It is 3 A 3x3 matrix represents the rotation matrix of the optical imaging sensor; 3 Translation matrix of 1; These are the coordinates of the target's center point in the camera coordinate system. These are the coordinates of the target's center point in the world coordinate system. Step 132: Transform the camera coordinate system into the image coordinate system. The coordinate transformation is given by the following formula: in, This is the intrinsic parameter matrix of the optical imaging sensor. These are the coordinates of the target's center point projected onto the image coordinate system. Represents a three-dimensional column vector of all zeros; Step 133: Transform the image coordinate system into the pixel coordinate system. The coordinate transformation is given by the following formula: in, and These are the width and height of the physical dimensions of each pixel in an optical imaging sensor, corresponding to a single photosensitive point. These are the coordinates of the center point of the target in a two-dimensional pixel coordinate system. The coordinates of the center pixel of the image; Step 134: Substitute the coordinate transformation formulas from steps 131, 132, and 133 into the equation to obtain the three-dimensional point coordinates. The formula for projecting from the world coordinate system to the pixel coordinate system of a two-dimensional image: ； in, For optical imaging sensors in Focal length along the axial direction, For optical imaging sensors in Focal length along the axial direction, This indicates the transpose of a vector or matrix.

6. The method for three-dimensional target annotation based on point cloud data-assisted two-dimensional images according to claim 5, characterized in that, In step 150, all annotation information of the three-dimensional target includes: Target category information; Two-dimensional annotation box information: ( ; The true location of the center point of the target ( , , ); Target size: ; Target direction ; The coordinates of the center point of the target in the pixel coordinate system of the two-dimensional image ( ); The coordinates of the eight vertices of the target's 3D bounding box: ( () () () () () () () ).

7. The method for three-dimensional target annotation based on point cloud data-assisted two-dimensional images according to claim 6, characterized in that, In step 150, the target depth is calculated using information from the two-dimensional image, including: Using the 2D bounding box information of the 2D image, obtain the height difference of the target in the pixel coordinate system: ; ; in, For the target in pixel coordinates The difference in height in the direction, For the target in pixel coordinates The difference in elevation in the direction; Calculate the target depth: , in, To determine the true height of the target.

8. The method for three-dimensional target annotation based on point cloud data-assisted two-dimensional images according to claim 7, characterized in that, In step 150, pose calculation is performed, including: Calculate the position of the target center in the image coordinate system. ): , ; Inverse camera coordinate system: , ; Using the target orientation in the camera coordinate system Cosine relation: Calculate the target orientation : ； Transform to coordinates in the world coordinate system: , ; Extracting rotation matrix by quadrant Relative target orientation in: ; Reacquire the target orientation in the world coordinate system: .

9. The method for three-dimensional target annotation based on point cloud data-assisted two-dimensional images according to claim 8, characterized in that, In step 150, if the point cloud data is incomplete, the process of completing the 3D target annotation also includes: Obtain the true location of the target center ( , , ), given by the following formula: ； Obtain the coordinates of the eight vertices of the target 3D bounding box in the image coordinate system. : ; in, Circling around the target The rotation matrix of the axis.

10. A three-dimensional target annotation device based on point cloud data-assisted two-dimensional images, characterized in that, include: The first module is used to jointly calibrate the lidar and the optical imaging sensor, acquire three-dimensional point cloud data and two-dimensional images of the target and background, obtain the extrinsic parameter matrix describing the spatial coordinate transformation relationship between the lidar and the optical imaging sensor, and record the intrinsic parameter matrix of the optical imaging sensor. The second module is used to annotate the point cloud data of the target, and obtain the center point coordinates, target orientation, and target size of the target; The third module is used to transform the world coordinate system of the target's point cloud data to the image coordinate system; The fourth module is used to annotate the target's information in the two-dimensional image, including the target's category information and the two-dimensional bounding box information; The fifth module is used to dynamically switch the annotation mode based on whether the point cloud data is complete, and complete the 3D target annotation; if the point cloud data is complete, it directly obtains all the annotation information of the 3D target based on the point cloud data of the target and the information in the 2D image. If the point cloud data is incomplete, the target depth is calculated using information from the 2D image, and the pose is solved. By re-acquiring the target orientation in the world coordinate system, the 3D target annotation is completed.

Images