A method, apparatus and electronic device for data annotation

Abstract

An embodiment of the present application discloses a method for data annotation, including: obtaining image data and point cloud data, where the annotation targets in the image data include the annotation targets in the first annotation target set and the second annotation target set, and the annotation targets in the first annotation target set are the annotation targets in the point cloud data; determining the three-dimensional envelope of each annotation target in the first annotation target set in the lidar coordinate system; obtaining the two-dimensional annotation result of the first annotation target set in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system and the three-dimensional envelope; determining the two-dimensional annotation result of the second annotation target set in the pixel coordinate system; determining the three-dimensional annotation result of the second annotation target set in the lidar point cloud coordinate system according to the conversion relationship between the pixel coordinate system and the lidar point cloud coordinate system and the two-dimensional annotation result of the second annotation target set; to improve the annotation ability of joint annotation for long-distance detection targets.

Description

Technical Field

[0001] The present invention relates to the field of computers, and particularly to a method, apparatus, and electronic device for data annotation. Background Art

[0002] With the development of technology, machine learning has been more and more widely applied. Machine learning refers to using a computer as a tool to simulate the human learning method and solve problems in scientific research or actual production practice. In recent years, as autonomous driving has gradually come into people's view, target detection in the process of autonomous driving through machine learning has received people's attention. Target detection refers to obtaining target objects of interest in the entire image or a large amount of data, usually achieved through a target detection model. Usually, in the process of obtaining the target detection model, existing data is annotated, and the annotated data forms a training dataset to train the model and optimize the model.

[0003] Currently, usually, joint annotation is performed on image data detected by a camera and point cloud data obtained by a lidar detector to reduce the workload of annotation. That is, the same targets in the image data and the point cloud data are annotated simultaneously. In the annotation result, the annotated target has both the information of the image data (such as color information) and the information of the point cloud data (such as depth information). Due to the different working principles of the two detectors, the camera and the lidar point cloud, the detection distance of the lidar system is less than that of the camera, resulting in difficulty in annotating objects to be detected at a relatively long distance during the joint annotation process.

[0004] Therefore, it is difficult to annotate detection targets at a long distance by using the above method for joint annotation. Summary of the Invention

[0005] In view of this, the present application provides a method, apparatus, and electronic device for data annotation to improve the ability of joint annotation to annotate detection targets at a long distance.

[0006] In a first aspect, the present application provides a method for data annotation, the method including:

[0007] Obtain image data obtained by a camera and point cloud data obtained by a lidar detector; wherein, the annotated targets in the image data include the annotated targets in the first annotated target set and the second annotated target set, and the annotated targets in the first annotated target set are the annotated targets in the point cloud data;

[0008] Calibrate the camera and the lidar detector to determine the conversion relationship between the camera coordinate system and the lidar coordinate system;

[0009] Determine the three-dimensional envelope of each labeled target in the first labeled target set in the lidar coordinate system;

[0010] According to the conversion relationship between the camera coordinate system and the lidar coordinate system and the three-dimensional envelope, obtain the two-dimensional labeling result of the first labeled target set in the pixel coordinate system;

[0011] Determine the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system according to the image data;

[0012] Determine the two-dimensional labeling result of the second labeled target set in the pixel coordinate system;

[0013] According to the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system, and the two-dimensional labeling result of the second labeled target set, determine the three-dimensional labeling result of the second labeled target set in the laser point cloud coordinate system.

[0014] In a possible implementation, the step of obtaining the two-dimensional labeling result of the first labeled target set in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system and the three-dimensional envelope includes:

[0015] Determine the two-dimensional envelope corresponding to the three-dimensional envelope in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system;

[0016] Determine the circumscribed quadrilateral of each two-dimensional envelope in the pixel coordinate system to obtain the two-dimensional labeling result of the first labeled target set.

[0017] In a possible implementation, the step of determining the two-dimensional envelope corresponding to the three-dimensional envelope in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system includes:

[0018] Determine the vertices of the three-dimensional envelope in the lidar coordinate system;

[0019] Determine the two-dimensional coordinates corresponding to the vertices of the three-dimensional envelope in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system;

[0020] Determine the two-dimensional envelope corresponding to the three-dimensional envelope in the pixel coordinate system according to the two-dimensional coordinates corresponding to the vertices of the three-dimensional envelope.

[0021] In a possible implementation, after determining the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system according to the image data, it further includes:

[0022] Determine the three-dimensional data corresponding to the two-dimensional labeling result of the first labeled target set in the lidar coordinate system;

[0023] Determine the deviation between the three-dimensional envelope and the three-dimensional data;

[0024] Determining the three-dimensional annotation result of the second annotation target set in the lidar coordinate system according to the conversion relationship between the pixel coordinate system and the lidar coordinate system, and the two-dimensional annotation result of the second annotation target set, includes:

[0025] Determining the first three-dimensional annotation result of the second annotation target set in the lidar coordinate system according to the conversion relationship between the pixel coordinate system and the lidar coordinate system, and the two-dimensional annotation result of the second annotation target set;

[0026] Correcting the first three-dimensional annotation result according to the deviation to obtain the three-dimensional annotation result of the second annotation target set.

[0027] In a second aspect, the present application provides a data annotation device, and the device includes:

[0028] A data acquisition unit, configured to acquire image data obtained by a camera and point cloud data obtained by a lidar detector; wherein, the annotation targets in the image data include the annotation targets in the first annotation target set and the second annotation target set, and the annotation targets in the first annotation target set are the annotation targets in the point cloud data;

[0029] A detector calibration unit, configured to: determine the calibration of the camera and the lidar detector, and determine the conversion relationship between the camera coordinate system and the lidar coordinate system;

[0030] A data annotation unit, configured to: determine the three-dimensional envelope of each annotation target in the first annotation target set in the lidar coordinate system; obtain the two-dimensional annotation result of the first annotation target set in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system and the three-dimensional envelope; determine the conversion relationship between the pixel coordinate system and the lidar coordinate system according to the image data; determine the two-dimensional annotation result of the second annotation target set in the pixel coordinate system; determine the three-dimensional annotation result of the second annotation target set in the lidar coordinate system according to the conversion relationship between the pixel coordinate system and the lidar coordinate system, and the two-dimensional annotation result of the second annotation target set.

[0031] In a possible implementation manner, the data annotation unit is specifically configured to:

[0032] Determine the two-dimensional envelope corresponding to the three-dimensional envelope in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system;

[0033] Determine the circumscribed quadrilateral of each two-dimensional envelope in the pixel coordinate system to obtain the two-dimensional annotation result of the first annotation target set.

[0034] In a possible implementation manner, the data annotation unit is specifically configured to:

[0035] Determine the vertices of the three-dimensional envelope in the lidar coordinate system;

[0036] Determine the two-dimensional coordinates corresponding to the vertices of the three-dimensional envelope in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system;

[0037] Determine the two-dimensional envelope corresponding to the three-dimensional envelope in the pixel coordinate system according to the two-dimensional coordinates corresponding to the vertices of the three-dimensional envelope.

[0038] In a possible implementation, the data annotation unit is specifically configured to:

[0039] Determine the three-dimensional data corresponding to the two-dimensional annotation result of the first annotation target set in the lidar coordinate system;

[0040] Determine the deviation between the three-dimensional inclusion and the three-dimensional data;

[0041] Determine the first three-dimensional annotation result of the second annotation target set in the lidar point cloud coordinate system according to the conversion relationship between the pixel coordinate system and the lidar point cloud coordinate system, and the two-dimensional annotation result of the second annotation target set;

[0042] Correct the first three-dimensional annotation result according to the deviation to obtain the three-dimensional annotation result of the second annotation target set.

[0043] In a third aspect, the present application provides an electronic device for data annotation. The electronic device includes a processor and a memory. Among them, the memory stores code, and the processor is used to call the code stored in the memory to implement the method described in any one of the above.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program for executing the method described in any one of the above.

[0045] Adopting the solution of the present application, the data of the annotation targets in the second annotation target set is supplemented into the point cloud data, thereby completing the joint annotation of the image data and the point cloud data. Since the annotation targets in the second annotation target set have a farther detection distance than the annotation targets in the first annotation target set, therefore, adopting the method of this embodiment can improve the annotation ability of the joint annotation for long-distance detection targets. Description of the Drawings

[0046] Figure 1 is a flowchart of the data annotation method provided by the embodiment of the present application;

[0047] Figure 2 is a schematic structural diagram of the data annotation device provided by the embodiment of the present application;

[0048] Figure 3 It is a schematic structural diagram of an electronic device for data annotation provided by an embodiment of the present application. Specific implementation manners

[0049] The image data is obtained by camera detection, and the point cloud data is obtained by lidar detector detection. Since the working principles of these two detectors, namely the camera and the lidar point cloud, are different, and the detection distance of the lidar system is less than that of the camera, it is difficult to annotate the measured object at a relatively long distance during the joint annotation process.

[0050] Based on this, in the embodiment of the present application provided by the applicant, first, the image data obtained by the camera and the point cloud data obtained by the lidar detector are respectively acquired. The annotation targets in the image data include the annotation targets in the first annotation target set and the second annotation target set. The annotation targets in the first annotation target set are the annotation targets in the point cloud data, and the annotation targets in the second annotation target set have a greater detection distance than the annotation targets in the first annotation target set; according to the conversion relationship between the camera coordinate system and the lidar coordinate system, and the three-dimensional envelope of each annotation target in the first annotation target set in the lidar coordinate system, the two-dimensional annotation result of the first annotation target set in the pixel coordinate system is obtained; according to the image data, the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system is determined; according to the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system, and the two-dimensional annotation result of the second annotation target set, the three-dimensional annotation result of the second annotation target set in the laser point cloud coordinate system is determined.

[0051] Adopting the solution of the present application, the data of the annotation targets in the second annotation target set is supplemented into the point cloud data, thereby completing the joint annotation of the image data and the point cloud data. Since the annotation targets in the second annotation target set have a farther detection distance than the annotation targets in the first annotation target set, therefore, adopting the method of this embodiment can improve the annotation ability of the joint annotation for the long-distance detection targets.

[0052] To facilitate understanding of the technical solution provided by the embodiment of the present application, the following describes a method, apparatus, and electronic device for data annotation provided by the embodiment of the present application with reference to the accompanying drawings.

[0053] Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Based on the embodiments in the present application, other embodiments obtained by those skilled in the art without making creative contributions all fall within the protection scope of the present application.

[0054] In the claims, specification, and drawings of this application, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion.

[0055] This application provides a method for data annotation.

[0056] Please refer to Figure 1 , Figure 1 which is a flowchart of the data annotation method provided by an embodiment of this application.

[0057] As Figure 1 shown, the data annotation method in the embodiment of this application includes S101 - S107.

[0058] S101. Obtain the image data detected by the camera and the point cloud data detected by the lidar detector. The annotation targets in the image data include the annotation targets in the first annotation target set and the second annotation target set. The annotation targets in the first annotation target set are the annotation targets in the point cloud data.

[0059] Both the first annotation target set and the second annotation target set contain one or more annotation targets.

[0060] The image data and the point cloud data are respectively the detection results of two detectors, namely the camera and the lidar detector.

[0061] The image data is data in the camera coordinate system, and the point cloud data is data in the lidar detector coordinate system.

[0062] Through camera detection, the two-dimensional position information of the object to be measured can be obtained. The image data includes at least the two-dimensional position information of the object to be measured, and the information of the object at each position point is represented by a gray value in the image, such as color information;

[0063] Through the lidar detector, the three-dimensional position information of the object to be measured can be obtained. The point cloud data includes at least the three-dimensional position information of the object to be measured, and can also include other information of the object at each position point, such as the number of echoes, etc.

[0064] The above two-dimensional position information and three-dimensional position information refer to the position information in the real physical world. Usually, the original data obtained by the lidar detector includes the angle information and distance information of the object to be measured. For the convenience of processing, the original data obtained by the lidar detector is converted into three-dimensional point coordinate information in the Cartesian coordinate system to obtain the point cloud data. The point cloud data includes the three-dimensional position information of the object to be measured, as well as information such as the reflection intensity and the number of echoes at each position point.

[0065] Specifically, the point cloud data can be the data in the coordinate system where the lidar detector is located, and this coordinate system is the lidar coordinate system. In the autonomous driving scenario, the lidar detector is located on the vehicle, and the point cloud data can be the data in the vehicle body coordinate system.

[0066] The labeled target refers to the object that exists in the image data / point cloud data and is labeled, that is, the object to be measured.

[0067] Generally, the labeling result is obtained through the labeling process. The labeling result includes one or more labeled targets, and the labels corresponding to the above one or more labeled targets.

[0068] For example, the labeled targets can include people, vehicles, and trees existing in the above image data and point cloud data, and the corresponding labels are "person", "vehicle", and "tree" respectively.

[0069] Among the detection results obtained by different detectors, the labeled targets may be represented by different forms of data. For example, for the same labeled target in the real world, in the image data, the labeled target may be represented by the two-dimensional position information of the object and the color information of the object corresponding to the two-dimensional position information; while in the point cloud data, the labeled target may be represented by the three-dimensional position information of the object.

[0070] Since the detection distance of the lidar is less than that of the camera, therefore, the labeled targets in the image data at least include the labeled targets in the point cloud data to achieve the joint labeling of the image data and the point cloud data.

[0071] Although the representation methods in the image data and in the point cloud data are different, the labeled targets corresponding to the real world are the same. For example, it is the same tree in the real world that is labeled.

[0072] That is to say, the labeled targets in the real world corresponding to the labeled targets in the image data at least include the labeled targets in the real world corresponding to the labeled targets in the point cloud data.

[0073] S102. Calibrate the camera and the lidar detector to obtain the conversion relationship between the camera coordinate system and the lidar coordinate system.

[0074] The conversion relationship between the camera coordinate system and the lidar coordinate system characterizes the corresponding relationship of points between the camera coordinate system and the lidar coordinate system.

[0075] That is to say, for a point in the camera coordinate system, the corresponding position of this point in the lidar coordinate system can be obtained by using the conversion relationship between the camera coordinate system and the lidar coordinate system; and, for a point in the lidar coordinate system, the corresponding position of this point in the camera coordinate system can be obtained by using the conversion relationship between the camera coordinate system and the lidar coordinate system.

[0076] The camera detects a two-dimensional image of the object to be measured. For the camera coordinate system, the origin is the light point of the camera, and the three directions of the coordinate axes are respectively two directions parallel to the two sides of the two-dimensional image and the direction perpendicular to the two-dimensional image.

[0077] S103. Determine the three-dimensional envelope of each annotation target in the first annotation target set in the lidar coordinate system.

[0078] That is, obtain the three-dimensional annotation result of the first annotation target set.

[0079] The annotation targets in the first annotation target set are the annotation targets in the point cloud data, and the first annotation target set contains one or more annotation targets.

[0080] The point cloud data includes at least the three-dimensional position information of the annotation targets in the first annotation target set.

[0081] According to the corresponding relationship between the lidar point cloud coordinate system and the Cartesian coordinate system, the corresponding data of the point cloud data in the Cartesian coordinate system can be obtained. The data in the Cartesian coordinate system is the position of the object to be measured in the real physical world.

[0082] Each annotation target in the annotation target set corresponds to a three-dimensional envelope, and the three-dimensional position information of the above envelope in the Cartesian coordinate system can be obtained, that is, the three-dimensional coordinates of the three-dimensional envelope in the Cartesian coordinate system.

[0083] The shape of each three-dimensional envelope is usually a three-dimensional polyhedron.

[0084] S104. According to the conversion relationship between the camera coordinate system and the lidar coordinate system, and the three-dimensional envelope, obtain the two-dimensional annotation result of the first annotation target set in the pixel coordinate system.

[0085] For a point in the lidar coordinate system, the corresponding position of this point in the camera coordinate system can be obtained by using the conversion relationship between the camera coordinate system and the lidar coordinate system.

[0086] According to the camera imaging principle, that is, the pinhole imaging principle, the corresponding relationship between the camera coordinate system and the pixel coordinate system can be obtained.

[0087] The origin of the pixel coordinate system is at the upper left corner of the image obtained by the camera, and the directions of the two coordinate axes of the pixel coordinate system are along the two sides of the image obtained by the camera respectively.

[0088] Therefore, the corresponding relationship between the lidar coordinate system and the pixel coordinate system can be obtained, so as to determine the data corresponding to the three-dimensional envelope in the pixel coordinate system and obtain the two-dimensional annotation result of the first annotation target set.

[0089] The two-dimensional annotation result of the first annotation target set is also the annotation result of each annotation target in the first annotation target set in the pixel coordinate system.

[0090] Since the annotation result of point cloud data is usually more accurate than that of image data, the above method is to optimize the two-dimensional annotation result through the three-dimensional annotation result, which can improve the accuracy of the two-dimensional annotation result.

[0091] That is, the annotation result of point cloud data is introduced into the image, which is used as a constraint condition to improve the accuracy of the image annotation result.

[0092] Specifically, since the annotation targets in the first annotation target set are the annotation targets in the point cloud data, and the point cloud data usually corresponds to the detection results with a small detection distance, the above method can improve the annotation results of the close-range annotation targets in the image data.

[0093] S105. Determine the conversion relationship between the pixel coordinate system and the lidar point cloud coordinate system according to the image data.

[0094] That is, obtain the conversion relationship from two-dimensional data to three-dimensional data.

[0095] The conversion relationship between the pixel coordinate system and the lidar point cloud coordinate system obtained here according to the image data is used to reduce the systematic error caused by detector calibration, so as to improve the accuracy of calibration.

[0096] This embodiment does not limit the implementation manner of obtaining three-dimensional data from two-dimensional annotation results according to image data, and any existing method can be adopted.

[0097] S106. Determine the two-dimensional annotation result of the second annotation target set in the pixel coordinate system.

[0098] The two-dimensional annotation result of the second annotation target set is also the two-dimensional annotation result of each annotation target in the second annotation target set.

[0099] S107. Determine the three-dimensional annotation result of the second annotation target set in the lidar point cloud coordinate system according to the conversion relationship between the pixel coordinate system and the lidar point cloud coordinate system, and the two-dimensional annotation result of the second annotation target set.

[0100] According to the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system, the corresponding relationship between each position point in the pixel coordinate system and the laser point cloud coordinate system can be obtained.

[0101] Since the camera detection distance is greater than the laser radar detection distance, the data of the object to be measured with a larger detection distance that is not in the point cloud data is converted to the laser radar coordinate system according to the conversion relationship between the pixel coordinate system and the laser radar coordinate system, that is, the point cloud data of the object to be measured with a larger detection distance is obtained.

[0102] Through the above method, the three-dimensional annotation results and the two-dimensional annotation results of the first annotation target set, and the three-dimensional annotation results and the two-dimensional annotation results of the second annotation target set are obtained.

[0103] The data of the labeled targets in the second labeled target set are added to the point cloud data, thereby completing the joint labeling of the image data and the point cloud data. Since the labeled targets in the second labeled target set have a longer detection distance than the labeled targets in the first labeled target set, the method of this embodiment can improve the labeling capability of the joint labeling for long-distance detection targets.

[0104] This application also provides another data annotation method.

[0105] The data annotation method in the embodiment of the present application includes S201-S211.

[0106] S201. Acquire image data detected by a camera and point cloud data detected by a laser radar detector; wherein the annotated targets in the image data include annotated targets in a first annotated target set and a second annotated target set, and the annotated targets in the first annotated target set are the annotated targets in the point cloud data.

[0107] S202, calibrate the camera and the laser radar detector to obtain a conversion relationship between the camera coordinate system and the laser radar coordinate system.

[0108] Furthermore, the conversion relationship between the camera coordinate system and the lidar coordinate system can be presented in the form of calibration parameters between the camera and the lidar detector.

[0109] S203. Determine, based on the point cloud data, the three-dimensional envelope of each labeled target in the first labeled target set in the laser radar coordinate system, and obtain a labeling result of the first labeled target set.

[0110] S204: Determine the vertices of the three-dimensional envelope of each labeled object in the first labeled object set.

[0111] Each labeled object in the first labeled object set corresponds to a three-dimensional envelope, and each three-dimensional envelope has multiple vertices.

[0112] Determine the vertices of the three-dimensional envelope of each labeled object in the first labeled object set, that is, determine the three-dimensional coordinates of the vertices of each three-dimensional envelope.

[0113] Both S203 - S204 are performed in the lidar coordinate system.

[0114] S205. According to the conversion relationship between the camera coordinate system and the lidar coordinate system, determine the two-dimensional coordinates of the vertices of each three-dimensional envelope in the pixel coordinate system.

[0115] For a point in the lidar coordinate system, using the conversion relationship between the camera coordinate system and the lidar coordinate system, the corresponding position of this point in the camera coordinate system can be obtained.

[0116] According to the camera imaging principle, that is, the pinhole imaging principle, the corresponding relationship between the camera coordinate system and the pixel coordinate system can be obtained.

[0117] The origin of the pixel coordinate system is at the upper left corner of the image obtained by the camera, and the directions of the two coordinate axes of the pixel coordinate system are respectively along the two sides of the image obtained by the camera.

[0118] Therefore, the corresponding relationship between the lidar coordinate system and the pixel coordinate system can be obtained, so as to determine the two-dimensional coordinates of the vertices of each three-dimensional envelope in the pixel coordinate system.

[0119] Furthermore, according to the corresponding relationship between the camera coordinate system and the image coordinate system, and the corresponding relationship between the image coordinate system and the pixel coordinate system, the corresponding relationship between the camera coordinate system and the pixel coordinate system can be obtained.

[0120] The image coordinate system is a two-dimensional coordinate system, the origin of the image coordinate system is the center of the image obtained by camera imaging, and the two directions of the image coordinate system are respectively the directions of the two sides of the image obtained by camera imaging.

[0121] The imaging principle of the camera is the pinhole imaging principle. Therefore, the corresponding relationship between the camera coordinate system and the image coordinate system can be obtained according to the pinhole imaging principle.

[0122] The directions of the coordinate axes of the pixel coordinate system and the image coordinate system are the same; the coordinate centers of the pixel coordinate system and the image coordinate system are different, and the center of the pixel coordinate system is located at the position of the point in the upper left corner of the image obtained by camera imaging.

[0123] S206. Determine the two-dimensional envelope formed by the two-dimensional coordinates of the vertices of each three-dimensional envelope in the pixel coordinate system.

[0124] S205 - S206 is to determine the two - dimensional envelope corresponding to the three - dimensional envelope in the lidar coordinate system in the pixel coordinate system.

[0125] Each three - dimensional envelope in the lidar point cloud coordinate system corresponds to a two - dimensional envelope in the pixel coordinate system.

[0126] S207. Determine the circumscribed quadrilateral of each two - dimensional envelope in the pixel coordinate system, and obtain the two - dimensional annotation result of each annotation target in the first annotation target set.

[0127] Each three - dimensional envelope corresponds to a two - dimensional envelope, and each two - dimensional envelope corresponds to a quadrilateral circumscribing the two - dimensional envelope.

[0128] The circumscribed quadrilateral of the two - dimensional envelope is the two - dimensional annotation target of the annotation target in the first annotation target set.

[0129] That is to say, for the annotation targets in the first annotation target set, the annotation result (three - dimensional polyhedron) of the point cloud data is mapped into the pixel coordinate system to obtain the annotation result (two - dimensional circumscribed quadrilateral) of the image data.

[0130] Since the annotation result of point cloud data is usually more accurate than that of image data, the above - mentioned method, that is, optimizing the two - dimensional annotation result through the three - dimensional annotation result, can improve the accuracy of the two - dimensional annotation result.

[0131] That is, the annotation result of the polygon of the point cloud data is introduced into the image, which improves the accuracy of the image annotation result as a constraint condition.

[0132] Specifically, since the annotation targets in the first annotation target set are the annotation targets in the point cloud data, and the point cloud data usually corresponds to the detection results with a relatively small detection distance, the above - mentioned method can improve the annotation results of the close - range annotation targets in the image data.

[0133] S208. Determine the conversion relationship between the pixel coordinate system and the lidar point cloud coordinate system according to the image data.

[0134] That is, obtain the conversion relationship from two - dimensional data to three - dimensional data.

[0135] This embodiment does not limit the implementation method of obtaining three - dimensional data from two - dimensional annotation results, and any method in the prior art can be adopted. For example, obtain the data of multiple frames before and after the above - mentioned image data, and according to the change gradient of the pixels between the front and rear frames, obtain the corresponding distance in the lidar point cloud coordinate system, so as to determine the conversion relationship between the pixel coordinate system and the lidar point cloud coordinate system.

[0136] The position information of the measured object represented by the point cloud data in the lidar coordinate system is the position of the measured object in the real physical world.

[0137] Therefore, the conversion relationship between the pixel coordinate system and the lidar point cloud coordinate system obtained in S208, that is, the conversion relationship between the pixel coordinate system and the real physical world coordinate system.

[0138] S209. Obtain the three-dimensional data corresponding to the two-dimensional annotation result in the lidar point cloud coordinate system.

[0139] The above two-dimensional annotation result is the two-dimensional annotation result of the first annotation target set.

[0140] That is, convert the two-dimensional data into the three-dimensional coordinate system to obtain three-dimensional data.

[0141] In S208, the conversion relationship between the pixel coordinate system and the lidar point cloud coordinate system is obtained. According to the above conversion relationship, the three-dimensional data in S209 can be obtained.

[0142] S209. Determine the deviation between the above three-dimensional envelope and the above three-dimensional data.

[0143] The three-dimensional envelope is the three-dimensional envelope of the annotation target in the first annotation target set in the lidar coordinate system, and each three-dimensional envelope corresponds to an annotation target in the first annotation target set;

[0144] The three-dimensional data of the two-dimensional annotation result in the lidar point cloud coordinate system corresponds to the two-dimensional annotation result, and each two-dimensional annotation result corresponds to a three-dimensional envelope.

[0145] Therefore, the above three-dimensional envelope and the above three-dimensional data correspond one by one.

[0146] Map the two-dimensional annotation result from the two-dimensional pixel coordinates to the three-dimensional lidar coordinate system to obtain the above three-dimensional data. S209 also compares the result obtained by the above mapping with the real three-dimensional envelope to obtain the deviation.

[0147] S210. Determine the two-dimensional annotation result of each annotation target in the second annotation target set in the pixel coordinate system to obtain the two-dimensional annotation result of the second annotation target set.

[0148] S211. According to the conversion relationship between the pixel coordinate system and the lidar coordinate system, determine the data corresponding to the two-dimensional annotation result of the second annotation target set in the lidar coordinate system, and correct the data corresponding to the two-dimensional annotation result of the second annotation target set in the lidar coordinate system according to the above deviation to obtain the three-dimensional annotation result of the second annotation target set.

[0149] S211 also adds data of more labeled targets to the point cloud data to obtain a labeling result. Since the detection range of the lidar detector is small and the detection range of the camera is large, the point cloud data added by S211 is the data of the labeled targets with a large detection range.

[0150] There may be a deviation in converting the two-dimensional data in the pixel coordinate system to the three-dimensional data in the lidar coordinate system, that is, the deviation obtained by S209. To improve the accuracy of the labeling result in the lidar coordinate system, after determining the data of the two-dimensional labeling result of the second set of labeled targets in the lidar coordinate system, the above deviation is superimposed to obtain the final three-dimensional labeling result of the second set of labeled targets.

[0151] Please refer to Figure 2 , Figure 2 which is a schematic structural diagram of the data labeling device provided by the embodiment of the present application.

[0152] As Figure 2 shown, the data labeling device 200 in the embodiment of the present application includes a data acquisition unit 201, a detector calibration unit 202, and a data labeling unit 203.

[0153] The data acquisition unit 201 is used to acquire the image data obtained by the camera and the point cloud data obtained by the lidar detector; wherein, the labeled targets in the image data include the labeled targets in the first set of labeled targets and the second set of labeled targets, and the labeled targets in the first set of labeled targets are the labeled targets in the point cloud data.

[0154] The detector calibration unit 202 is used to: determine the calibration of the camera and the lidar detector, and determine the conversion relationship between the camera coordinate system and the lidar coordinate system;

[0155] The data labeling unit 203 is used to: determine the three-dimensional envelope of each labeled target in the first set of labeled targets in the lidar coordinate system; obtain the two-dimensional labeling result of the first set of labeled targets in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system and the three-dimensional envelope; determine the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system according to the image data; determine the two-dimensional labeling result of the second set of labeled targets in the pixel coordinate system; and determine the three-dimensional labeling result of the second set of labeled targets in the laser point cloud coordinate system according to the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system and the two-dimensional labeling result of the second set of labeled targets.

[0156] The units included in the above data labeling device can achieve the same technical effects as those in the above embodiments. To avoid repetition, they will not be elaborated here.

[0157] Please refer to Figure 3 , Figure 3It is a schematic structural diagram of an electronic device for data annotation provided by an embodiment of the present application.

[0158] As Figure 3 shown, the electronic device 300 for data annotation in the embodiment of the present application includes a processor 301 and a memory 302. Among them, the memory stores code, and the processor is used to call the code stored in the memory to implement the execution of the method described in any of the above.

[0159] The units included in the above electronic device can achieve the same technical effects as those in the above embodiments. To avoid repetition, they will not be elaborated here.

[0160] In an embodiment of the present application, a computer-readable storage medium is further provided. The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the above data annotation method and can achieve the same technical effects. To avoid repetition, they will not be elaborated here. Among them, the computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc.

[0161] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for data annotation, characterized in that, The method includes: Obtaining image data obtained by a camera and point cloud data obtained by a lidar detector; wherein, the labeled objects in the image data include the labeled objects in the first labeled object set and the second labeled object set, the labeled objects in the first labeled object set are the labeled objects in the point cloud data, and the labeled objects in the second labeled object set have a greater detection distance than the labeled objects in the first labeled object set; Calibrating the camera and the lidar detector to determine the conversion relationship between the camera coordinate system and the lidar coordinate system; Determining the three-dimensional envelope of each labeled object in the first labeled object set in the lidar coordinate system; Obtaining the two-dimensional labeling result of the first labeled object set in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system and the three-dimensional envelope; Determining the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system according to the image data; Determining the two-dimensional labeling result of the second labeled object set in the pixel coordinate system; Determining the three-dimensional labeling result of the second labeled object set in the laser point cloud coordinate system according to the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system and the two-dimensional labeling result of the second labeled object set.

2. The method according to claim 1, characterized in that, The obtaining the two-dimensional labeling result of the first labeled object set in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system and the three-dimensional envelope includes: Determining the two-dimensional envelope corresponding to the three-dimensional envelope in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system; Determining the circumscribed quadrilateral of each two-dimensional envelope in the pixel coordinate system to obtain the two-dimensional labeling result of the first labeled object set.

3. The method according to claim 1, characterized in that, The determining the two-dimensional envelope corresponding to the three-dimensional envelope in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system includes: Determining the vertices of the three-dimensional envelope in the lidar coordinate system; Determining the two-dimensional coordinates corresponding to the vertices of the three-dimensional envelope in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system; Determining the two-dimensional envelope corresponding to the three-dimensional envelope in the pixel coordinate system according to the two-dimensional coordinates corresponding to the vertices of the three-dimensional envelope.

4. The method according to claim 1, characterized in that, After determining the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system according to the image data, it further includes: Determining the three-dimensional data corresponding to the two-dimensional labeling result of the first labeled object set in the lidar coordinate system; Determining the deviation between the three-dimensional envelope and the three-dimensional data; The determining the three-dimensional labeling result of the second labeled object set in the laser point cloud coordinate system according to the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system and the two-dimensional labeling result of the second labeled object set includes: Determining the first three-dimensional labeling result of the second labeled object set in the laser point cloud coordinate system according to the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system and the two-dimensional labeling result of the second labeled object set; Correcting the first three-dimensional labeling result according to the deviation to obtain the three-dimensional labeling result of the second labeled object set.

5. A device for data annotation, characterized in that, The device includes: A data acquisition unit for acquiring image data obtained by a camera and point cloud data obtained by a lidar detector; wherein, the labeled objects in the image data include the labeled objects in the first labeled object set and the second labeled object set, the labeled objects in the first labeled object set are the labeled objects in the point cloud data, and the labeled objects in the second labeled object set have a greater detection distance than the labeled objects in the first labeled object set; A detector calibration unit for: determining a calibrated camera and a lidar detector, and determining the conversion relationship between the camera coordinate system and the lidar coordinate system; A data annotation unit for: determining the three-dimensional envelope of each labeled object in the first labeled object set in the lidar coordinate system; obtaining the two-dimensional annotation result of the first labeled object set in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system and the three-dimensional envelope; determining the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system according to the image data; determining the two-dimensional annotation result of the second labeled object set in the pixel coordinate system; and determining the three-dimensional annotation result of the second labeled object set in the laser point cloud coordinate system according to the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system and the two-dimensional annotation result of the second labeled object set.

6. The device according to claim 5, characterized in that, The data annotation unit specifically is used for: Determining the two-dimensional envelope corresponding to the three-dimensional envelope in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system; Determining the circumscribed quadrilateral of each two-dimensional envelope in the pixel coordinate system to obtain the two-dimensional annotation result of the first labeled object set.

7. The device according to claim 5, characterized in that, The data annotation unit specifically is used for: Determining the vertices of the three-dimensional envelope in the lidar coordinate system; Determining the two-dimensional coordinates corresponding to the vertices of the three-dimensional envelope in the pixel coordinate system according to the conversion relationship between the camera coordinate system and the lidar coordinate system; Determining the two-dimensional envelope corresponding to the three-dimensional envelope in the pixel coordinate system according to the two-dimensional coordinates corresponding to the vertices of the three-dimensional envelope.

8. The device according to claim 5, characterized in that,The data annotation unit specifically is used for: Determining the three-dimensional data corresponding to the two-dimensional annotation result of the first labeled object set in the lidar coordinate system; Determining the deviation between the three-dimensional envelope and the three-dimensional data; Determining the first three-dimensional annotation result of the second labeled object set in the laser point cloud coordinate system according to the conversion relationship between the pixel coordinate system and the laser point cloud coordinate system and the two-dimensional annotation result of the second labeled object set; Correcting the first three-dimensional annotation result according to the deviation to obtain the three-dimensional annotation result of the second labeled object set.

9. An electronic device for data annotation, characterized in that, The electronic device includes a processor and a memory, wherein, the memory stores code, and the processor is used to call the code stored in the memory to implement the method according to any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1-4.

Images