Data labeling method and device, computer equipment and readable storage medium

Abstract

The invention relates to the technical field of data processing, in particular to a data labeling method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring a point cloud data frame acquired by point cloud acquisition equipment and an image frame acquired by image acquisition equipment for a target area; according to a first device parameter of the point cloud acquisition device and a second device parameter corresponding to the image acquisition device, determining a point cloud projection point corresponding to each point cloud data in the point cloud data frame in the image frame and a projection point position of the point cloud projection point; and if the projection point position is successfully matched with the corresponding pixel point position in the image frame, marking the point cloud projection point in the image frame.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data annotation method, apparatus, computer equipment and readable storage medium. Background Art

[0002] Perception training data is an important basis for autonomous driving perception algorithm training. The acquisition of training data is divided into two parts: data collection and data labeling. The method and effect of data labeling directly affect the algorithm training effect. A reasonable labeling method and system are the key to data labeling work and are also the key parts that need to be developed.

[0003] The perception data obtained by the lidar is in the form of point cloud, while the perception data obtained by the camera is in the form of image. How to standardize, streamline and unify the target annotation of the two to form an efficient annotation system is an urgent problem that needs to be solved. Summary of the invention

[0004] Based on this, it is necessary to provide a data labeling method, device, computer equipment and readable storage medium that can improve the accuracy of data labeling in response to the above technical problems.

[0005] In a first aspect, the present application provides a data annotation method, the method comprising:

[0006] Acquire a point cloud data frame collected by a point cloud acquisition device and an image frame collected by an image acquisition device for a target area;

[0007] Determine the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame according to the first device parameters of the point cloud acquisition device and the second device parameters corresponding to the image acquisition device;

[0008] If the projection point position matches the corresponding pixel point position in the image frame successfully, the point cloud projection point is marked in the image frame.

[0009] In one embodiment, determining the point cloud projection point and the projection point position of the point cloud projection point corresponding to each point cloud data in the point cloud data frame in the image frame according to the first device parameter of the point cloud acquisition device and the second device parameter corresponding to the image acquisition device includes:

[0010] Determine a coordinate system transformation relationship between a first coordinate system corresponding to the point cloud acquisition device and a second coordinate system corresponding to the image acquisition device according to a first device parameter of the point cloud acquisition device and a second device parameter corresponding to the image acquisition device;

[0011] According to the coordinate system transformation relationship and the point cloud coordinates of each point cloud data in the first coordinate system, the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame are determined.

[0012] In one of the embodiments, the first device parameter and the second device parameter both include device external parameters and device internal parameters; and the coordinate system transformation relationship includes a rotation transformation relationship and a translation transformation relationship.

[0013] In one of the embodiments, determining the point cloud projection point and the projection point position of the point cloud projection point corresponding to each point cloud data in the point cloud data frame in the image frame according to the coordinate system transformation relationship and the point cloud coordinates of each point cloud data in the first coordinate system includes:

[0014] According to the coordinate system transformation relationship, the point cloud coordinates of the point cloud data in the first coordinate system and the device internal parameters in the second device parameters, the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame are determined.

[0015] In one embodiment, the method further comprises:

[0016] Selecting a key area in the image frame and determining a region position corresponding to the key area;

[0017] Get the projection point position corresponding to the key area;

[0018] According to the distance measurement between the projection point position and the region position, it is determined whether the projection point position matches the corresponding pixel point position in the image frame successfully.

[0019] In one embodiment, determining whether the projection point position matches the corresponding pixel point position in the image frame successfully based on the distance measurement between the projection point position and the region position includes:

[0020] If the distance measure between the projection point position and the region position is less than or equal to the distance threshold, it is determined that the projection point position matches the corresponding pixel point position in the image frame successfully.

[0021] In a second aspect, the present application also provides a data labeling device, including:

[0022] An acquisition module, used to acquire point cloud data frames acquired by a point cloud acquisition device and image frames acquired by an image acquisition device for a target area;

[0023] A projection module, used to determine the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame according to the first device parameters of the point cloud acquisition device and the second device parameters corresponding to the image acquisition device;

[0024] The annotation module is used to annotate the point cloud projection point in the image frame if the projection point position matches the corresponding pixel point position in the image frame successfully.

[0025] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0026] Acquire a point cloud data frame collected by a point cloud acquisition device and an image frame collected by an image acquisition device for a target area;

[0027] Determine the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame according to the first device parameters of the point cloud acquisition device and the second device parameters corresponding to the image acquisition device;

[0028] If the projection point position matches the corresponding pixel point position in the image frame successfully, the point cloud projection point is marked in the image frame.

[0029] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0030] Acquire a point cloud data frame collected by a point cloud acquisition device and an image frame collected by an image acquisition device for a target area;

[0031] Determine the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame according to the first device parameters of the point cloud acquisition device and the second device parameters corresponding to the image acquisition device;

[0032] If the projection point position matches the corresponding pixel point position in the image frame successfully, the point cloud projection point is marked in the image frame.

[0033] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0034] Acquire a point cloud data frame collected by a point cloud acquisition device and an image frame collected by an image acquisition device for a target area;

[0035] Determine the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame according to the first device parameters of the point cloud acquisition device and the second device parameters corresponding to the image acquisition device;

[0036] If the projection point position matches the corresponding pixel point position in the image frame successfully, the point cloud projection point is marked in the image frame.

[0037] The above-mentioned data annotation method, device, computer device and readable storage medium realize the fusion of three-dimensional point cloud information and two-dimensional image information by combining the point cloud data frame with the image frame. This fusion enables the user to intuitively see the point cloud information in the three-dimensional space on the two-dimensional image, thereby enhancing the visualization and understanding of the data. By using the equipment parameters of the point cloud acquisition device and the image acquisition device, the point cloud data can be accurately projected onto the image frame to form point cloud projection points. This precise spatial positioning capability enables the position of the point cloud data to be accurately marked in the image frame, improving the accuracy and efficiency of the annotation. By marking the point cloud projection points in the image frame, the point cloud data can be conveniently further analyzed and processed. For example, specific objects or features in the target area can be quickly identified, or point cloud data can be classified, segmented, and other operations can be performed. This annotation method simplifies the data analysis process and improves processing efficiency. The technical solution can be widely used in various scenarios that require the combination of three-dimensional space information and two-dimensional image information, such as autonomous driving, robot navigation, three-dimensional reconstruction, remote sensing mapping and other fields. By fusing point cloud data and image data, more comprehensive and accurate information support can be provided for these fields, thereby enhancing the value and effect of the application. The robustness and reliability of the entire system can be improved through precise calibration of equipment parameters and accurate matching of point cloud data and image data. Even under complex environmental conditions, data can be stably acquired and processed to ensure the normal operation and accuracy of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] FIG1 is a schematic diagram of a flow chart of a data annotation method in one embodiment;

[0040] 2 is a flow chart of the steps of determining the point cloud projection points corresponding to each point cloud data in the point cloud data frame in the image frame and the projection point positions of the point cloud projection points in one embodiment;

[0041] FIG3 is a schematic flow chart of a step of determining an area position corresponding to a key area in one embodiment;

[0042] FIG4 is a structural block diagram of a data labeling device in one embodiment;

[0043] FIG. 5 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] In an exemplary embodiment, as shown in FIG1 , a data annotation method is provided, which is described by taking the method applied to a computer device as an example, and includes the following S101 to S103. Among them:

[0046] S101, acquiring a point cloud data frame acquired by a point cloud acquisition device and an image frame acquired by an image acquisition device for a target area.

[0047] Among them, select appropriate point cloud acquisition equipment, such as laser radar (LiDAR), 3D scanner or depth camera. These devices can emit laser or light pulses and receive the reflected signals to generate 3D point cloud data of the target area. Before collecting point cloud data, it is necessary to set the parameters of the point cloud acquisition equipment, including scanning range, resolution, acquisition frequency, etc. The setting of these parameters will directly affect the quality and accuracy of the point cloud data. Start the point cloud acquisition device and scan the target area. The device will emit laser or light pulses and receive the reflected signals, and generate point cloud data frames by processing these signals. The point cloud data frame contains the 3D coordinate information of each point in the target area.

[0048] Among them, select an appropriate image acquisition device, such as a camera, a camcorder or a smartphone camera. These devices can capture two-dimensional image information of the target area. Before acquiring the image, it is necessary to set the parameters of the image acquisition device, including focal length, exposure time, white balance, etc. The setting of these parameters will affect the clarity and color reproduction of the image. In order to ensure the spatiotemporal consistency between the point cloud data frame and the image frame, it is necessary to start the point cloud acquisition device and the image acquisition device for data acquisition at the same time or nearly at the same time. This can be achieved by using a synchronization trigger or timestamp. Start the image acquisition device and shoot the target area. The device will capture the two-dimensional image information of the target area and generate an image frame.

[0049] S102, determining the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame according to the first device parameters of the point cloud acquisition device and the second device parameters corresponding to the image acquisition device.

[0050] Among them, the first device parameters of the point cloud acquisition device refer to the internal parameters and external parameters of the point cloud acquisition device; the second device parameters corresponding to the image acquisition device refer to the internal parameters and external parameters of the point cloud acquisition device of the image acquisition device.

[0051] Take the point cloud acquisition device as an example:

[0052] Intrinsic parameters are parameters inherent to the point cloud acquisition device that describe its internal geometric and optical properties. These parameters are crucial for mapping points in three-dimensional space to a two-dimensional image plane. Intrinsic parameters usually include:

[0053] 1) Focal Length: A parameter that describes the lens's ability to converge light, affecting the size and clarity of the image.

[0054] 2) Principal Point: A point on the image plane, usually located near the center of the image, where the optical axis intersects the image plane.

[0055] 3) Distortion Coefficients: Parameters that describe lens distortion (such as radial distortion and tangential distortion), which affect the geometric accuracy of the image.

[0056] 4) Pixel Size: The physical size of a single pixel on an image sensor, which affects the resolution and accuracy of the image.

[0057] Furthermore, external parameters describe the relative position and posture between the point cloud acquisition device and a global coordinate system (such as the world coordinate system) or the image acquisition device coordinate system. These parameters usually include a rotation matrix (RotationMatrix) and a translation vector (Translation Vector):

[0058] Rotation matrix (R): A 3x3 matrix that describes the rotation angle of the device coordinate system relative to the reference coordinate system (such as the image acquisition device coordinate system).

[0059] Translation vector (T): A 3x1 vector that describes the translation distance of the origin of the device coordinate system relative to the origin of the reference coordinate system.

[0060] During the projection process, first, the 3D points in the point cloud data frame are transformed from the point cloud acquisition device coordinate system to the image acquisition device coordinate system using external parameters (rotation matrix and translation vector). Then, the transformed 3D points are projected onto the 2D image plane of the image frame using internal parameters (focal length, principal point, distortion coefficient, etc.). Finally, the projection points are corrected for distortion according to the distortion coefficient to eliminate the influence of lens distortion on the projection point position. Through this projection process, the point cloud projection points and their positions corresponding to each point cloud data in the point cloud data frame can be accurately determined in the image frame, providing a basis for subsequent data fusion, annotation, and analysis.

[0061] S103: If the position of the projection point successfully matches the position of the corresponding pixel point in the image frame, the projection point of the point cloud is marked in the image frame.

[0062] First, make sure that the coordinates of the projected points of the point cloud are expressed in the same coordinate system as the pixel coordinates of the image frame. This usually involves converting the coordinates of the projected points (which may be actual physical coordinates in meters or millimeters) into pixel coordinates (in pixels). Use an appropriate matching algorithm to compare the position of the projected points with the position of the corresponding pixel points in the image frame. This can be distance-based matching (such as nearest neighbor matching), feature-based matching (such as shape, color or texture features), or a combination of multiple methods. Set a matching threshold to determine the degree of match between the projected points and the pixel points. If the degree of match exceeds this threshold, the match is considered successful.

[0063] Select the appropriate annotation type based on the application requirements. For example, it can be a simple point annotation, a box annotation (such as a rectangular box or an ellipse box), or a more complex polygon annotation. Annotate the successfully matched projection points in the image frame and add the necessary annotation information. This can include the three-dimensional coordinates, color, size, category label, etc. of the point. Make sure that the annotation is clearly visible in the image frame for easy observation and analysis by the user. Different colors, shapes, or line widths can be used to distinguish different annotation types or categories. Store the annotated image frame as an image file or video stream for subsequent processing, analysis, or presentation. At the same time, the annotation information can also be stored or transmitted together with the image frame in the form of metadata.

[0064] In the above data annotation method, by combining the point cloud data frame with the image frame, the fusion of three-dimensional point cloud information and two-dimensional image information is realized. This fusion enables the user to intuitively see the point cloud information in the three-dimensional space on the two-dimensional image, thereby enhancing the visualization and understanding of the data. By using the equipment parameters of the point cloud acquisition device and the image acquisition device, the point cloud data can be accurately projected onto the image frame to form point cloud projection points. This precise spatial positioning capability enables the position of the point cloud data to be accurately marked in the image frame, improving the accuracy and efficiency of the annotation. By marking the point cloud projection points in the image frame, the point cloud data can be further analyzed and processed conveniently. For example, specific objects or features in the target area can be quickly identified, or point cloud data can be classified, segmented, and other operations can be performed. This annotation method simplifies the data analysis process and improves processing efficiency. The technical solution can be widely used in various scenarios that require the combination of three-dimensional space information and two-dimensional image information, such as autonomous driving, robot navigation, three-dimensional reconstruction, remote sensing mapping, and other fields. By fusing point cloud data and image data, more comprehensive and accurate information support can be provided for these fields, thereby enhancing the value and effect of the application. Through the precise calibration of equipment parameters and the accurate matching of point cloud data and image data, the robustness and reliability of the entire system can be improved. Even under complex environmental conditions, data can be stably acquired and processed to ensure the normal operation and accuracy of the system.

[0065] In an exemplary embodiment, as shown in FIG2 , determining the point cloud projection point and the projection point position of the point cloud projection point corresponding to each point cloud data in the point cloud data frame in the image frame according to the second device parameter corresponding to the first image acquisition device of the point cloud acquisition device includes:

[0066] S201, determining a coordinate system transformation relationship between a first coordinate system corresponding to the point cloud acquisition device and a second coordinate system corresponding to the image acquisition device according to first device parameters of the point cloud acquisition device and second device parameters corresponding to the image acquisition device.

[0067] Among them, the first device parameters of the point cloud acquisition device include internal parameters (such as focal length, principal point, distortion coefficient, etc.) and external parameters (such as rotation matrix, translation vector, etc.). The second device parameters of the image acquisition device usually involve the camera's intrinsic parameter matrix (including focal length, principal and standby parameters, image points, etc.) and extrinsic parameter matrix (describing the relationship between the camera and the world coordinate system).

[0068] Furthermore, the first coordinate system refers to the coordinate system associated with the point cloud acquisition device, which is usually the local coordinate system of the device itself. The second coordinate system refers to the coordinate system associated with the image acquisition device, which can be a camera coordinate system or an image coordinate system (pixel coordinate system). Using the external parameters of the point cloud acquisition device and the image acquisition device, the transformation relationship between the two coordinate systems is calculated. This usually involves the application of a rotation matrix and a translation vector to transform points from the first coordinate system to the second coordinate system.

[0069] S202, determining the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame according to the coordinate system transformation relationship and the point cloud coordinates of each point cloud data in the first coordinate system.

[0070] Specifically, for each point in the point cloud data frame, a coordinate system transformation relationship is applied to transform it from a first coordinate system to a second coordinate system, which generally involves matrix multiplication and vector addition operations.

[0071] Rotation: Use the rotation matrix to adjust the coordinate direction of the point to be consistent with the second coordinate system.

[0072] Translation: Use a translation vector to move the position of a point closer to the origin of the second coordinate system.

[0073] Perspective projection: A perspective projection calculation is performed in a second coordinate system using the internal parameters of the image acquisition device (such as focal length, principal point, etc.). This usually involves mapping 3D points onto a 2D image plane, taking into account the perspective effect of the lens.

[0074] Output projection point position: Calculate the projection point position of each point in the image frame, usually expressed in pixel coordinates. These projection point positions can be used for subsequent annotation, analysis or processing.

[0075] Among them, according to the coordinate system transformation relationship and the point cloud coordinates of each point cloud data in the first coordinate system, the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame are determined, including: according to the coordinate system transformation relationship, the point cloud coordinates of the point cloud data in the first coordinate system and the device internal parameters in the second device parameters, the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame are determined.

[0076] According to the external parameters of the point cloud acquisition device and the external parameters of each image acquisition device, the rotation matrix R from the laser radar coordinate system to the image acquisition device coordinate system under each image acquisition device is calculated. 3×3 , translation T 3×1 ;

[0077] Further, the three-dimensional coordinates of the laser radar point cloud are calculated in the coordinate system of the corresponding image acquisition device;

[0078] Afterwards, according to the image acquisition device internal parameter matrix corresponding to each image acquisition device, the pixel positions of the valid points in the point cloud in the image acquisition device coordinate system in its corresponding image are calculated in sequence by matrix multiplication.

[0079] The overall homogeneous calculation process is as follows:

[0080]

[0081] in, is the homogeneous form of the coordinates of the original point in the point cloud, R and T are the rotation matrix and translation from the laser radar coordinate system to the image acquisition device coordinate system under a certain image acquisition device, and P is the intrinsic parameter matrix of the image acquisition device. Finally, the pixel position of the point in the point cloud in the image is:

[0082]

[0083] Among them, Y width , Y height Indicates the horizontal and vertical pixel positions of the point in the point cloud projected on the image, Y i , i=1, 2, 3 represent the components of Y. Since the field of view of the point cloud is significantly different from that of the image acquisition device, the point cloud outside the field of view of the image acquisition device needs to be eliminated, and the points in the point cloud that do not meet the following conditions need to be eliminated:

[0084]

[0085] Among them, Y C Represents the coordinate homogeneous form of the point in the point cloud in the coordinate system of the image acquisition device corresponding to the image acquisition device, Y C3 It is its first component, ImageWight and ImageHeight represent the image width and height corresponding to the image acquisition device.

[0086] Furthermore, according to the projection position of the point cloud on the image, the point cloud data is drawn on the projected image frame in the form of 2D points, and the projection points are colored during the drawing process based on the forward vertical distance of the point cloud from the target projection image acquisition device, which facilitates the subsequent internal and external parameters of the sensor and synchronization verification.

[0087] In an exemplary embodiment, as shown in FIG3 , selecting a key area in an image frame and determining a region position corresponding to the key area includes:

[0088] S301, obtaining the projection point position corresponding to the key area.

[0089] Among them, one or more key areas are defined in the image frame. These areas may be based on the importance of the image content, the presence of specific objects, or user-specified areas of interest. In this embodiment, the key areas include but are not limited to image segmentation areas such as the ground, landmark obstacles, and buildings.

[0090] S302, determining whether the projection point position matches the corresponding pixel point position in the image frame successfully based on the distance measurement between the projection point position and the region position.

[0091] Specifically, based on the distance measurement between the projection point position and the regional position, it is determined whether the projection point position and the corresponding pixel point position in the image frame are matched successfully, including: if the distance measurement between the projection point position and the regional position is less than or equal to the distance threshold, it is determined that the projection point position and the corresponding pixel point position in the image frame are matched successfully.

[0092] Exemplarily, randomly select edge point cloud projection points corresponding to the key area, calculate the average horizontal distance from these projection points to the boundary of the image segmentation area, and then set a threshold to determine whether the projection effect meets the requirements. If so, it is determined that the internal and external parameters and synchronization conditions of the sensor meet the calibration requirements; otherwise, it is not satisfied, and consider adjusting the parameters or abandoning them. The point cloud frames and images to be annotated that have completed pre-marking verification are input into the annotation module. Annotate the targets presented in the point cloud according to the annotation identification number, category, status, visibility, three-dimensional bounding box and orientation requirements and label format requirements determined by the data requirements, and the annotation is completed in the original coordinate system of the point cloud, that is, the lidar coordinate system. The annotation method is as follows:

[0093] 1. According to the target object labeling, after all target objects are labeled in a scene consisting of continuous frames of the point cloud (about 20 seconds), the next object will be labeled.

[0094] 2. After opening a scene, browse it first, and then start to select the target object for annotation. Usually the data collected is 10 frames per second, and the interval between each frame is 100ms. Due to the short interval time and the small scene change, linear interpolation is performed based on the annotation results of 2 frames per second to complete the annotation of the remaining 8 frames.

[0095] 3. When marking the target object, start with the frame with the best view in the scene, such as a frame with a close distance and no obstruction. In this case, the size and direction of the object can be marked more accurately. After the image-assisted marking, you can use the copy / paste method to migrate the 3D bounding box to other frames in the scene. After adjusting the remaining two frames per second in the scene, you can use interpolation to mark other frames.

[0096] 4. The batch interpolation labeling algorithm may fail, such as not being able to find the object (tracking lost, etc.), incorrect direction, etc. At this time, manual adjustments are made to adjust the 3D bounding boxes of some target objects and then try to use the interpolation algorithm for batch labeling, repeating the above steps. Finally, delete the 3D bounding boxes of objects that do not exist (occluded, target objects disappeared, etc.).

[0097] 5. Check the trajectory of the target object in the entire scene. If there is any abnormality (such as a large change in direction, etc.), confirm it again.

[0098] 6. For small objects, such as people, it is not easy to distinguish the direction during the labeling process. According to the direction of their movement, the direction relative to the building / road surface, etc., auxiliary judgment is made. If it is difficult to track in crowded scenes, you can switch to 10Hz data for identification and labeling. The 10Hz data and 2Hz data share the labeling results, and switch back to 2Hz after the identification is completed.

[0099] 7. Save the labeling results according to the required label format requirements.

[0100] 8. Further, the obtained annotation results are verified by image projection. The 8 vertices of the three-dimensional rectangular box are projected and drawn onto the image corresponding to each camera according to the S2 method. According to the example in Figure 3, line segments are drawn on the image in the order of points 1-2-3-4, 5-6-7-8, 1-5, 2-6, 3-7, and 4-8. Finally, the bounding box generated on the image can completely surround the corresponding target to verify the point cloud annotation results. If there is no error, the label file is stored to complete the annotation. If there is an error, re-annotate.

[0101] In this embodiment, this implementation takes into account the changes in sensor external parameters caused by vehicle vibration and other reasons during the acquisition process, and then introduces a method and process of projecting the point cloud to the camera to verify the synchronization conditions of the sensor's internal and external parameters, which is the key technical point of this application.

[0102] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0103] Based on the same inventive concept, the embodiment of the present application also provides a data labeling device for implementing the data labeling method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data labeling device embodiments provided below can refer to the limitations on the data labeling method above, and will not be repeated here.

[0104] In an exemplary embodiment, as shown in FIG4 , a data annotation device is provided, including: an acquisition module 11, a projection module 12 and an annotation module 13, wherein:

[0105] An acquisition module 11 is used to acquire a point cloud data frame acquired by a point cloud acquisition device and an image frame acquired by an image acquisition device for a target area;

[0106] The projection module 12 is used to determine the point cloud projection point corresponding to each point cloud data in the point cloud data frame and the projection point position of the point cloud projection point in the image frame according to the first device parameter of the point cloud acquisition device and the second device parameter corresponding to the image acquisition device;

[0107] The marking module 13 is used to mark the point cloud projection point in the image frame if the projection point position matches the corresponding pixel point position in the image frame successfully.

[0108] Each module in the above data annotation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0109] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data of a data annotation method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data annotation method is implemented.

[0110] Those skilled in the art will understand that the structure shown in FIG. 5 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0111] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0112] Acquire a point cloud data frame collected by a point cloud acquisition device and an image frame collected by an image acquisition device for a target area;

[0113] Determine the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame according to the first device parameters of the point cloud acquisition device and the second device parameters corresponding to the image acquisition device;

[0114] If the projection point position matches the corresponding pixel point position in the image frame successfully, the point cloud projection point is marked in the image frame.

[0115] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0116] Acquire a point cloud data frame collected by a point cloud acquisition device and an image frame collected by an image acquisition device for a target area;

[0117] Determine the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame according to the first device parameters of the point cloud acquisition device and the second device parameters corresponding to the image acquisition device;

[0118] If the projection point position matches the corresponding pixel point position in the image frame successfully, the point cloud projection point is marked in the image frame.

[0119] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0120] Acquire a point cloud data frame collected by a point cloud acquisition device and an image frame collected by an image acquisition device for a target area;

[0121] Determine the point cloud projection points corresponding to each point cloud data in the point cloud data frame and the projection point positions of the point cloud projection points in the image frame according to the first device parameters of the point cloud acquisition device and the second device parameters corresponding to the image acquisition device;

[0122] If the projection point position matches the corresponding pixel point position in the image frame successfully, the point cloud projection point is marked in the image frame.

[0123] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0124] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0125] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data labeling method, characterized in that: The method comprises: Acquire a point cloud data frame collected by a point cloud acquisition device and an image frame collected by an image acquisition device for a target area; Determine, according to a first device parameter of a point cloud acquisition device and a second device parameter corresponding to the image acquisition device, a point cloud projection point corresponding to each point cloud data in the point cloud data frame in the image frame and a projection point position of the point cloud projection point; If the position of the projection point successfully matches the position of the corresponding pixel point in the image frame, the point cloud projection point is marked in the image frame.

2. The method according to claim 1, characterized in that The step of determining the point cloud projection points corresponding to each point cloud data in the point cloud data frame in the image frame and the projection point positions of the point cloud projection points according to the first device parameters of the point cloud acquisition device and the second device parameters corresponding to the image acquisition device comprises: Determine, according to a first device parameter of a point cloud acquisition device and a second device parameter corresponding to the image acquisition device, a coordinate system transformation relationship between a first coordinate system corresponding to the point cloud acquisition device and a second coordinate system corresponding to the image acquisition device; According to the coordinate system transformation relationship and the point cloud coordinates of each point cloud data in the first coordinate system, the point cloud projection point corresponding to each point cloud data in the point cloud data frame in the image frame and the projection point position of the point cloud projection point are determined.

3. The method according to claim 2, characterized in that The first device parameter and the second device parameter both include device external parameters and device internal parameters; the coordinate system transformation relationship includes a rotation transformation relationship and a translation transformation relationship.

4. The method according to claim 3, characterized in that The step of determining the point cloud projection point corresponding to each point cloud data in the point cloud data frame in the image frame and the projection point position of the point cloud projection point according to the coordinate system transformation relationship and the point cloud coordinates of each point cloud data in the first coordinate system comprises: According to the coordinate system transformation relationship, the point cloud coordinates of the point cloud data in the first coordinate system and the device internal parameters in the second device parameters, determine the point cloud projection points corresponding to each point cloud data in the point cloud data frame in the image frame and the projection point positions of the point cloud projection points.

5. The method according to claim 3, characterized in that: The method further comprises: Selecting a key area in the image frame, and determining an area position corresponding to the key area; Obtaining the projection point position corresponding to the key area; According to the distance measurement between the projection point position and the area position, it is determined whether the projection point position matches the corresponding pixel point position in the image frame successfully.

6. The method according to claim 5, characterized in that The determining, based on the distance measurement between the projection point position and the region position, whether the projection point position matches the corresponding pixel point position in the image frame successfully comprises: If the distance metric between the projection point position and the region position is less than or equal to a distance threshold, it is determined that the projection point position matches the corresponding pixel point position in the image frame successfully.

7. A data labeling device, characterized in that: The device comprises: An acquisition module, used to acquire point cloud data frames acquired by a point cloud acquisition device and image frames acquired by an image acquisition device for a target area; A projection module, used to determine the point cloud projection points corresponding to each point cloud data in the point cloud data frame in the image frame and the projection point positions of the point cloud projection points according to the first device parameters of the point cloud acquisition device and the second device parameters corresponding to the image acquisition device; The marking module is used to mark the point cloud projection point in the image frame if the projection point position matches the corresponding pixel point position in the image frame successfully.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images