Point cloud data annotation method, device and system

Abstract

The present application provides a point cloud data annotation method, device, and system, relating to the field of computer application technology. It can effectively solve the problem of inaccurate annotation of target objects when annotation is performed based solely on point cloud data. The method includes: obtaining initial point cloud data and two-dimensional pixel data of a first image corresponding to the initial point cloud data. At the same time, a first selection box and a second selection box are obtained, the first selection box including the target object in the two-dimensional pixel data, and the second selection box including the target object in the initial point cloud data. Thereafter, the annotation information of the target object in the initial point cloud data is determined based on the first selection box and the second selection box. The embodiments of the present application are applied to a computer system.

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and in particular to a point cloud data annotation method, device and system. Background Art

[0002] With the development of intelligent measurement technology, identifying target objects in the surrounding environment has become a key technology in engineering measurement applications. To detect target objects in the surrounding environment, most existing methods directly annotate the point cloud data of the environment acquired by LiDAR to obtain the target object's point cloud data annotation results, and then identify the target object. However, this method, which only annotates the point cloud data, may have the problem of inaccurate target object annotation. Summary of the Invention

[0003] The present application provides a point cloud data annotation method, device and system, which can effectively solve the problem of inaccurate annotation of target objects when annotation is performed based solely on point cloud data.

[0004] To achieve the above objectives, this application adopts the following technical solutions:

[0005] In a first aspect, the present application provides a point cloud data annotation method, comprising obtaining initial point cloud data and two-dimensional pixel data of a first image corresponding to the initial point cloud data. Simultaneously, obtaining a first selection box and a second selection box, wherein the first selection box includes a target object in the two-dimensional pixel data, and the second selection box includes the target object in the initial point cloud data. Subsequently, determining annotation information for the target object in the initial point cloud data based on the first selection box and the second selection box.

[0006] In this method, by combining 2D pixel data with point cloud data to determine the target object's annotation information, this method can address the potential for inaccurate object annotation based solely on point cloud data. It also improves the quality of object annotation and recognition rate within point cloud data.

[0007] Optionally, determining the annotation information of the target object in the initial point cloud data based on the first selected box and the second selected box includes: determining the first point cloud data, wherein the first point cloud data corresponds to the object in the first selected box; determining the second point cloud data in the first point cloud data, wherein the second point cloud data corresponds to the second selected box; determining a two-dimensional box corresponding to the second point cloud data; and determining the annotation information of the target object in the initial point cloud data based on the first selected box and the two-dimensional box.

[0008] Optionally, determining the first point cloud data includes: converting the initial point cloud data into initial two-dimensional data according to a preset coordinate transformation relationship; determining first two-dimensional data in the initial two-dimensional data; the first two-dimensional data being the two-dimensional data within a first selected box in the initial two-dimensional data; and determining the point cloud data corresponding to the first two-dimensional data as the first point cloud data.

[0009] Optionally, determining the two-dimensional frame corresponding to the second point cloud data includes: determining second two-dimensional data corresponding to the second point cloud data according to a preset coordinate transformation relationship, and determining the two-dimensional frame according to the second two-dimensional data.

[0010] Optionally, a two-dimensional frame is determined based on the second two-dimensional data, including: determining the maximum value u1 and the minimum value u2 of the first coordinate axis data in the pixel coordinate system, and the maximum value v1 and the minimum value v2 of the second coordinate axis data in the pixel coordinate system in the second two-dimensional data; and determining a rectangle formed with coordinates (u1, v1) and coordinates (u2, v2) as diagonal vertices as a two-dimensional frame.

[0011] Optionally, the annotation information of the target object in the point cloud data is determined based on the first selected box and the two-dimensional box, including: when the degree of overlap between the two-dimensional box and the first selected box is greater than or equal to a preset threshold, determining that the second point cloud data is the annotation information of the target object.

[0012] Optionally, based on the first selected box and the two-dimensional box, the annotation information of the target object in the point cloud data is determined, including: when the overlap between the two-dimensional box and the first selected box is less than a preset threshold, determining the external bounding box; the external bounding box is the minimum external bounding box that includes the first selected box and the two-dimensional box; and determining the point cloud data corresponding to the two-dimensional data in the external bounding box as the annotation information corresponding to the target.

[0013] Optionally, the first selected box contains the contour line of the target object in the pixel coordinate system; the second selected box contains the contour line of the target object in the lidar coordinate system.

[0014] Optionally, a second image is acquired; the second image is the next frame image of the first image; when it is determined that the position change data of the target object in the first image in the second image is within a preset range, the labeling information of the target object in the second image is determined based on the labeling information of the target object in the first image.

[0015] Optionally, obtaining initial point cloud data includes: obtaining original point cloud data corresponding to a first image; obtaining a third image when it is determined that the point cloud data of the target object in the original point cloud data is less than or equal to a first threshold; the third image is an image of the previous frame of the first image; obtaining point cloud data corresponding to the third image when it is determined that the position change data of the target object in the third image in the first image is within a preset range; and determining the initial point cloud data based on the point cloud data corresponding to the third image.

[0016] In a second aspect, the present application provides a point cloud data annotation device, comprising an acquisition unit and a processing unit. The acquisition unit is configured to acquire initial point cloud data and two-dimensional pixel data of a first image corresponding to the initial point cloud data. The acquisition unit is further configured to acquire a first selection box and a second selection box, wherein the first selection box includes a target object in the two-dimensional pixel data, and the second selection box includes the target object in the initial point cloud data. The processing unit is configured to determine annotation information of the target object in the initial point cloud data based on the first selection box and the second selection box.

[0017] Optionally, the processing unit is specifically configured to determine first point cloud data, the first point cloud data corresponding to an object within a first selected box. The processing unit is further configured to determine second point cloud data within the first point cloud data, the second point cloud data corresponding to a second selected box. The processing unit is further configured to determine a two-dimensional box corresponding to the second point cloud data. The processing unit is further configured to determine annotation information of the target object within the initial point cloud data based on the first selected box and the two-dimensional box.

[0018] Optionally, the processing unit is specifically configured to convert the initial point cloud data into initial two-dimensional data according to a preset coordinate transformation relationship. The processing unit is further configured to determine first two-dimensional data within the initial two-dimensional data; the first two-dimensional data is the two-dimensional data within a first selected box within the initial two-dimensional data. The processing unit is further configured to determine the point cloud data corresponding to the first two-dimensional data as the first point cloud data.

[0019] Optionally, the processing unit is specifically configured to determine second two-dimensional data corresponding to the second point cloud data based on a preset coordinate transformation relationship. The processing unit is further configured to determine a two-dimensional frame based on the second two-dimensional data;

[0020] Optionally, the processing unit is specifically configured to determine, in the second two-dimensional data, a maximum value u1 and a minimum value u2 of the first coordinate axis data in the pixel coordinate system, and a maximum value v1 and a minimum value v2 of the second coordinate axis data in the pixel coordinate system. The processing unit is further configured to determine a rectangle formed by the coordinates (u1, v1) and the coordinates (u2, v2) as diagonal vertices as a two-dimensional box.

[0021] Optionally, the processing unit is specifically configured to determine that the second point cloud data is annotation information of the target object when a degree of overlap between the two-dimensional box and the first selected box is greater than or equal to a preset threshold.

[0022] Optionally, the processing unit is specifically configured to determine a bounding box when the overlap between the two-dimensional box and the first selected box is less than a preset threshold; the bounding box is a minimum bounding box that includes the first selected box and the two-dimensional box. The processing unit is further configured to determine point cloud data corresponding to the two-dimensional data in the bounding box as annotation information corresponding to the target.

[0023] Optionally, the first selected box contains the contour line of the target object in the pixel coordinate system; the second selected box contains the contour line of the target object in the lidar coordinate system.

[0024] Optionally, the acquisition unit is further configured to acquire a second image, the second image being a frame image subsequent to the first image. The processing unit is configured to, upon determining that position change data of the target object in the first image acquired by the acquisition unit in the second image is within a preset range, determine the labeling information of the target object in the second image based on the labeling information of the target object in the first image.

[0025] Optionally, the acquisition unit is specifically used to acquire original point cloud data corresponding to the first image. The processing unit is used to determine that the point cloud data of the target object in the original point cloud data acquired by the acquisition unit is less than or equal to a first threshold. The acquisition unit is used to acquire a third image when the processing unit determines that the point cloud data of the target object in the original point cloud data is less than or equal to the first threshold; the third image is the previous frame image of the first image. The processing unit is used to determine that the position change data of the target object in the third image in the first image is within a preset range. The acquisition unit is used to acquire point cloud data corresponding to the third image when the processing unit determines that the position change data of the target object in the third image in the first image is within a preset range. The processing unit is used to determine initial point cloud data based on the point cloud data corresponding to the third image acquired by the acquisition unit.

[0026] In a third aspect, the present application provides a point cloud data annotation device, comprising a memory and a processor. The memory and processor are coupled. The memory is configured to store computer program code, the computer program code comprising computer instructions. When the processor executes the computer instructions, the point cloud data annotation device performs the point cloud data annotation method provided in the first aspect or any possible design embodiment of the first aspect.

[0027] In a fourth aspect, the present application provides a point cloud data annotation system, comprising: a laser radar, a point cloud data annotation device, and an image acquisition device. The laser radar is configured to acquire raw point cloud data, and the image acquisition device is configured to acquire images corresponding to the raw point cloud data. The point cloud data annotation device is configured to perform the point cloud data annotation method of the first aspect.

[0028] In a fifth aspect, the present application provides a chip system for use in a point cloud data annotation device; the chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via circuits; the interface circuits are configured to receive signals from the memory of the point cloud data annotation device and send signals to the processors, the signals including computer instructions stored in the memory. When the processors execute the computer instructions, the point cloud data annotation device executes the point cloud data annotation method provided in the first aspect or any possible design of the point cloud data annotation method provided in the first aspect.

[0029] In a sixth aspect, the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are run on a point cloud data annotation device, the point cloud data annotation device implements the point cloud data annotation method provided in the first aspect or the point cloud data annotation method described in any possible design method in the first aspect.

[0030] In the seventh aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are run on a point cloud data labeling device, the point cloud data labeling device executes the point cloud data labeling method provided in the first aspect or the point cloud data labeling method described in any possible design method in the first aspect.

[0031] It should be noted that the above-mentioned computer instructions may be stored in whole or in part on a computer-readable storage medium. The computer-readable storage medium may be packaged together with the processor of the point cloud data annotation device, or may be packaged separately from the processor of the point cloud data annotation device, and this application does not limit this.

[0032] The description of the second, third, fourth, fifth, sixth and seventh aspects in this application can refer to the detailed description of the first aspect and its various implementations; and the beneficial effects of the second, third, fourth, fifth, sixth and seventh aspects can refer to the analysis of the beneficial effects of the first aspect and its various implementations, which will not be repeated here.

[0033] In this application, the name of the point cloud data annotation device does not limit the device or functional modules themselves. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalents.

[0034] These and other aspects of the present application will become more readily apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1A schematic diagram of the structure of a point cloud data annotation system provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of the hardware structure of a communication device provided in an embodiment of the present application;

[0037] Figure 3 One of the flow diagrams of a point cloud data annotation method provided in an embodiment of the present application;

[0038] Figure 4 This is one of the schematic diagrams of the effect of a two-dimensional annotation frame provided in an embodiment of the present application;

[0039] Figure 5 This is one of the schematic diagrams of the effect of a three-dimensional annotation frame provided in an embodiment of the present application;

[0040] Figure 6 The second flowchart of a point cloud data annotation method provided in an embodiment of the present application;

[0041] Figure 7 This is a second schematic diagram of the effect of a three-dimensional annotation frame provided in an embodiment of the present application;

[0042] Figure 8 A schematic diagram of the projection relationship between camera coordinates and image coordinates provided in an embodiment of the present application;

[0043] Figure 9 A schematic diagram of the relationship between an image coordinate system and a pixel coordinate system provided in an embodiment of the present application;

[0044] Figure 10 This is a second schematic diagram of the effect of a three-dimensional annotation frame provided in an embodiment of the present application;

[0045] Figure 11 The third flowchart of a point cloud data annotation method provided in an embodiment of the present application;

[0046] Figure 12 A fourth flowchart of a point cloud data annotation method provided in an embodiment of the present application;

[0047] Figure 13 A schematic structural diagram of a point cloud data annotation device provided in an embodiment of the present application;

[0048] Figure 14 A schematic diagram of the structure of a computer program product for the point cloud data annotation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following describes in detail a point cloud data annotation method, device, and storage medium provided by an embodiment of the present application in conjunction with the accompanying drawings.

[0050] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0051] The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0052] Furthermore, the terms "including," "having," and any variations thereof, as used in the description of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules is not limited to the listed steps or modules, but may optionally include other steps or modules not listed, or may optionally include other steps or modules inherent to the process, method, product, or apparatus.

[0053] It should be noted that in the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in the embodiments of this application as "exemplarily" or "for example" should not be interpreted as being more preferred or advantageous than other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0054] In the description of the present application, unless otherwise specified, “plurality” means two or more.

[0055] It is understandable that in actual applications, the order of steps of the point cloud data annotation method provided in the embodiment of the present application can be adjusted, and the embodiment of the present application does not limit this.

[0056] Existing technologies for detecting target objects in the surrounding environment mostly label the point cloud data of the surrounding environment acquired by LiDAR to obtain the target object's point cloud data labeling results. However, when the point cloud data is scarce, there is a risk of missing the target object.

[0057] In view of this, the present invention provides a point cloud data annotation method that combines two-dimensional pixel data and point cloud data to comprehensively determine the annotation information of the target object, thereby greatly preventing the occurrence of missed annotations of the target object.

[0058] The point cloud data annotation method provided in the embodiment of the present application can be applied to the point cloud data annotation system. Figure 1The present invention provides a possible structure of a point cloud data annotation system, comprising: a laser radar 10, an image acquisition device 20, and a point cloud data annotation device 30. The laser radar 10 is used to acquire point cloud data. The data acquired by the image acquisition device 20 may be an image or video containing a target object. In the case of a video, a single frame of the video may be captured as a basis for data analysis and processing by the point cloud data annotation device 30. Alternatively, multiple images of the video may be captured, and one image may be selected from the captured images as a basis for data analysis and processing by the point cloud data annotation device 30.

[0059] In one possible implementation, Figure 1 As shown, the laser radar 10 is in communication with the point cloud data annotation device 30. The image acquisition device 20 is in communication with the point cloud data annotation device 30. The laser radar 10 and the image acquisition device 20 respectively send their collected data to the point cloud data annotation device 30.

[0060] In another possible implementation, the laser radar 10 can be connected to an image acquisition device 20. In actual applications, based on the functions of the laser radar 10 and the image acquisition device 20, one of the devices is selected to communicate with the image acquisition device 20. The data collected by both devices is then uploaded to the point cloud data annotation device 30.

[0061] In practical applications, the laser radar 10 and the image acquisition device 20 can be as follows: Figure 1 Alternatively, the two devices may be integrated into one device. Unless otherwise specified, the following contents of the embodiments of the present application are described by taking the laser radar 10 and the image acquisition device 20 as two independent devices.

[0062] The laser radar 10 can be a multi-line laser radar, such as a 16-line, 32-line, 40-line, or 64-line laser radar. Furthermore, the embodiments of the present application do not impose any restrictions on the specific installation location of the laser radar 10. For example, in the case of autonomous driving, the laser radar 10 is placed above the vehicle. Typically, the point cloud coordinates of the laser radar 10 are in the X-axis direction directly in front, the Y-axis direction to the left, and the Z-axis direction directly above.

[0063] The data acquisition device 20 includes a camera for shooting, such as a still camera or a video camera, and the camera can be an RGB camera. In the following, the data acquisition device 20 is described using a camera as an example.

[0064] The point cloud data annotation device 30 may be a computer device.

[0065] In one embodiment, an implementation scenario of the point cloud data acquired by the laser radar 10 includes: in the robot software platform (ros) environment in the Ubuntu system (Ubuntu), connecting the laser radar 10, recording the data packet of the laser radar data, and saving the laser radar data to the hard disk, and then converting the continuous laser radar data into point cloud data (PCD), collecting a frame of data from 5 adjacent frames of data to form the point cloud data that needs to be labeled.

[0066] Figure 2 The hardware structure of the laser radar 10, the image acquisition device 20 and the point cloud data annotation device 30 in the embodiment of the present application can refer to the following: Figure 2 The communication device includes a processor 41, a communication line 44 and at least one transceiver ( Figure 2 The description is merely illustrative and takes the transceiver 43 as an example).

[0067] The processor 41 may include one or more processing units. For example, the processor 41 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video processing unit (VPU) controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0068] The controller can be the nerve center and command center of the communication equipment. The controller can generate operation control signals based on instruction operation codes and timing signals to complete the control of instruction fetching and execution.

[0069] Processor 41 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 41 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 41. If processor 41 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 41's latency, and thus improves system efficiency.

[0070] In some embodiments, the processor 41 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.

[0071] The communication link 44 may include a pathway for transmitting information between the aforementioned components.

[0072] The transceiver 43 may be any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0073] Optionally, the communication device may further include a memory 42 .

[0074] The memory 42 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory may be independent and connected to the processor via a communication line 44. The memory may also be integrated with the processor.

[0075] The memory 42 is used to store computer-executable instructions for executing the solution of the present application, and the execution is controlled by the processor 41. The processor 41 is used to execute the computer-executable instructions stored in the memory 42, thereby implementing the point cloud data annotation method provided in the following embodiments of the present application.

[0076] Optionally, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, which is not specifically limited in the embodiments of the present application.

[0077] In a specific implementation, as an embodiment, the processor 41 may include one or more CPUs, such as Figure 2 CPU0 and CPU1 in.

[0078] In a specific implementation, as an embodiment, the communication device may include multiple processors, such as Figure 2 4 and 5. Each of these processors may be a single-CPU processor or a multi-CPU processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0079] The following will be combined Figures 1 to 2 A point cloud data annotation method provided in an embodiment of the present application is specifically described.

[0080] It should be noted that the message names between network elements or the names of parameters in the messages in the following embodiments of the present application are only examples, and other names may be used in specific implementations. The embodiments of the present application do not specifically limit this.

[0081] It should be pointed out that the various embodiments of the present application can refer to each other, for example, the same or similar steps, method embodiments, communication system embodiments and device embodiments can refer to each other without limitation.

[0082] Reference Figure 3 The point cloud data annotation method provided in the embodiment of the present application specifically includes the following steps:

[0083] S31. The point cloud data annotation device obtains initial point cloud data and two-dimensional pixel data of a first image corresponding to the initial point cloud data.

[0084] It should be noted that the pixel coordinate system of the 2D pixel data and the image coordinate system of the 2D image data are in a translational transformation relationship. Generally, 2D image data is converted into 2D pixel data through a translational transformation. This facilitates subsequent calculations. The specific conversion method can be found in Formula 2 below and will not be described here.

[0085] Specifically, the laser radar coordinate system can describe the relative position of the object and the laser radar, and the coordinates are expressed as [X L ,Y L ,Z L ], where the origin is the geometric center of the lidar, X L Axis horizontal forward, Y L Axis horizontal to the left, Z L The axis is vertically upward, which conforms to the right-hand coordinate system rule. The laser radar coordinate system and the camera coordinate system are in the same three-dimensional space. The main difference is that the coordinate origin and the directions of the axes are different, so a rigid body transformation is sufficient. Therefore, the point p[X L ,Y L ,Z L ] is converted to the camera coordinate system point p[X based on formula 1 C ,Y C ,Z C ].

[0086]

[0087] Among them, X C Horizontal right; Y C Horizontal upward; Z C Horizontal backward; R is a 3x3 rotation matrix, and T is a 3x1 translation matrix.

[0088] S32. The point cloud data annotation device obtains a first selected box and a second selected box.

[0089] The first selected box includes the target object in the two-dimensional pixel data, and the second selected box includes the target object in the initial point cloud data.

[0090] Optionally, the first selected box contains the contour line of the target object in the pixel coordinate system; the second selected box contains the contour line of the target object in the lidar coordinate system.

[0091] It should be noted that the first selection box contains the target object's outline in the pixel coordinate system, and the second selection box contains the target object's outline in the lidar coordinate system. In other words, the selection box contains the target object's outline in the corresponding coordinate system. There are two specific implementations: the first is to use the selection box as the target object's outline in the corresponding coordinate system; the second is to use the target object's outline within the selection box in the corresponding coordinate system. In practical applications, the shape of the selection box can be determined based on the target object's outline in the corresponding coordinate system.

[0092] Exemplarily, the selection box can be manually framed by mouse operation. Specifically, the user can first click the left mouse button, select the upper right vertex A of the target, then drag the mouse, select the lower left vertex B of the target, and then release the mouse to complete the framing of a target object. In other embodiments, the framing of the target object can also adopt other mouse operation methods, such as first selecting the upper left vertex of the target and then selecting the lower right vertex of the target, or first selecting the lower left vertex of the target and then selecting the upper right vertex of the target, or first selecting the lower right vertex of the target and then selecting the upper left vertex of the target. An example of the first selection box is as follows Figure 4 As shown, the target object is determined as Figure 4 The first selected box for car a is Figure 4 The rectangular frame 4a shown in FIG. An example of the second selected frame is as follows Figure 5 As shown, Figure 5 The second selected frame for the target object in the initial point cloud data is the stereo frame 5a.

[0093] The above-mentioned first selection box being a rectangular box and the second selection box being a three-dimensional box are merely exemplary. The present embodiment does not impose any limitation on the shapes of the first selection box and the second selection box.

[0094] S33. The point cloud data annotation device determines the annotation information of the target object in the initial point cloud data according to the first selected box and the second selected box.

[0095] Furthermore, in combination with the target object's annotation information, the target object's attribute information is set. The target object's attribute information includes the target object's location information, the target object category corresponding to the target object's location information, and the confidence level of the target object category. The target object's location information may include the target object's length, width, height, and three-dimensional coordinates of its center point. In one embodiment, the confidence level of the target object category is the confidence level of the category corresponding to the target object's location information. The target object category and the confidence level of the target object category can be obtained using a trained PointNet segmentation model. In the model output, each target object's location information corresponds to a confidence level of a different category. For example, for a target object, the confidence level of its category being car is 0.9, while the confidence level of its category being truck is 0.1.

[0096] For example, for S33, specific implementation methods may include but are not limited to the following two situations:

[0097] Case 1: When the degree of overlap between the two-dimensional frame and the first selected frame is greater than or equal to a preset threshold, the point cloud data annotation device determines that the second point cloud data is annotation information of the target object.

[0098] Case 2: If the overlap between the 2D bounding box and the first selected box is less than a preset threshold, the point cloud data annotation device determines a bounding box; the bounding box is the minimum bounding box that includes the first selected box and the 2D bounding box. The point cloud data corresponding to the 2D data within the bounding box is determined as the annotation information corresponding to the target.

[0099] Of course, the above two cases merely illustrate that the target object's annotation information is the second point cloud data described in Case 1 or the point cloud data corresponding to the two-dimensional data in the bounding box described in Case 2. However, the present embodiment does not impose any limitations on the target object's annotation information. For example, the target object's annotation information may also be the bounding box of the second point cloud data described in Case 1 or the bounding box of the point cloud data corresponding to the two-dimensional data described in Case 2.

[0100] In a specific implementation, the above-mentioned overlap can be determined in the following manner: first, determine the intersection between the two-dimensional data selected in the two-dimensional box and the two-dimensional data selected in the first selected box; and determine the union between the two-dimensional data selected in the two-dimensional box and the two-dimensional data selected in the first selected box. Then, based on the union and the intersection, determine the overlap between the two-dimensional box and the first selected box. The specific calculation method can be to divide the above-mentioned intersection by the above-mentioned union, and the obtained quotient is used as the above-mentioned overlap. Therefore, by using the intersection and union between the two-dimensional data selected in the two-dimensional box and the two-dimensional data selected in the first selected box, the overlap between the two-dimensional box and the first selected box can be accurately determined.

[0101] For the above two situations, using the overlap between the two-dimensional frame and the first selected frame as a basis for determining the labeling information corresponding to the target can improve the labeling accuracy of the target object.

[0102] In this method, by combining 2D data with point cloud data to determine the target object's annotation information, this method can address the potential for inaccurate object annotation based solely on point cloud data. It also improves the quality of target object annotation and recognition rate within point cloud data.

[0103] In one implementation, combining Figure 3 , refer to Figure 6 , the specific implementation of S33 includes the following steps:

[0104] S331. The point cloud data annotation device determines the first point cloud data.

[0105] The first point cloud data corresponds to the object in the first selected box.

[0106] For example, Figure 4 The first point cloud data corresponding to the object contained in the first selected box 4a (the object includes the target object car a) is Figure 5 The region 5b shown in FIG. 5 contains gray-white point cloud data.

[0107] Optionally, an association relationship is established between the first selection box and the second selection box. Specifically, after obtaining the association relationship between each first selection box and the second selection box, the user can view the association relationship between the first selection box and the second selection box on the pixel image corresponding to the point cloud data. Exemplarily, the association relationship between the first selection box and the second selection box is displayed on the image corresponding to the point cloud data. The associated first selection box and the second selection box are connected by a line. Different association relationships are displayed using different colors of connecting lines according to their numbers for easier viewing by the user.

[0108] S332. The point cloud data annotation device determines the second point cloud data in the first point cloud data.

[0109] The second point cloud data corresponds to the second selected box.

[0110] Optionally, by moving, rotating, scaling, and other operations on the second selection box in the first point cloud data, the second point cloud data corresponding to the target object in the first point cloud data is determined. In one implementation, the operations such as moving, rotating, scaling, and the like can be directly performed by the human side; or, they can be implemented by the device side. More preferably, in order to improve the recognition accuracy of the second point cloud data, the two methods can be combined to implement the operations such as moving, rotating, scaling, and the like of the second selection box. The specific effect diagram is as follows Figure 7 shown.

[0111] For a better understanding, for the target object in the image, the target object can be framed using a contour line. The conversion relationship between the point cloud coordinate system and the pixel coordinate system in the contour line is determined according to the rotation matrix R and the translation matrix T. The area marked in the contour line in the pixel data is projected onto the corresponding initial point cloud data containing the second point cloud data of the target object according to the conversion relationship of the rotation matrix R and the translation matrix T, and vice versa. In this way, the corresponding conversion of point cloud data and pixel data is achieved, which facilitates the reading and processing of the device or apparatus.

[0112] S333. The point cloud data annotation device determines a two-dimensional frame corresponding to the second point cloud data.

[0113] In one implementation, second two-dimensional data corresponding to the second point cloud data is determined according to a preset coordinate transformation relationship, and a two-dimensional frame is determined according to the second two-dimensional data.

[0114] For example, (1) determine the conversion relationship between the camera coordinate system and the image coordinate system, such as Figure 8 As shown, it shows the p[X C ,Y C ,ZC ] and the projection relationship between the point p(x,y) in the image coordinate system. C ,Y C ,Z C ]The coordinate relationship of the point p(x,y) converted to the image coordinate system is:

[0115]

[0116] Where f is the focal length of the camera.

[0117] Transforming Formula 2 into a matrix form is shown in Formula 3:

[0118]

[0119] (2) Determine the conversion relationship between the image coordinate system and the pixel coordinate system. Figure 9 The figure shows the relationship between a point p(x,y) in the image coordinate system and a point p(u,v) in the pixel coordinate system. The image coordinate system and the pixel coordinate system are in a translation transformation relationship, with O1 being the origin of the pixel coordinate system and O being the origin of the image coordinate system. The coordinates of point O in the pixel coordinate system are (u0,v0). Therefore, the transformation relationship between a point p(x,y) in the image coordinate system and a point p(u,v) in the pixel coordinate system is:

[0120]

[0121] Where dx and dy are the length and width of a single pixel in the image plane, respectively.

[0122] Converting Equation 3 into matrix form is shown in Equation 4:

[0123]

[0124] In summary, the pixel coordinate point p(u,v) and the laser radar coordinate point p[X L ,Y L ,Z L The conversion of ] is derived based on the above formulas 1, 3, and 5, and is specifically expressed as formula 6:

[0125]

[0126] Furthermore, the maximum value u1 and minimum value u2 of the first coordinate axis data in the pixel coordinate system, and the maximum value v1 and minimum value v2 of the second coordinate axis data in the pixel coordinate system are first determined in the second two-dimensional data. Then, the rectangle formed by the coordinates (u1, v1) and the coordinates (u2, v2) as the diagonal vertices is determined as a two-dimensional box.

[0127] For example, the rectangle formed by the coordinates (u1, v1) and (u2, v2) as diagonal vertices is determined as follows: Figure 10 The two-dimensional frame 10a shown in FIG.

[0128] In the embodiment of the present application, an outer rectangular frame containing the second two-dimensional data is selected as the two-dimensional frame according to the coordinate position of the second two-dimensional data, so that the point cloud data annotation information of the target object can be obtained more accurately.

[0129] S334. The point cloud data annotation device determines the annotation information of the target object in the initial point cloud data according to the first selected box and the two-dimensional box.

[0130] In this implementation, by determining the second point cloud data in the first point cloud data, and comprehensively determining the annotation information of the target object in the initial point cloud data based on the two-dimensional box obtained from the second point cloud data and the first selected box, the target object can be effectively and accurately identified.

[0131] In one implementation, combining Figure 6 , refer to Figure 11 , the specific implementation of S331 is:

[0132] S331a, the point cloud data annotation device converts the initial point cloud data into initial two-dimensional data according to a preset coordinate conversion relationship.

[0133] It should be noted that, based on Formula 6, the three-dimensional data of the initial point cloud data can be converted into initial two-dimensional data.

[0134] In one embodiment, the Open Graphics Library (OpenGL) can be used for visualization, and the initial point cloud data can be displayed in the three-dimensional coordinate system of OpenGL to obtain a corresponding point cloud distribution map in a two-dimensional image. The two-dimensional image can be a two-dimensional top view or a two-dimensional front view.

[0135] S331b. The point cloud data annotation device determines the first two-dimensional data in the initial two-dimensional data.

[0136] The first two-dimensional data is the two-dimensional data within the first selected frame in the initial two-dimensional data.

[0137] S331c. The point cloud data labeling device determines the point cloud data corresponding to the first two-dimensional data as first point cloud data.

[0138] In this implementation, the initial point cloud data is converted into two-dimensional pixel data, and the first selected frame is mapped to the two-dimensional pixel data to obtain two-dimensional data containing the target object. The point cloud data of the object contained in the first selected frame is then determined based on the two-dimensional data. This allows for more accurate identification of the point cloud data of the object in the first selected frame, providing a better basis for identifying the target object.

[0139] In one implementation, taking the position of the target object in the first image as a reference value, when the position change of the target object in the next frame of the first image is within a preset range, the annotation information of the target object in the next frame of the first image can be obtained by the annotation information of the target object in the first image. Figure 3 , refer to Figure 12 The point cloud data annotation method provided in the embodiment of the present application may also include S34 and S35.

[0140] S34. The point cloud data annotation device obtains a second image; the second image is the next frame image of the first image.

[0141] It should be noted that the embodiments of the present application do not impose any restrictions on the geographical locations of the image acquisition device when acquiring the first image and the second image. In other words, the image acquisition device can be in the same geographical location or in different geographical locations when acquiring the location data of the first image and the second image.

[0142] S35. When the point cloud data annotation device determines that the position change data of the target object in the first image in the second image is within a preset range, the point cloud data annotation device determines the annotation information of the target object in the second image based on the annotation information of the target object in the first image.

[0143] Exemplarily, the position change data may be a coordinate change of the target object. It should be noted that the coordinate change of the target object is only an exemplary description of the position change data, and the embodiment of the present application does not impose any limitation on the position change data.

[0144] Furthermore, one method of determining position change data of a target object in a first image in a second image is to determine first camera coordinates of the target object in the first image and second camera coordinates of the target object in the second image based on a camera origin, and determine a coordinate change amount of the target object based on a difference between the first camera coordinates and the second camera coordinates.

[0145] For example, taking the target object's annotation information as the target object's point cloud data, when the target object's position data is based on the first image and remains unchanged in the second image, the target object's point cloud data in the first image can be directly used in the target object's point cloud data in the second image, without having to re-determine the target object's annotation information in the second image.

[0146] When the target object's position data, based on the first image, is translated in the second image, the corresponding conversion formula is selected from the above formula based on the coordinate type of the position change data to convert the position change data into point cloud translation data. The point cloud data of the target object in the first image is then translated based on the point cloud translation data to determine the target object's annotation information in the second image. It should be noted that the point cloud translation data is 3D data in the LiDAR coordinate system.

[0147] In this implementation, when the target object's position change data in the first and second images is determined to be within a preset range, the target object's annotation information in the second image can be determined based on the annotation information in the first image. This improves the efficiency of object annotation in the second image and reduces the computational burden on the point cloud data annotation device.

[0148] In one implementation, considering that the point cloud data collected by the laser radar at the same time as the first image is sparse, using this point cloud data as the initial point cloud data may result in the problem of missing the target object or affecting the accuracy of the target object labeling. Figure 12 , the initial point cloud data in S31 can be obtained by the following steps:

[0149] S311. The point cloud data annotation device obtains original point cloud data corresponding to the first image.

[0150] S312. When determining that the point cloud data of the target object in the original point cloud data is less than or equal to the first threshold, the point cloud data annotation device obtains a third image; the third image is a frame image previous to the first image.

[0151] S313: When the point cloud data annotation device determines that the position change data of the target object in the third image in the first image is within a preset range, the point cloud data corresponding to the third image is acquired.

[0152] S314. The point cloud data annotation device determines initial point cloud data based on the point cloud data corresponding to the third image.

[0153] Specifically, when the position of the target object in the third image relative to the first image changes, the point cloud data corresponding to the third image is determined as the initial point cloud data. When the position of the target object in the third image relative to the first image changes, and the position change data is within a preset range, the position change data is converted using a corresponding conversion formula selected from the above formulas based on the coordinate type of the position change data to generate point cloud translation data. The point cloud data corresponding to the third image is translated based on the position change data, and the translated point cloud data corresponding to the third image is determined as the initial point cloud data.

[0154] The position change data is determined based on the coordinate position of the target object in the third image and the coordinate position of the target object in the first image.

[0155] Optionally, when it is determined that the point cloud data of the target object in the original point cloud data is greater than a preset threshold, the original point cloud data is determined as the initial point cloud data.

[0156] In this implementation, if it is determined that the point cloud data of the target object in the original point cloud data is less than or equal to a first threshold, and the position change data of the target object in the third image relative to the first image is within a preset range, the initial point cloud data is determined based on the point cloud data corresponding to the third image. This solves the problem that if the point cloud data collected by the lidar at the same time as the first image is sparse, the target object may be missed or the target object labeling accuracy may be affected when the point cloud data at that time is used as the initial point cloud data.

[0157] For a better understanding, an exemplary description is given of the method provided in the embodiment of the present application. Among them, a vehicle is taken as an example. First, the two-dimensional image data (i.e., the first image) of a preset frame in the video image captured by the video capture device and the original point cloud data containing the vehicle captured by the corresponding laser radar are obtained. The initial point cloud data is determined based on the original point cloud data (the specific implementation method refers to the corresponding contents of S311-S314 above, which will not be repeated here) so as to determine the stereo frame (i.e., the second selected frame) in the initial point cloud data. Then, the two-dimensional image data is converted into two-dimensional pixel data so as to determine the rectangular frame containing the vehicle (i.e., the first selected frame) in the two-dimensional pixel data. Thereafter, the point cloud data contained in the stereo frame is converted into two-dimensional pixel data, and the two-dimensional pixel data contained in the first selected frame (i.e., the first two-dimensional data) is determined in the two-dimensional pixel data. Further, the point cloud data corresponding to the first two-dimensional data (i.e., the first point cloud data) is determined; and the point cloud data of the vehicle in the first point cloud data (i.e., the second point cloud data) is determined based on the second selected frame. Further, the second point cloud data is converted into two-dimensional data (i.e., the second two-dimensional data). The minimum bounding box of the second two-dimensional data is determined as the two-dimensional box. Finally, the labeling information of the vehicle is determined according to the degree of overlap between the first selected box and the two-dimensional box.

[0158] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0159] like Figure 13 FIG. 3 is a schematic diagram of the structure of a point cloud data annotation device 30 provided in an embodiment of the present application. The point cloud data annotation device 30 is used to perform Figure 3 The point cloud data annotation method shown in FIG. 3 includes an acquisition unit 301 and a processing unit 302 .

[0160] Specifically, the acquisition unit 301 is used to acquire the initial point cloud data and the two-dimensional pixel data of the first image corresponding to the initial point cloud data. For example, the acquisition unit 301 can be used to implement the following Figure 3 S31 shown.

[0161] The acquisition unit 301 is further configured to acquire a first selected frame and a second selected frame, wherein the first selected frame includes the target object in the two-dimensional pixel data and the second selected frame includes the target object in the initial point cloud data. For example, the processing unit 302 may be configured to implement the following: Figure 3 S32 shown.

[0162] The processing unit 302 is used to determine the annotation information of the target object in the initial point cloud data according to the first selected box and the second selected box. For example, the processing unit 302 can be used to implement the following Figure 3 S33 shown.

[0163] In a possible implementation, the processing unit 302 is specifically configured to determine first point cloud data, where the first point cloud data corresponds to an object in the first selected box. For example, the processing unit 302 may be configured to implement the following: Figure 6 S331 shown.

[0164] The processing unit 302 is further configured to determine second point cloud data in the first point cloud data, where the second point cloud data corresponds to the second selected box. For example, the processing unit 302 may be configured to implement the following: Figure 6 S332 shown.

[0165] The processing unit 302 is further configured to determine a two-dimensional frame corresponding to the second point cloud data. For example, the processing unit 302 may be configured to implement the following Figure 6 S333 shown.

[0166] The processing unit 302 is further configured to determine the annotation information of the target object in the initial point cloud data according to the first selected frame and the two-dimensional frame. For example, the processing unit 302 can be used to implement the following Figure 6 S334 shown.

[0167] In a possible implementation, the processing unit 302 is specifically configured to convert the initial point cloud data into initial two-dimensional data according to a preset coordinate transformation relationship. For example, the processing unit 302 may be configured to implement the following: Figure 11 S331a shown.

[0168] The processing unit 302 is further configured to determine the first two-dimensional data in the initial two-dimensional data; the first two-dimensional data is the two-dimensional data within the first selected frame in the initial two-dimensional data. For example, the processing unit 302 may be configured to implement the following: Figure 11 S331b shown.

[0169] The processing unit 302 is further configured to determine the point cloud data corresponding to the first two-dimensional data as the first point cloud data. For example, the processing unit 302 may be configured to implement the following: Figure 11 S331c shown.

[0170] In a possible implementation, the processing unit 302 is specifically configured to determine second two-dimensional data corresponding to the second point cloud data according to a preset coordinate transformation relationship.

[0171] The processing unit 302 is further configured to determine a two-dimensional frame according to the second two-dimensional data.

[0172] In a possible implementation, the processing unit 302 is specifically configured to determine the maximum value u1 and minimum value u2 of the first coordinate axis data in the pixel coordinate system, and the maximum value v1 and minimum value v2 of the second coordinate axis data in the pixel coordinate system in the second two-dimensional data.

[0173] The processing unit 302 is further configured to determine a rectangle formed by the coordinates (u1, v1) and the coordinates (u2, v2) as diagonal vertices as a two-dimensional frame.

[0174] In a possible implementation, the processing unit 302 is specifically configured to determine that the second point cloud data is annotation information of the target object when a degree of overlap between the two-dimensional box and the first selected box is greater than or equal to a preset threshold.

[0175] In a possible implementation, the processing unit 302 is specifically configured to determine a bounding box when the overlap between the two-dimensional box and the first selected box is less than a preset threshold; the bounding box is a minimum bounding box including the first selected box and the two-dimensional box.

[0176] The processing unit 302 is further configured to determine the point cloud data corresponding to the two-dimensional data in the circumscribed frame as annotation information corresponding to the target.

[0177] In a possible implementation, the first selected box includes a contour line of the target object in a pixel coordinate system; the second selected box includes a contour line of the target object in a lidar coordinate system.

[0178] In a possible implementation, the acquisition unit 301 is further configured to acquire a second image; the second image is the next frame of the first image. For example, the processing unit 302 may be configured to implement the following: Figure 12 S34 shown.

[0179] The processing unit 302 is configured to determine the labeling information of the target object in the second image based on the labeling information of the target object in the first image when it is determined that the position change data of the target object in the first image acquired by the acquisition unit 301 in the second image is within a preset range. For example, the processing unit 302 can be used to implement the following Figure 12 S35 shown.

[0180] In an exemplary solution, the acquisition unit 301 is specifically used to acquire the original point cloud data corresponding to the first image. For example, the acquisition unit 301 can be used to implement the following Figure 12 S311 shown.

[0181] The processing unit 302 is configured to determine whether the point cloud data of the target object in the original point cloud data acquired by the acquisition unit 301 is less than or equal to a first threshold. For example, the processing unit 302 may be configured to implement the following: Figure 12 S312 shown.

[0182] The acquisition unit 301 is used to acquire a third image when the processing unit 302 determines that the point cloud data of the target object in the original point cloud data is less than or equal to the first threshold; the third image is the previous frame image of the first image. For example, the acquisition unit 301 can be used to implement the following Figure 12 S312 shown.

[0183] The processing unit 302 is used to determine that the position change data of the target object in the third image in the first image is within a preset range. For example, the processing unit 302 can be used to implement the following Figure 12 S313 shown.

[0184] The acquisition unit 301 is configured to acquire point cloud data corresponding to the third image when the processing unit 302 determines that the position change data of the target object in the third image is within a preset range in the first image. For example, the acquisition unit 301 can be used to implement the following Figure 12 S313 shown.

[0185] The processing unit 302 is configured to determine initial point cloud data based on the point cloud data corresponding to the third image acquired by the acquisition unit 301. For example, the processing unit 302 may be configured to implement the following: Figure 12 S314 shown.

[0186] Of course, the point cloud data annotation device 30 provided in the embodiment of the present application includes but is not limited to the above-mentioned modules. For example, the point cloud data annotation device 30 may also include a sending unit 303 and a storage unit 304. The sending unit 303 can be used to send relevant data in the point cloud data annotation device 30 to other devices to realize data exchange with other devices. The storage unit 304 can be used to store the program code of the point cloud data annotation device 30, and can also be used to store data generated by the point cloud data annotation device 30 during operation, such as data in write requests.

[0187] Here, the system architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0188] In some embodiments, the disclosed methods may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of manufacture.

[0189] Figure 14 A conceptual partial view of a computer program product provided by an embodiment of the present application is schematically shown, wherein the computer program product includes a computer program for executing a computer process on a computing device.

[0190] In one embodiment, the computer program product is provided using a signal bearing medium 410. The signal bearing medium 410 may include one or more program instructions that, when executed by one or more processors, may provide the above-described Figure 3 Thus, for example, reference to Figure 3 In the embodiment shown in , one or more features of S31-S33 may be undertaken by one or more instructions associated with the signal bearing medium 410. In addition, Figure 14 The program instructions in also describe example instructions.

[0191] In some examples, signal-bearing medium 410 may include computer-readable medium 411, such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, a memory, a read-only memory (ROM), a random access memory (RAM), and the like.

[0192] In some implementations, signal bearing medium 410 may include computer recordable medium 412 such as, but not limited to, memory, read / write (R / W) CD, R / W DVD, or the like.

[0193] In some embodiments, signal bearing medium 410 may include communication medium 413 such as, but not limited to, digital and / or analog communication media (eg, fiber optic cables, waveguides, wired communication links, wireless communication links, etc.).

[0194] Signal bearing medium 410 may be communicated by a wireless form of communication medium 413 (eg, a wireless communication medium conforming to the IEEE 802.41 standard or other transmission protocols). The one or more program instructions may be, for example, computer executable instructions or logic implemented instructions.

[0195] In some examples, such as for Figure 3 The described data writing device may be configured to provide various operations, functions, or actions in response to one or more program instructions via computer-readable media 411 , computer-recordable media 412 , and / or communication media 413 .

[0196] In addition, embodiments of the present application further provide a chip system, which is applied to a point cloud data annotation device; the chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via circuits; the interface circuits are configured to receive signals from the memory of the point cloud data annotation device and send signals to the processors, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the point cloud data annotation device executes the point cloud data annotation method provided in the first aspect or any possible design of the point cloud data annotation method provided in the first aspect.

[0197] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0199] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0201] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0202] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A point cloud data annotation method, characterized in that: include: Acquiring initial point cloud data and two-dimensional pixel data of a first image corresponding to the initial point cloud data; Acquire a first selected box and a second selected box, wherein the first selected box includes the target object in the two-dimensional pixel data, and the second selected box includes the target object in the initial point cloud data; Determining first point cloud data, where the first point cloud data corresponds to the object in the first selected frame; determining second point cloud data in the first point cloud data, where the second point cloud data corresponds to the second selected box; determining a two-dimensional frame corresponding to the second point cloud data; Determine the labeling information of the target object in the initial point cloud data according to the first selected box and the two-dimensional box.

2. The point cloud data annotation method according to claim 1, characterized in that: The determining of the first point cloud data includes: Converting the initial point cloud data into initial two-dimensional data according to a preset coordinate conversion relationship; Determining first two-dimensional data in the initial two-dimensional data; the first two-dimensional data is the two-dimensional data in the initial two-dimensional data within the first selected frame; The point cloud data corresponding to the first two-dimensional data is determined as the first point cloud data.

3. The point cloud data annotation method according to claim 1, characterized in that: Determining a two-dimensional frame corresponding to the second point cloud data includes: Determining second two-dimensional data corresponding to the second point cloud data according to a preset coordinate transformation relationship; The two-dimensional frame is determined according to the second two-dimensional data.

4. The point cloud data annotation method according to claim 3, characterized in that: The determining the two-dimensional frame according to the second two-dimensional data includes: Determine the maximum value u1 and the minimum value u2 of the first coordinate axis data in the pixel coordinate system, and the maximum value v1 and the minimum value v2 of the second coordinate axis data in the pixel coordinate system in the second two-dimensional data; A rectangle formed by the coordinates (u1, v1) and the coordinates (u2, v2) as diagonal vertices is determined as the two-dimensional frame.

5. The point cloud data annotation method according to claim 1, characterized in that: The determining, based on the first selected frame and the two-dimensional frame, the labeling information of the target object in the point cloud data includes: When the degree of overlap between the two-dimensional frame and the first selected frame is greater than or equal to a preset threshold, the second point cloud data is determined to be the annotation information of the target object.

6. The point cloud data annotation method according to claim 1, characterized in that: The determining, based on the first selected frame and the two-dimensional frame, the labeling information of the target object in the point cloud data includes: When the overlap between the two-dimensional frame and the first selected frame is less than a preset threshold, determining a bounding box; the bounding box is a minimum bounding box including the first selected frame and the two-dimensional frame; The point cloud data corresponding to the two-dimensional data in the circumscribed frame is determined as the annotation information corresponding to the target.

7. The point cloud data annotation method according to claim 1, characterized in that: The first selected box contains the contour line of the target object in the pixel coordinate system; the second selected box contains the contour line of the target object in the lidar coordinate system.

8. The point cloud data annotation method according to claim 1, characterized in that: Also includes: Acquire a second image; the second image is a next frame image of the first image; When it is determined that the position change data of the target object in the first image in the second image is within a preset range, the labeling information of the target object in the second image is determined based on the labeling information of the target object in the first image.

9. The point cloud data annotation method according to claim 1, characterized in that: The obtaining of initial point cloud data includes: Obtaining original point cloud data corresponding to the first image; When it is determined that the point cloud data of the target object in the original point cloud data is less than or equal to a first threshold, acquiring a third image; the third image is a frame image previous to the first image; When determining that the position change data of the target object in the third image in the first image is within a preset range, acquiring point cloud data corresponding to the third image; The initial point cloud data is determined based on the point cloud data corresponding to the third image.

10. A point cloud data annotation device, characterized in that: include: an acquisition unit, configured to acquire initial point cloud data and two-dimensional pixel data of a first image corresponding to the initial point cloud data; The acquisition unit is further configured to acquire a first selected box and a second selected box, wherein the first selected box includes the target object in the two-dimensional pixel data, and the second selected box includes the target object in the initial point cloud data; A processing unit is configured to determine first point cloud data, the first point cloud data corresponding to the object in the first selected box; and determine second point cloud data in the first point cloud data, the second point cloud data corresponding to the second selected box; Determine a two-dimensional box corresponding to the second point cloud data; and determine annotation information of the target object in the initial point cloud data based on the first selected box and the two-dimensional box.

11. A point cloud data annotation system, characterized in that: Including laser radar, point cloud data annotation device and image acquisition equipment; The laser radar is used to collect original point cloud data; The image acquisition device is used to acquire images corresponding to the original point cloud data; The point cloud data labeling device is used to execute the point cloud data labeling method according to any one of claims 1 to 9.

Images