Point cloud labeling method and device and storage medium

Abstract

The invention provides a point cloud labeling method and device and a storage medium. According to the main technical scheme, the method comprises the steps that information of an N-polygon marking frame of an image is obtained, N is a positive integer larger than 2, and the image is obtained by projecting point cloud to an image plane of a camera; determining N construction planes according to the position point of the camera and any two adjacent vertexes of the N-polygon marking frame; and determining a target point according to the directed distances from the points in the point cloud to the N construction planes. In this way, a user only needs to input the N-polygon labeling box on the image to determine the target point, point-by-point labeling is not needed, and the labeling speed and the labeling efficiency are greatly improved. And the target point is determined by using the directed distance from the point to the plane, so that the calculation complexity is greatly reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a point cloud annotation method, device and storage medium. Background Art

[0002] Point cloud is a collection of a large number of points in three-dimensional space obtained by detecting the surrounding environment. It is widely used in computer vision and machine learning. Each point in the point cloud contains three-dimensional coordinates and can further contain other attribute information such as color (RGB) and reflection intensity (Intensity).

[0003] Point cloud annotation is a process of labeling points in a three-dimensional point cloud with corresponding labels (such as type labels), that is, the process of determining the target points in the point cloud. Point cloud annotation technology has been increasingly widely used in the fields of autonomous driving, drones, virtual reality, etc., and is a very important branch of computer vision.

[0004] Currently, point cloud annotation is mostly done manually by annotating the point clouds point by point, which is slow and inefficient. Summary of the invention

[0005] In view of this, the present disclosure provides a point cloud annotation method, device and storage medium to improve the annotation speed and efficiency of point clouds.

[0006] According to a first aspect of the present disclosure, a point cloud annotation method is provided, the method comprising:

[0007] Obtain information of an N-sided annotation box input on an image, where N is a positive integer greater than 2, and the image is obtained by projecting the point cloud onto an image plane of a camera;

[0008] Determine N construction planes according to the position point of the camera and any two adjacent vertices of the N-gon annotation box; and

[0009] A target point is determined according to a directed distance from a point in the point cloud to the N construction planes.

[0010] According to a second aspect of the present disclosure, a point cloud annotation device is provided, the device comprising:

[0011] one or more processors, and

[0012] A memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the method described in the first aspect.

[0013] According to a third aspect of the present disclosure, a computer-readable storage medium storing a program is provided, wherein the program includes instructions, which, when executed by one or more processors of a computing device, cause the computing device to execute the method described in the first aspect above.

[0014] It can be seen from the above technical solution that the present disclosure can obtain the information of the N-gon annotation box input on the image, and determine the target point in the point cloud according to the directed distance from the point in the point cloud to the N construction planes. In this way, the user only needs to input the N-gon annotation box on the image to determine the target point, without the need to mark each point, which greatly improves the annotation speed and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A system architecture diagram applicable to the embodiments of the present disclosure;

[0017] Figure 2 A flow chart of a data annotation method provided in an embodiment of the present disclosure;

[0018] Figure 3 A schematic diagram of the principle of projecting a point cloud onto an image plane provided by an embodiment of the present disclosure;

[0019] Figure 4 An example image corresponding to a frame of point cloud provided in an embodiment of the present disclosure;

[0020] Figure 5 An example diagram of an N-gon annotation box input into an image provided by an embodiment of the present disclosure;

[0021] Figure 6 A schematic diagram of a construction plane provided for an embodiment of the present disclosure;

[0022] Figure 7 (a) and (b) are schematic diagrams of normal vectors in two directions on a construction plane provided by an embodiment of the present disclosure;

[0023] Figure 8 A schematic diagram of determining the directed distance from a point to a construction plane provided in an embodiment of the present disclosure;

[0024] Fig. 9 A schematic diagram of determining a normal vector of a construction plane provided by an embodiment of the present disclosure;

[0025] Fig.10 A schematic diagram of determining a normal vector of another construction plane provided by an embodiment of the present disclosure;

[0026] Fig.11 A flowchart for a point cloud annotation scenario provided by an embodiment of the present disclosure;

[0027] Fig.12 A modular structural block diagram of a point cloud annotation device provided in an embodiment of the present disclosure;

[0028] Fig.13 A schematic block diagram of a point cloud annotation device in the form of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. The singular forms "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0031] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0032] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0033] In order to facilitate the understanding of the embodiments of the present disclosure, firstly Figure 1 The system architecture applicable to the embodiments of the present disclosure is described. Figure 1 An exemplary system architecture diagram to which the embodiments of the present disclosure can be applied is shown in FIG. Figure 1As shown in , the system mainly includes: a point cloud acquisition device and a point cloud annotation device.

[0034] The point cloud acquisition device is used to detect the surrounding environment and acquire a point cloud that can reflect the surrounding environment information. The point cloud acquisition device can be implemented by, for example, a three-dimensional scanning device, a depth camera, a binocular camera, etc.

[0035] The 3D scanning device can obtain point clouds by scanning the surface of surrounding objects through laser or other optical methods. Its working principle is to convert the information of the object surface into point clouds by controlling the scanning angle of the scanner. Specifically, the 3D scanning device can be a Lidar (Light detection and ranging).

[0036] Depth cameras project light onto surrounding objects, collect depth information through the camera, and then convert it into a point cloud.

[0037] A binocular camera uses two cameras to capture images of surrounding objects from different positions and converts them into point clouds through triangulation calculations.

[0038] In addition to the above implementations, other types of point cloud acquisition devices may also be used, which are not listed here one by one.

[0039] The point cloud acquisition device can use different mounting platforms according to different application scenarios. For example, it can be set on platforms such as satellites, space shuttles, and space stations to detect the space environment and obtain point clouds. For another example, it can be set on platforms such as airplanes and drones to conduct a large-scale environmental detection at high altitudes and obtain point clouds. For another example, it can be set on the ground to detect the static environment on the ground (such as terrain, cultural relics, etc.) and obtain point clouds. For another example, it can be set on mobile platforms such as vehicles and robots to detect the surrounding environment on the ground (such as obstacles, etc.) during movement and obtain point clouds.

[0040] As one of the feasible ways, the point cloud annotation device can be set on the server side to provide annotation services to users through a user interface. The server can be a single server or a server cluster composed of multiple servers. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak service scalability in traditional physical hosts and virtual private servers (VPS) services. Users can use the client or browser on the user terminal to operate the above-mentioned user interface to annotate point cloud data. Figure 1 This implementation is shown in FIG.

[0041] As another feasible method, the point cloud annotation device may be arranged in a computer terminal with strong computing capability, and the user may annotate the point cloud data by operating through a user interface provided by the computer terminal.

[0042] The computer terminal or user terminal mentioned above may be, for example, a smart mobile terminal, a wearable device, a PC (Personal Computer), etc. Among them, smart mobile devices may include, for example, mobile phones, tablet computers, laptop computers, PDAs (Personal Digital Assistants), etc. Wearable devices may include, for example, smart watches, VR (Virtual Reality) devices, AR (Augmented Reality), mixed reality devices (i.e., devices that can support both virtual reality and augmented reality), etc.

[0043] It should be understood that Figure 1 The number of point cloud acquisition devices, point cloud annotation devices, user terminals, clients or browsers in the embodiment is only illustrative. According to the implementation requirements, there may be any number of point cloud acquisition devices, point cloud annotation devices, user terminals, clients or browsers.

[0044] Figure 2 A flow chart of a point cloud annotation method provided in an embodiment of the present disclosure. The method can be performed by Figure 1 The point cloud annotation device in the system shown is executed. Figure 2 As shown in , the method may include the following steps:

[0045] Step 201: Obtain information of an N-sided annotation box of an image, where N is a positive integer greater than 2, and the image is obtained by projecting a point cloud onto an image plane of a camera.

[0046] Step 203: Determine N construction planes according to the camera position point and any two adjacent vertices of the N-gon annotation box.

[0047] Step 205: Determine the target point according to the directed distances from the point in the point cloud to the N construction planes.

[0048] It can be seen from the above process that the present disclosure can obtain the information of the N-gon annotation box input on the image, and determine the target point in the point cloud according to the directed distance from the point in the point cloud to the N construction planes. In this way, the user only needs to input the N-gon annotation box on the image to determine the target point, without the need to mark each point, which greatly improves the annotation speed and efficiency.

[0049] The following describes each step in the above process in detail with reference to the embodiments. First, the above step 201, namely "obtaining the information of the N-gon annotation box input on the image", is described in detail.

[0050] The point cloud annotation device first needs to obtain the point cloud to be annotated. The point cloud annotation device can directly or indirectly obtain point cloud data from the point cloud acquisition device, store the point cloud data and provide it to the user (such as annotator), so that the user can use the annotation service provided by the point cloud annotation device to implement annotation for the point cloud. The point cloud data involved in the embodiment of the present disclosure is data containing point clouds. Indirect methods may include, for example: the point cloud acquisition device uploads the acquired point cloud data to a server or storage device on the service end, and the point cloud annotation device obtains one frame of the point cloud from the server or storage device as the point cloud to be annotated. Another example is: the point cloud acquisition device provides the acquired point cloud data to the user terminal, and when the user uses the point cloud annotation service on the user terminal, one frame of the point cloud is provided to the point cloud annotation device as the point cloud to be annotated.

[0051] After obtaining the point cloud to be annotated, the point cloud annotation device can provide a user interface. If the point cloud annotation device is set on the server side, the user interface can be sent to the user side such as the client, browser, etc., and the user side will display the user interface to the user. That is, the C (Client) / S (Server) architecture can be adopted, and the B (Browser) / S architecture can also be adopted.

[0052] If the point cloud annotation device is set on a computer terminal, the point cloud annotation device can directly display the user interface to the user through the display device of the computer terminal.

[0053] The user interface may be a graphical user interface or a web page, which may also be referred to as a web page, and may be a web page written based on HyperText Markup Language (HTML), i.e., an HTML page, or a web page written based on HTML and Java, i.e., a Java Server Page (JSP), or a web page written in other languages, which is not particularly limited in this embodiment.

[0054] The image of the point cloud to be annotated is displayed on the above user interface. In other words, the image of the point cloud to be annotated is rendered on the user interface. The image is a two-dimensional image, which is obtained by projecting the point cloud onto the image plane of the camera, and converting the three-dimensional coordinates of all points in the point cloud to the coordinate system of the camera's image plane. The process of rendering an image can be regarded as simulating the process of a camera "shooting" a point cloud to obtain an image. Figure 3As shown in , a camera is simulated, and light is emitted from the position of the camera to the point cloud at a certain viewing angle, and finally projected onto the image plane to obtain an image containing the projection points of each point cloud.

[0055] Image rendering usually requires powerful processing capabilities and can be performed using a GPU (Graphics Processing Unit). The GPU uses parallel processing and dedicated hardware to quickly and efficiently perform image rendering and reduce the pressure on the CPU (Central Processing Unit).

[0056] The image displayed on the user interface can be a two-dimensional image projected based on the default camera position and viewing angle information, or a two-dimensional image projected based on the camera position and viewing angle information preset by the user. The camera can be a camera that actually collects point cloud data, or a simulated camera.

[0057] As one of the more preferred implementation methods, the user can input an image adjustment instruction to the point cloud annotation device, and the image adjustment instruction includes camera position and / or viewing angle information, that is, the adjusted target camera position and / or viewing angle information. Among them, the input of the image adjustment instruction can be implemented by means such as shortcut keys, preset mouse operations, preset gesture operations, or triggering specific components on the user interface. In response to the input image adjustment instruction, the point cloud annotation device can reproject the point cloud according to the camera position and / or viewing angle information included in the image adjustment instruction, obtain the adjusted image, and display the adjusted image on the user interface. In this way, the user can make the above adjustments as needed and observe the image on the user interface, so as to find the image corresponding to the camera position and / or viewing angle that is convenient for annotation, and perform point cloud annotation on this image.

[0058] After the above image is displayed through the user interface, since the image contains the projection points of the point cloud on the image plane, such as Figure 4 As shown in . The user can select the target point on the image by observing the image. Specifically, the user can select the target point by inputting an N-sided annotation box on the image, where N is a positive integer greater than 2. For example, a triangular annotation box, a quadrilateral annotation box, a pentagonal annotation box, etc.

[0059] The user may enter an N-sided annotation box in the following ways, but is not limited to:

[0060] The first method: generate an N-gon annotation box on the image by using shortcut keys, then move the position of the N-gon annotation box by using a mouse or gestures, and adjust the size and shape of the N-gon annotation box by dragging the vertices of the N-gon annotation box.

[0061] The second method is to generate an N-sided polygonal annotation box on the image using the mouse. For example, the N-sided polygonal annotation box is generated by continuously inputting the positions of N vertices on the image using the mouse. For another example, the N-sided polygonal annotation box is generated by drawing N connected edges on the image using the mouse. The position of the N-sided polygonal annotation box can be further moved using the mouse, and the size and shape of the N-sided polygonal annotation box can be adjusted by dragging the vertices of the N-sided polygonal annotation box.

[0062] The third method: Generate an N-sided annotation box on the image through a preset gesture operation. For example, select N vertex positions on the image with a finger or a touch screen pen to generate an N-sided annotation box. Another example is to draw N connected edges on the image with a finger or a touch screen pen to generate an N-sided annotation box. The position of the N-sided annotation box can be further moved by a finger or a touch screen pen, and the size and shape of the N-sided annotation box can be adjusted by dragging the vertices of the N-sided annotation box.

[0063] The fourth method: an N-gon annotation box is generated on the image through components provided on the user interface, for example, a toolbar is provided on the user interface, and the toolbar contains components such as a triangle annotation box, a quadrilateral annotation box, and a pentagonal annotation box for the user to select, and the user can select a component to generate an annotation box of the corresponding shape on the image. The position of the N-gon annotation box can be further moved by a mouse or gesture, and the size and shape of the N-gon annotation box can be adjusted by dragging the vertices of the N-gon annotation box.

[0064] Other methods may also be used to input an N-gon annotation box on an image, which are not listed here one by one.

[0065] For example, Figure 4 The image is obtained by projecting a frame of the traffic environment point cloud onto the camera's image plane. The image is presented to the user through the user interface. The user needs to mark the points in the point cloud that belong to street lights, that is, label them with street lights. User observation Figure 4 After the image is shown, an N-gon annotation box can be input on the image. The figure takes a quadrilateral annotation box as an example. The quadrilateral annotation box selects the projection points belonging to the street lamp in the image, such as Figure 5 as shown in .

[0066] The N-gon annotation box input on the image has N vertices, and the positions of the N vertices actually determine the range of the N-gon annotation box. In order to facilitate the determination of the target point in the subsequent steps, the coordinates of the N vertices of the N-gon annotation box can be converted to the point cloud coordinate system where the point cloud is located, where the point cloud coordinate system can use the world coordinate system. In addition to this method, the coordinates of the N vertices and the coordinates of the point cloud can also be converted to the same three-dimensional coordinate system. However, the former is preferred in order to reduce the amount of calculation.

[0067] Specifically, since the annotation operation is performed on a two-dimensional image obtained by projecting the point cloud, after drawing the N-sided annotation box, the coordinates of the N vertices of the N-sided annotation box need to be converted to the point cloud coordinate system. During the conversion, it is usually necessary to first convert the coordinates of the N vertices from the screen coordinate system to the device coordinate system, and then convert them from the device coordinate system to the point cloud coordinate system (such as the world coordinate system). Among them, the screen coordinate system refers to the coordinates based on which the user clicks or touches on the screen. The screen coordinate system is a two-dimensional coordinate system, for example: the position of the point on the screen is represented by the X and Y axis coordinates. The device coordinate system is a three-dimensional coordinate system, which has one more dimension than the screen coordinate system (such as: the Z axis, and usually from the screen to the inside). When the screen coordinate system is converted to this coordinate system, the X and Y axis coordinates are usually converted according to certain rules, and then the Z axis coordinate takes a preset value. For example, assuming that the height of the canvas in the screen coordinate system is h and the width is w, and there is a point (x, y) in the coordinate system, and the coordinates of this point in the device coordinate system are (x1, y1, z1), then x1 = (x / w)*2-1, y1 = -(y / h)*2+1, where the value ranges of x1, y1 and z1 are all [-1, 1]. The point cloud coordinate system is also a three-dimensional coordinate system, which can be understood as the coordinate system used when the point cloud is collected and stored, usually the world coordinate system. When the device coordinate system needs to be converted to this coordinate system, it can be converted through inverse projection.

[0068] The above step 203, namely "determining N construction planes according to the position point of the camera and any two adjacent vertices of the N-gon annotation box", is described in detail below in conjunction with the embodiments.

[0069] In this step, N construction planes are constructed using the camera position points and any two adjacent vertices of the N-sided annotation box. Taking the quadrilateral annotation box as an example, Figure 6 As shown in , assuming that the coordinate of the camera position in the point cloud coordinate system is p0, the four vertices of the quadrilateral annotation box are converted to the coordinates of p1, p2, p3 and p4 in the point cloud coordinate system. Then the four construction planes are:

[0070] Construction plane 1: the plane where points p0, p1 and p2 are located;

[0071] Construction plane 2: the plane where points p0, p2 and p3 are located;

[0072] Construction plane 3: the plane where points p0, p3 and p4 are located;

[0073] Construct plane 4: the plane where points p0, p4, and p1 lie.

[0074] The above step 204, namely "determining the target point according to the directed distances from the points in the point cloud to the N construction planes", is described in detail below in conjunction with the embodiments.

[0075] In the field of computer vision, a signed distance (i.e., a distance with a sign) is usually used to represent a specific meaning. In the embodiment of the present disclosure, a signed distance from a point to a plane is used to represent the positional relationship between a point and a plane. The signed distance from a point to a plane is usually related to the normal vector of the plane. Therefore, in this step, the normal vectors of the N construction planes can be used to first determine the signed distances from the points in the point cloud to the N construction planes.

[0076] The so-called normal vector is a concept in spatial analytic geometry, which refers to a non-zero vector perpendicular to a plane. The normal vector has two possible directions, taking the above construction plane 1 as an example. Figure 7 As shown in , there is a point P on construction plane 1, and the normal vector passing through point P has two directions: Figure 7 The direction of the normal vector shown in (a) is away from the quadrilateral annotation box. Figure 7 The direction of the normal vector shown in (b) is toward the direction of the quadrilateral annotation box. Figure 7 The dashed arrow in the middle represents the normal vector passing through point P on the construction plane where p0, p1 and p2 are located. It should be noted here that since the construction plane is usually not perpendicular to the quadrilateral annotation box, the direction of the normal vector must have relative "towards" and "away from". In other words, the above-mentioned "towards" and "away from" the quadrilateral annotation box are relative concepts.

[0077] In the embodiment of the present disclosure, when determining the directed distance from a point in a point cloud to a construction plane, an implementable method is provided: first determine the vector from a point on the construction plane to a point in the point cloud; then perform dot multiplication of the vector with the normal vector of the construction plane to obtain the directed distance from the point in the point cloud to the construction plane. In addition to this method, other methods may also be used to determine the directed distance from a point in a point cloud to a construction plane.

[0078] Still taking the above construction plane 1 as an example, Figure 8 As shown in , when calculating the directed distance from point Q in the point cloud to the construction plane 1, first calculate the vector PQ, that is, the vector from point P to point Q, as shown in Figure 8 As shown by the solid arrow in the figure. Then dot-multiply the vector PQ with the normal vector N (shown by the dotted arrow in the figure) to obtain the signed distance d from point Q in the point cloud to the construction plane 1. If the direction of vector PQ is consistent with that of normal vector N, the value of d will usually be greater than 0; if Q is on construction plane 1, the value of d will usually be equal to 0; if the direction of vector PQ is opposite to that of normal vector N, the value of d will usually be less than 0. It can be seen that the signed distance d can be used to determine which side of construction plane 1 point Q is located on. For point Q, the above method is used to calculate its signed distances to construction planes 2, 3 and 4 respectively, so as to determine whether point Q is on Figure 6If point Q is within the viewing cone area enclosed by p0, p1, p2, p3 and p4 (including the case where point Q is on a certain construction plane of the viewing cone), then point Q can be considered as the target point; otherwise, point Q is considered not to be the target point.

[0079] In view of this, if the normal vector takes the direction toward the N-gon annotation box, and the signed distances of the points in the point cloud to the N construction planes are all greater than or equal to 0, it means that the point is located inside the viewing cone formed by the camera position and the N vertices (included on a certain construction plane of the viewing cone), and the point is determined to be the target point. If the normal vector takes the direction away from the N-gon annotation box, and the signed distances of the points in the point cloud to the N construction planes are all less than or equal to 0, it means that the point is located inside the viewing cone formed by the camera position and the N vertices (included on a certain construction plane of the viewing cone), and the point is determined to be the target point.

[0080] It can be seen that the determination of the direction of the normal vector is very important for the determination of the above-mentioned target point. The embodiment of the present disclosure provides a method for determining the normal vector. If the normal vector takes the direction toward the N-gon annotation box, the normal vector of the construction plane is obtained by the outer product of the first vector and the second vector. If the normal vector takes the direction away from the N-gon annotation box, the normal vector of the construction plane is obtained by the outer product of the second vector and the first vector. Among them, the first vector is the vector pointing to the first vertex from the position point of the camera on the construction plane, and the second vector is the vector pointing to the second vertex from the position point of the camera on the construction plane. The first vertex and the second vertex are two adjacent vertices of the N-gon annotation box in the clockwise direction from the camera's perspective.

[0081] by Fig. 9 Taking the normal vector shown in the figure as the direction toward the quadrilateral annotation box, the normal vector N1 of the construction plane 1 (i.e., the plane where p0, p1, and p2 are located) can be:

[0082] N1=A×B

[0083] Among them, vector A is the vector pointing from p0 to p1, and vector B is the vector pointing from p0 to p2.

[0084] by Fig.10 Taking the normal vector shown in the figure as the direction toward the quadrilateral annotation box as an example, the normal vector N2 of the construction plane 2 (i.e., the plane where p0, p2, and p3 are located) can be:

[0085] N2=B×C

[0086] Among them, vector C is the vector pointing from p0 to p3.

[0087] By analogy, the normal vectors of the four construction planes can be determined. In view of the similarity of the principles, the calculation method of the normal vector when the normal vector deviates from the quadrilateral annotation frame is not described here by example.

[0088] When performing point cloud annotation processing, the GPU usually first projects the point cloud data and renders it to the user interface for the user to perform annotation operations. Then the CPU performs calculations based on the user's annotation operations and determines the target point cloud. The target point determination process involved in the above steps 203 and 205 is usually performed by the CPU. This target point determination method eliminates the need for the CPU to perform complex projection processing between three-dimensional and two-dimensional points. The target point can be determined by simply calculating the directed distance from the point to the construction plane, which greatly reduces the computational complexity, reduces the amount of calculations that the CPU needs to perform, and improves the point cloud annotation efficiency.

[0089] As one of the achievable methods, after determining the target point, the user can input a label creation instruction through a shortcut key, a preset mouse operation, a preset gesture operation, or triggering a specific component on the user interface. The point cloud annotation device responds to the label creation instruction, creates a target label, and stores the target label in correspondence with the determined target point in the point cloud.

[0090] As another achievable method, the user may also input a label creation instruction by means of shortcut keys, preset mouse operations, preset gesture operations, or triggering a specific component on the user interface before inputting the N-gon annotation box. The point cloud annotation device responds to the label creation instruction, creates a target label, and after determining the target point in the point cloud, stores the target label in correspondence with the determined target point in the point cloud.

[0091] In addition, the creation of target tags can also be performed at other times, which are not listed here.

[0092] The target labels involved in the embodiments of the present disclosure may be category labels, color labels, and the like. The category labels are used to indicate the object category corresponding to each point in the point cloud, and different category labels may be set according to different fields. For example, in the field of autonomous driving, the collected point cloud mainly reflects the data of the traffic environment, so object categories such as vehicles, pedestrians, traffic signs, traffic lights, trees, etc. may be set. Each category may further include multiple subcategories. For example, vehicles may include subcategories such as cars, trucks, and buses. In the medical field, the point cloud mainly reflects the data of the body model, so organ categories such as the heart, lungs, and liver may be set.

[0093] In this way, the determined target points can be "labeled" with target labels, that is, the target labels are stored in correspondence with the determined target points in the point cloud. If selecting an N-gon annotation box on the image once (i.e., inputting the information of the N-gon annotation box on the image) cannot complete the labeling of all target points, the user can select multiple times and "label" the target points determined by the multiple selections with target labels.

[0094] Furthermore, during the above-mentioned multiple selections, the user can input an image adjustment instruction to the point cloud annotation device, and the image adjustment instruction includes camera position and / or viewing angle information, that is, the adjusted target camera position and / or viewing angle information. In response to the input image adjustment instruction, the point cloud annotation device can reproject the point cloud according to the camera position and / or viewing angle information included in the image adjustment instruction, obtain the adjusted image, and display the adjusted image on the user interface. In other words, the user can adjust the above-mentioned image during the multiple selections, and complete the target point annotation of a target label from images of multiple viewing angles.

[0095] After completing the target point marking of one target label, you can continue to mark the target points of the next target label. The marking process of each target label is similar and will not be repeated here.

[0096] For the marked target points, the user can input editing instructions through shortcut keys, preset mouse operations, preset gesture operations, or triggering specific components on the user interface to edit the marked target points. In response to the input editing instructions, the point cloud annotation device can edit the target points stored corresponding to the target labels. The editing process includes: adding target points, removing target points, repeating the last editing process, or undoing the last editing process.

[0097] For example, if the user wants to add a target point to the target label, the user can select the target point by clicking, selecting a box, etc., and then trigger the add component provided on the user interface to trigger the edit instruction. The point cloud annotation device adds the newly selected target point to the target points stored corresponding to the target label. For another example, if the user wants to delete a target point from the target label, the user can select the target point by clicking, selecting a box, etc., and then trigger the delete component provided on the user interface to trigger the edit instruction. The point cloud annotation device deletes the newly selected target point from the target points stored corresponding to the target label.

[0098] The above method provided by the embodiment of the present disclosure can be applied to a variety of application scenarios, including but not limited to autonomous driving scenarios, map construction scenarios, smart transportation scenarios, AI medical scenarios, and the like.

[0099] Taking the autonomous driving scenario as an example, autonomous driving vehicles rely on high-quality training data to acquire driving decision-making capabilities, that is, using training data to train driving decision models. The higher the quality of the training data, the more accurate the driving decision model and the safer the vehicle. Autonomous driving vehicles are equipped with three-dimensional scanning devices such as lidar and millimeter-wave radar to obtain point cloud data of the vehicle's surrounding environment. It is usually necessary to annotate the acquired point cloud data, that is, annotate points belonging to each type (for example, annotate points belonging to each type of obstacle), so as to train the driving decision model.

[0100] Taking a frame of point cloud obtained from the scene as an example, after the GPU image rendering, an image of the frame of point cloud is obtained (that is, the image is obtained by projecting the frame of point cloud onto the image plane of the camera), which is called a point cloud image in this embodiment. Assume that Figure 4 For this scenario, a specific point cloud annotation process provided by the embodiment of the present disclosure is as follows:

[0101] In step 1101, the server sends the point cloud data to be annotated to the point cloud annotation device of the client, and provides a user interface to the user through the client, wherein the user interface includes a point cloud image corresponding to the point cloud data to be annotated.

[0102] In step 1103, the user triggers the label creation component provided on the user interface to send a label creation instruction to the point cloud annotation device, and the instruction includes information of the target label. Accordingly, in step 1105, in response to the label creation instruction, the target label is created and sent to the server for storage.

[0103] Creating a target tag can be seen as creating a container object corresponding to the tag. The container object is used to store the target point information that needs to be labeled with the tag. The container object is used to realize the associated storage of the target tag and the target point. For example, if a user wants to Figure 4 The street light label is annotated in the image shown, and a container object is created for the street light label. The container object is used to store the target point corresponding to the street light.

[0104] After the label is created, in step 1107, the user can select the midpoint of the point cloud on the point cloud image through the user interface, that is, the user observes the point cloud image and enters an N-sided annotation box on the point cloud image, and tries to include the points corresponding to the street lamp in the N-sided annotation box. The client sends the information of the N-sided annotation box to the point cloud annotation device. During this period, the user can trigger the annotation box adjustment instruction by moving, dragging, etc., so as to adjust the position, size, shape, etc. of the N-sided annotation box.

[0105] In step 1109, the point cloud annotation device uses Figure 2The process shown determines the target point, and stores the determined target point in association with the target tag, that is, in the container object of the street light tag. Figure 2 The implementation details in the process shown can be found in the relevant records in the previous embodiments, which will not be repeated here.

[0106] The user can repeat the above selection process to continue to determine the target point and store the newly determined target point in the container object of the street light label. During this period, the user can input an image adjustment instruction to the point cloud annotation device to adjust the distance (distance from the camera) and viewing angle of the point cloud image displayed on the user interface.

[0107] The user can also edit the target point stored in the target tag by inputting an editing command, such as adding a target point, removing a target point, repeating the last editing process, or undoing the last editing process.

[0108] If the point cloud annotation of the street light label is completed, the user can confirm the completion of the annotation of the street light label through the user interface. The process can also go to step 1103 to create a next target label to perform point cloud annotation of the next label.

[0109] It can be seen that the point cloud annotation device provided in the embodiment of the present disclosure actually provides a point cloud annotation tool for users. Users only need to input an N-gon annotation box on the image to realize the annotation of target points without the need to annotate point by point, which greatly improves the annotation speed and efficiency.

[0110] The above-mentioned point cloud annotation tool determines the target point through the signed distance from the point in the point cloud to the construction plane. It only needs to use the camera position and the vertices of the N-gon annotation box to determine the construction plane and calculate the signed distance from the point to the plane. The CPU does not need to perform projection between the three-dimensional space and the two-dimensional plane for each point in the point cloud, which greatly reduces the calculation complexity, reduces the calculation pressure of the CPU, and improves the annotation efficiency.

[0111] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] According to an embodiment of another aspect, a point cloud annotation device is provided. Fig.12 FIG. 2 is a schematic block diagram of a point cloud annotation device in the form of a computer program according to an embodiment. Fig.12As shown, the device 1200 includes: a labeling frame acquisition unit 1201, a plane construction unit 1202 and a target point determination unit 1203, and may further include a point cloud acquisition unit 1204, an image rendering unit 1205, a label creation unit 1206, a target point storage unit 1207 and a target point editing unit 1208. The main functions of each component unit are as follows:

[0113] The annotation box acquisition unit 1201 is configured to acquire information of an N-sided annotation box input on an image, where N is a positive integer greater than 2, and the image is obtained by projecting the point cloud onto an image plane of a camera.

[0114] The plane construction unit 1202 is configured to determine N construction planes according to the position point of the camera and any two adjacent vertices of the N-gon annotation box.

[0115] The target point determination unit 1203 is configured to determine the target point according to the directed distances from the points in the point cloud to the N construction planes.

[0116] Furthermore, the point cloud acquisition unit 1204 may first acquire the point cloud to be annotated, and then the image rendering unit 1205 may project the point cloud to be annotated onto the image plane of the camera to obtain the image, and present it to the user through the user interface.

[0117] Furthermore, the plane construction unit 1202 may be further configured to: transform the vertex coordinates of the N-gon annotation box into a point cloud coordinate system of the point cloud.

[0118] As one of the feasible methods, the target point determination unit 1203 can be specifically configured as follows: using the normal vectors of N construction planes, determine the signed distances of the points in the point cloud to the N construction planes respectively; if the normal vector takes a direction toward the N-gonal annotation box, and the signed distances of the points in the point cloud to the N construction planes are all greater than or equal to 0, then the point is determined to be the target point; or, if the normal vector takes a direction away from the N-gonal annotation box, and the signed distances of the points in the point cloud to the N construction planes are all less than or equal to 0, then the point is determined to be the target point.

[0119] Among them, when the target point determination unit 1203 uses the normal vectors of N construction planes to determine the signed distances from a point in the point cloud to the N construction planes, it can be specifically configured as: determining the vector pointing from a point on the construction plane to a point in the point cloud; performing a dot multiplication of the vector and the normal vector of the construction plane to obtain the signed distance from the point in the point cloud to the construction plane.

[0120] As one of the feasible methods, if the normal vector takes a direction toward the N-gon annotation box, the normal vector of the construction plane is obtained by the outer product of the first vector and the second vector; if the normal vector takes a direction away from the N-gon annotation box, the normal vector of the construction plane is obtained by the outer product of the second vector and the first vector; wherein the first vector is a vector pointing from the position point of the camera on the construction plane to the first vertex, the second vector is a vector pointing from the position point of the camera on the construction plane to the second vertex, and the first vertex and the second vertex are two adjacent vertices in the clockwise direction of the N-gon annotation box.

[0121] Furthermore, the label creation unit 1206 is configured to create a target label in response to an input label creation instruction.

[0122] The target point storage unit 1207 is configured to store the target tag and the target point in the determined point cloud in correspondence.

[0123] As one of the achievable methods, the target point editing unit 1208 is configured to edit the target point stored corresponding to the target tag in response to an input editing instruction, and the editing process includes: adding a target point, removing a target point, repeating the last editing process, or undoing the last editing process.

[0124] As another achievable manner, the image rendering unit 1205 may be further configured to, in response to an input image adjustment instruction, reproject the point cloud according to the camera position and / or viewing angle information included in the image adjustment instruction to obtain an adjusted image.

[0125] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0126] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The program includes instructions, and when the instructions are executed by one or more processors of a computing device, the steps of the method described in any one of the aforementioned method embodiments are executed.

[0127] The point cloud annotation device provided in the embodiment of the present disclosure may be implemented in the form of an electronic device. In this case, Fig.13 As shown in , the point cloud annotation device includes one or more processors 1301 and a memory 1302 for storing one or more programs, which are executed by the one or more processors 1301 to implement the method flow shown in the above embodiments of the present disclosure and / or program units corresponding to each unit in the device.

[0128] The various components are interconnected using different buses and can be mounted on a common motherboard or in other ways as needed. The processor 1301 can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the user interface on an external input / output device (such as a display device coupled to the interface). In other embodiments, if desired, multiple processors and / or multiple buses can be used with multiple memories and multiple memories.

[0129] The processor 1301 may include one or more single-core processors or multi-core processors. The processor 1301 may include any combination of general-purpose processors or dedicated processors (such as image processors, application processors, baseband processors, etc.).

[0130] The memory 1302 is a computer-readable storage medium provided by the present disclosure, which can be used to store non-transient software programs, non-transient computer executable programs and units, such as the following in the embodiments of the present disclosure: Figure 2 The processor 1301 executes the non-transient software programs, instructions and units stored in the memory 1302, thereby executing the above method embodiments. Figure 2 Programs, instructions and units corresponding to the point cloud annotation method shown.

[0131] The point cloud annotation device may further include: an input device 1303 and an output device 1304. The processor 1301, the memory 1302, the input device 1303 and the output device 1304 may be connected via a bus or other means. Fig.13 The example of connecting through bus is taken in the following.

[0132] The input device 1303 can receive input digital or character information, and generate signal input related to user settings and function control of the point cloud annotation device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1304 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0133] The above-mentioned programs (also referred to as software, software applications, or codes) include machine instructions for programmable processors, and these computer programs can be implemented using object-oriented programming languages, assembly or machine languages.

[0134] With the development of time and technology, the meaning of medium is becoming more and more extensive, and the propagation path of computer programs is no longer limited to tangible media, and can also be downloaded directly from the network, etc. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can be used but not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by or in combination with instruction execution systems, devices or devices.

[0135] The technical solution provided by the present disclosure is described in detail above. The principles and implementation methods of the present disclosure are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present disclosure. At the same time, for those skilled in the art, according to the idea of ​​the present disclosure, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present disclosure.

[0136] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A point cloud annotation method, characterized in that: The method comprises: Obtain information of an N-sided annotation box of an image, where N is a positive integer greater than 2, and the image is obtained by projecting the point cloud onto an image plane of a camera; Determine N construction planes according to the position point of the camera and any two adjacent vertices of the N-gon annotation box; and A target point is determined according to a directed distance from a point in the point cloud to the N construction planes.

2. The method according to claim 1, characterized in that The method further comprises: The vertex coordinates of the N-gon annotation box are converted to the point cloud coordinate system of the point cloud.

3. The method according to claim 1, characterized in that Determining the target point according to the directed distances from the point in the point cloud to the N construction planes comprises: Using the normal vectors of the N construction planes, determining the directed distances from the points in the point cloud to the N construction planes respectively; If the normal vector is oriented toward the N-gon annotation box, and the directed distances of the points in the point cloud to the N construction planes are all greater than or equal to 0, then the point is determined to be the target point; or, If the normal vector takes a direction away from the N-gon annotation box, and the directed distances of the point in the point cloud to the N construction planes are all less than or equal to 0, then the point is determined to be the target point.

4. The method according to claim 3, characterized in that Determining the directed distances from the points in the point cloud to the N construction planes respectively using the normal vectors of the N construction planes includes: Determine a vector from a point on the construction plane to a point in the point cloud; The vector is multiplied by the normal vector of the construction plane to obtain the directed distance from the point in the point cloud to the construction plane.

5. The method according to claim 3, characterized in that: If the normal vector is oriented toward the N-gon annotation box, the normal vector of the construction plane is obtained by the outer product of the first vector and the second vector; If the normal vector is in a direction away from the N-gon annotation box, the normal vector of the construction plane is obtained by the outer product of the second vector and the first vector; Among them, the first vector is a vector pointing from the camera's position point on the construction plane to the first vertex, the second vector is a vector pointing from the camera's position point on the construction plane to the second vertex, and the first vertex and the second vertex are two adjacent vertices of the N-gon annotation box in the clockwise direction from the camera's perspective.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: In response to the input label creation instruction, create a target label; The target tag is stored correspondingly to the determined target point in the point cloud.

7. The method according to claim 6, characterized in that The method further comprises: In response to the input editing instruction, the target point stored corresponding to the target tag is edited, and the editing process includes: adding a target point, removing a target point, repeating the last editing process, or undoing the last editing process.

8. The method according to any one of claims 1 to 5, characterized in that The method further comprises: In response to the input image adjustment instruction, the point cloud is re-projected according to the camera position and / or viewing angle information included in the image adjustment instruction to obtain an adjusted image.

9. A point cloud annotation device, characterized in that: The device comprises: one or more processors, and A memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a program, the program comprising instructions, which, when executed by one or more processors of a computing device, cause the computing device to perform the method according to any one of claims 1 to 8.

Images