Point cloud annotation method, device, equipment, medium and program product

Abstract

The present disclosure provides a point cloud annotation method, device, equipment, medium and product, which relate to the field of computer technology, especially to artificial intelligence, autonomous driving, computer vision and deep learning technology. The specific implementation scheme is: according to each point cloud frame in a continuous point cloud frame sequence, a three-dimensional outer frame is generated for the object to be annotated in the point cloud frame; the three-dimensional outer frame is a rectangular parallelepiped; the fit between the three-dimensional outer frame in the point cloud frame and the object to be annotated is evaluated; the key point cloud frame is determined from the continuous point cloud frame sequence according to the fit determined by the evaluation; according to the posture relationship between the objects to be annotated in each point cloud frame, the three-dimensional outer frame in the key point cloud frame is adjusted in posture, and then synchronized to the non-key point cloud frame as the three-dimensional outer frame of the object to be annotated. The embodiment of the present disclosure can improve the accuracy and efficiency of point cloud annotation.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to artificial intelligence, autonomous driving, computer vision and deep learning technology. Background Art

[0002] 3D point clouds are important data for autonomous driving technology to perceive the surrounding environment. In order to train a model that can identify obstacles from 3D point clouds, a large number of labeled 3D point cloud samples are required, that is, the outer bounding box and attributes of the object are marked in the 3D point cloud as training model samples.

[0003] In the prior art, in order to accurately annotate the bounding box of an object in a 3D point cloud, it is necessary to annotate multiple consecutive frames of point clouds where the object appears one by one, which is a time-consuming and complicated task. Usually, it relies on manual frame-by-frame annotation, or simple automated tools to generate the bounding box and then manually adjust it. It can be seen that the prior art requires a lot of manpower costs to annotate point clouds. Summary of the invention

[0004] The present disclosure provides a point cloud annotation method, apparatus, device, medium and program product.

[0005] According to one aspect of the present disclosure, a point cloud annotation method is provided, comprising:

[0006] According to each point cloud frame in the continuous point cloud frame sequence, a three-dimensional outer bounding box is generated for each object to be annotated in the point cloud frame;

[0007] Evaluating the fit between the three-dimensional bounding box in the point cloud frame and the object to be annotated;

[0008] Determining a key point cloud frame from the continuous point cloud frame sequence according to the evaluated fit;

[0009] According to the posture relationship between the objects to be annotated in each point cloud frame, the three-dimensional outer frame in the key point cloud frame is adjusted in posture and then synchronized to the non-key point cloud frame as the three-dimensional outer frame of the object to be annotated.

[0010] According to another aspect of the present disclosure, a point cloud annotation device is provided, comprising:

[0011] The outer frame generation module is used to generate a three-dimensional outer frame for the object to be annotated in the point cloud frame according to each point cloud frame in the continuous point cloud frame sequence;

[0012] A fit evaluation module, used to evaluate the fit between the three-dimensional outer bounding box in the point cloud frame and the object to be annotated;

[0013] A key frame determination module, used to determine a key point cloud frame from the continuous point cloud frame sequence according to the evaluated fit degree;

[0014] The outer frame synchronization module is used to adjust the posture of the three-dimensional outer frame in the key point cloud frame according to the posture relationship between the objects to be annotated in each point cloud frame, and synchronize it to the non-key point cloud frame as the three-dimensional outer frame of the object to be annotated.

[0015] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the point cloud annotation method described in any embodiment of the present disclosure.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the point cloud annotation method described in any embodiment of the present disclosure.

[0020] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the point cloud annotation method described in any embodiment of the present disclosure is implemented.

[0021] The technical solution of the disclosed embodiment can determine the key point cloud frames from a continuous point cloud frame sequence through the evaluation of fit, and synchronize the three-dimensional outer frame in the key point cloud frame to other point cloud frames, thereby greatly reducing the time cost of manual labeling.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0024] Figure 1 is a schematic diagram of an operation interface of a point cloud annotation method provided according to an embodiment of the present disclosure;

[0025] Figure 2 is a flow chart of a point cloud annotation method provided according to an embodiment of the present disclosure;

[0026] Figure 3A is a flow chart of a point cloud annotation method provided according to an embodiment of the present disclosure;

[0027] Figure 3B It is a two-dimensional outer frame annotation schematic diagram provided according to an embodiment of the present disclosure;

[0028] Figure 4 is a flow chart of a point cloud annotation method provided according to an embodiment of the present disclosure;

[0029] Figure 5 is a schematic diagram of a fitting result provided according to an embodiment of the present disclosure;

[0030] Fig. 6A is a flow chart of a point cloud annotation method provided according to an embodiment of the present disclosure;

[0031] Figure 6B is a schematic diagram of obtaining a visible surface according to an embodiment of the present disclosure;

[0032] Figure 7 is a structural schematic diagram of a point cloud annotation device disclosed in an embodiment of the present disclosure;

[0033] Figure 8 It is a block diagram of an electronic device used to implement the point cloud annotation method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0035] In related technologies, point clouds are generally a large number of point cloud points collected by lidar, which are distributed in three-dimensional space to form point cloud frames. The point cloud frame acquisition source used to collect point cloud data is usually a collection vehicle, which collects point cloud data of the surrounding environment while stationary or moving. Based on point cloud data, various objects in the surrounding environment can be identified. In order to assist the environmental perception needs of autonomous driving, it is generally necessary to identify objects that serve as obstacles. The main categories may include pedestrians, cars, trucks or other obstacles. By marking the outline of the object in the point cloud frame and indicating the object identification, category and other attribute information, a labeled point cloud frame is formed, which can be used as a sample to train the perception model.

[0036] When performing point cloud annotation, manual participation is often required, such as Figure 1FIG. 1 is a schematic diagram of an operation interface of a point cloud annotation method provided by an embodiment of the present disclosure. Multiple windows may be displayed in the annotation interface, including a point cloud frame window 101, at least one projection surface window 102, a function menu 103, an operation control 104, etc. Figure 1 As shown, the point cloud frame window 101 shows the distribution of point cloud points in the three-dimensional point cloud frame from an angle. The point cloud frame acquisition source 101a can be used as the center, and the dense point cloud set around it generally corresponds to an object 101b. If you choose to mark an object in the point cloud frame, you can project the point cloud set in the area where the object is located to the set projection plane. The projection plane is set according to the needs. It can be optionally including a top view plane, a side view plane and a main view plane, which are parallel to the plane of the object's own coordinate system of the object to be marked. Specifically, the object's own coordinate system is an XYZ coordinate system, which is a right-handed coordinate system with the center of the object as the origin and the front as the positive direction of the X-axis. The top view plane is parallel to the XY plane, the side view plane is parallel to the XZ plane, and the main view plane is parallel to the YZ plane. The user can input or adjust the projection outer frame of the object in the projection plane, so that the three-dimensional outer frame of the object in the three-dimensional space can be calculated based on each projection outer frame. The three-dimensional outer frame is a cuboid, and its bottom surface is parallel to the XY plane. Of course, by using different three-dimensional outer frame generation algorithms, outer frames of different shapes can be obtained. For example, other polyhedrons can also form outer frames, such as other three-dimensional outer frames surrounded by rectangular surfaces.

[0037] Most objects in the real world are rigid, and their general shape and size do not change. For example, the size and shape of the three-dimensional outer box of various motor vehicles and traffic signs can be considered to remain unchanged. For objects such as pedestrians, their outlines may change or not change under different annotation requirements. In the embodiment of the present disclosure, point cloud annotation is performed based on the premise that the object outline remains unchanged, and a specific implementation scheme is given.

[0038] Figure 2 This is a flow chart of a point cloud annotation method provided according to an embodiment of the present disclosure. The embodiment of the present disclosure can be applied to the case where a human-computer interactive annotation interface is used to automatically annotate objects in continuous point cloud frames or to assist manual annotation. The method can be executed by a point cloud annotation device, which can be implemented in hardware and / or software and can be configured in an electronic device with computing and storage capabilities, such as a server or a personal computer. Figure 2 , the method specifically includes the following:

[0039] S201, generating a three-dimensional outer bounding box for an object to be annotated in each point cloud frame in a continuous point cloud frame sequence;

[0040] S202, evaluating the fit between the three-dimensional outer bounding box in the point cloud frame and the object to be annotated;

[0041] S203, determining a key point cloud frame from a sequence of continuous point cloud frames according to the degree of fit determined by the evaluation;

[0042] S204, adjusting the posture of the 3D outer frame in the key point cloud frame according to the posture relationship between the objects to be annotated in each point cloud frame, and synchronizing it to the 3D outer frame in the non-key point cloud frame as the object to be annotated.

[0043] Among them, the collection vehicle generally continuously collects point cloud data of the surrounding environment, so the same object usually appears continuously in multiple point cloud frames and needs to be labeled. Therefore, when a certain object is used as the current object to be labeled, an identification ID can be first established for the object to be labeled, and multiple point cloud frames including the object to be labeled can be obtained as a continuous point cloud frame sequence. Then, a three-dimensional outer bounding box can be generated for the object to be labeled in the point cloud frame based on the set outer bounding box generation algorithm; the three-dimensional outer bounding box is a cuboid. The outer bounding box generation algorithm, for example, is an axis-aligned bounding box algorithm (Axis-Aligned Bounding Box, AABB) or an oriented bounding box algorithm (Oriented Bounding Box, OBB). The generated outer bounding box is a cuboid, and the cuboid has some fixed relationship with the object's own coordinate system. For example, the bottom of the outer bounding box is generally parallel to the ground, the side of the outer bounding box is generally perpendicular to the ground, and the outer bounding box generally only has an angle around the axis perpendicular to the ground (Z axis). The initially generated 3D bounding box is often not accurate enough and will be affected by many factors such as the noisy point cloud or missing point cloud around the object.

[0044] Furthermore, it is necessary to select key point cloud frames from the continuous point cloud frame sequence. The key point cloud frame is the point cloud frame with the highest fit between the three-dimensional outer bounding box and the object contour. Therefore, the fit of the three-dimensional outer bounding box initially generated in each point cloud frame can be evaluated, and then the key point cloud frame can be selected based on the fit. Preferably, a three-dimensional outer bounding box is generated for each point cloud frame and the fit is evaluated, but optionally, other auxiliary strategies can be used to screen some point cloud frames for fit evaluation. The so-called fit of the outer bounding box refers to the degree of match between the outer bounding box and the actual distribution of the point cloud data. Among them, the three-dimensional outer bounding box that can surround the point cloud set of the object and has the smallest size is generally considered to have the best fit. When different algorithms are used to evaluate the fit, the specific method and amount of data to characterize the fit are different. Since there may be point cloud points of other objects around the object, or the collected point cloud points may be inaccurate, or the point cloud points may be missing due to the change of the object's posture or the object being far away from the collection vehicle, these are equivalent to noise, which will cause differences in the three-dimensional outer frames of the object formed in different point cloud frames.

[0045] After the key point cloud frame is determined, since the three-dimensional outer frame in the key point cloud frame has the best fit, the three-dimensional outer frame of the key point cloud frame can be synchronously copied to other non-key point cloud frames without changing its size and shape.

[0046] According to the pose relationship between the objects to be annotated in each point cloud frame, the 3D outer frame in the key point cloud frame is adjusted in pose and then synchronized to the 3D outer frame in the non-key point cloud frame as the object to be annotated. Of course, as the acquisition source or object moves, the pose of the object relative to the acquisition source will change, so it is also necessary to adjust the pose of the 3D outer frame in the key point cloud frame according to the pose relationship between the objects to be annotated in each point cloud frame, so as to match the position and pose of the object in the non-key point cloud frame.

[0047] The technical solution of the disclosed embodiment can determine the key point cloud frames from a continuous point cloud frame sequence through the evaluation of fit, and synchronize the three-dimensional outer frame in the key point cloud frame to other point cloud frames, so that the object outer frame in each point cloud frame has a high degree of fit, thereby reducing the workload required for manual secondary adjustment, thereby greatly reducing the time cost of manual labeling.

[0048] On the basis of the above embodiment, the object to be annotated and its corresponding continuous point cloud frame sequence can be predetermined. Specifically, before generating a three-dimensional outer frame for the object to be annotated in the point cloud frame according to each point cloud frame in the continuous point cloud frame sequence, the method further includes:

[0049] The point clouds in each point cloud frame are clustered, and a continuous point cloud frame sequence including the same object to be labeled is determined according to the position and shape of the clustered point cloud set; wherein each point cloud frame in the continuous point cloud frame sequence includes the object to be labeled.

[0050] The point clouds in each point cloud frame are clustered. Clustering refers to dividing the point cloud data into different categories according to certain features. Each point contains information such as position and reflection intensity. The purpose of clustering is to divide the point cloud data into groups with similar features to form at least one clustered point cloud set. The clustering method includes but is not limited to: K-means algorithm, density-based clustering algorithm, hierarchical clustering algorithm, Gaussian mixture clustering algorithm, Euclidean clustering algorithm, density subtraction clustering algorithm or adaptive density clustering algorithm, etc., which is not limited in the embodiment of the present disclosure. A clustered point cloud set describes the position and shape corresponding to an object to be labeled. For example, the object to be labeled can be a vehicle, a garbage bin or a pedestrian. The position corresponding to the object to be labeled can be determined according to the spatial position coordinates of each point in the clustered point cloud set, or the center point corresponding to the outline formed by each point in the clustered point cloud set can be determined, and the position corresponding to the object to be labeled can be determined according to the center point. The object to be labeled can be three-dimensional, and the outline information of a three-dimensional object exists in at least one point cloud frame, and each point cloud frame is a continuous point cloud frame. Therefore, according to the position and shape of the clustered point cloud set, at least one point cloud frame including the same object to be annotated can be determined, and each point cloud frame can be converted into a continuous point cloud frame sequence. After clustering, point cloud sets corresponding to multiple objects may be obtained, and point cloud annotation can be performed separately for the continuous point cloud frames of each object.

[0051] By clustering the point clouds in each point cloud frame and determining the continuous point cloud frame sequence including the same object to be labeled according to the position and shape of the clustered point cloud set, only the continuous point cloud frame sequence corresponding to the same object to be labeled is retained, reducing the attention to the remaining point cloud frames corresponding to the object, reducing the data volume of the point cloud frames to be analyzed, and improving the efficiency of point cloud labeling.

[0052] Figure 3A This is a flow chart of a point cloud annotation method provided by another embodiment of the present disclosure. Based on the above embodiment, this embodiment further optimizes the specific scheme for generating a three-dimensional outer bounding box. Figure 3A As shown:

[0053] S310, generating a three-dimensional outer bounding box for each object to be annotated in the point cloud frame according to each point cloud frame in the continuous point cloud frame sequence; the three-dimensional outer bounding box is a cuboid;

[0054] The above operation S310 specifically includes the following:

[0055] S311, projecting a point cloud set of the object to be annotated in the point cloud frame onto at least one projection surface to form a projected point cloud set;

[0056] S312, within a set angle range, rotating the projected point cloud set along a first direction at a set interval angle to obtain a plurality of rotated point cloud sets;

[0057] S313, generating a two-dimensional outer bounding box for each rotation point cloud set;

[0058] S314, calculating the loss value of each two-dimensional outer bounding box based on at least one loss function;

[0059] S315, selecting a target two-dimensional outer frame that meets the fitting condition on the projection surface according to the loss values ​​of each two-dimensional outer frame;

[0060] S316, rotating the target two-dimensional outer frame on each selected projection surface in the opposite direction of the first direction to restore to the initial angle;

[0061] S317, generating a three-dimensional outer bounding box of the object to be labeled according to the two-dimensional outer bounding boxes of each target after being rotated in the opposite direction;

[0062] S320, evaluating the fit between the three-dimensional bounding box in the point cloud frame and the object to be annotated;

[0063] S330, determining a key point cloud frame from a sequence of continuous point cloud frames according to the degree of fit determined by the evaluation;

[0064] S340, according to the posture relationship between the objects to be annotated in each point cloud frame, adjust the posture of the three-dimensional outer frame in the key point cloud frame, and synchronize it to the non-key point cloud frame as the three-dimensional outer frame of the object to be annotated.

[0065] The operation of generating a three-dimensional outer frame is to make the three-dimensional outer frame more closely match the orientation of the object. The three-dimensional outer frame generated by a general algorithm is usually parallel to the plane of the object's own XYZ coordinate system. The orientation of the initially generated three-dimensional outer frame is often inaccurate, with a small range of deviation (-10 to 10 degrees), which makes the three-dimensional outer frame fit poorly. The technical solution of this embodiment can effectively solve this problem.

[0066] The point cloud set of the object to be annotated in the point cloud frame is projected onto at least one projection surface, that is, the point cloud set corresponding to the object to be annotated in the point cloud frame is projected onto the top view surface, the side view surface and the main view surface. The operations on any projection surface are similar. The spatial coordinates of each point in the point cloud set of the object to be annotated and the spatial coordinates of each projection plane are known, and the position of each point on the plane after projection can be calculated by the geometric relationship from the point to the plane. For each projection surface, the projection point cloud set can be obtained according to the projection points of each point corresponding to the object to be annotated projected onto the projection surface. The projection point cloud set corresponding to the top view surface, the projection point cloud set corresponding to the side view surface and the projection point cloud set corresponding to the main view surface can be obtained. Preferably, taking the three-dimensional outer frame of a cuboid as an example, it can be projected onto three sets of parallel planes of the cuboid, namely the top view surface, the side view surface and the main view surface. Those skilled in the art can understand that if the three-dimensional outer frame is a polyhedron of other shapes, the surface of the outer frame of other shapes can also be used as a parallel plane of the projection surface for projection.

[0067] A projected point cloud set is located in the same plane, that is, the center of the outline of the object formed by the projected point cloud set is the rotation center, and a rotation of 360 degrees is started from any position in the projection plane. The object's own coordinate system is the XYZ coordinate system, the top view plane is parallel to the XY plane, the side view plane is parallel to the XZ plane, and the main view plane is parallel to the YZ plane. Optionally, the top view plane and the side view plane select a point in the direction of the X axis as the starting position for rotation, and the main view plane selects a point in the direction of the Y axis as the starting position for rotation. In any projection plane, it is optional to divide 360 ​​degrees into four areas, that is, 90 degrees is an angle interval range, and the three planes can be divided into 4 rotation angle intervals. The rotation interval angle can be preset. In each angle interval range, for example, δ is the search range, and the optional angle value range of δ is -90 to 90 degrees. The projected point cloud sets are rotated one by one along the first direction according to the set interval angle θ, rotated once, and all points scanned by this rotation are determined as a rotation point cloud set, and multiple rotation point cloud sets can be obtained. The first direction can be clockwise. The value of the interval angle θ can be set to 1 degree, etc. Figure 3B The figure shows a schematic diagram of the projection of the object to be annotated on the top view. The set of all points on the plane is the projection point cloud set. The center of the outline of the object formed by the projection point cloud set is taken as the rotation center. The projection point cloud set is rotated clockwise by an angle of θ to generate a projection two-dimensional outer frame.

[0068] The two-dimensional bounding box can be a two-dimensional rectangular box, which can surround all points in a rotated point cloud set. Edge points can be extracted from the point cloud in the rotated point cloud set. For each edge point, select the next neighboring point adjacent to it to form a straight line, and use the straight line and the straight line perpendicular to it to determine the direction of the minimum bounding rectangle, and calculate a minimum two-dimensional bounding rectangle. By traversing all edge points, a two-dimensional bounding box corresponding to a rotated point cloud set can be obtained. For each rotated point cloud set, the two-dimensional bounding box corresponding to each rotated point cloud set is generated in the above manner. The rotated point cloud set and the two-dimensional bounding box correspond one to one, such as Figure 3B shown.

[0069] The loss function is used to estimate the difference. The loss value of the two-dimensional outer frame corresponding to each rotation point cloud set can be calculated by at least one loss function. According to the loss value of each two-dimensional outer frame, the two-dimensional outer frame that meets the fitting condition on the projection surface is selected as the target two-dimensional outer frame. The fitting condition can be that each loss value is the smallest. The target two-dimensional outer frame is screened from the two-dimensional outer frames corresponding to each rotation point cloud set. Then, the rotation angle of the target two-dimensional outer frame is adjusted, and the target two-dimensional outer frame on each selected projection surface is rotated in the opposite direction of the first direction to restore to the initial angle, that is, each target two-dimensional outer frame is rotated to the starting rotation position corresponding to each projection surface. If the first direction is clockwise, the opposite direction of the first direction is counterclockwise.

[0070] Furthermore, the target two-dimensional bounding boxes on different projection surfaces corresponding to the same object to be labeled can be reversely calculated according to the projection direction to obtain the three-dimensional bounding box of the object to be labeled.

[0071] The technical solution of the embodiment of the present disclosure rotates the point cloud set on different projection planes so that the formed outer bounding box fits the orientation of the object.

[0072] In the above embodiment, a variety of methods may be used to calculate the loss value (Loss). Optionally, the loss value of each two-dimensional outer bounding box is calculated based on at least one loss function, including at least one of the following:

[0073] The area of ​​each two-dimensional outer frame is calculated as the first loss value; the sum of the distances between each point cloud point and the nearest border in the two-dimensional outer frame is calculated as the second loss value; the sum of the variances of the point cloud sets near the two borders opposite to the two-dimensional outer frame is calculated as the third loss value.

[0074] Specifically, at least one loss function can be used to calculate the loss value of each two-dimensional outer frame from three aspects, for example, the area of ​​each two-dimensional outer frame, the sum of the distances between each point cloud point and the nearest border in the two-dimensional outer frame, and the sum of the variances of the point cloud sets near the two borders opposite to the two-dimensional outer frame. The more the orientation of the two-dimensional outer frame matches the object, the smaller the area of ​​the two-dimensional outer frame formed; the sum of the distances between the point cloud points and the edges of the two-dimensional outer frame reflects the relationship between each point cloud point and the edge. The smaller the sum of the distances, the higher the fit of the outer frame. This loss function can test the local fit. For example, when the vehicle has a rearview mirror, the contour of the local position may affect the fit of the generated two-dimensional frame. The sum of the distances can reduce the local impact; the sum of the variances of the point cloud sets near the two borders opposite to the two-dimensional outer frame, for example, can be the variance of the point sets closest to the left and right, plus the variance of the point sets closest to the top and bottom. The calculation of this loss can make the overall fit of the two-dimensional frame and the vehicle better.

[0075] By judging the position and posture relationship of the point cloud points of each two-dimensional outer frame, the degree of fit of each two-dimensional outer frame on the projection surface is determined, and the loss value of each two-dimensional outer frame is calculated based on multi-dimensional data, thereby improving the comprehensiveness and accuracy of the loss value calculation.

[0076] Optionally, according to the loss values ​​of each two-dimensional outer frame, selecting a target two-dimensional outer frame that meets the fitting condition on the projection surface includes:

[0077] A two-dimensional outer frame that meets at least one of the following fitting conditions is selected as the target two-dimensional outer frame: the first loss value is the smallest; the second loss value is the smallest; and the third loss value is the smallest.

[0078] Specifically, the smaller the area of ​​the calculated two-dimensional outer frame is, the closer the two-dimensional outer frame is to the area of ​​the object to be annotated, and the smaller the first loss value corresponding to the two-dimensional outer frame is; the smaller the sum of the distances between each point cloud point and the nearest border in the two-dimensional outer frame is, the less the impact of the local protrusion of the object on the two-dimensional frame can be reduced; the smaller the sum of the variances of the point cloud sets near the two borders opposite to the two-dimensional outer frame is, the higher the fit between the two-dimensional frame and the overall edge of the object. Therefore, the preset fit satisfaction condition can be the minimum first loss value, the minimum second loss value and / or the minimum third loss value. According to the first loss value, the second loss value and the third loss value corresponding to each two-dimensional outer frame, they are sorted respectively, and the two-dimensional outer frame that meets at least one fit condition is used as the target two-dimensional outer frame.

[0079] By presetting multiple fitting conditions, each two-dimensional outer frame is screened, and the two-dimensional outer frame that best fits the projection surface is obtained as the target two-dimensional outer frame corresponding to the object to be labeled. The two-dimensional outer frame with the highest fit is determined, thereby improving the screening accuracy of the two-dimensional outer frame.

[0080] Figure 4 This is a flow chart of a point cloud annotation method provided by another embodiment of the present disclosure. Based on the above embodiments, this embodiment further provides a specific solution for evaluating the degree of fit. Figure 4 As shown:

[0081] S410, generating a three-dimensional outer bounding box for each object to be annotated in the point cloud frame according to each point cloud frame in the continuous point cloud frame sequence; the three-dimensional outer bounding box is a cuboid;

[0082] S420, evaluating the fit between the three-dimensional outer bounding box in the point cloud frame and the object to be annotated;

[0083] The above operation S420 specifically includes the following:

[0084] S421, using an L-shape fitting algorithm to calculate at least one fitting right-angled edge of the point cloud set on at least one projection surface; wherein the projection surface includes a top view surface, a side view surface, and a main view surface, which are respectively parallel to the plane of the object's own coordinate system;

[0085] S422, calculating the distance difference between the right-angled side of the projected outer bounding box of the three-dimensional outer bounding box in the point cloud frame on the projection surface and each fitted right-angled side as the fit degree.

[0086] S430, determining a key point cloud frame from a sequence of continuous point cloud frames according to the degree of fit determined by the evaluation;

[0087] S440, according to the posture relationship between the objects to be annotated in each point cloud frame, adjust the posture of the three-dimensional outer frame in the key point cloud frame, and synchronize it to the non-key point cloud frame as the three-dimensional outer frame of the object to be annotated.

[0088] In order to mitigate the impact of appearance changes, an L-shape feature consisting of a corner point and two lines can reduce unexpected errors caused by appearance changes. The L-shape fitting algorithm is used to process two-dimensional point cloud data. It is based on a geometric shape matching idea. Its core is to find a structure similar to an "L" shape in a two-dimensional point cloud. In order to match the fit, the edge points of the two-dimensional point cloud can be found and fitted to generate multiple "L"-shaped fitting right-angled edges. The fit of the projected outer frame of the point cloud frame is evaluated based on these fitting right-angled edges. Figure 5 As shown, the projection bounding box of the object to be annotated on a certain projection surface and several fitted right-angled edges generated based on the fitting of the edge point cloud points are shown. The angle difference between each right-angled edge of the projection bounding box and the nearest fitted right-angled edge can be calculated respectively, and the gap between the projection bounding box and the fitted right-angled edge can be obtained by the average or accumulation method as the fit. The gap between the right-angle points can be determined by the Euclidean distance between the right-angle points. A three-dimensional bounding box has a fit. The greater the fit, the more accurate the position area of ​​the object to be annotated by the three-dimensional bounding box. The fit corresponding to each three-dimensional bounding box can be compared, and the fit between the three-dimensional bounding box in the point cloud frame and the object to be annotated is evaluated, and the point cloud frame corresponding to the maximum fit is selected as the key point cloud frame.

[0089] The technical solution of the disclosed embodiment obtains at least one fitting right-angled edge of the point cloud set on the projection surface through an L-shape fitting algorithm, and calculates the angle difference between the right-angled edge of the two-dimensional outer bounding box on the projection surface and each fitting right-angled edge. It can be determined whether the two-dimensional outer bounding box accurately marks the area of ​​the object to be labeled on the projection surface, which is conducive to obtaining the most accurate two-dimensional outer bounding box for the object to be labeled.

[0090] On the basis of the solutions provided in the above embodiments, in addition to selecting key point cloud frames based on the degree of fit, auxiliary strategies may also be used for selection.

[0091] Optionally, a point cloud annotation method also includes: using at least one of the following auxiliary screening strategies to sort the point cloud frames: sorting each point cloud frame according to the distance between the object to be annotated and the acquisition source in the point cloud frame; sorting each point cloud frame according to the number and density of the point cloud set of the object to be annotated; sorting each point cloud frame according to the area size of the fitting box of the point cloud set of the object to be annotated.

[0092] Specifically, the closer the relative distance between the object to be annotated and the acquisition source is, the more complete the size and acquisition completeness of the object to be annotated are, and the greater the probability that the point cloud frame where the object to be annotated is located is determined as a key point cloud frame. Therefore, the point cloud frames can be sorted according to the distance between the object to be annotated and the acquisition source in the point cloud frame.

[0093] In the key point cloud frame, the scanning completeness of the object to be labeled is higher, the point cloud is denser, and the corresponding point cloud points are also more. Each point cloud frame can be sorted according to the number and density of the point cloud sets of the object to be labeled.

[0094] When the object to be annotated is blocked or the scan is incomplete, the point cloud frames can be sorted according to the area of ​​the fitting frame of the object to be annotated. The larger the area of ​​the fitting frame, the more likely the corresponding point cloud frame is to be determined as a key point cloud frame.

[0095] Based on the above principle, as an auxiliary screening strategy, the distance between the object to be annotated corresponding to each point cloud frame and the acquisition source in the point cloud frame, the number and density of the point cloud set of the object to be annotated, and the area of ​​the fitting frame of the point cloud set of the object to be annotated are calculated. The weights corresponding to the values ​​calculated by the three auxiliary screening strategies can be preset, and the point cloud frames can be sorted according to the size of the value corresponding to the auxiliary screening strategy with the largest weight value. At least one auxiliary screening strategy can be selected from the three auxiliary screening strategies to calculate the value corresponding to each point cloud frame, and the point cloud frames can be sorted according to the value.

[0096] When multiple point cloud frames are selected based on the fit, the selected point cloud frames can be sorted based on the auxiliary screening strategy, so as to select the point cloud frame with the highest order as the key point cloud frame. Alternatively, all point cloud frames can be sorted based on the auxiliary screening strategy, and then the fit is calculated from the top several point cloud frames, and then the key point cloud frame is determined based on the fit.

[0097] By screening the two-dimensional bounding boxes corresponding to the objects to be labeled through a multi-dimensional screening strategy, the target two-dimensional bounding box can be determined, thereby determining the key point cloud frame corresponding to the objects to be labeled, thereby improving the accuracy of key point cloud frame determination.

[0098] Optionally, the degree of fit between the three-dimensional bounding box in the point cloud frame and the object to be labeled is evaluated, including: selecting a set number of point cloud frames or point cloud frames that meet set conditions from the sorted point cloud frames; and evaluating the degree of fit between the three-dimensional bounding box in the point cloud frame and the object to be labeled.

[0099] Specifically, after the point cloud frames are sorted using at least one auxiliary screening strategy, a set number or set conditions may be preset, and the embodiments of the present disclosure are not limited to this. For example, a set number of point cloud frames may be selected from the sorted point cloud frames according to the set number. Or a set condition may be preset, and the set condition may be at least one of a threshold value less than the distance between the object to be labeled and the acquisition source in the point cloud frame, a threshold value greater than or equal to the number and density of the point cloud set of the object to be labeled, or a threshold value greater than the area of ​​the fitting frame of the point cloud set of the object to be labeled, or all the set conditions must be met to filter out at least one point cloud frame. For the point cloud frame, the fit between the three-dimensional outer frame in the point cloud frame and the object to be labeled is evaluated, and the point cloud frame corresponding to the three-dimensional outer frame with the largest fit is determined as the key point cloud frame corresponding to the object to be labeled.

[0100] Each point cloud frame is filtered by presetting the filtering quantity and filtering conditions, thereby reducing the number of point cloud frames to be analyzed and improving the efficiency of data processing of point cloud frames.

[0101] Fig. 6A This is a flowchart of a point cloud annotation method provided by another embodiment of the present disclosure. Based on the previous embodiments, this embodiment optimizes the outer frame synchronization method.

[0102] S610, generating a three-dimensional outer bounding box for each object to be annotated in the point cloud frame according to each point cloud frame in the continuous point cloud frame sequence; the three-dimensional outer bounding box is a cuboid;

[0103] S620, evaluating the fit between the three-dimensional bounding box in the point cloud frame and the object to be annotated;

[0104] S630, determining a key point cloud frame from a sequence of continuous point cloud frames according to the degree of fit determined by the evaluation;

[0105] S640, adjusting the posture of the 3D outer frame in the key point cloud frame according to the posture relationship between the objects to be annotated in each point cloud frame, and synchronizing it to the 3D outer frame in the non-key point cloud frame as the object to be annotated;

[0106] The above operation S640 specifically includes the following:

[0107] S641, identifying at least two mutually perpendicular visible edges of the object to be annotated in the projection plane of each point cloud frame; wherein the visible edge is an edge of the object to be annotated that is visible from a viewing angle relative to a point cloud frame acquisition source;

[0108] S642, adjusting the posture of the three-dimensional outer frame in the key point cloud frame according to the posture relationship between the visible edges of the object to be annotated in each point cloud frame;

[0109] S643, copying the three-dimensional bounding box after the posture adjustment to each non-key frame as the three-dimensional bounding box of the object to be labeled.

[0110] Among them, the point cloud frame acquisition source can scan the point cloud data of the object within a certain angle range, and the angle range is determined as the visible viewing angle of the point cloud frame. If a certain area of ​​the object is not within the angle range that the point cloud frame acquisition source can collect, the area of ​​the object is determined as an invisible area. The visible edge can be the edge of the visible viewing angle of the object to be annotated when the point cloud frame acquisition source collects the point cloud data of the object. According to the L-shape fitting algorithm, at least two mutually perpendicular visible edges of the object to be annotated in the projection surface of each point cloud frame can be identified, and the position and posture information of the visible edges can be obtained. According to the position and posture information of the visible edges, the edge of the three-dimensional outer frame of the object to be annotated in the key point cloud frame that is located on the visible viewing angle is aligned with the visible edge, the three-dimensional outer frame of the object to be annotated in the key point cloud frame is translated and rotated, and the position and posture of the three-dimensional outer frame in the key point cloud frame are adjusted to make the position of the three-dimensional outer frame corresponding to the object to be annotated more accurate. In this process, since the size of the object's 3D bounding box remains unchanged, the 3D bounding box with adjusted posture is copied to each non-key frame as the 3D bounding box of the object to be annotated, and then the position of the 3D bounding box in the key point cloud frame in the non-key frame is determined.

[0111] By determining the position of the visible edge corresponding to the object to be labeled, the position of the 3D outer frame corresponding to the object to be labeled is adjusted to make the adjusted 3D outer frame more closely fit the object to be labeled. According to the rigidity of the object, the 3D outer frame with adjusted posture is copied to each non-key frame, and the position information of the 3D outer frame in the non-key frame is synchronized to improve the consistency of the data.

[0112] Optionally, identifying at least two mutually perpendicular visible edges of the object to be labeled in the projection surface of each point cloud frame includes: unifying the coordinate system of the point cloud frame acquisition source with the coordinate system of the object to be labeled; connecting the center of the point cloud frame acquisition source and the center of the object to be labeled to form a first line; calculating the angle between the first line and the horizontal axis of the coordinate system; if the angle is within a set angle range, determining that the two mutually perpendicular borders close to the point cloud frame acquisition source in the projection envelope of the point cloud frame projection surface of the object to be labeled are visible edges.

[0113] The orientation of the object may be different from that of the collection vehicle, so the origin and XY axis of the object to be annotated and the collection vehicle will be different. Unify the coordinate system of the point cloud frame acquisition source with the coordinate system of the object to be annotated, and convert the coordinate system of the point cloud frame acquisition source into the coordinate system of the object to be annotated through the conversion relationship between the coordinate systems, so as to determine the corresponding relationship between the position of the point cloud frame acquisition source and the object to be annotated in the same coordinate system. Connect the center of the point cloud frame acquisition source and the center of the object to be annotated, and determine the connecting line of the two centers as the first connecting line. According to the geometric relationship, calculate the angle between the first connecting line and the pointing direction of the horizontal axis (X axis) of the coordinate system of the point cloud frame acquisition source. The scanning angle of the point cloud frame acquisition source that can obtain point cloud data can be preset. The scanning angle can be determined according to the acquisition device corresponding to the point cloud frame acquisition source, or it can be preset according to industry experience. If the angle is within the set angle range, it is determined that the two mutually perpendicular borders close to the point cloud frame acquisition source in the projection outer frame of the projection surface of the point cloud frame of the object to be annotated are visible edges. For example, the relative position of the acquisition source coordinates and the fitted right-angled edge calculated by the fitting algorithm is used to determine whether the right-angled edge is a visible surface. After the acquisition source is rotated to the coordinate system of the object to be annotated, the visible surface can be determined by the angle between the line connecting the origin (the center of the object to be annotated) and the center point of the acquisition source and the forward direction of the object's X-axis. If the angle is in the range of 0 to 90 degrees, the forward and left faces of the object to be annotated can be preliminarily determined to be visible surfaces. Figure 6B shown.

[0114] By using the coordinate system of the point cloud frame acquisition source and the pose relationship of the object to be annotated, the visible surface of the object to be annotated is determined, and the visible edge is determined based on the visible surface, so that the position of the three-dimensional outer frame can be adjusted according to the position of the visible edge. The side of the visible edge can usually collect more point cloud data better, so the outer frame formed on the side of the visible edge is more accurate. The three-dimensional outer frame is adjusted based on the visible edge, which improves the accuracy of the three-dimensional outer frame position. The technical solution of the disclosed embodiment can effectively assist manual annotation work and reduce labor costs.

[0115] Optionally, a point cloud annotation method also includes: displaying a projection bounding box of the object to be annotated on at least one projection surface of one or more point cloud frames in the annotation interface; in response to a position adjustment made by a user on any projection bounding box, recalculating the position of the three-dimensional bounding box while keeping the size of the three-dimensional bounding box unchanged, and synchronously updating it to other point cloud frames.

[0116] Specifically, the point cloud set of the object to be annotated on the point cloud frame can be projected on at least one projection surface, and the projection surface can be a front view surface, a side view surface, and a top view surface. Figure 1As shown. The point cloud information of the projection envelope on the projection surface can be displayed on the annotation interface to visualize the point cloud information of the projection envelope. The user can adjust the position of any projection envelope according to user needs, and the adjustment method can be: mouse adjustment, keyboard adjustment, adjustment according to user voice, and adjustment according to preset adjustment text specifications. While keeping the size of the 3D envelope unchanged, obtain the coordinate position of the 3D envelope, recalculate the position of the 3D envelope, and copy it to other point cloud frames.

[0117] By manually adjusting the position of each projection outer frame again, the point cloud annotation results are more in line with the annotation requirements, reducing the point cloud annotation error and improving the accuracy of point cloud annotation.

[0118] Optionally, a point cloud annotation method further includes: performing jump recognition on the attributes of the object to be annotated in the point cloud frame, and prompting the user when an attribute jump is recognized.

[0119] Among them, the attributes of the object to be labeled may be descriptive information of the type of the object to be labeled. The attributes of the object to be labeled may be the name identifier, type identifier, quality description information and volume description text of the object to be labeled. The object to be labeled can be described by the attributes of the object to be labeled. In a point cloud frame sequence of an object to be labeled, if at least one attribute information different from the attribute of the object to be labeled appears in each point cloud frame sequence, it indicates that the attribute of the object in the point cloud frame has jumped, which is often not in line with the actual situation. Therefore, when an attribute jump is identified, a prompt is given to the user, and the prompt method includes but is not limited to: information prompt, flash prompt, or pop-up prompt, etc., which is not limited to the embodiment of the present disclosure.

[0120] By identifying the attribute jumps corresponding to the objects to be annotated in the point cloud frame and providing timely user prompts, the consistency of the objects to be annotated in the point cloud frame sequence can be ensured, the impact of point cloud noise can be reduced, and the accuracy of point cloud annotation can be improved.

[0121] According to an embodiment of the present disclosure, Figure 7 is a structural diagram of a point cloud annotation device in an embodiment of the present disclosure, and the embodiment of the present disclosure is applicable to the case of annotating point clouds. The device is implemented by software and / or hardware, and is specifically configured in an electronic device with certain data computing capabilities.

[0122] like Figure 7 The point cloud annotation device 700 shown in the figure includes: an outer frame generation module 701, a fit evaluation module 702, a key frame determination module 703 and an outer frame synchronization module 704.

[0123] The outer frame generation module 701 is used to generate a three-dimensional outer frame for the object to be annotated in the point cloud frame according to each point cloud frame in the continuous point cloud frame sequence;

[0124] A fit evaluation module 702 is used to evaluate the fit between the three-dimensional bounding box in the point cloud frame and the object to be annotated;

[0125] A key frame determination module 703 is used to determine a key point cloud frame from a continuous point cloud frame sequence according to the evaluated fit degree;

[0126] The outer frame synchronization module 704 is used to adjust the posture of the 3D outer frame in the key point cloud frame according to the posture relationship between the objects to be annotated in each point cloud frame, and synchronize it to the non-key point cloud frame as the 3D outer frame of the object to be annotated.

[0127] According to the technical solution disclosed in the present invention, by clustering the point clouds in each point cloud frame, and determining the continuous point cloud frame sequence including the same object to be labeled according to the position and shape of the clustered point cloud set, only the continuous point cloud frame sequence corresponding to the same object to be labeled is retained, and the attention to the remaining point cloud frames corresponding to the object is reduced, thereby reducing the data volume of the point cloud frames to be analyzed and improving the efficiency of point cloud labeling.

[0128] Optionally, the three-dimensional outer frame is a rectangular parallelepiped. As mentioned above, each object has its own coordinate system, and the above specifically introduces the vehicle's own XYZ coordinate system. Based on the set outer frame generation algorithm, a three-dimensional outer frame can be generated for the object. Preferably, a three-dimensional outer frame of a rectangular parallelepiped surrounding the object point cloud is generated based on the object's XYZ coordinate system, and the bottom surface of the rectangular parallelepiped is parallel to the XY plane. Of course, those skilled in the art will understand that based on the setting method of other coordinate systems and the outer frame generation algorithm, other polyhedron outer frames can also be generated to establish a corresponding relationship with the object coordinate system.

[0129] Furthermore, the fit assessment module includes:

[0130] A right-angle side calculation unit, used to calculate at least one fitting right-angle side of the point cloud set on at least one projection surface by using an L-shape fitting algorithm; wherein the projection surface includes a top view surface, a side view surface and a main view surface, which are parallel to the plane of the object's own coordinate system respectively;

[0131] The gap calculation unit is used to calculate the angle difference between the right-angle side of the projected outer frame of the three-dimensional outer frame in the point cloud frame on the projection surface and each fitting right-angle side as the fit degree.

[0132] Furthermore, the point cloud annotation device further includes:

[0133] The continuous frame determination module is used to cluster the point clouds in each point cloud frame, and determine a continuous point cloud frame sequence including the same object to be labeled according to the position and shape of the clustered point cloud set; wherein each point cloud frame in the continuous point cloud frame sequence includes the object to be labeled.

[0134] Furthermore, the outsourcing frame generation module includes:

[0135] A point cloud projection unit, used to project a point cloud set of an object to be annotated in a point cloud frame onto at least one projection surface to form a projected point cloud set;

[0136] A point cloud set acquisition unit, used to rotate the projected point cloud set along a first direction at a set interval angle within a set angle range to obtain a plurality of rotated point cloud sets;

[0137] A two-dimensional frame generating unit, used to generate a two-dimensional outer bounding box for each rotation point cloud set;

[0138] A loss value calculation unit, used to calculate the loss value of each two-dimensional outer bounding box based on at least one loss function;

[0139] A target frame selection unit, used for selecting a target two-dimensional outer frame that meets the fitting condition on the projection surface according to the loss values ​​of each two-dimensional outer frame;

[0140] An outer frame rotation unit, used to rotate the target two-dimensional outer frame on each selected projection surface in the opposite direction of the first direction to restore it to an initial angle;

[0141] The three-dimensional frame generating unit is used to generate a three-dimensional outer frame of the object to be labeled according to each target two-dimensional outer frame rotated in the opposite direction.

[0142] Furthermore, the loss value calculation unit includes:

[0143] An area calculation subunit, used to calculate the area of ​​each two-dimensional outer frame as a first loss value;

[0144] A distance calculation subunit, used to calculate the sum of the distances between each point cloud point and the nearest bounding box in the two-dimensional outer box as a second loss value;

[0145] The variance calculation subunit is used to calculate the sum of the variances of the point clouds near the two borders relative to the two-dimensional outer box as the third loss value.

[0146] Furthermore, the target frame selection unit is used to:

[0147] Select a 2D outer bounding box that meets at least one of the following fitting conditions as the target 2D outer bounding box:

[0148] The first loss value is the smallest;

[0149] The second loss value is the smallest;

[0150] The third loss value is the smallest.

[0151] Furthermore, the point cloud annotation device further includes:

[0152] The point cloud frame sorting module is used to sort the point cloud frames using at least one of the following auxiliary screening strategies:

[0153] Sort each point cloud frame according to the distance between the object to be annotated and the acquisition source in the point cloud frame;

[0154] Sort each point cloud frame according to the number and density of the point cloud set of the object to be annotated;

[0155] The point cloud frames are sorted according to the area of ​​the fitting frame of the point cloud set of the object to be annotated.

[0156] Furthermore, the fit assessment module includes:

[0157] A point cloud frame screening unit, used for selecting a set number of point cloud frames or point cloud frames that meet set conditions from the sorted point cloud frames;

[0158] The three-dimensional box evaluation unit is used to evaluate the fit between the three-dimensional outer box in the point cloud frame and the object to be annotated.

[0159] Furthermore, the outsourced frame synchronization module includes:

[0160] A visible edge recognition unit, used to recognize at least two mutually perpendicular visible edges of the object to be annotated in the projection plane of each point cloud frame; wherein the visible edge is an edge of the object to be annotated that is visible from a viewing angle relative to the acquisition source of the point cloud frame;

[0161] A posture adjustment unit, used for adjusting the posture of the three-dimensional outer frame in the key point cloud frame according to the posture relationship between the visible edges of the object to be annotated in each point cloud frame;

[0162] The three-dimensional frame copying unit is used to copy the three-dimensional outer frame after the posture adjustment to each non-key frame as the three-dimensional outer frame of the object to be labeled.

[0163] Furthermore, the visible edge identification module is used to:

[0164] Unify the coordinate system of the point cloud frame acquisition source with the coordinate system of the object to be annotated;

[0165] Connecting the center of the point cloud frame acquisition source and the center of the object to be annotated to form a first connection line;

[0166] Calculate the angle between the first connecting line and the horizontal axis of the coordinate system;

[0167] If the angle is within the set angle range, it is determined that the two mutually perpendicular borders of the object to be annotated in the projection envelope of the point cloud frame projection surface and close to the point cloud frame acquisition source are visible edges.

[0168] Furthermore, the point cloud annotation device further includes:

[0169] An outer frame display module, used to display the projection outer frame of the object to be annotated on at least one projection surface of one or more point cloud frames in the annotation interface;

[0170] The position synchronization module is used to respond to the position adjustment made by the user in any projection outer frame, recalculate the position of the three-dimensional outer frame while keeping the size of the three-dimensional outer frame unchanged, and synchronously update it to other point cloud frames.

[0171] Furthermore, the point cloud annotation device further includes:

[0172] The jump prompt module is used to identify jumps in the attributes of the object to be annotated in the point cloud frame, and to prompt the user when an attribute jump is identified.

[0173] The above-mentioned point cloud annotation device can execute the point cloud annotation method provided by any embodiment of the present disclosure, and has corresponding functional modules and beneficial effects for executing the point cloud annotation method.

[0174] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0175] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0176] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0177] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0178] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0179] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as method XXX. For example, in some embodiments, method XXX may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the method XXX described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform method XXX in any other appropriate manner (e.g., by means of firmware).

[0180] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0182] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0184] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0185] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.

[0186] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0187] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for technical applications such as artificial intelligence and blockchain, as well as model training.

[0188] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0189] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A point cloud annotation method, comprising: According to each point cloud frame in the continuous point cloud frame sequence, a three-dimensional outer bounding box is generated for each object to be annotated in the point cloud frame; Evaluating the fit between the three-dimensional bounding box in the point cloud frame and the object to be annotated; Determining a key point cloud frame from the continuous point cloud frame sequence according to the evaluated fit; According to the posture relationship between the objects to be annotated in each point cloud frame, the three-dimensional outer frame in the key point cloud frame is adjusted in posture and then synchronized to the non-key point cloud frame as the three-dimensional outer frame of the object to be annotated.

2. The method according to claim 1, wherein: The three-dimensional outer frame is a cuboid; The degree of fit between the three-dimensional bounding box in the point cloud frame and the object to be annotated is evaluated, including: Using an L-shape fitting algorithm to calculate at least one fitting right-angle edge of the point cloud set on at least one projection surface; wherein the projection surface includes a top view surface, a side view surface, and a main view surface, which are parallel to the plane of the object's own coordinate system; The angle difference between the right-angle side of the projected outer frame of the three-dimensional outer frame in the point cloud frame on the projection surface and each of the fitted right-angle sides is calculated as the fit degree.

3. The method according to claim 1, before generating a three-dimensional outer bounding box for the object to be annotated in each point cloud frame in the continuous point cloud frame sequence, further comprises: The point clouds in each point cloud frame are clustered, and a continuous point cloud frame sequence including the same object to be labeled is determined according to the position and shape of the clustered point cloud set; wherein each point cloud frame in the continuous point cloud frame sequence includes the object to be labeled.

4. The method according to claim 1, wherein: According to each point cloud frame in the continuous point cloud frame sequence, a three-dimensional outer bounding box is generated for the object to be annotated in the point cloud frame, including: Projecting a point cloud set of the object to be annotated in the point cloud frame onto at least one projection surface to form a projected point cloud set; Within a set angle range, the projected point cloud set is rotated along a first direction at a set interval angle to obtain a plurality of rotated point cloud sets; Generating a two-dimensional outer bounding box for each of the rotation point cloud sets; Based on at least one loss function, calculating a loss value of each two-dimensional outer bounding box; According to the loss values ​​of each two-dimensional outer frame, a target two-dimensional outer frame that meets the fitting conditions on the projection surface is selected; Rotate the target two-dimensional outer frame on each selected projection surface in the opposite direction of the first direction to restore it to the initial angle; The three-dimensional bounding box of the object to be labeled is generated according to the two-dimensional bounding boxes of each target after being rotated in the opposite direction.

5. The method according to claim 4, wherein: The calculating the loss value of each two-dimensional outer bounding box based on at least one loss function includes at least one of the following: Calculate the area of ​​each two-dimensional outer frame as the first loss value; Calculate the sum of the distances between each point cloud point and the nearest border in the two-dimensional outer frame as the second loss value; The sum of the variances of the point clouds near the two borders relative to the two-dimensional outer frame is calculated as the third loss value.

6. The method according to claim 5, wherein: According to the loss value of each two-dimensional outer frame, a target two-dimensional outer frame that meets the fitting conditions on the projection surface is selected, including: Select a 2D outer bounding box that meets at least one of the following fitting conditions as the target 2D outer bounding box: The first loss value is the smallest; The second loss value is the smallest; The third loss value is the smallest.

7. The method according to claim 1, further comprising: At least one of the following auxiliary filtering strategies is used to sort the point cloud frames: Sort each point cloud frame according to the distance between the object to be annotated and the acquisition source in the point cloud frame; Sort each point cloud frame according to the number and density of the point cloud set of the object to be annotated; The point cloud frames are sorted according to the area of ​​the fitting frame of the point cloud set of the object to be annotated.

8. The method according to claim 7, wherein: The degree of fit between the three-dimensional bounding box in the point cloud frame and the object to be annotated is evaluated, including: Select a set number of point cloud frames or point cloud frames that meet set conditions from the sorted point cloud frames; The fit between the three-dimensional bounding box in the point cloud frame and the object to be labeled is evaluated.

9. The method according to claim 1 or 2, wherein: According to the posture relationship between the objects to be annotated in each point cloud frame, the three-dimensional outer frame in the key point cloud frame is adjusted in posture and synchronized to the three-dimensional outer frame in the non-key point cloud frame as the object to be annotated, including: Identify at least two mutually perpendicular visible edges of the object to be annotated in the projection plane of each point cloud frame; wherein the visible edge is an edge of the object to be annotated that is visible from a viewing angle relative to a point cloud frame acquisition source; According to the posture relationship between the visible edges of the object to be annotated in each point cloud frame, the posture of the three-dimensional outer frame in the key point cloud frame is adjusted; The 3D bounding box after pose adjustment is copied to each non-key frame as the 3D bounding box of the object to be labeled.

10. The method according to claim 9, wherein: The step of identifying at least two mutually perpendicular visible edges of the object to be annotated in the projection plane of each point cloud frame includes: Unifying the coordinate system of the point cloud frame acquisition source with the coordinate system of the object to be annotated; Connecting the center of the point cloud frame acquisition source and the center of the object to be annotated to form a first connecting line; Calculating the angle between the first connecting line and the horizontal axis of the coordinate system; If the angle is within the set angle range, it is determined that two mutually perpendicular borders of the object to be annotated in the projection envelope frame of the point cloud frame projection surface and close to the point cloud frame acquisition source are visible edges.

11. The method according to claim 1, further comprising: Displaying a projection envelope of the object to be annotated on at least one projection surface of one or more point cloud frames in the annotation interface; In response to the position adjustment made by the user on any projected bounding box, the position of the three-dimensional bounding box is recalculated while keeping the size of the three-dimensional bounding box unchanged, and is synchronously updated to other point cloud frames.

12. The method according to claim 1, further comprising: The attributes of the object to be annotated in the point cloud frame are recognized to jump, and the user is prompted when an attribute jump is recognized.

13. A point cloud annotation device, comprising: The outer frame generation module is used to generate a three-dimensional outer frame for the object to be annotated in the point cloud frame according to each point cloud frame in the continuous point cloud frame sequence; A fit evaluation module, used to evaluate the fit between the three-dimensional outer bounding box in the point cloud frame and the object to be annotated; A key frame determination module, used for determining a key point cloud frame from the continuous point cloud frame sequence according to the evaluated fit degree; The outer frame synchronization module is used to adjust the posture of the three-dimensional outer frame in the key point cloud frame according to the posture relationship between the objects to be annotated in each point cloud frame, and synchronize it to the non-key point cloud frame as the three-dimensional outer frame of the object to be annotated.

14. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the point cloud annotation method described in any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the point cloud annotation method according to any one of claims 1-12.

16. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the point cloud annotation method according to any one of claims 1-12.

Images