Point cloud data labeling method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a point cloud data labeling method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of photoelectricity. The method comprises the following steps: acquiring a point cloud sequence acquired by acquisition equipment deployed on a target vehicle, wherein the point cloud sequence comprises a plurality of point cloud frames; identifying a static object and a dynamic object in the point cloud sequence; for each static object, determining a first point cloud frame group related to the static object from the point cloud sequence, randomly selecting one point cloud frame from the first point cloud frame group, and marking the static object in the point cloud frame; and for each dynamic object, determining a second point cloud frame group related to the dynamic object from the point cloud sequence, selecting a part of point cloud frames from the second point cloud frame group, and labeling the dynamic object in each point cloud frame in the part of point cloud frames. According to the embodiment of the invention, the problems that single-frame labeling is too slow and dynamic objects are difficult to process by fixed-frame-splicing labeling are avoided.

Description

Technical Field

[0001] The present application relates to the field of optoelectronic technology. Specifically, the present application relates to a point cloud data annotation method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] Point cloud data is three-dimensional data obtained by sensors such as LiDAR, which provides rich spatial information. Annotated point cloud data can be used to train deep learning models to recognize and classify objects in a three-dimensional environment, which is particularly important for self-driving cars to recognize pedestrians, vehicles, and road signs. It provides rich spatial information. However, the complexity and high-dimensional characteristics of point cloud data make its annotation a challenge. Summary of the invention

[0003] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for annotating point cloud data, which can solve the above-mentioned problems of the prior art. The technical solution is as follows:

[0004] According to one aspect of an embodiment of the present application, a method for labeling point cloud data is provided, the method comprising:

[0005] Obtaining a point cloud sequence collected by a collection device deployed on a target vehicle, wherein the point cloud sequence includes a plurality of point cloud frames;

[0006] identifying static objects and dynamic objects in the point cloud sequence;

[0007] For each static object, determining a first point cloud frame group related to the static object from the point cloud sequence, selecting a point cloud frame from the first point cloud frame group, and marking the static object in the point cloud frame;

[0008] For each dynamic object, a second point cloud frame group related to the dynamic object is determined from the point cloud sequence, a portion of point cloud frames is selected from the second point cloud frame group, and the dynamic object is annotated in each point cloud frame of the portion of point cloud frames.

[0009] According to another aspect of an embodiment of the present application, a point cloud data annotation device is provided, the device comprising:

[0010] A point cloud acquisition module, used to obtain a point cloud sequence collected by a collection device deployed on a target vehicle, wherein the point cloud sequence includes a plurality of point cloud frames;

[0011] An object recognition module, used to recognize static objects and dynamic objects in the point cloud sequence;

[0012] a static processing module, configured to determine, for each static object, a first point cloud frame group related to the static object from the point cloud sequence, select a point cloud frame from the first point cloud frame group, and mark the static object in the point cloud frame;

[0013] A dynamic processing module is used to determine, for each dynamic object, a second point cloud frame group related to the dynamic object from the point cloud sequence, select a portion of point cloud frames from the second point cloud frame group, and mark the dynamic object in each point cloud frame in the portion of point cloud frames.

[0014] According to another aspect of an embodiment of the present application, an electronic device is provided. The electronic device includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above method.

[0015] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0016] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0017] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0018] For static objects, after marking the object's annotation box in a displayed point cloud frame, since the static object does not move relative to the acquisition device, after marking a point cloud frame, the annotation box can be automatically included in each point cloud frame from the start time value to the end time. For dynamic objects, this application does not mark all point cloud frames where the dynamic object is located, but only marks a small number of point cloud frames, avoiding the problem of slow single-frame annotation and difficulty in handling dynamic objects with fixed frame annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0020] Figure 1 A schematic diagram of the system architecture for implementing a method for annotating point cloud data provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a process for labeling point cloud data provided in an embodiment of the present application;

[0022] Figure 3A schematic diagram of an interface for playing a point cloud sequence provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a process for marking objects in a point cloud frame provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a first expansion frame and a coordinate system provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of a process for labeling point cloud data provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the structure of a point cloud data annotation device provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0029] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "an" and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we call an element "connected" or "coupled" to another element, the one element can be directly connected or coupled to the other element, or it can refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here may include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0030] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0031] The point cloud data annotation method provided in this application can be applied to the field of intelligent transportation, and specifically to the field of dynamic driving of intelligent transportation.

[0032] First, several terms involved in this application are introduced and explained:

[0033] Intelligent Traffic System (ITS), also known as Intelligent Transportation System, is an effective and comprehensive application of advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) to transportation, service control and vehicle manufacturing, strengthening the connection between vehicles, roads and users, thus forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy. Or;

[0034] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), referred to as IVICS, is a development direction of Intelligent Transportation Systems (ITS). IVICS uses advanced wireless communications and new generation Internet technologies to implement all-round dynamic real-time information interaction between vehicles and roads, and conducts active vehicle safety control and road cooperative management based on the collection and integration of dynamic traffic information in all time and space, fully realizing the effective coordination of people, vehicles and roads, ensuring traffic safety, and improving traffic efficiency, thus forming a safe, efficient and environmentally friendly road traffic system.

[0035] Autonomous driving, also known as driverless or self-driving cars, is a technology that uses various sensors, controllers, computer vision and artificial intelligence technologies to achieve autonomous driving of vehicles. Autonomous driving cars are able to complete driving tasks without human intervention, achieving safe and efficient travel from starting point to destination.

[0036] Point cloud data is three-dimensional data obtained by sensors such as LiDAR, which provides rich spatial information. Annotated point cloud data can be used to train deep learning models to recognize and classify objects in a three-dimensional environment, which is particularly important for self-driving cars to recognize pedestrians, vehicles, and road signs. It provides rich spatial information. However, the complexity and high-dimensional characteristics of point cloud data make its annotation a challenge.

[0037] Current methods of manually annotating point cloud data have some significant shortcomings:

[0038] Point cloud 3D box annotation requires manual adjustment of the box size, orientation, and position, which is labor-intensive and complex, and cannot guarantee a perfect fit between the annotation box and the point cloud.

[0039] The point cloud data annotation method, device, electronic device, computer-readable storage medium and computer program product provided in this application are intended to solve the above technical problems in the prior art.

[0040] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0041] Figure 1 A schematic diagram of the system architecture for implementing the point cloud data annotation method provided in an embodiment of the present application, wherein the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other servers. The point cloud data annotation method is executed by the terminal 102 or the server 104 alone, or by the terminal 102 and the server 104 in collaboration. In some embodiments, the point cloud data annotation method is executed by terminal 102, which obtains a point cloud sequence collected by an acquisition device deployed on a target vehicle, wherein the point cloud sequence includes multiple point cloud frames; identifies static objects and dynamic objects in the point cloud sequence; for each static object, determines a first point cloud frame group related to the static object from the point cloud sequence, selects one point cloud frame from the first point cloud frame group, and annotates the static object in the point cloud frame; for each dynamic object, determines a second point cloud frame group related to the dynamic object from the point cloud sequence, selects a portion of point cloud frames from the second point cloud frame group, and annotates the dynamic object in each point cloud frame in the portion of point cloud frames.

[0042] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 102 and the server 104 can be directly or indirectly connected via wired or wireless communications, and this application does not limit this.

[0043] The present application provides a method for labeling point cloud data. Figure 2 As shown, the method includes:

[0044] S101. Obtain a point cloud sequence collected by a collection device deployed on a target vehicle, wherein the point cloud sequence includes a plurality of point cloud frames.

[0045] In the embodiment of the present application, a collection device can be deployed on the target vehicle, and the collection device can be a laser radar. The collection device collects data for a period of time to obtain a point cloud sequence. It can be understood that the point cloud sequence is a plurality of point cloud frames arranged in chronological order according to the collection time.

[0046] The embodiment of the present application does not limit the type of target vehicle, for example, it can be a mine car. Mine cars are mainly used in specific scenarios such as mines and engineering projects, and usually operate in non-road environments, such as narrow-gauge railways inside mines or rugged mine roads. These environments often have high requirements for the durability, carrying capacity and off-road performance of the vehicle. Compared with ordinary vehicles, mine cars are usually larger in size, wider in body, and more complicated to operate.

[0047] The target vehicle in the embodiment of the present application can obtain a point cloud sequence by the acquisition device while it is driving, or it can obtain a point cloud sequence by the acquisition device when it is stopped, and the embodiment of the present application does not make specific limitations.

[0048] S102, identifying static objects and dynamic objects in the point cloud sequence;

[0049] It can be understood that the static object in the embodiment of the present application refers to an object that is stationary relative to the acquisition device, and the dynamic object also refers to an object that is moving relative to the acquisition device.

[0050] The embodiment of the present application can use artificial intelligence to identify each point cloud frame in a point cloud sequence, determine the object in the point cloud frame, and further determine whether the object is a static object or a dynamic object based on whether the position of the object changes significantly in multiple consecutive frames.

[0051] It should be understood that, in a point cloud sequence, the same object may be a static object in some point cloud frames and a dynamic object in other point cloud frames.

[0052] S103, for a static object, determining a first point cloud frame group related to the static object from the point cloud sequence, selecting a point cloud frame from the first point cloud frame group, and marking the static object in the point cloud frame;

[0053] For a dynamic object, a second point cloud frame group related to the dynamic object is determined from the point cloud sequence, a portion of point cloud frames is selected from the second point cloud frame group, and the dynamic object is marked in each point cloud frame in the portion of point cloud frames.

[0054] See also Figure 3 , which exemplarily shows a schematic diagram of an interface for playing a point cloud sequence provided in an embodiment of the present application. As shown in the figure, the interface provides a playback area 301 for playing the point cloud sequence and a playback progress bar 302. The staff can drag the sliding control on the playback progress bar 302. It can be seen that there are two sliding controls in the embodiment of the present application, wherein the first sliding control 303 is used to control the starting point cloud frame for playback, and the second sliding control 304 is used to control the starting point cloud frame for playback. The interface also displays various objects pre-identified by artificial intelligence and the point cloud frame group 305 corresponding to each object. As shown in the figure, the present application recognizes that object 1 appears in frames 1 to 35 of the point cloud sequence, object 2 appears in frames 24 to 60 of the point cloud sequence, and object 3 appears in frames 61 to 82 of the point cloud sequence.

[0055] For static objects in the embodiment of the present application, if their positions do not change within the complete time range of the acquisition, the display time range can be set to the complete range and annotated centrally, so that the static object annotation results of all timestamps can be obtained through one annotation.

[0056] For dynamic objects whose positions change rapidly within the complete time range of the acquisition, the embodiment of the present application can set the display time range to single-frame or a small number of frame annotations, so that the dynamic objects can be accurately annotated.

[0057] For static objects, after marking the object's annotation box in a displayed point cloud frame, since the static object does not move relative to the acquisition device, after marking a point cloud frame, the annotation box can be automatically included in each point cloud frame from the start time value to the end time. For dynamic objects, this application does not mark all point cloud frames where the dynamic object is located, but only marks a small number of point cloud frames, avoiding the problem of slow single-frame annotation and difficulty in handling dynamic objects with fixed frame annotation.

[0058] Based on the above embodiments, as an optional embodiment, for any one of the static objects and the dynamic objects, the object is marked in the corresponding point cloud frame, such as Figure 4 As shown, including:

[0059] S201, marking an initial marking frame;

[0060] S202, adjusting the size and position of the annotation frame;

[0061] S203, adjusting the orientation angle of the annotation frame;

[0062] S204, judging whether the adjusted annotation box meets the stopping condition, if it meets the stopping condition, determining to obtain the final annotation box, if it does not meet the stopping condition, returning to execute step S203.

[0063] When annotating objects in a point cloud frame, the embodiment of the present application adopts an iterative method to continuously adjust the size, position and orientation angle of the annotation box, and determines whether the stopping condition is met at the end of each round of iteration. The iterative stopping condition of the embodiment of the present application can obtain the smallest annotation box that contains all the point clouds of the object, or it can reach a preset number of iterations, which is not specifically limited in the embodiment of the present application.

[0064] In some embodiments, the embodiment of the present application labels the object in the corresponding point cloud frame, including:

[0065] Set the size, position and orientation of the initial annotation box;

[0066] Perform a preset number of iterations according to the initial annotation frame:

[0067] Each round of iterative operations includes:

[0068] Expanding the annotation frame of this round of iteration according to a preset ratio to obtain the first expansion frame of this round of iteration, wherein the annotation frame of the first round of iteration is the initial annotation frame;

[0069] According to the distribution of the point cloud in the first expansion frame, the size of the target vehicle and the first center point, the size and position of the first expansion frame are adjusted to obtain a second expansion frame of this iteration;

[0070] Determine multiple angle offsets, and for each angle offset, adjust the orientation angle of the second expansion frame according to the angle offset, and adjust the size and position of the second expansion frame after the orientation angle is adjusted according to the distribution of the point cloud in the second expansion frame after the orientation angle is adjusted, to obtain a candidate frame corresponding to the angle offset;

[0071] The number of sampling points that meet the preset conditions in the candidate frames corresponding to each angle offset is determined, and the candidate frame with the largest number is used as the labeling frame for the next iteration.

[0072] The embodiment of the present application can determine an initial annotation box according to the position of the object. The size, position and orientation angle of the initial annotation box do not need too many restrictions, and it only needs to be able to roughly match the object. Subsequently, through continuous iterative adjustment, a annotation box that meets the iteration stop condition (for example, the minimum annotation box that includes all point clouds of the object) is finally obtained.

[0073] During each iterative operation of the embodiment of the present application, the originally iterated annotation box is first expanded according to a preset ratio. It should be understood that when expanding, only the size of the annotation box, that is, the length, width and height dimensions are expanded, and the center point and orientation angle of the annotation box are not adjusted. The embodiment of the present application does not limit the size of the preset ratio. For example, it can be 2, that is, the length, width and height are expanded to twice the original.

[0074] The expanded annotation box is called the first expanded box. The embodiment of the present application further adjusts the size and position of the expanded box according to the respective conditions of the point cloud in the first expanded box, such as the distance between each sampling point and each side of the first expanded box, etc., combined with the size of the target vehicle and the first center point, to obtain the second expanded box of this iteration. The first center point is also the center point of the target vehicle. The first center point can be used to measure the distance between each sampling point and the target vehicle.

[0075] In one round of iteration, the embodiment of the present application first adjusts the size and position of the first expansion frame, and then begins to determine the orientation angle that needs to be adjusted in this round of iteration. When adjusting the orientation angle, the embodiment of the present application first determines multiple angle offsets, and the range of the angle offset of the embodiment of the present application can cover from -30° to 30°. For each angle offset, the embodiment of the present application first adjusts the orientation angle of the second expansion frame with the angle offset, and then, according to the distribution of the point cloud in the second expansion frame after adjusting the orientation angle, the size and position of the second expansion frame after adjusting the orientation angle are adjusted again to obtain the candidate frame corresponding to the angle offset, that is, in one round of iteration, the embodiment of the present application adjusts the size and position of the annotation frame twice and the orientation angle once. After the embodiment of the present application determines the candidate frame corresponding to each angle offset, it will count the number of sampling points that meet the preset conditions in each candidate frame, and the candidate frame with the largest number will be used as the annotation frame for the next round of iteration. The embodiment of the present application can quickly identify the annotation frame of the object.

[0076] Based on the above embodiments, as an optional embodiment, adjusting the size of the expansion frame according to the distribution of the point cloud in the first expansion frame, the size of the target vehicle and the first center point includes:

[0077] Determine, according to the distribution of the point cloud in the first expansion frame, a first target point in the point cloud that is closest to each side surface of the first expansion frame;

[0078] For any pair of parallel side surfaces in the first expansion frame, the length of the side perpendicular to the pair of parallel side surfaces in the second expansion frame is determined according to the distance between the first target points closest to each of the pair of parallel side surfaces and the magnitude relationship of the first length.

[0079] It should be understood that the first expansion frame of the embodiment of the present application is a three-dimensional cube or cuboid. For each sampling point located inside the first expansion frame, the embodiment of the present application calculates the distance between the sampling point and each side, thereby screening out the first target point closest to each side in the first expansion frame. Therefore, there are 6 first target points. Each side of the first expansion frame corresponds to a first target point, and the first target point corresponding to each side is the sampling point in the first expansion frame that is closest to the side.

[0080] See also Figure 5 , which exemplarily shows a schematic diagram of the first expansion frame and the coordinate system provided by the embodiment of the present application. The embodiment of the present application defines the six sides of the first expansion frame as the upper side, the lower side, the left side, the right side, the front side and the rear side. It should be understood that the upper side and the lower side are a pair of parallel sides, the left side and the right side are a pair of parallel sides, and the front side and the rear side are a pair of parallel sides. Figure 5 It can also be seen that the coordinate axes of the coordinate system are constructed along the edges of the first expansion frame, the positive direction of the x-axis is from the rear side surface to the front side surface, the positive direction of the y-axis is from the left side surface to the right side surface, and the positive direction of the z-axis is from the lower side surface to the upper side surface.

[0081] For each pair of parallel side surfaces, an embodiment of the present application determines the distance between the first target points closest to each side surface in the pair of parallel side surfaces. Taking the front side surface and the rear side surface as an example, the distance between the first target point closest to the front side surface and the first target point closest to the rear side surface is calculated. Based on the relationship between the distance and the first length, the length of the side perpendicular to the front and rear side surfaces in the second expansion frame is determined.

[0082] It should be noted that the embodiment of the present application determines a preset anchor frame according to the size of the target vehicle, and the length, width and height of the anchor frame correspond to the length, width and height of the target vehicle respectively. The first length is the length of the side corresponding to the pair of parallel sides in the preset anchor frame. The embodiment of the present application will pre-define the sides corresponding to each pair of parallel sides in the preset anchor frame, and thus adjust the length of the side perpendicular to the pair of parallel sides in the second expansion frame based on the relationship between the length of the side and the size of the above-mentioned distance.

[0083] Based on the above embodiments, as an optional embodiment, determining the length of the side perpendicular to the pair of parallel side surfaces in the second expansion frame according to the distance between the first target points closest to each side surface of the pair of parallel side surfaces and the magnitude relationship of the first length includes:

[0084] If the distance between the first target points closest to each of the pair of parallel side surfaces is greater than the first length, the first length is used as the length of the side perpendicular to the pair of parallel side surfaces in the second expansion frame;

[0085] If the distance between the first target points closest to each side of the pair of parallel side surfaces is not greater than the first length, the distance between the first target points closest to each side of the pair of parallel side surfaces is used as the length of the side perpendicular to the pair of parallel side surfaces in the second expansion frame.

[0086] by Figure 5 Taking the first expansion frame shown as an example, the x-axis distance between the first target point on the front and rear sides in the coordinate system of the first expansion frame is calculated as the candidate length of the second expansion frame. If the preset anchor frame length is greater than the candidate length, the anchor frame length is used as the length l of the second expansion frame. Otherwise, the candidate length is used as the length l of the second expansion frame.

[0087] Calculate the y-axis distance of the first target point on the left and right sides in the coordinate system of the first expansion frame as the candidate width of the second expansion frame. If the preset anchor frame width is greater than the candidate width, the anchor frame width is used as the width w of the second expansion frame. Otherwise, the candidate width is used as the width w of the second expansion frame.

[0088] Calculate the z-axis distance of the first target point on the upper and lower sides in the coordinate system of the first expansion frame as the candidate height of the second expansion frame. If the preset anchor frame height is greater than the candidate height, the anchor frame height is used as the height h of the second expansion frame; otherwise, the candidate height is used as the height h of the second expansion frame.

[0089] Based on the above embodiments, as an optional embodiment, adjusting the position of the expansion frame according to the distribution of the point cloud in the first expansion frame, the size of the target vehicle and the first center point includes:

[0090] For any pair of parallel side surfaces in the first expansion frame, one side surface of the pair of parallel side surfaces is used as the first side surface, and the other side surface is used as the second side surface;

[0091] If the first target point closest to the first side surface is closer to the first center point than the first target point closest to the second side surface, the second center point of the first expansion frame is moved a first distance toward the second side surface.

[0092] by Figure 5Taking the first expansion frame shown in the figure as an example, the distances from the first target point on the front and rear sides to the center of the vehicle are compared. If the distance from the first target point on the front side to the center of the vehicle is closer, the x-axis coordinate of the center point of the second expansion frame is obtained by moving l / 2 from the first target point on the front side to the negative x-axis. Otherwise, the x-axis coordinate of the center point is obtained by moving l / 2 from the first target point on the rear side to the positive x-axis.

[0093] Compare the distances from the first target point on the left and right sides to the center of the vehicle. If the distance from the first target point on the left side to the center of the vehicle is closer, move w / 2 from the first target point on the left side to the positive direction of the y-axis to obtain the y-axis coordinate of the center point of the second expansion frame. Otherwise, move w / 2 from the first target point on the right side to the negative direction of the y-axis to obtain the y-axis coordinate of the center point.

[0094] The first target point corresponding to the lower side surface is moved in the positive direction of the z-axis by h / 2 to obtain the z-axis coordinate of the center point of the second expansion frame.

[0095] Based on the above embodiments, as an optional embodiment, according to the distribution of the point cloud in the second expansion frame after the orientation angle is adjusted, the size and position of the second expansion frame after the orientation angle is adjusted are adjusted to obtain a candidate frame corresponding to the angle offset, including:

[0096] Constructing a coordinate system along the edge of the second expansion frame after adjusting the orientation angle;

[0097] Determine the maximum and minimum components of the point cloud on each coordinate axis of the coordinate system according to the distribution of the point cloud in the second expansion frame after adjusting the orientation angle;

[0098] For each edge in the second expansion frame after adjusting the orientation angle, adjusting the length of the edge to a second length, where the second length is the maximum and minimum components on a coordinate axis parallel to the edge;

[0099] For the component of the second center point of the second expanded frame after the orientation angle is adjusted on any coordinate axis, the component is adjusted based on the second length corresponding to the coordinate axis to obtain a candidate frame corresponding to the angle offset.

[0100] The embodiment of the present application can traverse the point cloud to obtain the maximum x-axis component xmax, the minimum x-axis component xmin, the maximum y-axis component ymax, the minimum y-axis component ymin, the maximum z-axis component zmax, and the minimum z-axis component zmin;

[0101] The length of the candidate box is updated to xmax-xmin, the width is updated to ymax-ymin, and the height is updated to zmax-zmin. In addition, the x-axis component of the center point of the candidate box is changed to x+(xmax-xmin), the y-axis component is changed to y+(ymax-ymin), and the z-axis component is changed to z+(zmax-zmin), where x, y, and z are the x-axis, y-axis, and z-axis components of the second center point of the second expansion box, respectively.

[0102] Based on the above embodiments, as an optional embodiment, the preset condition refers to that the distance between the sampling point and each side of the candidate frame is less than the preset threshold. After obtaining the candidate frames corresponding to each angle offset, the embodiment of the present application counts the number of points whose distances from the candidate frame to each side of the candidate frame are less than the preset threshold, and uses the candidate frame with the most points that meet the conditions as the labeling frame for the next iteration.

[0103] Based on the above embodiments, as an optional embodiment, each round of iterative operation further includes:

[0104] If it is determined that the annotation box of the next iteration includes all point clouds, the annotation box of the next iteration is used as the minimum annotation box, and the iteration is stopped;

[0105] If it is determined that the annotation box of the next iteration does not include all point clouds, the next iteration operation is performed.

[0106] See also Figure 6 , which exemplarily shows a schematic diagram of a process of marking an object in a point cloud frame according to an embodiment of the present application, as shown in the figure, including:

[0107] S301, setting the size, position and orientation angle of the initial annotation box;

[0108] S302, obtaining a labeling box of this round of iterative operation, wherein the labeling box of the first round of iterative operation is an initial labeling box;

[0109] S303, expanding the annotation frame of this iteration according to a preset ratio to obtain a first expansion frame of this iteration;

[0110] S304, determining, according to the distribution of the point cloud in the first expansion frame, first target points in the point cloud that are closest to each side surface of the first expansion frame;

[0111] S305: for any pair of parallel side surfaces in the first expansion frame, determine the length of a side perpendicular to the pair of parallel side surfaces in the second expansion frame according to the distance between the first target points closest to each side surface in the pair of parallel side surfaces and the magnitude relationship of the first length;

[0112] S306: For any pair of parallel side surfaces in the first expansion frame, one side surface in the pair of parallel side surfaces is used as a first side surface, and the other side surface is used as a second side surface;

[0113] S307: If the first target point closest to the first side surface is closer to the first center point than the first target point closest to the second side surface, move the second center point of the first expansion frame toward the second side surface by a first distance to obtain a second expansion frame of this iteration;

[0114] S308: determining a plurality of angle offsets, and for each angle offset, adjusting the orientation angle of the second expansion frame according to the angle offset;

[0115] S309, constructing a coordinate system along the edge of the second expansion frame after adjusting the orientation angle;

[0116] S310, determining the maximum and minimum components of the point cloud on each coordinate axis of the coordinate system according to the distribution of the point cloud in the second expansion frame after adjusting the orientation angle;

[0117] S311, for each edge in the second expansion frame after adjusting the orientation angle, adjusting the length of the edge to a second length, where the second length is a difference between a maximum component and a minimum component on a coordinate axis parallel to the edge;

[0118] S312: For a component of the second center point of the second expanded frame after the orientation angle is adjusted on any coordinate axis, adjust the component based on a second length corresponding to the coordinate axis to obtain a candidate frame corresponding to the angle offset;

[0119] S313, determining the number of sampling points in the candidate frames corresponding to the angle offsets, the number of which is less than a preset threshold from each side of the candidate frame, and using the candidate frame with the largest number as the labeling frame for the next iteration;

[0120] S314, judging whether the iterative operation has reached a preset number of times, if not, executing step S302, if reached, taking the labeling box of the next iteration as the final labeling box.

[0121] The present application embodiment provides a point cloud data annotation device, such as Figure 7 As shown, the point cloud data annotation device may include: a point cloud acquisition module 110, an object recognition module 120, a static processing module 130 and a dynamic processing module 140, wherein:

[0122] The point cloud acquisition module 110 is used to obtain a point cloud sequence collected by a collection device deployed on a target vehicle, wherein the point cloud sequence includes a plurality of point cloud frames;

[0123] An object recognition module 120, configured to recognize static objects and dynamic objects in the point cloud sequence;

[0124] The static processing module 130 is used to determine, for each static object, a first point cloud frame group related to the static object from the point cloud sequence, select a point cloud frame from the first point cloud frame group, and mark the static object in the point cloud frame;

[0125] The dynamic processing module 140 is used to determine, for each dynamic object, a second point cloud frame group related to the dynamic object from the point cloud sequence, select a portion of point cloud frames from the second point cloud frame group, and mark the dynamic object in each point cloud frame in the portion of point cloud frames.

[0126] The device of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0127] In an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory. The processor executes the above-mentioned computer program to implement the steps of the method for labeling point cloud data. Compared with the related art, the following can be achieved: for static objects, after the embodiment of the present application labels the annotation box of the object in a displayed point cloud frame, since the static object has not moved relative to the acquisition device, after a point cloud frame is labeled, the annotation box can be automatically included in each point cloud frame with a start time value and an end time. For dynamic objects, the present application does not label all point cloud frames where the dynamic object is located, but only labels a small number of point cloud frames, avoiding the problem that single-frame labeling is too slow and fixed-frame labeling is difficult to handle dynamic objects.

[0128] In an alternative embodiment, an electronic device is provided, such as Figure 8 As shown, Figure 8 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0129] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0130] The bus 4002 may include a path for transmitting information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 4002 is represented by only one thick line in the figure, but this does not mean that there is only one bus or one type of bus.

[0131] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0132] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0133] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0134] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0135] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0136] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0137] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. A method for labeling point cloud data, characterized in that: include: Obtaining a point cloud sequence collected by a collection device deployed on a target vehicle, wherein the point cloud sequence includes a plurality of point cloud frames; identifying static objects and dynamic objects in the point cloud sequence; For each static object, determining a first point cloud frame group related to the static object from the point cloud sequence, selecting a point cloud frame from the first point cloud frame group, marking the static object in the point cloud frame, and synchronizing the position of the marking frame to each point cloud frame in the first point cloud frame group; For each dynamic object, a second point cloud frame group related to the dynamic object is determined from the point cloud sequence, a portion of point cloud frames is selected from the second point cloud frame group, and the dynamic object is marked in each point cloud frame in the portion of point cloud frames.

2. The method according to claim 1, characterized in that For any of the static objects and dynamic objects, the object is labeled in the corresponding point cloud frame, including: Set the size, position and orientation of the initial annotation box; Perform at least one round of iteration according to the initial annotation frame until a preset iteration stop condition is met: Each round of iterative operations includes: Expanding the annotation frame of this round of iteration according to a preset ratio to obtain the first expansion frame of this round of iteration, wherein the annotation frame of the first round of iteration is the initial annotation frame; According to the distribution of the point cloud in the first expansion frame, the size of the target vehicle and the first center point, the size and position of the first expansion frame are adjusted to obtain a second expansion frame of this iteration; Determine multiple angle offsets, and for each angle offset, adjust the orientation angle of the second expansion frame according to the angle offset, and adjust the size and position of the second expansion frame after the orientation angle is adjusted according to the distribution of the point cloud in the second expansion frame after the orientation angle is adjusted, to obtain a candidate frame corresponding to the angle offset; The number of sampling points that meet the preset conditions in the candidate frames corresponding to each angle offset is determined, and the candidate frame with the largest number is used as the labeling frame for the next iteration.

3. The method according to claim 2, characterized in that The adjusting the size of the first expansion frame according to the distribution of the point cloud in the first expansion frame, the size of the target vehicle and the first center point includes: Determine, according to the distribution of the point cloud in the first expansion frame, a first target point in the point cloud that is closest to each side surface of the first expansion frame; For any pair of parallel side surfaces in the first expansion frame, determine the length of a side perpendicular to the pair of parallel side surfaces in the second expansion frame according to the distance between the first target points closest to each side surface in the pair of parallel side surfaces and the magnitude relationship of the first length; The first length is the length of a side in a preset anchor frame corresponding to the pair of parallel side faces, and the preset anchor frame is determined according to the size of the target vehicle.

4. The method according to claim 3, characterized in that: Determining the length of the side perpendicular to the pair of parallel side surfaces in the second expansion frame according to the distance between the first target points closest to each side surface of the pair of parallel side surfaces and the magnitude relationship of the first length includes: If the distance between the first target points closest to each of the pair of parallel side surfaces is greater than the first length, the first length is used as the length of the side perpendicular to the pair of parallel side surfaces in the second expansion frame; If the distance between the first target points closest to each side of the pair of parallel side surfaces is not greater than the first length, the distance between the first target points closest to each side of the pair of parallel side surfaces is used as the length of the side perpendicular to the pair of parallel side surfaces in the second expansion frame.

5. The method according to claim 3, characterized in that: The adjusting the position of the first expansion frame according to the distribution of the point cloud in the first expansion frame, the size of the target vehicle and the first center point includes: For any pair of parallel side surfaces in the first expansion frame, one side surface of the pair of parallel side surfaces is used as the first side surface, and the other side surface is used as the second side surface; If the first target point closest to the first side surface is closer to the first center point than the first target point closest to the second side surface, moving the second center point of the first expansion frame toward the second side surface by a first distance; The first distance is half the length of a side of the second expansion frame that is perpendicular to the pair of parallel side surfaces.

6. The method according to claim 2, characterized in that The step of adjusting the size and position of the second expansion frame after the orientation angle is adjusted according to the distribution of the point cloud in the second expansion frame after the orientation angle is adjusted to obtain a candidate frame corresponding to the angle offset includes: Constructing a coordinate system along the edge of the second expansion frame after adjusting the orientation angle; Determine the maximum and minimum components of the point cloud on each coordinate axis of the coordinate system according to the distribution of the point cloud in the second expansion frame after adjusting the orientation angle; For each edge in the second expansion frame after adjusting the orientation angle, adjusting the length of the edge to a second length, where the second length is a difference between a maximum and a minimum component on a coordinate axis parallel to the edge; For the component of the second center point of the second expanded frame after the orientation angle is adjusted on any coordinate axis, the component is adjusted based on the second length corresponding to the coordinate axis to obtain a candidate frame corresponding to the angle offset.

7. The method according to claim 2 or 6, characterized in that: The preset condition is that the distance between the sampling point and each side surface in the candidate frame is less than a preset threshold.

8. A point cloud data annotation device, characterized in that: include: A point cloud acquisition module, used to obtain a point cloud sequence collected by a collection device deployed on a target vehicle, wherein the point cloud sequence includes a plurality of point cloud frames; An object recognition module, used to recognize static objects and dynamic objects in the point cloud sequence; a static processing module, configured to, for each static object, determine a first point cloud frame group related to the static object from the point cloud sequence, select a point cloud frame from the first point cloud frame group, annotate the static object in the point cloud frame, and synchronize the position of the annotation frame to each point cloud frame in the first point cloud frame group; A dynamic processing module is used to determine, for each dynamic object, a second point cloud frame group related to the dynamic object from the point cloud sequence, select a portion of point cloud frames from the second point cloud frame group, and mark the dynamic object in each point cloud frame in the portion of point cloud frames.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the point cloud data annotation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the point cloud data annotation method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the point cloud data annotation method according to any one of claims 1 to 7 is implemented.

Images