Point cloud labeling method and device, equipment and medium

Abstract

The invention provides a point cloud labeling method and device, equipment and a medium, and relates to the field of image processing, in particular to the technical field of computer vision, artificial intelligence, automatic driving and intelligent traffic. The specific implementation scheme is as follows: obtaining point cloud data of a first frame in continuous multi-frame point cloud data; determining a labeling type according to the obtained target feature of the first frame; the annotation type comprises a superimposed frame annotation, a predicted frame annotation or a single frame annotation; according to the labeling type, determining to-be-labeled point cloud data; determining a labeling operation according to the labeling type; and in the to-be-labeled point cloud data, based on the labeling operation, determining a labeling result of the selected point. According to the embodiment of the invention, the point cloud labeling efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, specifically to the fields of computer vision, artificial intelligence, autonomous driving and intelligent transportation technology, and in particular to a point cloud annotation method, device, equipment and medium. Background Art

[0002] Devices with autonomous navigation capabilities need to be able to automatically identify surrounding objects while moving. In order to achieve accurate recognition, a large number of point clouds need to be collected for training, and the point clouds need to be annotated before training.

[0003] Especially with the development of autonomous driving systems, higher requirements are placed on the efficiency of point cloud annotation. Segmentation and box annotation of obstacles in point clouds are typical tasks in the annotation process. Summary of the invention

[0004] The present invention provides a point cloud annotation method, device, equipment and medium.

[0005] According to one aspect of the present disclosure, a point cloud annotation method is provided, comprising:

[0006] Obtain the point cloud data of the first frame among multiple consecutive frames of point cloud data;

[0007] Determine a labeling type according to the acquired target features of the first frame; the labeling type includes: superimposed frame labeling, predicted frame labeling or single frame labeling; the target features include: point cloud features and / or scene features;

[0008] Determining the point cloud data to be annotated according to the annotation type;

[0009] Determine a marking operation according to the marking type;

[0010] In the point cloud data to be annotated, based on the annotation operation, an annotation result of a selected point is determined.

[0011] According to one aspect of the present disclosure, a point cloud annotation device is provided, comprising:

[0012] A first point cloud acquisition module, used to acquire point cloud data of a first frame among multiple consecutive frames of point cloud data;

[0013] A labeling type determination module, used to determine the labeling type according to the target features of the first frame acquired; the labeling type includes: superimposed frame labeling, predicted frame labeling or single frame labeling; the target features include: point cloud features and / or scene features;

[0014] A marked point cloud determination module, used to determine the point cloud data to be marked according to the marking type;

[0015] A marking operation determination module, used to determine a marking operation according to the marking type;

[0016] The labeling result determination module is used to determine the labeling result of the selected point in the point cloud data to be labeled based on the labeling operation.

[0017] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the point cloud annotation method described in any embodiment of the present disclosure.

[0021] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the point cloud annotation method described in any embodiment of the present disclosure.

[0022] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the point cloud annotation method described in any embodiment of the present disclosure is implemented.

[0023] The disclosed embodiments can improve the efficiency and accuracy of point cloud annotation.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0026] Figure 1 is a flow chart of a point cloud annotation method disclosed in an embodiment of the present disclosure;

[0027] Figure 2 is a flow chart of another point cloud annotation method disclosed in an embodiment of the present disclosure;

[0028] Figure 3 is a flow chart of another point cloud annotation method disclosed in an embodiment of the present disclosure;

[0029] Figure 4 is a schematic diagram of point cluster fine-tuning according to an embodiment of the present disclosure;

[0030] Figure 5 is a scene graph of a point cloud annotation method disclosed in an embodiment of the present disclosure;

[0031] Figure 6 is a schematic diagram of point cloud range selection according to an embodiment of the present disclosure;

[0032] Figure 7 is a schematic diagram of point cloud data after adjacent frame registration according to an embodiment of the present disclosure;

[0033] Figure 8 is a schematic diagram of the structure of a point cloud annotation device disclosed in an embodiment of the present disclosure;

[0034] Fig. 9 It is a block diagram of an electronic device according to the point cloud annotation method disclosed in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0036] Figure 1 This is a flow chart of a point cloud annotation method disclosed in an embodiment of the present disclosure. This embodiment can be applied to the case of annotating multiple frames of point cloud data. The method of this embodiment can be executed by a point cloud annotation device, which can be implemented in software and / or hardware and is specifically configured in an electronic device with certain data computing capabilities, which can be a client device or a server device.

[0037] S101, acquiring point cloud data of a first frame among multiple consecutive frames of point cloud data.

[0038] The continuous multiple frames of point cloud data may refer to at least two frames of point cloud data collected continuously by the same device. The continuous multiple frames are continuous and ordered in time. The first frame may be any frame in the continuous multiple frames. In some embodiments, the continuous multiple frames of point cloud data collected by a vehicle may be obtained, wherein the vehicle may be in a driving state, or the vehicle may be in a stationary state.

[0039] S102. Determine a labeling type according to the acquired target features of the first frame; the labeling type includes: superimposed frame labeling, predicted frame labeling or single frame labeling; the target features include: point cloud features and / or scene features.

[0040] The point cloud data of the first frame is usually the displayed point cloud data. The point cloud data of the first frame may or may not have an annotation result. When there is no annotation result in the point cloud data of the first frame, the user can annotate the point cloud data of the first frame. When there is an annotation result in the point cloud data of the first frame, the user can modify or re-annotate the annotation result of the point cloud data of the first frame. The target feature is used to describe the key information of the first frame. Specifically, the target feature is used to describe the information of the annotation method adapted for the first frame. The point cloud feature may refer to the feature extracted from the point cloud data itself. The scene feature may refer to the scene-related feature extracted from the scene where multiple consecutive frames are located in combination with the point cloud data. The annotation type is used to determine the annotation method. Superimposed frame annotation may refer to superimposing at least two frames of point cloud data and annotating the superimposed point cloud data together manually. Predicted frame annotation may refer to annotating at least one frame of point cloud data based on an already annotated frame of point cloud data. Single frame annotation may refer to annotating one frame of point cloud data manually.

[0041] In some embodiments, the annotation type corresponding to the target feature can be determined based on the point cloud features and / or scene features and the feature judgment rules corresponding to each annotation type. In some embodiments, the weight or score of each annotation type for each feature can be determined based on the point cloud features and / or scene features, the scores of each annotation type can be calculated and compared, and the annotation type corresponding to the target feature can be screened out. In some embodiments, a annotation method prediction model that predicts the annotation type based on the target feature can be pre-trained, the target feature can be input into the annotation method prediction model for processing, and the annotation type can be output. In addition, the user can also select the annotation type.

[0042] S103. Determine point cloud data to be annotated according to the annotation type; the point cloud data to be annotated at least includes the point cloud data of the first frame.

[0043] Among them, the point cloud data to be annotated for different annotation types are different. The point cloud data to be annotated includes the point cloud data of the first frame and / or the point cloud data of the frame after the first frame. In some embodiments, the point cloud data to be annotated corresponding to the single frame annotation is the point cloud data of the first frame; the point cloud data to be annotated corresponding to the superimposed frame annotation is the point cloud data of the first frame and the point cloud data of at least one frame after the first frame; the point cloud data to be annotated corresponding to the predicted frame annotation includes the point cloud data of at least one frame after the first frame.

[0044] S104: Determine a labeling operation for the point cloud data to be labeled according to the labeling type.

[0045] Among them, the annotation operation is used to determine the category of at least one point. The annotation operation can be input by the user or generated based on the annotated point cloud data. In some embodiments, both single-frame annotation and superimposed frame annotation are user annotation methods. Accordingly, for single-frame annotation and superimposed frame annotation, the annotation operation input by the user is obtained and used as the annotation operation of the point cloud data to be annotated. Predicted frame annotation is a predicted annotation method. Accordingly, for predicted frame annotation, the annotation operation of the point cloud data to be annotated is determined based on the annotation results of the first frame or the annotation results of other annotated frames.

[0046] In an optional embodiment, determining the annotation operation of the point cloud data to be annotated according to the annotation type includes: when the annotation type includes superimposed frame annotation or single frame annotation, displaying the point cloud data to be annotated, and obtaining the user's annotation operation for points within the connected domain selected in the displayed point cloud data.

[0047] Among them, the connected domain range is used to limit the annotated points, and usually the points within the connected domain range have the same annotation result. The annotation result may include the category of the point. In some embodiments, in a vehicle driving scene, the category of the point may include: cars, pedestrians, lane lines, indicator lights, signs, and trees. In some embodiments, in a robot transportation scene, the category of the point may include: cars, pedestrians, transported items, shelves, walls or decorations. In some embodiments, in a robot cleaning scene, the category of the point may include: pedestrians, cleaning supplies, cleaning vehicles, cabinets, beds, sofas, tables and chairs, or electrical appliances.

[0048] Among them, the number of labeling operations can be at least one, and one labeling operation usually specifies a connected domain range, and different labeling operations specify different connected domain ranges. One labeling operation is used to label the points within a connected domain range with the same category, and to determine the labeled category as the labeling result of the points within the connected domain range. The points within the connected domain range are labeled, specifically, the category is added to the relevant data of the points within the connected domain range. In addition, the color of the category can be set and added to the relevant data of the point. According to the color of the category, the point is re-rendered, so that points of different categories are displayed in different colors, and the category of each point can be intuitively seen.

[0049] The point cloud data to be annotated is rendered on the annotation page, and the user inputs the annotation operation on the annotation page. Specifically, the user draws the range of the connected domain, and the user inputs the category of the point cloud within the connected domain, which is equivalent to the user determining the point cloud within the connected domain as an object and determining the category of the object, thereby annotating the point cloud within the connected domain. The annotation page can be a page of an annotation application or a page of an annotation web page, etc.

[0050] It can be seen that by providing the annotation types of superimposed frame annotation and single frame annotation, the user can input the annotation operation on the page displaying the point cloud data to realize the annotation of the point cloud data to be annotated, thereby increasing the diversity and flexibility of the annotation methods.

[0051] S105 . Determine, in the point cloud data to be annotated, an annotation result of a selected point based on the annotation operation.

[0052] The annotation operation specifies the position of at least one point and the category of each point. The selected point may refer to the point specified in the annotation operation. The category of the selected point may be added to the relevant data of the selected point according to the annotation operation.

[0053] According to the technical solution disclosed in the present invention, during the process of labeling multiple consecutive frames of point cloud data, the labeling type is determined according to the target features of the first frame, and based on the labeling type, the point cloud data to be labeled and the labeling operation are determined, and the point cloud data to be labeled are labeled using corresponding labeling operations, so as to achieve fast and accurate labeling of point clouds. It can adapt to different scenes and the characteristics of different point clouds, select the best labeling type to label the point clouds, and improve the labeling efficiency of the point clouds while also improving the labeling accuracy of the point clouds.

[0054] Figure 2 It is a flowchart of another point cloud annotation method disclosed in accordance with an embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution, and can be combined with the above optional implementation modes. The method of determining the annotation type based on the target features of the first frame obtained is specifically as follows: determining the point cloud features of the first frame based on the point cloud data of the first frame, the point cloud features including: the number of point clouds and / or the size of the point cloud area; determining the scene features of the first frame based on the point cloud scene in which the first frame is located, the scene features including the complexity and / or the first degree of change; determining the annotation type based at least on the point cloud features and the scene features.

[0055] S201, obtaining point cloud data of a first frame among multiple consecutive frames of point cloud data.

[0056] S202. Determine point cloud features of the first frame according to the point cloud data of the first frame, where the point cloud features include: point cloud quantity and / or point cloud area size.

[0057] Among them, if the number of point clouds in a certain frame is small, the reason may be occlusion or sensor cover. The number of point clouds is small, and the category accuracy of the predicted frame annotation and the single frame annotation will decrease accordingly. Superimposed frame annotation is preferred. Superimposed frames can make up for the disadvantage of insufficient point clouds and improve the annotation accuracy. The point cloud area size may include the area or volume covered by the point cloud in three-dimensional space. Similar to the number of point clouds. The point cloud area size is small, and it is difficult to accurately distinguish the category of the point cloud. Superimposed frame annotation is preferred. The point cloud area size is large, and predicted frame annotation and single frame annotation are preferred. At least one of the number of point clouds and area size is counted to obtain point cloud features.

[0058] S203. Determine scene features of the first frame according to the point cloud scene in which the first frame is located, where the scene features include a complexity level and / or a first variation level.

[0059] The complexity is used to describe the complexity of the content of the first frame, and the complexity may include the number of categories and the density of categories. The first degree of variation is used to describe the movement of the object in the first frame. Generally, the higher the complexity and / or the higher the first degree of variation, the more sophisticated the annotation method is required, preferably the superimposed frame annotation. The lower the complexity and / or the lower the first degree of variation, the more suitable the coarse-grained and fast annotation method is, preferably the predicted frame annotation.

[0060] In an optional embodiment, the method of determining the scene features of the first frame based on the point cloud scene in which the first frame is located includes: detecting the environment type corresponding to the first frame based on the acquisition position and acquisition time of the first frame; the environment type includes a complex type or a simple type; obtaining the number of dynamic participants included in the first frame; determining the complexity of the first frame based on the environment type and the number of dynamic participants corresponding to the first frame; calculating the point cloud intersection-union ratio between the first frame and the next frame adjacent to the first frame; determining the first change degree of the first frame based on the point cloud intersection-union ratio; determining the scene features of the first frame based on the complexity and the first change degree of the first frame.

[0061] Among them, the environment type may refer to the environment corresponding to the acquisition position and acquisition time. When acquiring multiple consecutive frames, the acquisition time of each frame and the positioning information of the device when acquiring each frame are acquired at the same time, and the positioning information is determined as the acquisition position of the corresponding frame. In the scene of vehicle driving, the environment type may be a road section type. The complex type may refer to that the annotated objects included in the environment are richer, more dense, and difficult to distinguish. The simple type may refer to that the environment is more spacious and significantly distinguishes the annotated objects. In some embodiments, the point cloud scene is a vehicle driving scene, and the traffic density corresponding to the acquisition position corresponding to the first frame at the acquisition time is determined according to the acquisition position, acquisition time and pre-collected traffic density data of the first frame. According to the traffic density, acquisition time and acquisition position corresponding to the first frame, it is determined that the road section corresponding to the first frame is complex and congested, and accordingly, the environment type is a complex type. According to the traffic density, acquisition time and acquisition position corresponding to the first frame, it is determined that the road section corresponding to the first frame is simple and empty, and accordingly, the environment type is a simple type. For example, if the acquisition time is during peak hours, the acquisition location is at least one of a multi-road confluence point and high traffic density, it indicates that the road section corresponding to the first frame is complex and congested; if the acquisition time is during off-peak hours, the acquisition location is at least one of a flat national highway and low traffic density, it indicates that the road section corresponding to the first frame is simple and empty. In some embodiments, if the environment type is complex, it is preferred to use superimposed frame annotation to increase details. If the environment type is simple, the next dynamic of the point cloud can be quickly predicted, and predicted frame annotation is usually used.

[0062] The dynamic participants may refer to objects belonging to the categories identified in the point cloud data. In some embodiments, the point cloud scene is a vehicle driving scene, and the dynamic participants may include at least one of vehicles, pedestrians, and non-motorized vehicles. Generally, the more dynamic participants there are, the higher the complexity; the fewer dynamic participants there are, the lower the complexity.

[0063] The complexity of the first frame is determined according to the environment type and the number of dynamic participants. If the environment type is complex, the complexity of the first frame is high; the greater the number of dynamic participants, the higher the complexity of the first frame; if the environment type is simple, the complexity of the first frame is low; the smaller the number of dynamic participants, the lower the complexity of the first frame. The complexity of the environment type can be described by numerical values, and the environment type and dynamic participants can be weighted to obtain the complexity.

[0064] Among them, the point cloud intersection-union ratio may refer to the movement speed of the point cloud of the previous and next frames. Usually, the movement speed of the point cloud is fast, that is, the first degree of change is high, the distance between the same point cloud in the previous frame and the next frame is larger, and correspondingly, the overlapping area between the areas formed by multiple point clouds is small, that is, the point cloud intersection-union ratio is small; the movement speed of the point cloud is slow, that is, the first degree of change is low, the distance between the same point cloud in the previous frame and the next frame is smaller, and correspondingly, the overlapping area between the areas formed by multiple point clouds is large, that is, the point cloud intersection-union ratio is large. In some embodiments, the point cloud scene is a vehicle driving scene, and the ratio of the intersection and union of the point clouds of the previous and next two frames, that is, the point cloud intersection-union ratio R, is used to determine whether the first frame is in a driving scene. In the parking scene, the vehicle drives slower, and correspondingly, the movement of each object in the first frame is slower, and the point cloud intersection-union ratio is large, while the vehicle drives faster in the driving scene, and correspondingly, the movement of each object in the first frame is faster, and the point cloud intersection-union ratio is small. The point cloud intersection-union ratio R can be calculated based on the following formula:

[0065]

[0066] Among them, P t is the point cloud of the first frame, P t+1 is the point cloud of the next frame adjacent to the first frame. The point cloud intersection-and-union ratio is inversely proportional to the first degree of change, and the reciprocal or opposite number of the point cloud intersection-and-union ratio can be determined as the first degree of change.

[0067] In driving scenes, there are many dynamic vehicles, and the effect of predicting the next frame fluctuates greatly. It is preferred to use superimposed frame annotation to achieve range annotation. In parking scenes, static vehicles account for a large proportion. Using predicted frame annotation, the next frame can be predicted and combined with posture information to obtain a more accurate annotation result.

[0068] It can be seen that by determining the complexity according to the environment type and the number of dynamic participants, and determining the scene features according to the first degree of change determined by the point cloud intersection ratio, the scene information of the first frame can be enriched, the content of the scene in the target feature can be increased, the annotation type can be determined based on the real scene, and the detection accuracy of the annotation types of different scenes can be improved.

[0069] S204. Determine a labeling type at least according to the point cloud features and the scene features; the labeling type includes: superimposed frame labeling, predicted frame labeling or single frame labeling.

[0070] Among them, the annotation type can be determined only based on the point cloud features and scene features; or the annotation type can be determined based on other content combined with the point cloud features and scene features. For example, the annotation type can be determined based on whether there are already annotated frames, combined with the point cloud features and scene features. In fact, the prediction frame annotation requires the existence of at least one annotated frame. In the case of an already annotated frame, the annotation type is determined based on the point cloud features and scene features; in the case of no annotated frames, single frame annotation or superimposed frame annotation is determined based on the point cloud features and scene features.

[0071] In some embodiments, weighted calculation is performed on various contents in the point cloud features and scene features to obtain a score, and the annotation type is determined based on the score. A correspondence between the score and the annotation type can be preset, and the annotation type is determined based on the score and the correspondence.

[0072] S205: Determine the point cloud data to be annotated according to the annotation type.

[0073] S206: Determine a marking operation according to the marking type.

[0074] S207 . Determine, in the point cloud data to be annotated, an annotation result of a selected point based on the annotation operation.

[0075] In an optional embodiment, the labeling result includes: the object category in the vehicle driving scene; the complexity level includes the traffic complexity level, and the first change level includes the vehicle speed change level.

[0076] Among them, the object category may include at least one of the following: vehicles, pedestrians, lane lines, indicator lights, signs and trees, etc. The traffic complexity can describe the complexity of traffic congestion and traffic intersections. The first degree of change can describe the degree of change in the speed of the current vehicle. The current vehicle may refer to the vehicle that collects point cloud data. The first degree of change is limited to the degree of change in speed. The annotation type can be determined based on the real-time motion state of the vehicle, and the environmental information collected by the vehicle in real time is further combined with the annotation operation, which can increase the information of the real scene during the annotation process and increase the adaptability between the annotation type and the scene. The point cloud data is limited to the point cloud data collected by the current vehicle, and the point cloud data collected by the current vehicle is annotated. The subsequent model training is performed according to the annotation results, so that the model has the function of object recognition in the point cloud collected by the real-time driving vehicle. The trained model is applied to the autonomous driving vehicle system to ensure the accuracy of obstacle recognition of the autonomous driving vehicle.

[0077] It can be seen that by configuring the application scenario as a vehicle driving scenario, limiting the annotation results to object categories, and further limiting the complexity to the traffic complexity, and limiting the first degree of change to the degree of change in vehicle speed, the annotation of point cloud data in vehicle driving scenarios can be achieved, the annotation speed of vehicle driving scenes can be improved, the workload and time of manual annotation can be greatly reduced, and the annotation quality in vehicle driving scenes can be increased. Then, based on the annotation results, the obstacle recognition capability of the autonomous driving system can be enhanced, which can improve the driving safety of the autonomous driving system.

[0078] According to the technical solution disclosed in the present invention, by configuring the specific content of the point cloud features and the specific content of the scene features, the target features of the point cloud data can be enriched, so that the annotation type can be determined based on the target features. The subdivision granularity of the annotation type can be increased to flexibly adapt to a variety of application scenarios and improve the detection accuracy of the annotation type.

[0079] Figure 3 It is a flowchart of another point cloud annotation method disclosed in accordance with an embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution and can be combined with the above optional implementations. The point cloud data to be annotated is determined according to the annotation type, which is specifically: when the annotation type is multi-frame annotation, at least one target frame is determined according to the target features of the first frame; the multi-frame annotation includes superimposed frame annotation or predicted frame annotation; the point cloud data to be annotated is determined according to the point cloud data of the first frame and the point cloud data of each of the target frames.

[0080] S301, acquiring point cloud data of a first frame among multiple consecutive frames of point cloud data.

[0081] S302. Determine a labeling type according to the acquired target features of the first frame; the labeling type includes: superimposed frame labeling, predicted frame labeling or single frame labeling; the target features include: point cloud features and / or scene features.

[0082] S303: When the annotation type is multi-frame annotation, determine at least one target frame according to the target features of the first frame; the multi-frame annotation includes superimposed frame annotation or predicted frame annotation.

[0083] Among them, multi-frame annotation can refer to an annotation type that can annotate point cloud data of at least two frames at the same time. Overlap frame annotation can annotate the first frame and at least one target frame at the same time. Prediction frame annotation can annotate only one target frame or multiple target frames at the same time. Target features are also used to determine frames that can be annotated at the same time. Target frames can refer to frames that can be annotated at the same time. The range of the target frame can be determined based on the target features, and the frames within the range can be taken as the target frames starting from the first frame.

[0084] S304: Determine the point cloud data to be labeled according to the point cloud data of the first frame and the point cloud data of each of the target frames.

[0085] The point cloud data of each target frame is determined as the point cloud data to be labeled. Alternatively, the point cloud data of the first frame and the point cloud data of each target frame are determined as the point cloud data to be labeled.

[0086] S305: Determine a marking operation according to the marking type.

[0087] S306: Determine the annotation result of the selected point in the point cloud data to be annotated based on the annotation operation.

[0088] When annotating, users can select a range on the point cloud to help users lock the target annotation area.

[0089] According to the technical solution disclosed in the present invention, when the annotation type is multi-frame annotation, the target frames that can be annotated simultaneously can be determined according to the target features of the first frame, and the point cloud data to be annotated can be determined in the point cloud data of the first frame and the point cloud data of each target frame according to the annotation type, and the upper limit of frames that can be accurately annotated can be screened out in the unannotated point cloud data, thereby reducing the annotation results with low accuracy and improving the accuracy of the annotation results.

[0090] In an optional embodiment, when the annotation type is multi-frame annotation, at least one target frame is determined according to the point cloud features of the first frame, including: when the annotation type is superimposed frame annotation, a second frame of the target number is determined according to the target features of the first frame; the multi-frame annotation includes superimposed frame annotation, and the target frame includes the second frame; in the point cloud data to be annotated, after determining the annotation results of the selected points based on the annotation operation, the method also includes: adding the annotation results of the points in the point cloud data to be annotated belonging to the target frame to the point cloud data of the target frame.

[0091] Among them, the target features are also used to determine the number of targets. Starting from the first frame, obtain the second frames of the continuous target number as the target frame. The point cloud data to be annotated includes a first frame and at least one target frame. At this time, the annotation results of the point cloud data to be annotated actually include the annotation results of the original points of the first frame, and the annotation results of the points mapped to the first frame by the target frame. The annotation results of the points belonging to the target frame may refer to the annotation results of the points mapped to the first frame among all the annotation results of the superimposed frame annotation. In fact, the annotation results of the points belonging to the target frame are generated in the first frame. The annotation process of the target frame is equivalent to obtaining the annotation results of the points mapped from the target frame to the first frame, and adding the annotation results of the mapped points in the first frame to the target frame. In the superimposed frame annotation process, it is necessary to map the annotation results of the points mapped from the target frame to the first frame back to the target frame to obtain the annotation results of the target frame. In some embodiments, the point cloud data of the target frame can be mapped to the first frame according to the transformation matrix between the first frame and the target frame. After the annotation is completed, the annotation results of the point cloud data belonging to the target frame are mapped to the target frame according to the transformation matrix to obtain the annotation results of the target frame; or after the annotation is completed, the annotation results of the points mapped to the first frame are obtained, and the points are queried in the target frame, and the annotation results are added to the relevant data of the point.

[0092] In some embodiments, the point cloud features include point cloud density and / or inter-frame variation. The scene features include complexity and / or scene dynamics. The target features include point cloud features and / or scene features.

[0093] According to the point cloud features, increasing the number of targets in sparse point cloud areas can enhance data quality; reducing overlapping frames in high-density point cloud areas to control computing costs. This is equivalent to the greater the point cloud density, the smaller the number of targets; the smaller the point cloud density, the larger the number of targets. Analyze the degree of change between point cloud frames, such as calculating the mean displacement of the point cloud between frames and / or the rate of change of velocity, etc., to obtain the degree of change between frames. If the degree of change between frames is large, by reducing the number of targets, it is possible to avoid annotation errors caused by dynamic point cloud blur. This is equivalent to the greater the degree of change between frames, the smaller the number of targets; the smaller the degree of change between frames, the larger the number of targets.

[0094] According to the scene characteristics, usually in complex traffic intersections and / or scenes with many vehicles, such as construction areas, the number of targets is increased to obtain higher detail accuracy; the higher the complexity, the larger the number of targets; the lower the complexity, the smaller the number of targets. In highly dynamic scenes such as highways, the number of targets is reduced to retain real-time performance. The more dynamic the scene, the smaller the number of targets; the less dynamic the scene, the larger the number of targets.

[0095] Based on the above content, each parameter in the target feature can be weighted to obtain a score of the first frame, and the number of targets can be determined based on the score. For example, a score range corresponding to at least one number of targets can be preset, and the score range to which the score belongs can be detected to obtain the number of targets corresponding to the score range.

[0096] The point cloud data of each second frame may be superimposed and displayed on the point cloud data of the displayed first frame according to a transformation matrix between at least one second frame and the target frame, thereby realizing superimposed display of the point cloud data of the first frame and each second frame.

[0097] The process of overlaying frames is to overlay the first frame and the target frame for display and annotation. The main purpose of overlaying frames is to overlay the point cloud data of consecutive frames on the front-end annotation platform to form a 4D (three-dimensional space coordinates and time dimension) scene with timestamp dimension. It is convenient for annotation users to intuitively observe the motion trajectory and morphological changes of objects in consecutive frames. First, the point cloud data of consecutive frames are preprocessed on the back end, including at least one of denoising, filtering and downsampling, so as to reduce the load of front-end rendering and improve the rendering quality. At the same time, the point cloud data are sorted according to the timestamp to ensure that they can be overlaid in the correct order on the front end. Create a three-dimensional scene on the front end and overlay the point cloud data of consecutive frames into the scene in sequence according to the timeline. Different frames of point clouds are distinguished by setting color coding. Implement the timeline control to allow annotation users to view point cloud data at different time points through the timeline.

[0098] The process of de-framing is to split the annotation results of the first frame and the target frame superimposed together, and map the annotation results of the target frame back to the target frame. Among them, the position of the consecutive frames on the time axis can be determined according to the acquisition time of the consecutive frames, and the identification information of each frame can be generated in sequence in the order of the time axis, for example, the number of each frame is generated. Correspondingly, the main purpose of de-framing is to synchronize the annotation results with the corresponding frames and time points in real time during the annotation process. After the annotation is completed, the superimposed point cloud data is split according to the time axis and frame number, and the annotation results are used as labels of the point cloud data, which are associated with the frame number of the point cloud data, and the labeled point cloud data of each frame number can be exported.

[0099] It can be seen that by determining the target number of overlapping frames according to the target features of the first frame, the number of frames that can be accurately superimposed and annotated can be determined. While the overlapping frames improve the annotation efficiency, the accuracy of the overlapping frames is also improved. By de-framing the annotation results of the target frames, the annotation accuracy and efficiency of the target frames can be improved.

[0100] In an optional embodiment, when the annotation type is multi-frame annotation, at least one target frame is determined according to the target features of the first frame, including: when the annotation type is predicted frame annotation, the third frame in the target time range is determined according to the target features of the first frame; the multi-frame annotation includes predicted frame annotation, and the target frame includes the third frame.

[0101] The target feature is also used to determine the target time range. Starting from the time of the first frame, the third frame within the time range is obtained as the target frame. The point cloud data to be annotated includes at least one target frame. In some embodiments, the point cloud data of the first frame can be mapped to the target frame according to the transformation matrix between the first frame and the target frame, and the annotation result of the point cloud data of the target frame can be predicted based on the annotation result of the point cloud data of the first frame.

[0102] After annotating the third frame based on the annotation operation, you can select any third frame and display the point cloud data of the third frame and its annotation results. The user can adjust or re-annotate the predicted annotation results, or use the third frame as the new first frame and predict the annotation results for subsequent frames based on the new first frame.

[0103] It can be seen that by determining the target time range of the predicted frame according to the target features of the first frame, the time range for accurate prediction of the annotation can be determined, which improves the annotation efficiency while also improving the accuracy of the predicted annotation results.

[0104] In an optional embodiment, determining the third frame of the target time range based on the target features of the first frame includes: obtaining the frame rate of the first frame; obtaining a second degree of change of the annotated object between the first frame and an adjacent frame; determining the point cloud features of the first frame based on the frame rate and the second degree of change; determining the scene features of the first frame based on the point cloud scene in which the first frame is located; determining the target time range based on the point cloud features and scene features of the first frame; taking the time of the first frame as the starting time, obtaining frames other than the first frame within the target time range as the third frame.

[0105] Among them, the point cloud features also include frame rate and / or a second degree of change. The frame rate may refer to the frame rate of the sensor. The annotated object may refer to the object to which the annotated category belongs. The second degree of change may refer to the degree of change of the annotated object from the base frame to the first frame. In fact, the frame rate represents the update frequency of the point cloud. The prediction time range is adjusted according to the frame rate of the sensor. For point cloud data collected by sensors with higher frame rates, frames with a shorter time range can be predicted. The second degree of change is used to indicate environmental changes. If the point cloud changes slightly, frames with a longer time range can be predicted; if the point cloud changes significantly, the prediction time range is appropriately reduced. The change in the point cloud is determined based on the categories pre-identified in the previous and next frames in the pre-identification results and the number of point sets in each category. The second degree of change Δ can be calculated based on the following formula t :

[0106]

[0107] By calculating the weighted sum of all category point cloud changes, the second degree of change between frames Δ is reflected t Prediction time range T: Dynamically adjust the prediction time range based on the degree of change. The smaller the change, the closer the time range is to the time limit T. max , the greater the change, the shorter the time range. C represents the set of dynamic participant categories, and c represents the category in set C. The weight w of the cth category c : Assign different weights to different categories. For example, in traffic scenes, changes in vehicles and pedestrians are more important than changes in the background. is the point cloud of the cth category in the first frame, is the point cloud of the cth category in the next frame adjacent to the first frame. In addition, Here, t+1 may be replaced by the numbers of other frames that are sequentially subsequent to the first frame. represents the difference between the point cloud of the cth category in the first frame and the point cloud of the cth category in the next frame adjacent to the first frame. For example, to predict only the annotation result of the fifth frame after the first frame, t+1 can be replaced by t+5. threshold is a control parameter, and the second change degree Δ t Exceeding the threshold Δ threshold For frames, single frame annotation is required to obtain the annotation results.

[0108] The corresponding environment type can be determined according to the point cloud scene, thereby determining the complexity and the first degree of change. Generally, for static scenes, such as parking lots or low-speed areas, longer time range frames can be predicted. For dynamic scenes, such as highways or complex intersections, the prediction time range should be short to capture dynamic changes. This means that the higher the complexity and / or the higher the first degree of change, the shorter the prediction time range, and the lower the complexity and / or the lower the first degree of change, the longer the prediction time range.

[0109] It can be seen that by determining the point cloud features according to the frame rate and the second degree of change, determining the scene features according to the point cloud scene, and determining the time range according to the point cloud features and the scene features, it is possible to determine the time range of predictable frames based on the characteristics of the point cloud itself and the real scene. This allows the frame annotation results to have high accuracy while increasing the number of predicted frames, reducing the labor cost of annotation, and improving annotation efficiency.

[0110] In an optional embodiment, determining the annotation operation according to the annotation type includes: when the annotation type is predicted frame annotation, mapping the point cloud data of the first frame to the third frame according to the transformation matrix between the first frame and the third frame to obtain the mapped point cloud of the third frame; for each point in the point cloud data of the third frame, determining the neighboring points of the point in the mapped point cloud; and determining the annotation operation of the point according to the annotation results of the neighboring points of the point.

[0111] Among them, the first frame is a frame in which the point cloud data has accurate annotation results. The transformation matrix is ​​used to realize the coordinate conversion between the point clouds between frames. The mapped point cloud may refer to the point cloud data of the first frame mapped to the third frame, and the mapped point cloud is different from the original point cloud data of the third frame. The neighboring point may refer to the point in the mapped point cloud around the original point of the third frame. The number of neighboring points is at least one, and the number of neighboring points can be pre-set. Based on the annotation results of each neighboring point, the most representative annotation result can be determined as the annotation result of the point, and an annotation operation for forming the annotation result of the point can be generated.

[0112] In some embodiments, the first frame is the first frame, and the annotated point cloud of the first frame is converted to the coordinate system of the Nth frame to establish a spatial correspondence between the two frames, so that the annotation information (such as labels and attributes, etc.) in the first frame is spatially aligned with the point cloud data of the Nth frame.

[0113] It can be seen that by mapping the first frame to the third frame, a mapping point cloud is obtained. According to the mapping point cloud around the original points in the third frame and its annotation results, the annotation results of the original points in the third frame are determined. The accurate annotation results can be used to predict the annotation results of the points in the unlabeled frame, and the predicted annotation results can be realized to improve the accuracy and efficiency of the annotation results.

[0114] In an optional embodiment, determining the labeling result of the point based on the labeling results of the neighboring points of the point includes: calculating the distance between the point and each of its neighboring points; calculating the confidence of the labeling result of each of the neighboring points; determining the weight of each of the neighboring points based on the distance between the point and each of its neighboring points and the confidence of the labeling result of each of the neighboring points; determining the labeling result of the point based on the weight of each of the neighboring points and the labeling result of each of the neighboring points.

[0115] Among them, the confidence of the annotation results of the neighboring points is used to determine the reliability of the annotation results of the neighboring points. The confidence can be input by the user when the user annotates the points in the first frame, and the confidence is 1 at this time; or, when predicting the annotation results of the points in the first frame, the predicted confidence is output at the same time, and the confidence is the confidence of the predicted output, and the value range is (0,1). In fact, a point cloud segmentation model can be pre-trained to provide an initial annotation reference for the annotated user. A relatively accurate initial prediction can help the annotated user quickly locate the annotated point cloud, and the prediction results of the modified model are more efficient than direct annotation, which can reduce the errors and omissions that may be caused by manual direct annotation, thereby improving the quality and consistency of the annotation data. To this end, a large model is used to perform the point cloud segmentation task, and a pre-annotated label field is added to the first frame point cloud of the continuous frame scene as a pre-identified annotation result. In addition, the large model outputs the annotation results of each point and the confidence corresponding to the annotation results.

[0116] Among them, the distance between the neighboring point and the point of the third frame is used to describe the correlation between the neighboring point and the point of the third frame. The weight of the neighboring point is used to describe the importance of the annotation result of the neighboring point to the annotation result of the point of the third frame. According to the weight of the neighboring point, the annotation result of the neighboring point is weighted and calculated to obtain the annotation result of the point of the third frame. Among them, the neighboring point can be understood as the point related to the point of the third frame, and the weight can be understood as the degree of correlation between the neighboring point and the point of the third frame, so that the annotation result of the point of the third frame is closer to the annotation result of the related neighboring point.

[0117] In some embodiments, each point in the Nth frame searches for a certain number of neighboring points in the mapped point cloud, and predicts the label information of the point based on the label information of these neighboring points, where the label information is the annotation result. In order to improve the efficiency of neighbor point search, a KD tree (K-dimensional tree) is constructed on the point cloud of the Nth frame. It helps to quickly locate the neighboring points around each point. Based on the label information of the found neighboring points, a weighted average is used to predict the label of the current point. The weight is allocated in proportion to the distance of the neighboring points and the confidence of the label.

[0118] It can be seen that the weight of the neighboring point is determined by the distance between the point and the neighboring point, and the confidence of the annotation results of the neighboring point. The annotation result of the point is determined in combination with the annotation results of the neighboring point. The related points around the point can be found, and the annotation result of the point can be determined according to the degree of correlation. The related annotation results in the annotation results of the point are increased, thereby improving the prediction accuracy of the annotation results.

[0119] In an optional embodiment, the method further includes: determining discrete points in the mapping point cloud based on the mapping point cloud and the point cloud data of the third frame; determining the neighboring points of each point in the point cloud data of the third frame in the mapping point cloud, including: determining the neighboring points of the point in the mapping point cloud other than discrete points for each point in the point cloud data of the third frame.

[0120] Among them, the discrete points in the mapping point cloud can refer to the points of the mapping point cloud that are free from each object. Discrete points can be understood as redundant points or noise points, etc. The points in the point cloud data of the third frame are close to the discrete points, but the discrete points do not belong to any object, so the discrete points have little auxiliary effect on the annotation results of the points in the third frame. The discrete points can be removed from the mapping point cloud to increase the effectiveness and representativeness of the neighboring points.

[0121] In some embodiments, discrete points can be determined by using Gaussian filtering, for example, points outside the mean ±3 standard deviations in the Gaussian distribution are taken as discrete points. In some embodiments, the mapping point cloud and the point cloud data of the third frame are clustered to obtain a clustering result; based on the clustering result, the discrete points in the mapping point cloud are determined. Clustering is used to aggregate the mapping point cloud and the point cloud data of the third frame of the same object.

[0122] It can be seen that by eliminating discrete points in the mapped point cloud and ensuring that the neighboring points are different from the discrete points, the effectiveness of the neighboring points can be improved, which helps to maintain the spatial continuity of the annotation results.

[0123] In an optional embodiment, in the point cloud data to be annotated, after annotating the points within the connected domain selected in the annotation operation and obtaining the annotation results of the points within the selected connected domain, the method further includes:

[0124] When the annotation type selected by the user is overlay frame modification, the point cloud data and annotation results of at least one modified frame are displayed; according to the user's modification operation on the displayed annotation results, the annotation results of the points within the selected connected domain are modified.

[0125] like Figure 4 As shown, after overlaying the frames, the user can quickly locate the point cluster for fine-tuning by clicking the options on the right. Figure 4It is the point cloud data displayed after at least one frame is superimposed in the first frame, that is, after the frames are overlapped. The annotation results are displayed on the right side of the page, and different categories are displayed in different colors. For example, the color of the point cloud of the ground category is the first green, the color of the point cloud of the curb category is red, the color of the point cloud of the passenger car category is the first pink, and the color of the point cloud of the two-wheeled vehicle category is the second pink; the color of the point cloud of the street light and / or pole category is the second green, the color of the point cloud of the high-reflective sign category is cyan, and the color of the point cloud of the vegetation category is the third green. Users can select any category in the point cluster workspace, and the point cluster of the selected category will be rendered in the corresponding color. For example, in Figure 4 In the figure, the passenger car category is selected. In the point cloud data on the left, the first pink points are all points of the passenger car category.

[0126] When the annotation type selected by the user is single-frame modification, the point cloud data and annotation results of the modified frame are displayed; according to the user's modification operation on the displayed annotation results, the annotation results of the points within the selected connected domain are modified.

[0127] In some embodiments, modifying the annotation results of points within the selected connected domain according to a modification operation of a user on a displayed annotation result includes: receiving a modification operation of the same type of annotation result from a user; selecting point cloud data to be modified of the same type of annotation result; and modifying the annotation results of points within the selected connected domain in the point cloud data to be modified according to the modification operation. The modified frame may include a single-frame annotated frame, a stacked-frame annotated frame, or a predicted-frame annotated frame.

[0128] In an application scenario, such as Figure 5 As shown, the point cloud annotation method also includes: obtaining point cloud data of multiple consecutive frames; determining whether the point cloud data of each frame has annotation information; if there is annotation information, determining the annotation information as the annotation result of the point cloud data of each frame. Determine whether the annotation user has modified it. If not, send the point cloud data and annotation results of each frame to the review user for review. After the review and verification, the annotation results can be delivered, specifically, the point cloud data with annotation results are used as deliverable data. If modified, determine the annotation type as annotation modification, render and display the point cloud data and annotation results of the frame specified by the annotation user; modify the annotation results of the points within the selected connected domain according to the modification operation of the annotation user on the displayed annotation results.

[0129] If there is no annotation information, point cloud registration can be performed, specifically by calculating the descriptor vector of the point cloud data of each frame; based on the descriptor vector of the point cloud data of each frame, the point cloud data of each frame is registered to obtain the transformation matrix between the point cloud data of each frame, thereby realizing point cloud registration.

[0130] After determining the transformation matrix, determine the annotation type according to the target features of the first frame obtained; determine the point cloud data to be annotated according to the annotation type; determine the annotation operation of the point cloud data to be annotated according to the annotation type; and determine the annotation result of the selected point in the point cloud data to be annotated based on the annotation operation. When the annotation type is single-frame annotation and superimposed frame annotation, the annotation user annotates in the displayed point cloud data to obtain the annotation result of the point cloud data. Among them, the displayed point cloud data can be point cloud data of a single frame or superimposed multiple frames. When the annotation type is predicted frame annotation, the annotation user can annotate the first frame, or the annotation user can select the recognition mode, use the pre-trained segmentation model to segment the first frame, obtain the pre-recognition result, realize the first frame pre-annotation, and iteratively predict the annotation results of the remaining frames based on the first frame to realize the predicted frame annotation. The annotation user can select any frame or any superimposed n frames for display, and correct the displayed point cloud data and the predicted annotation results.

[0131] In the displayed point cloud data, Figure 6 As shown, the point cloud is range-selected during annotation to help users lock the target annotation area. The range of the displayed point cloud data can be scaled by selecting the range on the x-axis, y-axis, and z-axis, for example, Figure 6 In the , the selected displayed point cloud range is x-axis (-116.27~7.45), y-axis (-82.07~69.74) and z-axis (-13.19~24.73). Point clouds outside the range are filtered and not displayed.

[0132] After all frames are labeled, the user can also choose to modify the annotations to modify the annotation results of each frame.

[0133] After the modification is completed, the point cloud data and annotation results of each frame will be sent to the review user for review, and the annotation results can be delivered after review and verification.

[0134] The methods for determining the descriptor include:

[0135] For each feature point in a frame of point cloud data, the three vectors of the coordinate system generated by the plane fitting of its surrounding points and the normal vector constitute a coordinate system with the key point as the origin, called LRF (Local Reference Frame); for each center point, a patch with a radius of r is extracted, and a part of the points in the patch is used to calculate the corresponding LRF.

[0136] Use LRF to normalize the patch and randomly sample n points from it;

[0137] The coordinates of these n points are represented by the patch center and normalized to obtain a patch of unit radius.

[0138] The patch-normalized point cloud is input into the deep learning model for processing, and the feature vector is output as the descriptor. The L2-normalization layer is used to output the descriptor with unit norm. The deep learning model is trained with a twin network, and the weights are shared between the twin networks, which is equivalent to calculating a deep learning model twice, so that the difference in the descriptors of a pair of internal points in the two point clouds is minimized.

[0139] In fact, when calculating the descriptor of the point cloud data, which is used as the basis for identifying the spatial information of key points, the registration point range is limited to 20 meters in front, behind, left, right, and 5 meters in the top and bottom of the current vehicle (the vehicle that collects multiple frames of point cloud data). This range has achieved relatively stable results in multiple static annotation scenes.

[0140] The point cloud is registered using the descriptor information. First, the FGR (Fast Global Registration) method is used to perform coarse registration of the point cloud to obtain an initial rigid body transformation (including rotation and translation). Then, the point cloud is finely registered using the point-to-plane ICP (Iterative Closest Point). The optimal registration method based on the least squares method repeatedly selects corresponding point pairs and calculates the optimal rigid body transformation until the convergence accuracy requirement is met, and the final adjacent frame transformation matrix is ​​obtained. By calculating the pose between all adjacent frames, the 4D reconstruction of a continuous frame scene is completed. The adjacent frames after registration are as follows: Figure 7 As shown, the point clouds of the same frame are displayed in a uniform color, and the point clouds of different frames are displayed in different colors.

[0141] With the development of autonomous driving technology, there is an increasing demand for high-quality, large-scale annotated data. The disclosed embodiments rely on a large number of data annotation services, which can improve the efficiency of the annotation services, reduce the annotation costs, and adapt to the annotation needs of users with different levels of annotation, which can effectively enhance the quality of data annotation.

[0142] According to an embodiment of the present disclosure, Figure 8 is a structural diagram of a point cloud annotation device in an embodiment of the present disclosure, and the present disclosure embodiment is applicable to the case of annotating multiple frames of point cloud data. The device is implemented by software and / or hardware, and is specifically configured in an electronic device with certain data computing capabilities.

[0143] like Figure 8The point cloud annotation device 800 shown in the figure comprises: a first point cloud acquisition module 801, an annotation type determination module 802, an annotation point cloud determination module 803, an annotation operation determination module 804 and an annotation result determination module 805.

[0144] The first point cloud acquisition module 801 is used to acquire the point cloud data of the first frame among the continuous multiple frames of point cloud data;

[0145] The annotation type determination module 802 is used to determine the annotation type according to the target features of the first frame acquired; the annotation type includes: superimposed frame annotation, predicted frame annotation or single frame annotation; the target features include: point cloud features and / or scene features;

[0146] The annotated point cloud determination module 803 is used to determine the point cloud data to be annotated according to the annotation type; the point cloud data to be annotated includes at least the point cloud data of the first frame;

[0147] A labeling operation determination module 804 is used to determine a labeling operation for the point cloud data to be labeled according to the labeling type;

[0148] The labeling result determination module 805 is used to determine the labeling result of the selected point in the point cloud data to be labeled based on the labeling operation.

[0149] According to the technical solution disclosed in the present invention, during the process of labeling multiple consecutive frames of point cloud data, the labeling type is determined according to the target features of the first frame, and based on the labeling type, the point cloud data to be labeled and the labeling operation are determined, and the point cloud data to be labeled are labeled using corresponding labeling operations, so as to achieve fast and accurate labeling of point clouds. It can adapt to different scenes and the characteristics of different point clouds, select the best labeling type to label the point clouds, and improve the labeling efficiency of the point clouds while also improving the labeling accuracy of the point clouds.

[0150] Optionally, the annotation type determination module 802 includes:

[0151] A point cloud feature determination unit, configured to determine point cloud features of the first frame according to the point cloud data of the first frame, wherein the point cloud features include: the number of point clouds and / or the size of a point cloud region;

[0152] a scene feature determination unit, configured to determine a scene feature of the first frame according to a point cloud scene in which the first frame is located, wherein the scene feature includes a complexity degree and / or a first variation degree;

[0153] The annotation type determination unit is used to determine the annotation type at least according to the point cloud features and the scene features.

[0154] Optionally, the scene feature determination unit includes:

[0155] An environment type determination subunit, configured to detect an environment type corresponding to the first frame according to a collection position and a collection time of the first frame; the environment type includes a complex type or a simple type;

[0156] A participant number acquisition subunit, used to acquire the number of dynamic participants included in the first frame;

[0157] A complexity determination subunit, configured to determine the complexity of the first frame according to the environment type and the number of dynamic participants corresponding to the first frame;

[0158] a point cloud intersection-and-union ratio calculation subunit, configured to calculate a point cloud intersection-and-union ratio between the first frame and a next frame adjacent to the first frame;

[0159] A first change detection subunit, configured to determine a first change degree of the first frame according to the point cloud intersection-to-union ratio;

[0160] The scene degree calculation subunit is used to determine the scene feature of the first frame according to the complexity and the first change degree of the first frame.

[0161] Optionally, the annotated point cloud determination module 803 includes:

[0162] A target frame detection unit, configured to determine at least one target frame according to target features of the first frame when the annotation type is multi-frame annotation; the multi-frame annotation includes superimposed frame annotation or predicted frame annotation;

[0163] The annotated point cloud detection unit is used to determine the point cloud data to be annotated based on the point cloud data of the first frame and the point cloud data of each of the target frames.

[0164] Optionally, the target frame detection unit includes:

[0165] A second frame determination subunit is used to determine a second frame of the target quantity according to the target features of the first frame when the annotation type is superimposed frame annotation; the multi-frame annotation includes superimposed frame annotation, and the target frame includes the second frame;

[0166] The device also includes:

[0167] The de-framing module is used to determine the annotation results of the selected points in the point cloud data to be annotated based on the annotation operation, and then add the annotation results of the points in the point cloud data to be annotated that belong to the target frame to the point cloud data of the target frame.

[0168] Optionally, the target frame detection unit includes:

[0169] The third frame determination subunit is used to determine the third frame of the target time range according to the point cloud features of the first frame when the annotation type is the predicted frame annotation; the multi-frame annotation includes the predicted frame annotation, and the target frame includes the third frame.

[0170] Optionally, the third frame determination subunit is used to:

[0171] Obtaining the frame rate of the first frame;

[0172] Acquire a second change degree of the annotated object between the first frame and an adjacent frame;

[0173] determining point cloud features of the first frame according to the frame rate and the second degree of change;

[0174] Determining a scene feature of the first frame according to a point cloud scene in which the first frame is located;

[0175] Determine a target time range according to the point cloud features and scene features of the first frame;

[0176] Taking the time of the first frame as the starting time, frames other than the first frame within the target time range are acquired as the third frame.

[0177] Optionally, the labeling result includes: the category of objects in the vehicle driving scene; the complexity level includes the traffic complexity level, and the first change level includes the vehicle speed change level.

[0178] Optionally, the marking operation determination module 804 includes:

[0179] The area annotation unit is used to display the point cloud data to be annotated when the annotation type includes superimposed frame annotation or single frame annotation, and obtain the annotation operation of the user for the points located within the connected domain selected in the displayed point cloud data.

[0180] Optionally, the marking operation determination module 804 includes:

[0181] a point cloud mapping unit, configured to map the point cloud data of the first frame to the third frame according to a transformation matrix between the first frame and the third frame to obtain a mapped point cloud of the third frame when the annotation type is a predicted frame annotation;

[0182] A neighbor point determination unit, configured to determine, for each point in the point cloud data of the third frame, a neighbor point of the point in the mapped point cloud;

[0183] The neighbor labeling unit is used to determine the labeling operation of the point according to the labeling results of the neighboring points of the point.

[0184] Optionally, the marking result determination module 805 includes:

[0185] A distance calculation unit, used to calculate the distance between the point and each neighboring point of the point;

[0186] A confidence calculation unit, used to calculate the confidence of the labeling result of each of the neighboring points;

[0187] A weight determination unit, configured to determine the weight of each neighboring point according to the distance between the point and each neighboring point of the point and the confidence of the annotation result of each neighboring point;

[0188] The labeling result determining unit is used to determine the labeling result of the point according to the weight of each of the neighboring points and the labeling result of each of the neighboring points.

[0189] Optionally, the point cloud annotation device further includes:

[0190] A discrete point detection module, used to determine discrete points in the mapping point cloud according to the mapping point cloud and the point cloud data of the third frame;

[0191] The neighbor point determination unit includes:

[0192] The discrete elimination subunit is used to determine, for each point in the point cloud data of the third frame, the neighboring points of the point among the points other than the discrete points in the mapped point cloud.

[0193] The above-mentioned point cloud annotation device can execute the point cloud annotation method provided by any embodiment of the present disclosure, and has corresponding functional modules and beneficial effects for executing the point cloud annotation method.

[0194] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0195] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0196] Fig. 9 A schematic area diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0197] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the device 900 instructions can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0198] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0199] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the point cloud annotation method. For example, in some embodiments, the point cloud annotation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the point cloud annotation method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the point cloud annotation method in any other appropriate manner (e.g., by means of firmware).

[0200] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard objects (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0201] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / instructions specified in the flow chart and / or the area diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0202] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0203] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0204] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0205] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.

[0206] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0207] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, instruction systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as model training.

[0208] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0209] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A point cloud annotation method, comprising: Obtain the point cloud data of the first frame among multiple consecutive frames of point cloud data; Determining a labeling type according to the acquired target features of the first frame; The annotation types include: superimposed frame annotation, predicted frame annotation or single frame annotation; the target features include: point cloud features and / or scene features; Determining the point cloud data to be annotated according to the annotation type; Determine a marking operation according to the marking type; In the point cloud data to be annotated, based on the annotation operation, an annotation result of a selected point is determined.

2. The method according to claim 1, wherein: The step of determining the annotation type according to the acquired target feature of the first frame includes: Determine, according to the point cloud data of the first frame, point cloud features of the first frame, wherein the point cloud features include: the number of point clouds and / or the size of a point cloud region; Determining, according to the point cloud scene in which the first frame is located, scene features of the first frame, wherein the scene features include a complexity degree and / or a first variation degree; The annotation type is determined at least according to the point cloud features and the scene features.

3. The method according to claim 2, wherein: The determining, according to the point cloud scene in which the first frame is located, a scene feature of the first frame includes: Detecting an environment type corresponding to the first frame according to the acquisition position and acquisition time of the first frame; the environment type includes a complex type or a simple type; Obtaining the number of dynamic participants included in the first frame; Determining the complexity of the first frame according to the environment type and the number of dynamic participants corresponding to the first frame; Calculating a point cloud intersection-over-union ratio between the first frame and a next frame adjacent to the first frame; Determining a first degree of change of the first frame according to the point cloud intersection-to-union ratio; The scene feature of the first frame is determined according to the complexity and the first variation degree of the first frame.

4. The method according to claim 1, wherein: The step of determining the point cloud data to be annotated according to the annotation type includes: When the annotation type is multi-frame annotation, at least one target frame is determined according to the target features of the first frame; the multi-frame annotation includes superimposed frame annotation or predicted frame annotation; Point cloud data to be marked are determined according to the point cloud data of the first frame and the point cloud data of each of the target frames.

5. The method according to claim 4, wherein: When the annotation type is multi-frame annotation, determining at least one target frame according to the target feature of the first frame includes: When the annotation type is superimposed frame annotation, a second frame of target quantity is determined according to the target feature of the first frame; the multi-frame annotation includes superimposed frame annotation, and the target frame includes the second frame; After determining the annotation result of the selected point in the point cloud data to be annotated based on the annotation operation, the method further includes: The marking results of the points in the point cloud data to be marked that belong to the target frame are added to the point cloud data of the target frame.

6. The method according to claim 4, wherein: When the annotation type is multi-frame annotation, determining at least one target frame according to the target feature of the first frame includes: When the annotation type is predicted frame annotation, a third frame in a target time range is determined according to target features of the first frame; the multi-frame annotation includes predicted frame annotation, and the target frames include the third frame.

7. The method according to claim 6, wherein: The step of determining a third frame within a target time range according to the target feature of the first frame includes: Obtaining the frame rate of the first frame; Acquire a second change degree of the annotated object between the first frame and an adjacent frame; determining point cloud features of the first frame according to the frame rate and the second degree of change; Determining a scene feature of the first frame according to a point cloud scene in which the first frame is located; Determine a target time range according to the point cloud features and scene features of the first frame; Taking the time of the first frame as the starting time, frames other than the first frame within the target time range are acquired as the third frame.

8. The method according to claim 2, wherein: The labeling result includes: the object category in the vehicle driving scene; the complexity level includes the traffic complexity level, and the first change level includes the vehicle speed change level.

9. The method according to claim 1, wherein: The step of determining the annotation operation according to the annotation type includes: When the annotation type includes superimposed frame annotation or single frame annotation, the point cloud data to be annotated is displayed, and an annotation operation of the user on points within a connected domain selected in the displayed point cloud data is obtained.

10. The method according to claim 6, wherein: The step of determining the annotation operation according to the annotation type includes: When the annotation type is a predicted frame annotation, mapping the point cloud data of the first frame to the third frame according to a transformation matrix between the first frame and the third frame to obtain a mapped point cloud of the third frame; For each point in the point cloud data of the third frame, determining a neighboring point of the point in the mapped point cloud; The labeling operation of the point is determined according to the labeling results of the neighboring points of the point.

11. The method according to claim 10, wherein: The step of determining the labeling result of the point according to the labeling results of the neighboring points of the point includes: Calculate the distance between the point and each of its neighboring points; Calculating the confidence of the labeling result of each of the neighboring points; Determine the weight of each neighboring point according to the distance between the point and each neighboring point of the point and the confidence of the annotation result of each neighboring point; The labeling result of the point is determined according to the weight of each of the neighboring points and the labeling result of each of the neighboring points.

12. The method according to claim 10, further comprising: Determine discrete points in the mapping point cloud according to the mapping point cloud and the point cloud data of the third frame; The step of determining, for each point in the point cloud data of the third frame, a neighboring point of the point in the mapped point cloud comprises: For each point in the point cloud data of the third frame, determine the neighboring points of the point among the points other than the discrete points in the mapped point cloud.

13. A point cloud annotation device, comprising: A first point cloud acquisition module, used to acquire point cloud data of a first frame among multiple consecutive frames of point cloud data; A label type determination module, used to determine the label type according to the target features acquired in the first frame; The annotation types include: superimposed frame annotation, predicted frame annotation or single frame annotation; the target features include: point cloud features and / or scene features; A marked point cloud determination module, used to determine the point cloud data to be marked according to the marking type; A marking operation determination module, used to determine a marking operation according to the marking type; The labeling result determination module is used to determine the labeling result of the selected point in the point cloud data to be labeled based on the labeling operation.

14. The device according to claim 13, wherein: The annotation type determination module includes: A point cloud feature determination unit, configured to determine point cloud features of the first frame according to the point cloud data of the first frame, wherein the point cloud features include: the number of point clouds and / or the size of a point cloud region; a scene feature determination unit, configured to determine a scene feature of the first frame according to a point cloud scene in which the first frame is located, wherein the scene feature includes a complexity degree and / or a first variation degree; The annotation type determination unit is used to determine the annotation type at least according to the point cloud features and the scene features.

15. The device according to claim 14, wherein: The scene feature determination unit includes: An environment type determination subunit, configured to detect an environment type corresponding to the first frame according to a collection position and a collection time of the first frame; the environment type includes a complex type or a simple type; A participant number acquisition subunit, used to acquire the number of dynamic participants included in the first frame; A complexity determination subunit, configured to determine the complexity of the first frame according to the environment type and the number of dynamic participants corresponding to the first frame; a point cloud intersection-and-union ratio calculation subunit, configured to calculate a point cloud intersection-and-union ratio between the first frame and a next frame adjacent to the first frame; A first change detection subunit, configured to determine a first change degree of the first frame according to the point cloud intersection-to-union ratio; The scene degree calculation subunit is used to determine the scene feature of the first frame according to the complexity and the first change degree of the first frame.

16. The device according to claim 13, wherein: The annotated point cloud determination module comprises: A target frame detection unit, configured to determine at least one target frame according to target features of the first frame when the annotation type is multi-frame annotation; the multi-frame annotation includes superimposed frame annotation or predicted frame annotation; The annotated point cloud detection unit is used to determine the point cloud data to be annotated based on the point cloud data of the first frame and the point cloud data of each of the target frames.

17. The device according to claim 16, wherein: The target frame detection unit comprises: A second frame determination subunit is used to determine a second frame of the target quantity according to the target features of the first frame when the annotation type is superimposed frame annotation; the multi-frame annotation includes superimposed frame annotation, and the target frame includes the second frame; The device also includes: The de-framing module is used to determine the annotation results of the selected points in the point cloud data to be annotated based on the annotation operation, and then add the annotation results of the points in the point cloud data to be annotated that belong to the target frame to the point cloud data of the target frame.

18. The device according to claim 16, wherein: The target frame detection unit comprises: The third frame determination subunit is used to determine the third frame of the target time range according to the point cloud features of the first frame when the annotation type is the predicted frame annotation; the multi-frame annotation includes the predicted frame annotation, and the target frame includes the third frame.

19. The device according to claim 18, wherein: The third frame determination subunit is used for: Obtaining the frame rate of the first frame; Acquire a second change degree of the annotated object between the first frame and an adjacent frame; determining point cloud features of the first frame according to the frame rate and the second degree of change; Determining a scene feature of the first frame according to a point cloud scene in which the first frame is located; Determine a target time range according to the point cloud features and scene features of the first frame; Taking the time of the first frame as the starting time, frames other than the first frame within the target time range are acquired as the third frame.

20. The device according to claim 14, wherein: The labeling result includes: the object category in the vehicle driving scene; the complexity level includes the traffic complexity level, and the first change level includes the vehicle speed change level.

21. The device according to claim 13, wherein: The marking operation determination module includes: The area annotation unit is used to display the point cloud data to be annotated when the annotation type includes superimposed frame annotation or single frame annotation, and obtain the annotation operation of the user for the points located within the connected domain range selected in the displayed point cloud data.

22. The device according to claim 18, wherein The marking operation determination module includes: a point cloud mapping unit, configured to map the point cloud data of the first frame to the third frame according to a transformation matrix between the first frame and the third frame to obtain a mapped point cloud of the third frame when the annotation type is a predicted frame annotation; A neighbor point determination unit, configured to determine, for each point in the point cloud data of the third frame, a neighbor point of the point in the mapped point cloud; The neighbor labeling unit is used to determine the labeling operation of the point according to the labeling results of the neighboring points of the point.

23. The device according to claim 22, wherein: The marking result determination module comprises: A distance calculation unit, used to calculate the distance between the point and each neighboring point of the point; A confidence calculation unit, used to calculate the confidence of the labeling result of each of the neighboring points; A weight determination unit, configured to determine the weight of each neighboring point according to the distance between the point and each neighboring point of the point and the confidence of the annotation result of each neighboring point; The labeling result determining unit is used to determine the labeling result of the point according to the weight of each of the neighboring points and the labeling result of each of the neighboring points.

24. The apparatus of claim 22, further comprising: A discrete point detection module, used to determine discrete points in the mapping point cloud according to the mapping point cloud and the point cloud data of the third frame; The neighbor point determination unit includes: The discrete elimination subunit is used to determine, for each point in the point cloud data of the third frame, the neighboring points of the point among the points other than the discrete points in the mapped point cloud.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the point cloud annotation method described in any one of claims 1-12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the point cloud annotation method according to any one of claims 1-12.

27. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the point cloud annotation method according to any one of claims 1 to 12 is implemented.

Images