Labeling method, electronic equipment and vehicle

Abstract

The invention discloses an annotation method, electronic equipment and a vehicle, and belongs to the technical field of data annotation. The method comprises the following steps: determining a tracking sequence of a target in an annotated scene according to multi-frame point cloud data of the annotated scene; optimizing the tracking sequence of the target in at least one of the geometric dimension, the position dimension and the confidence degree dimension; and labeling the target according to the optimized tracking sequence to obtain a labeling result of the target. According to the invention, the accuracy of labeling is improved.

Description

Technical Field

[0001] This application belongs to the field of data annotation technology, and in particular relates to an annotation method, electronic equipment, and vehicle. Background Technology

[0002] Environmental perception in autonomous driving technology relies on 3D object detection technology. 3D point cloud object annotation is used in multiple stages, including algorithm training, high-precision map construction, and real-time driving assistance, and its accuracy directly affects the system's perception capabilities. Annotation methods can be divided into manual annotation and automated annotation. Manual annotation is inefficient and difficult to meet large-scale demands, so automated annotation methods are often used. However, automated annotation methods in related technologies are usually limited by sparse point clouds or weak sensor noise capabilities, resulting in insufficient annotation accuracy. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the related art. To this end, this application proposes a labeling method, electronic device, and vehicle, which improves the accuracy of labeling.

[0004] Firstly, this application provides an annotation method, which includes: Based on multi-frame point cloud data of the labeled scene, determine the tracking sequence of the target in the labeled scene; The tracking sequence is optimized in at least one of the following dimensions: geometric dimension, location dimension, and confidence dimension; The target is labeled according to the optimized tracking sequence to obtain the labeling result of the target.

[0005] In the above technical solution, after determining the tracking sequence of the target in the annotation scene based on multi-frame point cloud data of the annotation scene, the tracking sequence is optimized in at least one dimension to avoid distortion of subsequent annotation results due to insufficient accuracy of the tracking sequence, thus providing an accurate data foundation for annotation. Moreover, the annotation process can be automated to complete the entire process from tracking sequence construction and optimization to annotation, reducing manual intervention in the annotation process. The target is annotated based on the optimized tracking sequence to obtain the target annotation result. This realizes the integration of multi-dimensional optimization into the annotation process, which helps to improve the low accuracy problem caused by traditional automated annotation due to sparse point clouds, dynamic interference of targets, and insufficient optimization in a single dimension, and helps to improve the accuracy and reliability of annotation.

[0006] Secondly, this application provides a labeling device, which includes: The tracking module is used to determine the tracking sequence of the target in the labeled scene based on multi-frame point cloud data of the labeled scene; An adjustment module is used to optimize the tracking sequence in at least one of the geometric dimension, the position dimension, and the confidence dimension; The annotation module is used to annotate the target based on the optimized tracking sequence to obtain the annotation result of the target.

[0007] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the annotation method described in the first aspect above.

[0008] Fourthly, this application provides a vehicle that includes a marking device as described in the second aspect, or an electronic device as described in the third aspect.

[0009] Fifthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the annotation method as described in the first aspect above.

[0010] In a sixth aspect, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the annotation method as described in the first aspect.

[0011] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the annotation method described in the first aspect above.

[0012] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0013] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating the annotation method provided in some embodiments of this application; Figure 2 This is one of the structural schematic diagrams of the annotation system provided in some embodiments of this application; Figure 3 This is a second schematic diagram of the structure of the annotation system provided in some embodiments of this application; Figure 4 This is a schematic diagram of the structure of the labeling device provided in some embodiments of this application; Figure 5 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application.

[0014] Explanation of reference numerals in the attached figures: 400: Annotation device; 401: Tracking module; 402: Adjustment module; 403: Annotation module; 500: Electronic device; 501: Processor; 502: Memory. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0016] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0017] The annotation method, electronic device, and vehicle provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0018] The annotation method provided in this application can be executed by an electronic device or a functional module or entity within an electronic device that implements the annotation method. The electronic devices mentioned in this application include, but are not limited to, vehicle control units (VCUs) and electronic control units (ECUs). The vehicle can be a gasoline-powered vehicle, a natural gas-powered vehicle, or a new energy vehicle. New energy vehicles can be pure electric vehicles, hybrid electric vehicles, or range-extended electric vehicles. The annotation method provided in this application is illustrated below using an electronic device as the executing entity.

[0019] Figure 1 This is a flowchart illustrating the annotation method provided in some embodiments of this application. For example... Figure 1 As shown, the annotation method includes steps 110, 120 and 130.

[0020] Step 110: Determine the tracking sequence of the target in the labeled scene based on the multi-frame point cloud data of the labeled scene.

[0021] It is understood that the annotation scenario may include one or more targets. The annotation method provided in this application embodiment is illustrated below with a single target as an example.

[0022] A labeled scenario refers to a spatial range or specific area where targets need to be labeled. For example, in autonomous driving scenarios (such as urban roads, highways, indoor areas, etc.), the labeled scenario can include targets such as vehicles, pedestrians, and obstacles.

[0023] Point cloud data refers to three-dimensional spatial data collected by sensors such as LiDAR, which can characterize spatial information within a labeled scene. Multi-frame point cloud data can be multiple sets of point cloud data continuously collected at a preset frequency or fixed time interval, and can serve as the data basis for target detection and tracking, used to capture the motion trajectory or attitude changes of targets.

[0024] For example, after acquiring multi-frame point cloud data of annotated scene, preprocessing operations such as denoising and cropping can be performed. Then, a 3D target detection algorithm is executed on the preprocessed multi-frame point cloud data to detect targets frame by frame, and output the 3D detection boxes of all targets in each frame. Then, a multi-target tracking algorithm is used to assign an identity identifier (ID) to each detection box based on indicators such as the position similarity and appearance feature similarity of the detection boxes. All detection boxes corresponding to the unified identity identifier are associated in chronological order to form a target tracking sequence.

[0025] Step 120: Optimize the tracking sequence in at least one of the geometric dimension, position dimension, and confidence dimension.

[0026] In some embodiments, the tracking information of the target's tracking sequence is adjusted to optimize the target's tracking sequence in at least one of the geometric, positional, and confidence dimensions. Specifically, the tracking information of the tracking sequence includes various parameters that constitute the tracking sequence. For example, the target's tracking sequence is typically formed by arranging the 3D detection boxes corresponding to the same target in multiple frames of point cloud data according to the acquisition time sequence; the tracking information of the tracking sequence may include the geometric parameters, confidence parameters, and identification identifiers corresponding to each 3D detection box.

[0027] For tracking sequences of a single target, optimization can be performed using a single dimension or a combination of multiple dimensions. For example, attention mechanisms can be used to adjust the geometric dimensions of the 3D bounding boxes in the tracking sequence to correct contour deviations in the geometric dimension, resulting in a higher degree of matching between the 3D bounding boxes and the target. Alternatively, Kalman filtering can be combined to adjust the positional information of the 3D bounding boxes in the tracking sequence based on the target's motion state, thereby optimizing and correcting spatial positioning deviations in the positional dimension and improving the target's positioning accuracy in the scene. It is understandable that multi-dimensional optimization of the tracking sequence can improve its accuracy and reliability, helping to solve problems in traditional tracking and annotation caused by insufficient sensor accuracy or point cloud coefficients in dynamic scenes, thus improving the accuracy of target tracking and annotation.

[0028] Step 130: Label the target according to the optimized tracking sequence to obtain the target labeling results.

[0029] The annotation results refer to accurate descriptive information related to the target generated based on the optimized tracking sequence annotations. For example, this may include the target's type (e.g., whether it belongs to a vehicle or a pedestrian), dynamic and static attributes, and the parameters of the 3D bounding box corresponding to the target.

[0030] The target is labeled based on the optimized tracking sequence to obtain the labeled results. For example, automated labeling tools combined with pre-trained detection models can be used to automatically generate labels, improving the automation level of target labeling. In some embodiments, the target is distinguished between dynamic and static targets based on the optimized tracking sequence, and differentiated labeling is used for dynamic and static targets. Dynamic targets retain their temporal trajectories to meet the needs of target motion state analysis; while static targets can output only the optimal bounding box, improving labeling efficiency and accuracy.

[0031] The annotation method provided in this application, after determining the tracking sequence of the target in the annotation scene based on multi-frame point cloud data, optimizes the tracking sequence in at least one dimension to avoid distortion of subsequent annotation results due to insufficient tracking sequence accuracy, thus providing an accurate data foundation for annotation. Moreover, the annotation process can automatically complete the entire process from tracking sequence construction and optimization to annotation, reducing manual intervention in the annotation process. The target is annotated based on the optimized tracking sequence to obtain the target annotation result. This realizes the integration of multi-dimensional optimization into the annotation process, which helps to improve the low accuracy problem caused by traditional automated annotation due to sparse point clouds, dynamic interference of targets, and insufficient single-dimensional optimization, and helps to improve the accuracy and reliability of annotation.

[0032] In some embodiments of this application, the target tracking sequence is composed of a temporal set of multiple frames of 3D detection boxes of the target; The tracking sequence is optimized in at least one of the following dimensions: geometric dimension, location dimension, and confidence dimension, including at least one of the following: Adjust the geometric dimensions of each 3D detection box to optimize the tracking sequence in the geometric dimension; Adjust the position of each 3D bounding box within the labeled scene to optimize the tracking sequence in terms of position; or, Adjust the confidence level of each 3D detection box to optimize the tracking sequence in terms of confidence level.

[0033] As can be understood, a 3D bounding box refers to a cubic structure used to represent the spatial extent of a target in point cloud data of an annotated scene, and is used to construct a target tracking sequence. A 3D bounding box includes various configurable parameters, such as geometric parameters (e.g., length, width, and height), positional parameters (e.g., center 3D coordinates, orientation), and confidence parameters (e.g., matching reliability score). In some embodiments, the 3D bounding boxes of a target in multiple frames within the target tracking sequence share the same identifier. This unique identifier ensures that the bounding boxes in the tracking sequence belong to the same target.

[0034] For example, the tracking sequence can be obtained by using a multi-target tracking algorithm, assigning a unique identifier to each frame's detection box based on the positional similarity of the detection boxes and the similarity of the point cloud features; and organizing all three-dimensional detection boxes corresponding to the same identifier into an ordered set according to the acquisition time sequence of the point cloud data of each frame; that is, the tracking sequence includes the four-dimensional information of the target (three-dimensional space + time dimension).

[0035] In some embodiments, the tracking sequences of all targets in the labeled scene are traversed. For each tracking sequence, at least one dimension of tracking information is selected for adjustment to achieve tracking sequence optimization. The specific dimension selection can be flexibly configured according to the needs of the scene (such as focusing on geometric / positional optimization for static targets, and superimposing confidence optimization for dynamic targets). This can break through the limitations of traditional single-dimensional optimization, adapt to the labeling needs of different scenes, and improve the versatility of the method.

[0036] Adjusting the geometric dimensions of each 3D detection box, such as adjusting the length, width, and height, can correct the deviation between the detection box and the true geometric contour of the target, thereby improving the fitting accuracy of the detection box to the target shape.

[0037] Adjusting the position of each 3D bounding box in the labeled scene can involve adjusting the center coordinates, orientation, and other positional parameters of the 3D bounding box in the labeled scene, correcting the spatial positioning deviation of the bounding box, and the core goal is to improve the positioning accuracy of the target in the scene.

[0038] Adjusting the confidence level of each 3D detection box can be achieved by adjusting the confidence score of the 3D detection box, quantifying the matching reliability between the detection box and the target, thereby filtering out high-confidence valid boxes and eliminating low-confidence detection boxes, thus improving the reliability of the tracking sequence.

[0039] The annotation method provided in this application adjusts the parameters of the three-dimensional detection box in the target tracking sequence to optimize the tracking sequence in at least one of the geometric, positional, and confidence dimensions. This decomposes the multi-dimensional optimization of the tracking sequence into three adjustment directions, which can be flexibly selected according to the target type. It can overcome the limitations of traditional single-dimensional optimization, adapt to the annotation needs of different scenarios, and improve the versatility of the method. In turn, it improves the flexibility and accuracy of target annotation.

[0040] In some embodiments of this application, the geometric dimensions of each 3D detection box are adjusted to optimize the tracking sequence in the geometric dimension, including: For each 3D detection box, the point cloud data corresponding to the target within the 3D detection box is enhanced, and the geometric features of the target are extracted based on the enhanced point cloud data. A geometric context model corresponding to the 3D detection box is established through an attention mechanism. The geometric context model represents the geometric relationship between the 3D detection box and the labeled scene. Using the geometric features of the target as input, the optimized geometric dimensions of the 3D detection box are predicted based on the geometric context model. Adjust the geometry of the 3D detection frame according to the optimized geometry.

[0041] Enhancement processing can be performed on the point cloud data corresponding to the target. For example, it can be done by geometric transformation (such as rotation or translation) to enhance the point cloud data, or by increasing the point cloud density by difference or generating new points based on sampling enhancement.

[0042] Extracting the geometric features of the target based on the enhanced point cloud data can be achieved using point cloud feature extraction algorithms (such as fast point feature histograms or point cloud feature extraction networks). Geometric features are extracted from both the original point cloud data and the aligned enhanced point cloud data separately. Then, the features from the original and enhanced point cloud data are fused using methods such as max pooling and average pooling to obtain a more comprehensive set of target geometric features. The fused geometric features balance the realism of the original point cloud data with the diversity of the enhanced point cloud data, effectively avoiding feature distortion caused by local missing points in a single frame, and improving the ability of geometric features to represent the true shape of the target. Specifically, the geometric features of the target characterize its geometric shape.

[0043] For example, an embedded attention mechanism using a Transformer architecture can be employed. This mechanism uses the geometric features of the detection box as the query vector and the geometric information of the scene background as the key and value vectors, allowing the geometric context model to focus on the geometric relationship between the detection box and the labeled scene. The attention mechanism enables the model to accurately capture the key geometric associations between the detection box and the scene, avoiding biases caused by judging the size of the detection box in isolation based on the target size. The geometric context model provides a scene-level reference for size optimization, overcoming the limitations of relying solely on the target's own point cloud to determine size, and improving the rationality and accuracy of size prediction.

[0044] Using the geometric features of the target as input, the optimized geometric dimensions of the 3D detection box are predicted based on the geometric context model. For example, the fused geometric features of the target can be converted into a tensor format that the model can recognize and input into the trained geometric context model. The geometric context model combines the input geometric features with the "detection box-scene geometric relationship" it has learned to output the predicted parameters of the 3D detection box, such as length, width, and height.

[0045] In the annotation method provided in this application embodiment, the point cloud data corresponding to the target within each 3D detection box is enhanced, which helps to improve the problem of sparse point cloud data and single viewpoint in a single frame, making the extracted geometric features more comprehensive. The attention mechanism enables the geometric context model to accurately capture the key geometric relationship between the detection box and the scene, avoiding the deviation caused by judging the size of the detection box in isolation based on the target size, and improving the rationality and accuracy of the detection box size prediction.

[0046] In some embodiments of this application, the positions of each 3D detection box in the labeled scene are adjusted to optimize the tracking sequence in the position dimension, including: For each 3D detection box, the point cloud data corresponding to the target within the 3D detection box is enhanced, and the position features of the target are extracted based on the enhanced point cloud data. A positional context model corresponding to the 3D detection box is established through an attention mechanism. The positional context model represents the positional relationship between the 3D detection box and the labeled scene. Using the target's positional features as input, and based on a positional context model, the offset between the position of the 3D detection box and the actual position of the target is predicted. Adjust the position of the 3D detection box in the labeled scene according to the offset.

[0047] Enhancement processing can be performed on the point cloud data corresponding to the target. For example, it can be done by geometric transformation (such as rotation or translation) to enhance the point cloud data, or by increasing the point cloud density by difference or generating new points based on sampling enhancement.

[0048] Extracting the geometric features of the target based on the enhanced point cloud data can be achieved using point cloud feature extraction algorithms (such as fast point feature histograms or point cloud feature extraction networks). Geometric features are extracted from both the original point cloud data and the aligned enhanced point cloud data separately. Then, the features from the original and enhanced point cloud data are fused using methods such as max pooling and average pooling to obtain a more comprehensive set of target geometric features. The fused geometric features balance the realism of the original point cloud data with the diversity of the enhanced point cloud data, effectively avoiding feature distortion caused by local missing points in a single frame, and improving the ability of geometric features to represent the true shape of the target. Specifically, the geometric features of the target characterize its geometric shape.

[0049] A location context model is a prediction model trained through an attention mechanism. It is used to characterize the spatial constraints between the 3D detection box and the labeled scene (such as the relative position of the vehicle detection box and the lane line, the belonging relationship between the pedestrian detection box and the sidewalk, the distance constraints between the target and the adjacent obstacles, etc.), and provides scene-level spatial reference for location offset prediction.

[0050] Using the target's positional features as input, a location context model predicts the offset between the 3D bounding box's position and the target's actual position. For example, the target's positional features can be converted into a tensor format recognizable by the model before being input into the trained location context model. The location context model combines the input positional features with its learned spatial relationship between the bounding box and the labeled scene, outputting the predicted offset parameter. The offset refers to the deviation between the original coordinates and orientation of the 3D bounding box and the target's true (or optimal) position, and may include, for example, 3D coordinate offset and orientation angle offset.

[0051] The annotation method provided in this application enhances the point cloud data corresponding to the target within the 3D detection box, which helps to improve the problem of sparse point cloud data and single viewpoint in a single frame, making the extracted position features more comprehensive. The attention mechanism enables the position context model to accurately capture the position association between the detection box and the scene, breaking through the limitation of relying solely on the target's own point cloud to determine the position, improving the rationality and accuracy of offset prediction, and improving the rationality and accuracy of detection box size prediction.

[0052] In some embodiments of this application, the point cloud data corresponding to the target within the 3D detection box is enhanced, including: Establish a local coordinate system based on the 3D detection box; Perform geometric transformations on the point cloud data corresponding to the target within the 3D detection box to obtain the transformed point cloud data; The transformed point cloud data is aligned to the local coordinate system to integrate the point cloud data and the transformed point cloud data, resulting in enhanced point cloud data.

[0053] As can be understood, a local coordinate system refers to a coordinate system established based on a single 3D bounding box. For example, it could have the geometric center of the bounding box as the origin, with the major axis, width axis, and height axis of the bounding box corresponding to the X-axis, Y-axis, and Z-axis of the coordinate system, respectively. A local coordinate system can be used to provide a unified local reference for processing point clouds within the bounding box, such as geometric transformations, helping to eliminate the interference of target pose differences on feature extraction.

[0054] Geometric transformations are applied to the point cloud data corresponding to the target within the 3D detection bounding box. These transformations can include translation, rotation, scaling, etc. Generating diverse transformed point cloud data through geometric transformations helps improve the robustness of subsequent feature extraction. Specifically, the transformed point cloud data, generated after geometric transformation of the point cloud data corresponding to the target, can enhance the geometric recognizability of the target within the 3D detection bounding box, address the issues of sparse point cloud data and limited perspective in single frames, and enable more comprehensive extraction of geometric features.

[0055] It is understandable that since geometric transformations may cause the enhanced point cloud to deviate from the local coordinate system reference, the transformed point cloud data that has deviated can be realigned to the local coordinate system through coordinate alignment, so that the transformed point cloud data and the original point cloud are in the same local coordinate system; while preserving the multi-view features brought about by the transformation, the coordinate reference is unified through alignment.

[0056] The annotation method provided in this application establishes a local coordinate system based on a 3D detection box. After transforming the point cloud data of the target within the detection box, the transformed point cloud is aligned to the local coordinate system. This allows for effective integration of the point cloud data with the transformed point cloud data, avoiding coordinate deviations from interfering with subsequent feature extraction. The integrated, enhanced point cloud data is then used to generate diverse point cloud data through geometric transformations such as translation and rotation. This simulates the point cloud distribution of the target under different viewpoints and shapes, enhancing the geometric recognizability of the target and resulting in more comprehensive extracted geometric or positional features. In some embodiments of this application, the confidence level of each 3D detection box is adjusted to optimize the tracking sequence in terms of confidence level, including: For each 3D detection box, the first confidence score of the 3D detection box is determined based on the matching degree between the 3D detection box and the tracking sequence; Based on the 3D detection box and the point cloud data corresponding to the target within the 3D detection box, predict the intersection-union ratio (IU) of the optimized 3D detection box and the point cloud data corresponding to the target, and map the IU to the second confidence score of the 3D detection box. The optimized confidence of the 3D detection box is obtained based on the first confidence score and the second confidence score. Adjust the confidence level of the 3D detection box according to the optimized confidence level of the 3D detection box.

[0057] The matching degree between the 3D detection box and the tracking sequence is used to characterize the correlation between a single detection box and other frame detection boxes in the same sequence. Specifically, it can include spatial overlap, feature similarity, or temporal coherence.

[0058] The first confidence score is a confidence score obtained based on the matching degree between the detection box and the tracking sequence. It reflects the reliability of the current detection box in the temporal dimension, that is, whether the detection box belongs to the current target tracking sequence. For example, for the current detection box to be evaluated, the matching degree index between its detection box and the 3D detection boxes in adjacent frames (such as the first 3 frames and the last 3 frames) in the sequence can be calculated, such as spatial overlap, point cloud feature cosine similarity, etc., and fused to obtain a comprehensive matching degree; and the first confidence score of the current detection box can be obtained by mapping the comprehensive matching degree according to a preset mapping relationship.

[0059] Based on the 3D detection box and the point cloud data corresponding to the target within the 3D detection box, predict the intersection-union ratio (IU) of the optimized 3D detection box and the point cloud data corresponding to the target. For example, an IU regression subnetwork can be constructed, with the geometric parameters and position parameters of the current detection box and the preprocessed features of the target as input, to predict the IU of the optimized detection box (after geometric and positional optimization) and the target point cloud data.

[0060] After determining the intersection-union ratio (IUR) of the optimized 3D detection bounding box and the point cloud data corresponding to the target, the predicted IUR can be directly mapped to the second confidence score according to the preset mapping strategy.

[0061] Intersection over Union (IoU) refers to the ratio of the intersection area to the union area of ​​two regions. In this embodiment, it can reflect the degree of fit between the 3D detection box and the target.

[0062] After calculating the first confidence score and the second confidence score, the optimized confidence score of the 3D detection box can be obtained by weighted fusion or geometric mean fusion. In some embodiments, a minimum confidence threshold is set, and 3D detection boxes below the threshold are marked as abnormal detection boxes and removed.

[0063] The annotation method provided in this application determines the first confidence score of the 3D detection box based on the matching degree between the 3D detection box and the tracking sequence; and predicts the intersection-union ratio of the optimized 3D detection box and the point cloud data corresponding to the target based on the 3D detection box and the point cloud data corresponding to the target, and maps it to the second confidence score of the 3D detection box; it realizes the evaluation of the confidence of the 3D detection box from two dimensions of temporal reliability and spatial accuracy, improves the limitations of traditional single-dimensional confidence evaluation, and helps to obtain a more comprehensive confidence of the 3D detection box.

[0064] In some embodiments of this application, the target is labeled based on the optimized tracking sequence to obtain the target labeling result, including: Distinguish between static and dynamic targets; If the target is a dynamic target, based on the optimized tracking sequence, the 3D detection box of the target is predicted frame by frame in the multi-frame point cloud data of the labeled scene to obtain the target's labeling result; If the target is a static target, the 3D detection box with the highest confidence in the optimized tracking sequence is used as the initial detection box, and the other 3D detection boxes in the tracking sequence are merged into the initial detection box to obtain the target annotation result.

[0065] Distinguishing between dynamic and static targets is the process of classifying targets into dynamic and static targets based on their positional change characteristics in multi-frame point cloud data. Dynamic targets are those whose positions change over time in the labeled scene, such as moving vehicles and pedestrians; static targets are those whose positions remain largely unchanged over time in the labeled scene, such as parked vehicles and fixed obstacles.

[0066] In some embodiments, distinguishing between static and dynamic targets includes: differentiating targets based on positional changes between 3D detection boxes in the optimized tracking sequence. For example, this can involve extracting positional parameters of the 3D detection boxes for each frame from the optimized tracking sequence, calculating the center coordinate offset or orientation angle offset for 3D detection boxes in adjacent frames, and calculating a comprehensive 3D coordinate offset using methods such as Euclidean distance; comparing the comprehensive offset with a preset threshold to distinguish between static and dynamic targets. The preset threshold can be a dynamic threshold, dynamically adjusted based on the scene characteristics of the labeled scene.

[0067] For dynamic targets, based on the optimized tracking sequence, the 3D detection box of the target is predicted frame by frame in the multi-frame point cloud data of the labeled scene; or the Kalman filter algorithm can be used to analyze the motion characteristics of the dynamic target, fit the motion model of the target, and then combine the motion model with the multi-frame point cloud data to predict the 3D detection box parameters of the target in the current frame based on the optimized detection box parameters and motion model of the previous frame.

[0068] Understandably, when the target is a dynamic target, the target annotation results can retain the temporal information of the 3D detection boxes in each frame, which can meet the needs of dynamic target motion state analysis in dynamic target tracking scenarios such as autonomous driving algorithms.

[0069] For static targets, the detection box with the highest confidence score in the tracking sequence can be selected as the initial detection box. Then, using the confidence score as the weight, the weighted average of the geometric parameters (length, width, height) and positional parameters (center coordinates, orientation) of other 3D detection boxes in the tracking sequence is calculated. The weighted average parameter values ​​are then updated to the initial detection box to obtain the fused detection box.

[0070] The annotation method provided in this application distinguishes between dynamic and static targets for differentiated annotation. For dynamic targets, based on the optimized tracking sequence, the 3D detection box of the target is predicted frame by frame in the multi-frame point cloud data of the annotation scene, so that the annotation result of the target can retain the temporal information of the 3D detection box of each frame, thereby meeting the needs of dynamic target motion state analysis. For static targets, the 3D detection box with the highest confidence in the optimized tracking sequence is used as the initial detection box, and the other 3D detection boxes in the tracking sequence are fused into the initial detection box. Combined with the weighted fusion of multi-frame detection boxes, the local bias caused by the sparsity and noise of single-frame point clouds can be eliminated, and the accuracy of the fused detection box is better than that of the single-frame detection box.

[0071] In some embodiments of this application, the target is distinguished between dynamic and static states, including: Based on the optimized tracking sequence, the positional change features of the target's 3D detection box in the time dimension are extracted; Based on the characteristics of positional changes, the target is determined to be either a dynamic target or a static target.

[0072] It is understood that the optimized tracking sequence refers to a tracking sequence optimized in at least one of the geometric dimension, position dimension, and confidence dimension, including multi-frame 3D detection boxes assembled according to a temporal sequence. Position change features refer to the quantitative features of the position of the target's 3D detection box in the labeled scene changing over time in the temporal sequence of multi-frame point cloud data. Examples may include indicators such as position offset between different detection boxes, position change variance, or position change rate.

[0073] For example, from the optimized tracking sequence, position parameters of the 3D detection boxes (such as the geometric center coordinates of the 3D detection boxes or the coordinates of the four bottom corner points of the 3D detection boxes) are extracted in time sequence. Based on the position parameters of each 3D detection box, quantized features of the target's 3D detection box position change are calculated as position change features. For example, the Euclidean distance between the center coordinates of two adjacent frames of detection boxes is calculated as the offset, and the variance of the offset is calculated. In some embodiments, the quantized features are integrated into a target position change feature vector, which serves as the basis for subsequent dynamic and static differentiation.

[0074] Determining whether a target is dynamic or static based on its position change characteristics can be achieved by comparing the position change characteristics with a preset feature threshold corresponding to the labeled scene; alternatively, it can be by calculating the average offset of the 3D detection boxes across multiple consecutive frames and using the average to apply a threshold to determine whether the target is dynamic or static. In some embodiments, the position change characteristics can also be input into a pre-trained dynamic / static discrimination model to obtain the dynamic / static discrimination result of the target.

[0075] The annotation method provided in this application extracts the position change features of the target's 3D detection box in the time dimension based on the optimized tracking sequence. This realizes the transformation of the 3D detection box position change over time into a quantified feature index, which can more accurately characterize the position change of the 3D detection box. This improves the accuracy of determining whether the target is a dynamic or static target based on the position change features, so that dynamic and static targets can be annotated differently in the future.

[0076] In some embodiments of this application, determining the tracking sequence of the target in the labeled scene based on multi-frame point cloud data of the labeled scene includes: Acquire multi-frame point cloud data of the labeled scene according to the time sequence, and determine the 3D detection boxes in the multi-frame point cloud data. Each frame of point cloud data includes one or more 3D detection boxes. Based on the 3D detection bounding boxes in multi-frame point cloud data, determine the tracking sequence of one or more targets in the labeled scene.

[0077] For example, multiple frames of point cloud data of the labeled scene are acquired multiple times using sensors such as LiDAR at a preset acquisition frequency to ensure the temporal continuity of the point cloud data. In some embodiments, preprocessing such as denoising, cropping, and point cloud downsampling are also performed on each frame of raw point cloud data.

[0078] To determine the 3D detection boxes in multi-frame point cloud data, a 3D target detection algorithm can be used to perform target detection on each frame of preprocessed point cloud data, outputting the 3D detection boxes and parameters of all targets in each frame, so that each target in a single frame of point cloud data corresponds to a detection box.

[0079] Based on 3D bounding boxes in multi-frame point cloud data, the tracking sequence for each target is determined. This can be achieved by extracting appearance and motion features from the 3D bounding boxes in each frame, followed by multi-target tracking association. For example, the spatial intersection-union ratio and appearance feature similarity between the current frame's bounding box and the previous frame's bounding box are calculated and compared with a preset threshold. If the threshold is met, the bounding boxes are identified as belonging to the same target and assigned the same identifier. By organizing all bounding boxes corresponding to each identifier according to the acquisition time sequence, the tracking sequence for each target can be formed.

[0080] The annotation method provided in this application acquires multi-frame point cloud data of the annotation scene in a time sequence and determines the 3D detection boxes in the multi-frame point cloud data. The time sequence acquisition ensures the temporal continuity of the point cloud data and provides temporal dimension support for subsequent target tracking. Based on the 3D detection boxes in the multi-frame point cloud data, the tracking sequence of each target is determined. The finally generated temporal tracking sequence realizes the cross-frame association of the 3D detection boxes of the same target, integrates temporal information into the target detection process, and provides a data foundation for subsequent multi-dimensional optimization.

[0081] In some embodiments of this application, acquiring multi-frame point cloud data of a labeled scene in a time sequence and determining 3D detection boxes in the multi-frame point cloud data includes: Using multi-frame point cloud data of the labeled scene as combined input, the point cloud density information of each frame of point cloud data is determined by point cloud density perception. Based on the combined multi-frame point cloud data, perform time series enhancement to generate multiple enhanced point cloud samples; Based on multiple enhanced point cloud samples and point cloud density information of each frame of point cloud data, target detection is performed on multiple frames of point cloud data to determine the 3D detection box in each frame of point cloud data.

[0082] In this embodiment, the multi-frame point cloud data of the labeled scene is used as a whole sequence as input. Compared with single-frame independent input, the temporal correlation of multi-frame data can be used to assist in subsequent density perception and enhanced sample generation, which helps to improve processing accuracy.

[0083] Point cloud density sensing refers to the technique of quantifying and calculating the density of point cloud data in each frame. For example, by counting the number of points in a unit space, the sparsity or density of point clouds in different frames and regions can be determined. Point cloud density information, as the output of point cloud density sensing, can specifically include the global average density and local region density of each frame's point cloud, providing a density adaptation basis for subsequent target detection. For example, five frames of point cloud data can be stitched together into a complete sequence according to their acquisition time and uniformly aligned to the global coordinate system of the labeled scene. Density calculation methods can be used, such as voxel grid or region of interest (ROI) grid point feature extraction. For instance, each frame of point cloud can be divided into a grid of fixed size, and the number of points in each grid can be counted to calculate the global average density of each frame's point cloud.

[0084] Performing time-series augmentation to generate multiple augmented point cloud samples can be achieved through Test Time Augmentation (TTA). For example, TTA can be used to perform diverse transformations on the input data during the testing phase of object detection, generating multiple augmented samples. Augmented point cloud samples refer to diverse point cloud data generated through TTA operations, which may include temporally stitched samples of multi-frame point clouds, cropped or scaled samples of single-frame point clouds, and fused augmented samples of cross-frame point clouds, etc.

[0085] Based on multiple enhanced point cloud samples and point cloud density information from each frame of point cloud data, target detection is performed on multi-frame point cloud data. For example, an adaptive detection strategy can be adopted based on point cloud density information; high-density frames can be directly processed using 3D detection algorithms, while medium- and low-density frames can be processed with enhanced samples to assist detection. A 3D target detection model supporting multiple inputs can be used, taking enhanced samples and original point cloud data as inputs, adjusting the detection weights based on density information, and outputting the detection results for each enhanced sample and the original multi-frame point cloud data.

[0086] The annotation method provided in this application uses multi-frame point cloud data of the annotation scene as a combined input, which can avoid detection bias caused by processing a single frame independently. Based on the combined multi-frame point cloud data, time series enhancement is performed to generate multiple enhanced point cloud samples, and target detection is performed based on the multiple enhanced point cloud samples and the point cloud density information of each frame of point cloud data. This realizes the use of multi-frame correlation to supplement the temporal information of dynamic targets for time series enhancement, which helps to improve the coherence of dynamic target detection. Based on the point cloud density information of the point cloud data, a differentiated detection strategy can be executed to adapt to point clouds of different densities, thereby improving the accuracy of determining the 3D detection box.

[0087] In some embodiments of this application, determining a tracking sequence for one or more targets in a labeled scene based on 3D detection boxes in multi-frame point cloud data includes: Obtain the confidence score of 3D bounding boxes in multi-frame point cloud data; The 3D target detection boxes are filtered based on their confidence level, and the tracking sequence for each target is determined based on the filtered 3D detection boxes.

[0088] For example, the confidence of the 3D detection box can be obtained by processing the point cloud data of each frame through a pre-trained 3D target detection model, or by calculating the matching degree between the point cloud data in the 3D detection box and the point cloud of the target template (such as using the iterative nearest point algorithm to calculate the overlap). This application does not make any specific limitations on this.

[0089] 3D detection boxes can be filtered based on their confidence level. This can be done by setting a confidence threshold and prioritizing the use of high-confidence 3D detection boxes to determine the target tracking sequence, or by using low-confidence detection boxes to supplement and improve the target tracking sequence when no matching high-confidence detection boxes are available.

[0090] For the filtered 3D detection boxes, the tracking sequence can be determined by association in various ways. For example, a data association algorithm (such as the Hungarian algorithm) can be used to match 3D detection boxes across point cloud data frames to the same target. Alternatively, a method based on cross-union ratio (CUP) and positional continuity can be used to calculate the CUP between each 3D detection box in the current frame and the last detection box in the target tracking sequence of the previous frame. If the CUP is greater than a preset CUP threshold and the center distance between the two detection boxes is less than a preset distance threshold, then the detection box in the current frame is added to the tracking sequence of the corresponding target.

[0091] In the annotation method provided in this application, after obtaining the confidence of multiple frames of detection boxes, the three-dimensional detection boxes are first screened based on the confidence. The high-confidence detection boxes with higher reliability are used first to construct the tracking sequence, reducing the interference of false detection boxes on the tracking sequence. When there are no high-confidence detection boxes to match, low-confidence detection boxes are used to supplement, which improves the accuracy and robustness of generating the tracking sequence.

[0092] In some embodiments of this application, the 3D target detection boxes are filtered based on their confidence levels, and the tracking sequence for each target is determined based on the filtered 3D detection boxes, including: The 3D detection boxes are filtered based on confidence level to obtain high-scoring and low-scoring detection boxes in the 3D detection boxes; According to the time sequence of acquiring multiple frames of point cloud data, traverse each frame of point cloud data and match the high-resolution detection boxes in the point cloud data with the tracking sequence of each target; If a target tracking sequence matches a high-resolution detection box, update the target tracking sequence based on the high-resolution detection box.

[0093] For example, for each 3D detection box, the confidence level of the 3D detection box can be compared with a preset confidence threshold to determine whether the 3D detection box is a high-scoring or low-scoring detection box. Among them, high-scoring detection boxes have stronger reliability than low-scoring detection boxes and can be given priority for matching and updating the tracking sequence; low-scoring detection boxes have weaker reliability and are only used to supplement matching when high-scoring detection boxes fail to match.

[0094] Traversing the point cloud data of the labeled scene in chronological order means processing the detection boxes within each frame according to the acquisition time sequence of the multi-frame point cloud data of the labeled scene; calculating the similarity between the detection box of the current frame and the existing target tracking sequence (the set of detection boxes of the previous frames) to determine whether the detection box belongs to the target corresponding to the tracking sequence, which can be based on 3D spatial overlap and feature similarity. For successfully matched detection boxes, they are added to the tracking sequence of the corresponding target to update the tracking sequence so that the tracking sequence can reflect the position and attitude changes of the target in real time.

[0095] The annotation method provided in this application divides the 3D detection box into high-scoring detection boxes and low-scoring detection boxes according to the confidence level of the 3D detection box, and uses high-scoring detection boxes to match the tracking sequence to ensure tracking accuracy and improve the accuracy of generating the tracking sequence.

[0096] In some embodiments of this application, the method further includes: After the high-resolution detection boxes in the point cloud data are matched, if there is a target tracking sequence that has not been updated, the low-resolution detection boxes in the point cloud data will be matched with the unupdated tracking sequence. If an unupdated tracking sequence matches a low-scoring detection box, update the unupdated tracking sequence based on the low-scoring detection box.

[0097] After the high-resolution detection box is matched, if there is a tracking sequence that does not find a matching detection box, it is necessary to supplement the matching with low-resolution detection boxes to maintain the continuity of the sequence. Supplementing with low-resolution boxes to maintain the continuity of the tracking sequence can avoid the inability to match when the target reappears after a brief occlusion.

[0098] The annotation method provided in this application uses low-scoring detection boxes to supplement the matching for tracking sequences where no matching high-scoring detection boxes exist, thereby ensuring continuity. The robustness of the generated tracking sequences is improved through hierarchical matching.

[0099] In some embodiments of this application, the method further includes: According to the time sequence of acquiring multiple frames of point cloud data, the point cloud data is traversed in reverse order, and an inverse tracking sequence is generated that is the opposite of the tracking sequence of each target. Update the tracking sequence of the corresponding target based on the inverse tracking sequence of each target.

[0100] As can be understood, reverse traversal refers to a traversal method that reverses the acquisition time sequence of multi-frame point cloud data, processing point cloud data frame by frame. The reverse tracking sequence is a target tracking sequence generated based on reverse traversal, with the timing reversed from the original tracking sequence. The detection boxes within the sequence are arranged sequentially from the last frame to the first frame, providing a tracking reference with reverse timing. Compared to traditional unidirectional tracking (forward only), it can detect and resolve potential timing deviations during tracking.

[0101] The tracking sequence of each target is updated based on its inverse tracking sequence. This can be achieved by fusing the bounding box parameters of the original tracking sequence (forward) and the inverse tracking sequence (backward) to correct deviations in the original sequence (such as changes in bounding box identifiers or parameter distortion caused by occlusion), thereby generating a more accurate and coherent tracking sequence. Specifically, the original tracking sequence and inverse tracking sequence of each target can be associated according to the target's identifier. For each target, a weighted average method is used to fuse the bounding box parameters (such as position coordinates and geometric dimensions) of the corresponding frames in the bidirectional sequence to correct parameters with excessive deviations in the original sequence.

[0102] In some embodiments, following the temporal sequence of acquiring multi-frame point cloud data of the labeled scene, the point cloud data is traversed in reverse order, and an inverse tracking sequence is generated that is the opposite of the tracking sequence of each target, including: According to the time sequence of acquiring multi-frame point cloud data of the labeled scene, the point cloud data of each frame is traversed in reverse order, and the high-resolution detection boxes in the point cloud data are matched with the tracking sequence of each target. If a target tracking sequence matches a high-resolution detection box, update the target tracking sequence based on the high-resolution detection box. After the high-resolution detection boxes in the point cloud data are matched, if there is a target tracking sequence that has not been updated, the low-resolution detection boxes in the point cloud data will be matched with the unupdated tracking sequence. If an unupdated tracking sequence matches a low-scoring detection box, update the unupdated tracking sequence based on the low-scoring detection box.

[0103] The annotation method provided in this application generates an inverse tracking sequence that is the reverse of the tracking sequence of each target, according to the reverse temporal order of the multi-frame point cloud data of the annotation scene. The inverse tracking sequence is then updated according to the inverse tracking sequence of each target. The inverse tracking sequence provides a tracking reference with reverse temporal order, which helps to discover and solve the temporal deviation problem that may occur during the tracking process, thereby improving the accuracy of determining the target tracking sequence.

[0104] Figure 2 This is one of the structural schematic diagrams of the annotation system provided in some embodiments of this application. For example... Figure 2 As shown, the annotation system includes: a 3D target detection subsystem, an offline multi-target tracking subsystem, a target information refinement subsystem, an automatic target annotation subsystem, and a computing and storage resource scheduling subsystem; The 3D target detection subsystem is used to acquire multi-frame point cloud data of the labeled scene in a time sequence and determine the 3D detection boxes in the multi-frame point cloud data. Each frame of point cloud data includes one or more 3D detection boxes. An offline multi-target tracking subsystem is used to determine the tracking sequence of each target based on the 3D detection boxes in multi-frame point cloud data; The target information refinement subsystem is used to optimize the tracking sequence of each target in the labeled scene in at least one of the geometric dimension, position dimension and confidence dimension; The automatic target labeling subsystem is used to label targets based on the optimized tracking sequence and obtain the target labeling results; The computation and storage resource scheduling subsystem is used to handle memory allocation, storage allocation, and computation resource allocation and scheduling issues during the runtime of each subsystem in the annotation system.

[0105] Figure 3 This is a second schematic diagram of the structure of the annotation system provided in some embodiments of this application. For example... Figure 3 As shown, the 3D target detection subsystem is used for: Using multi-frame point cloud data of the labeled scene as combined input, the point cloud density information of each frame of point cloud data is determined by point cloud density perception. Based on the combined multi-frame point cloud data, perform time series enhancement to generate multiple enhanced point cloud samples; Based on multiple enhanced point cloud samples and point cloud density information of each frame of point cloud data, target detection is performed on multiple frames of point cloud data to determine the three-dimensional detection box (3D target detection information) in each frame of point cloud data.

[0106] The offline multi-target tracking subsystem is used for: Based on the 3D detection bounding boxes in multi-frame point cloud data, the tracking sequence of each target is determined; In the target tracking sequence, the identity identifiers corresponding to the multi-frame 3D detection bounding boxes of the target are the same.

[0107] The target information refining subsystem is used for: Adjust the geometric dimensions of each 3D detection box to optimize the target tracking sequence in the geometric dimension; Adjust the position of each 3D bounding box within the labeled scene to optimize the target tracking sequence in terms of position; or, Adjust the confidence level of each 3D bounding box to optimize the target tracking sequence in terms of confidence level.

[0108] The automatic target labeling subsystem is used for: Distinguish between static and dynamic targets; If the target is a dynamic target, based on the optimized tracking sequence, the 3D detection box of the target is predicted frame by frame in the multi-frame point cloud data of the labeled scene to obtain the target's labeling result; If the target is a static target, the 3D detection box with the highest confidence in the optimized tracking sequence is used as the initial detection box, and the other 3D detection boxes in the tracking sequence are merged into the initial detection box to obtain the target annotation result.

[0109] The annotation method provided in this application can be executed by an annotation device. This application uses an annotation device executing the annotation method as an example to illustrate the annotation device provided in this application.

[0110] Figure 4 This is a schematic diagram of the structure of a labeling device provided in some embodiments of this application. For example... Figure 4 As shown, the labeling device 400 includes: The tracking module 401 is used to determine the tracking sequence of the target in the labeled scene based on the multi-frame point cloud data of the labeled scene; The adjustment module 402 is used to optimize the tracking sequence in at least one of the geometric dimension, the position dimension, and the confidence dimension; The annotation module 403 is used to annotate the target based on the optimized tracking sequence to obtain the annotation results of the target.

[0111] In some embodiments, the target tracking sequence is composed of a temporal set of multiple frames of 3D detection bounding boxes of the target; the adjustment module 402 is used to perform at least one of the following: Adjust the geometric dimensions of each 3D detection box to optimize the tracking sequence in the geometric dimension; Adjust the position of each 3D bounding box within the labeled scene to optimize the tracking sequence in terms of position; or, Adjust the confidence level of each 3D detection box to optimize the tracking sequence in terms of confidence level.

[0112] In some embodiments, the geometric dimensions of each 3D detection box are adjusted to optimize the tracking sequence in a geometrical dimension, including: For each 3D detection box, the point cloud data corresponding to the target within the 3D detection box is enhanced, and the geometric features of the target are extracted based on the enhanced point cloud data. A geometric context model corresponding to the 3D detection box is established through an attention mechanism. The geometric context model represents the geometric relationship between the 3D detection box and the labeled scene. Using the geometric features of the target as input, the optimized geometric dimensions of the 3D detection box are predicted based on the geometric context model. Adjust the geometry of the 3D detection frame according to the optimized geometry.

[0113] In some embodiments, the position of each 3D detection box in the labeled scene is adjusted to optimize the tracking sequence in the position dimension, including: For each 3D detection box, the point cloud data corresponding to the target within the 3D detection box is enhanced, and the position features of the target are extracted based on the enhanced point cloud data. A positional context model corresponding to the 3D detection box is established through an attention mechanism. The positional context model represents the positional relationship between the 3D detection box and the labeled scene. Using the target's positional features as input, and based on a positional context model, the offset between the position of the 3D detection box and the actual position of the target is predicted. Adjust the position of the 3D detection box in the labeled scene according to the offset.

[0114] In some embodiments, the point cloud data corresponding to the target within the 3D detection box is enhanced, including: Establish a local coordinate system based on the 3D detection box; Perform geometric transformations on the point cloud data corresponding to the target within the 3D detection box to obtain the transformed point cloud data; The transformed point cloud data is aligned to the local coordinate system to integrate the point cloud data and the transformed point cloud data, resulting in enhanced point cloud data.

[0115] In some embodiments, the confidence level of each 3D detection box is adjusted to optimize the tracking sequence in terms of confidence level, including: For each 3D detection box, the first confidence score of the 3D detection box is determined based on the matching degree between the 3D detection box and the tracking sequence; Based on the 3D detection box and the point cloud data corresponding to the target within the 3D detection box, predict the intersection-union ratio (IU) of the optimized 3D detection box and the point cloud data corresponding to the target, and map the IU to the second confidence score of the 3D detection box. The optimized confidence of the 3D detection box is obtained based on the first confidence score and the second confidence score. Adjust the confidence level of the 3D detection box according to the optimized confidence level of the 3D detection box.

[0116] In some embodiments, the annotation module 403 is used for: Distinguish between static and dynamic targets; If the target is a dynamic target, based on the optimized tracking sequence, the 3D detection box of the target is predicted frame by frame in the multi-frame point cloud data of the labeled scene to obtain the target's labeling result; If the target is a static target, the 3D detection box with the highest confidence in the optimized tracking sequence is used as the initial detection box, and the other 3D detection boxes in the tracking sequence are merged into the initial detection box to obtain the target annotation result.

[0117] In some embodiments, the target is distinguished between dynamic and static states, including: Based on the optimized tracking sequence, the positional change features of the target's 3D detection box in the time dimension are extracted; Based on the characteristics of positional changes, the target is determined to be either a dynamic target or a static target.

[0118] In some embodiments, the tracking module 401 is used for: Acquire multi-frame point cloud data of the labeled scene according to the time sequence, and determine the 3D detection boxes in the multi-frame point cloud data. Each frame of point cloud data includes one or more 3D detection boxes. Based on the 3D detection bounding boxes in multi-frame point cloud data, determine the tracking sequence of one or more targets in the labeled scene.

[0119] In some embodiments, the tracking sequence of each target is determined based on the 3D detection bounding boxes in multi-frame point cloud data, including: Obtain the confidence score of 3D bounding boxes in multi-frame point cloud data; The 3D target detection boxes are filtered based on their confidence level, and the tracking sequence for each target is determined based on the filtered 3D detection boxes.

[0120] In some embodiments, 3D bounding boxes are filtered based on their confidence levels, and a tracking sequence for one or more targets in the labeled scene is determined based on the filtered 3D bounding boxes, including... The 3D detection boxes are filtered based on confidence level to obtain high-scoring and low-scoring detection boxes in the 3D detection boxes; According to the time sequence of acquiring multiple frames of point cloud data, traverse each frame of point cloud data and match the high-resolution detection boxes in the point cloud data with the tracking sequence of each target; If a target tracking sequence matches a high-resolution detection box, update the target tracking sequence based on the high-resolution detection box.

[0121] In some embodiments, the tracking module 401 is also used for: After the high-resolution detection boxes in the point cloud data are matched, if there is a target tracking sequence that has not been updated, the low-resolution detection boxes in the point cloud data will be matched with the unupdated tracking sequence. If an unupdated tracking sequence matches a low-scoring detection box, update the unupdated tracking sequence based on the low-scoring detection box.

[0122] In some embodiments, the identity identifiers corresponding to the target's multi-frame 3D detection bounding boxes in the target tracking sequence are the same.

[0123] The labeling device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the device.

[0124] The labeling device in this application embodiment can be a device with an operating system. The operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems. This application embodiment does not specifically limit the specific operating system.

[0125] The annotation device provided in this application embodiment can implement all the processes implemented in the above annotation method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0126] Figure 5 These are schematic diagrams of the structure of an electronic device provided in some embodiments of this application. In some embodiments, such as Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, it implements the various processes of the above-mentioned labeled method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0127] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0128] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described annotation method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0129] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0130] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described annotation method.

[0131] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0132] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0133] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0134] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0136] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0137] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0138] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A labeling method, characterized in that, include: Based on multi-frame point cloud data of the labeled scene, determine the tracking sequence of the target in the labeled scene; The tracking sequence is optimized in at least one of the following dimensions: geometric dimension, location dimension, and confidence dimension; The target is labeled according to the optimized tracking sequence to obtain the labeling result of the target.

2. The annotation method according to claim 1, characterized in that, The target tracking sequence is composed of multiple frames of 3D detection boxes of the target arranged in a temporal order; Optimizing the tracking sequence in at least one of the geometric, positional, and confidence dimensions includes at least one of the following: Adjust the geometric dimensions of each of the three-dimensional detection boxes to optimize the tracking sequence in the geometric dimensions; Adjust the position of each of the 3D detection boxes in the labeled scene to optimize the tracking sequence in the position dimension; or, The confidence level of each of the three-dimensional detection boxes is adjusted to optimize the tracking sequence in the confidence level dimension.

3. The annotation method according to claim 2, characterized in that, The adjustment of the geometric dimensions of each of the three-dimensional detection boxes to optimize the tracking sequence in the geometric dimensions includes: For each of the three-dimensional detection boxes, the point cloud data corresponding to the target within the three-dimensional detection box is enhanced, and the geometric features of the target are extracted based on the enhanced point cloud data. A geometric context model corresponding to the 3D detection box is established through an attention mechanism. The geometric context model represents the geometric relationship between the 3D detection box and the labeled scene. Using the geometric features of the target as input, and based on the geometric context model, the optimized geometric dimensions of the 3D detection box are predicted; Adjust the geometry of the 3D detection frame according to the optimized geometry.

4. The annotation method according to claim 2, characterized in that, The step of adjusting the position of each of the three-dimensional detection boxes in the labeled scene to optimize the tracking sequence in the position dimension includes: For each of the three-dimensional detection boxes, the point cloud data corresponding to the target within the three-dimensional detection box is enhanced, and the position features of the target are extracted based on the enhanced point cloud data. A positional context model corresponding to the 3D detection box is established through an attention mechanism, and the positional context model represents the positional relationship between the 3D detection box and the labeled scene. Using the target's positional features as input, and based on the positional context model, the offset between the position of the 3D detection box and the actual position of the target is predicted; Adjust the position of the 3D detection box in the labeled scene according to the offset.

5. The annotation method according to claim 3 or 4, characterized in that, The enhancement processing of the point cloud data corresponding to the target within the 3D detection box includes: A local coordinate system is established based on the three-dimensional detection frame; Perform a geometric transformation on the point cloud data corresponding to the target within the three-dimensional detection frame to obtain the transformed point cloud data; The transformed point cloud data is aligned to the local coordinate system to integrate the point cloud data and the transformed point cloud data, resulting in enhanced point cloud data.

6. The annotation method according to claim 2, characterized in that, The step of adjusting the confidence level of each of the three-dimensional detection boxes to optimize the tracking sequence along the confidence level dimension includes: For each of the three-dimensional detection boxes, a first confidence score for the three-dimensional detection box is determined based on the matching degree between the three-dimensional detection box and the tracking sequence; Based on the 3D detection box and the point cloud data corresponding to the target within the 3D detection box, predict the intersection-union ratio (IU / R) of the optimized 3D detection box and the point cloud data corresponding to the target, and map the IU / R to the second confidence score of the 3D detection box. The optimized confidence of the 3D detection box is obtained based on the first confidence score and the second confidence score. The confidence level of the 3D detection box is adjusted according to the optimized confidence level of the 3D detection box.

7. The annotation method according to claim 1, characterized in that, The step of labeling the target based on the optimized tracking sequence to obtain the labeling result of the target includes: The target is distinguished between static and dynamic states; If the target is a dynamic target, based on the optimized tracking sequence, the 3D detection box of the target is predicted frame by frame in the multi-frame point cloud data of the labeled scene to obtain the labeling result of the target; If the target is a static target, the 3D detection box with the highest confidence in the optimized tracking sequence is used as the initial detection box, and the other 3D detection boxes in the tracking sequence are merged into the initial detection box to obtain the annotation result of the target.

8. The annotation method according to claim 7, characterized in that, The process of distinguishing between static and dynamic states of the target includes: Based on the optimized tracking sequence, the positional change features of the target's 3D detection box in the time dimension are extracted; Based on the location change characteristics, the target is determined to be either a dynamic target or a static target.

9. The annotation method according to claim 1, characterized in that, The step of determining the tracking sequence of the target in the labeled scene based on multi-frame point cloud data of the labeled scene includes: The annotation scene is acquired in time sequence, and three-dimensional detection boxes are determined in the multi-frame point cloud data. Each frame of point cloud data includes one or more three-dimensional detection boxes. Based on the 3D detection bounding boxes in the multi-frame point cloud data, a tracking sequence for one or more targets in the labeled scene is determined.

10. The annotation method according to claim 9, characterized in that, The step of determining a tracking sequence for one or more targets in the labeled scene based on the 3D detection bounding boxes in the multi-frame point cloud data includes: Obtain the confidence score of 3D bounding boxes in multi-frame point cloud data; The three-dimensional detection boxes are filtered based on their confidence level, and the tracking sequence of each target is determined based on the filtered three-dimensional detection boxes.

11. The annotation method according to claim 10, characterized in that, The process involves filtering the 3D target detection boxes based on their confidence levels and determining the tracking sequence for each target based on the filtered 3D detection boxes. The three-dimensional detection boxes are filtered based on the confidence level to obtain high-scoring and low-scoring detection boxes in the three-dimensional detection boxes; According to the time sequence of acquiring the multi-frame point cloud data, traverse each frame of point cloud data and match the high-resolution detection boxes in the point cloud data with the tracking sequence of each target; If a target tracking sequence matches a high-resolution detection box, the target tracking sequence is updated based on the high-resolution detection box.

12. The annotation method according to claim 11, characterized in that, The method further includes: After the high-resolution detection boxes in the point cloud data are matched, if there is a target tracking sequence that has not been updated, the low-resolution detection boxes in the point cloud data are matched with the unupdated tracking sequence. If an outdated tracking sequence matches a low-scoring detection box, the outdated tracking sequence is updated based on the low-scoring detection box.

13. The annotation method according to any one of claims 9 to 12, characterized in that, The method further includes: According to the time sequence of acquiring the multi-frame point cloud data, the point cloud data is traversed in reverse order, and an inverse tracking sequence is generated that is opposite to the tracking sequence time sequence of each target. Update the tracking sequence of the corresponding target based on the inverse tracking sequence of each target.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the annotation method as described in any one of claims 1 to 13.

15. A vehicle, characterized in that, Including the electronic device as described in claim 14.

Images