Automatic Optimization Method and Device for 3D Box Labels in 4D Dynamic Object Interactive Annotation

Abstract

This application discloses an automatic optimization method and apparatus for 3D bounding box labels in 4D dynamic object interactive annotation, comprising: acquiring all key 3D bounding boxes and their corresponding dynamic object point sets, and all interpolated 3D bounding boxes and their corresponding dynamic object point sets in a dynamic object sequence under a global coordinate system; calculating the principal directions of the two dynamic object point sets and calculating their differences, adjusting the direction of the interpolated 3D bounding boxes according to the differences to obtain the interpolated 3D bounding boxes with optimized directions; calculating the density distribution characteristics and geometric distance distribution characteristics of the interpolated 3D bounding boxes and key 3D bounding boxes and their corresponding dynamic object point sets, and constructing a composite loss function; iteratively optimizing the center point position of the interpolated 3D bounding boxes with optimized directions, fusing the composite loss function in each iteration to obtain the optimal center point position of the interpolated 3D bounding boxes, and then updating the center point position. Compared with previous solutions, this invention significantly improves the accuracy of interpolated 3D bounding boxes and further reduces manual workload.

Description

Technical Field

[0001] This application relates to the field of data annotation technology, and in particular to a method and apparatus for automatic optimization of 3D bounding box labels in 4D dynamic object interactive annotation. Background Technology

[0002] With the rapid development of autonomous driving, intelligent robots, and other fields, the demand for spatiotemporal state perception and modeling of dynamic objects is becoming increasingly urgent. 4D dynamic object interactive annotation technology (i.e., continuous annotation in 3D space + time dimensions) is a core component in constructing high-precision dynamic scene datasets, and its annotation quality directly affects the performance of downstream algorithms such as object detection and trajectory prediction. Previously, the most efficient 3D bounding box annotation method for dynamic objects adopted a "manual annotation of keyframes + temporal interpolation generation" approach. This involved selecting sparse keyframes in a time series to annotate 3D boundaries, and then automatically generating 3D bounding box labels for intermediate frames using a linear interpolation algorithm. However, dynamic objects in real-world scenes often exhibit variable speed motion, abrupt shape changes, and complex interactions with the environment, leading to significant limitations in traditional interpolation methods.

[0003] The applicant's previous technical solution (application number: CN202411062156.2) registered key 3D bounding boxes with interpolated 3D bounding boxes in point clouds, and optimized the position and orientation of the interpolated 3D bounding boxes based on the point cloud registration results, effectively improving the annotation accuracy of dynamic objects. However, in practical applications, it was found that some dynamic object point sets in the point cloud data are sparse, resulting in inaccurate point cloud registration results. This caused a systematic deviation in the optimization results of the solution, still requiring manual frame-by-frame fine-tuning, which significantly limited further improvement in annotation efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for automatically optimizing 3D bounding box labels in 4D dynamic object interactive annotation, so as to solve the problem that the previous scheme of optimizing the position and orientation of interpolated 3D bounding boxes based on point cloud registration results is easily affected by the point set density, leading to optimization failure.

[0005] According to a first aspect of the embodiments of this application, an automatic optimization method for 3D bounding box labels in a 4D dynamic object sequence is provided, comprising:

[0006] Obtain all key 3D bounding boxes, all interpolated 3D bounding boxes, the set of dynamic object points corresponding to the key 3D bounding boxes, and the set of dynamic object points corresponding to the interpolated 3D bounding boxes in the 4D dynamic object sequence under the global coordinate system;

[0007] Calculate the principal directions of the dynamic object point sets corresponding to the interpolated 3D bounding box and the key 3D bounding box respectively to obtain the interpolation principal direction and the key principal direction;

[0008] Calculate the difference between the interpolation principal direction and the key principal direction, and adjust the orientation of the interpolated 3D box according to the difference to obtain the interpolated 3D box with optimized orientation;

[0009] Calculate the density distribution characteristics and geometric distance distribution characteristics of the dynamic object point sets corresponding to the interpolated 3D bounding box and the key 3D bounding box after the optimization direction, and construct a composite loss function;

[0010] An iterative algorithm with dual-mode dynamic momentum terms is used to iteratively optimize the center point position of the interpolated 3D box after optimization. The composite loss function is fused in each iteration. Optimization stops when the value of the composite loss function is lower than the threshold or the number of consecutive unimproved times reaches the maximum allowable value, thus obtaining the optimal center point position of the interpolated 3D box.

[0011] The interpolation 3D box label optimization is completed by updating the center point position of the interpolated 3D box after optimizing the direction based on the optimal center point position.

[0012] According to a second aspect of the embodiments of this application, an automatic optimization device for 3D bounding box labels in a 4D dynamic object sequence is provided, comprising:

[0013] The data acquisition module is used to acquire all key 3D boxes, all interpolated 3D boxes, the set of dynamic object points corresponding to the key 3D boxes, and the set of dynamic object points corresponding to the interpolated 3D boxes in the 4D dynamic object sequence under the global coordinate system.

[0014] The main direction calculation module is used to calculate the main direction of the dynamic object point set corresponding to the interpolated 3D frame and the key 3D frame respectively, so as to obtain the interpolated main direction and the key main direction.

[0015] The orientation optimization module is used to calculate the difference between the interpolation principal orientation and the key principal orientation, and adjust the orientation of the interpolated 3D box according to the difference to obtain the interpolated 3D box with optimized orientation.

[0016] The composite loss function construction module is used to calculate the density distribution characteristics and geometric distance distribution characteristics of the dynamic object point set corresponding to the interpolated 3D box and the key 3D box after the optimization direction, respectively, and construct the composite loss function.

[0017] The optimal center point solution module is used to iteratively optimize the center point position of the interpolated 3D box after optimization direction using an iterative algorithm with dual-mode dynamic momentum term. Each iteration integrates the composite loss function. Optimization stops when the composite loss function value is lower than the threshold or the number of consecutive unimproved times reaches the maximum allowable value, thus obtaining the optimal center point position of the interpolated 3D box.

[0018] The center point optimization module is used to update the center point position of the interpolated 3D box after optimization based on the optimal center point position of the interpolated 3D box to complete the optimization of the interpolated 3D box label.

[0019] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising:

[0020] One or more processors;

[0021] Memory, used to store one or more programs;

[0022] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0023] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0024] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:

[0025] This application optimizes the inverse direction of the interpolated 3D bounding box by combining the calculation of the principal direction of the interpolated 3D bounding box relative to the corresponding dynamic object point set and the calculation of the density distribution characteristics and geometric distance distribution characteristics of the interpolated 3D bounding box relative to the corresponding dynamic object point set and the key 3D bounding box relative to the corresponding dynamic object point set. This scheme eliminates the optimization failure problem caused by the sparsity of the point set. The optimized interpolated 3D bounding box directly meets the annotation requirements, and the annotator does not need to make further fine-tuning to the optimized interpolated 3D bounding box, which significantly improves the annotation efficiency of 3D bounding boxes. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating an automatic optimization method for 3D bounding box labels in 4D dynamic object interactive annotation according to an exemplary embodiment.

[0027] Figure 2 This is a rendering of a key 3D frame of a dynamic sequence of objects, according to an exemplary embodiment.

[0028] Figure 3 This is a rendering of a 3D frame of a dynamic object sequence interpolation according to an exemplary embodiment.

[0029] Figure 4 This is an example illustration of the optimized sequence of dynamic objects in the global coordinate system.

[0030] Figure 5 This is an example illustration of the optimized interpolated 3D frame.

[0031] Figure 6This is a comparison diagram of the existing solution and the interpolation 3D frame optimization results of this application, according to an exemplary embodiment.

[0032] Figure 7 This is a block diagram illustrating an automatic optimization device for 3D bounding box labels in 4D dynamic object interactive annotation according to an exemplary embodiment.

[0033] Figure 8 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0034] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0035] Figure 1 This is a flowchart illustrating an automatic optimization method for 3D bounding box labels in 4D dynamic object interactive annotation, according to an exemplary embodiment. (Reference) Figure 1 The automatic optimization method for 3D bounding box labels in 4D dynamic object interactive annotation provided by this invention may include:

[0036] S1: Obtain all key 3D bounding boxes, all interpolated 3D bounding boxes, the set of dynamic object points corresponding to the key 3D bounding boxes, and the set of dynamic object points corresponding to the interpolated 3D bounding boxes in the 4D dynamic object sequence in the global coordinate system; this step includes the following sub-steps:

[0037] Specifically, 4D dynamic objects refer to objects whose position can change constantly in the global coordinate system, such as pedestrians and vehicles in road scenes, and pets in indoor scenes.

[0038] The data is obtained through the following sub-steps:

[0039] A1: For a time series of point cloud data, each frame of the point cloud is transformed to global coordinates through its corresponding pose. The pose consists of a rotation matrix R and a translation vector t.

[0040] R = ;

[0041]

[0042] The local point cloud is transformed by rotation matrix R and translation vector t to obtain the global point cloud:

[0043]

[0044] in The coordinates of the point in the global coordinate system. These are the coordinates of a point in the local coordinate system.

[0045] A2: For a certain dynamic object, the start, end and inflection points of the trajectory in the global point cloud are selected as keyframes, and key 3D bounding boxes are manually annotated.

[0046] Specifically, in the global point cloud, for a 4D dynamic object sequence, the trajectory can be distinguished into the following two motion modes:

[0047] Mode 1: Linear motion mode;

[0048] Specifically, if the trajectory of a moving object is linear, the annotator only needs to select the first and last frames as keyframes.

[0049] Mode 2: Nonlinear motion mode;

[0050] Specifically, if the trajectory of a moving object is non-linear, in addition to selecting the first and last frames as keyframes, the annotator also needs to select the midpoint of the arc of the trajectory as a keyframe.

[0051] like Figure 2 As shown, frames 1, 30, 60, and 90 in this dynamic object sequence are selected keyframes, in which key 3D bounding boxes are manually annotated.

[0052] A3: Based on manually annotated key 3D bounding boxes, interpolate other frames in the dynamic object sequence to obtain all interpolated 3D bounding boxes;

[0053] Specifically, after selecting a keyframe, the annotator annotates a 3D bounding box based on the shape of the dynamic object sequence in that frame, ensuring that the position, size, and orientation of the 3D bounding box fit the dynamic object, thus obtaining the key 3D bounding box; based on the key 3D bounding box, interpolation is performed on the remaining frames to obtain interpolated 3D bounding boxes.

[0054] The key 3D frame and the interpolated 3D frame consist of three parts: position, orientation, and size;

[0055] (1) Position interpolation:

[0056] Specifically, the lack of data points causes the entire data sequence to be discontinuous, leading to inaccurate fitting of the target trajectory. The interpolation of the 3D bounding box position can be divided into the following sub-steps:

[0057] B1: Using linear interpolation to compensate for missing normal frames, for trajectory T, give its value in... Time and The trajectory bounding box at time t, after performing linear interpolation of the trajectory at time t, the interpolated bounding box of the trajectory T can be calculated using Equation 1:

[0058]

[0059] in The interpolation frame at time t. express Key 3D frame of the moment express The key 3D frame of the moment, and .

[0060] In a single motion mode, linear interpolation is simple and efficient, and can quickly locate the possible position of the 3D box within the allowable error range, ensuring good accuracy while maintaining operational efficiency.

[0061] B2: After obtaining the linear interpolation results, Gaussian process regression is used to model the nonlinear motion of the trajectory. The Gaussian smooth interpolation model for trajectory T is shown below:

[0062]

[0063] in, Indicates the frame number. Indicates the first The position coordinate variable at the frame (i.e.) ), This represents all 3D bounding boxes in the sequence, and This represents Gaussian noise. Given a length of... linear interpolation trajectory By fitting function To solve the problem of nonlinear motion modeling, we assume it follows a Gaussian distribution:

[0064]

[0065] in, It is the radial basis kernel function. Based on the properties of Gaussian processes, a new prediction set is given. its smoothed position It can be predicted using the following formula:

[0066]

[0067] in, Based on The covariance function. Furthermore, the hyperparameters... To control the smoothness of the trajectory, we simply set it to be the same as the interpolation length. The corresponding function is shown in the following equation:

[0068]

[0069] Among them, based on the experimental results, Set it to 10.

[0070] By correcting the data missing problem in Gaussian process regression, Gaussian process regression can achieve more accurate trajectory fitting for linear interpolation in the case of nonlinear trajectories.

[0071] Although linear interpolation is simple and efficient, it lacks motion information, which limits the 3D box from reverting to its accurate position. Gaussian processes, on the other hand, can estimate the uncertainty of the results of linear interpolation to adapt to complex data distributions.

[0072] (2) Directional interpolation:

[0073] Specifically, even if the moving object is undergoing non-linear motion, constant velocity interpolation is still required. The orientation interpolation of the 3D bounding box can be divided into the following sub-steps:

[0074] C1: Quaternions that determine the orientation of the two key 3D bounding boxes;

[0075]

[0076] Where q represents a quaternion of a 3D bounding box direction, and the subscript represents the index of the key 3D bounding box.

[0077] C2: Perform a dot product of the quaternions in the key 3D bounding box direction;

[0078]

[0079] C3: Use arccosine to calculate the actual included angle;

[0080]

[0081] C4: Calculate the direction interpolation result;

[0082]

[0083] Where t is an interpolation parameter between 0 and 1, representing the interpolation position of the 3D box.

[0084] These steps ensure that the direction changes at a constant rate. Since the direction changes of dynamic objects in the real world are smooth, this direction interpolation scheme can satisfy the direction changes of dynamic objects in the real world.

[0085] (3) Size inheritance:

[0086] Specifically, the application scenarios of this invention are mainly, but not limited to, road dynamic object annotation. The size of the dynamic object will not change, so the size of the interpolated 3D box is inherited from the key 3D box.

[0087] By applying different interpolation schemes based on the characteristics of different 3D bounding boxes, the accuracy of interpolated 3D bounding boxes is effectively improved, providing more robust data information for subsequent data processing.

[0088] like Figure 3 As shown, all frames except frames 1, 30, 60 and 90 are interpolated frames, and the 3D boxes in them are interpolated 3D boxes obtained in the manner described above. The figure shows the interpolated 3D box of frame 34.

[0089] A4: Increase the size of the key 3D bounding box, and the set of points contained therein shall be used as the potential dynamic object point set corresponding to the key 3D bounding box; increase the size of the interpolation 3D bounding box, and the set of points contained therein shall be used as the potential dynamic object point set corresponding to the interpolation 3D bounding box.

[0090] Specifically, the key 3D bounding box and the interpolated 3D bounding box are located in the global coordinate system and are transformed to the local coordinate system through corresponding pose transformations, as shown in the following formula:

[0091]

[0092] in Let R be the inverse of the rotation matrix R.

[0093] Since the interpolated 3D frame may not fully contain the dynamic object point set, the size of the interpolated 3D frame is doubled after transformation to local coordinates to obtain the potential dynamic object point set.

[0094] In the global coordinate system, there is dynamic object information for each frame of point cloud. The trajectories of different dynamic objects will overlap in the global coordinate system, making it impossible to distinguish the dynamic object point set. Therefore, the key 3D bounding box and the interpolated 3D bounding box are transformed into the local coordinate system through the corresponding pose, which eliminates the influence of different dynamic object trajectories. After transforming to the local coordinate system, the size of the interpolated 3D bounding box can be enlarged to ensure that the dynamic object point set is within the 3D bounding box as much as possible.

[0095] A5: Cluster the potential dynamic object point set corresponding to the key 3D frame and the potential dynamic object point set corresponding to the interpolation 3D frame respectively to obtain a multi-cluster point set. The cluster point set with the most points is taken as the dynamic object point set corresponding to the key 3D frame and the dynamic object point set corresponding to the interpolation 3D frame.

[0096] Specifically, the neighborhood radius r and the minimum number of points minPts are set as the initial parameters for clustering, and clustering is performed according to the following steps:

[0097] D1: Identify core points: For each point in the dataset, calculate the number of its neighboring points within the range r. If the number of neighboring points of a point is greater than or equal to minPts, then the point is marked as a core point.

[0098] D2: Expanding Clusters: Select an unvisited core point, create a new cluster and mark it as visited. Add all points in the r-neighborhood of that core point to that cluster. For each point in the cluster, if it is an unvisited core point, also add the points in its r-neighborhood to the cluster. Repeat this process until no more points can be added to the current cluster.

[0099] D3: Handling Boundary Points and Noise Points: If a point is neither a core point nor within the r-neighborhood of a core point, it is labeled as a noise point. If a point is within the r-neighborhood of a core point but is not itself a core point, it is called a boundary point, and boundary points belong to the corresponding clusters.

[0100] D4: Clustering Result Output: Repeat the above steps until all points have been processed, and finally output multiple clusters in the potential dynamic object point set. Select the cluster with the most points as the point set of the dynamic object to obtain the dynamic object point set corresponding to the key 3D box and the dynamic object point set corresponding to the interpolation 3D box.

[0101] S2: Calculate the principal directions of the dynamic object point sets corresponding to the interpolation 3D frame and the key 3D frame respectively to obtain the interpolation principal direction and the key principal direction;

[0102] Specifically, the principal direction of the dynamic object point set corresponding to the interpolated 3D bounding box and the key 3D bounding box is calculated in the same way, and can be calculated through the following sub-steps:

[0103] E1: For dynamic point sets Design a two-factor fusion weight function:

[0104]

[0105] in: For spatial decay term, For height distribution, The initial coordinates of the frame center are: Point The vertical height value, The spatial attenuation coefficient, , Let be the mean and standard deviation of the height of the point set.

[0106] The weighted design employs a two-factor coupling mechanism to achieve collaborative perception of spatial distribution and structural features. The spatial attenuation term strengthens the influence of the central region points, conforming to the saliency of the central features of dynamic object point sets, while the height distribution term highlights the structural features of typical height layers and suppresses interference from anomalous outliers.

[0107] E2: Calculation of effective centers of dynamic point sets based on fusion weights:

[0108]

[0109] This calculation formula uses an adaptive weighting mechanism to shift the effective center towards the high-confidence region, significantly improving robustness to partial occlusion.

[0110] E3: Construct the weighted covariance matrix and perform eigenvalue decomposition. The weighted covariance matrix is ​​defined as follows:

[0111]

[0112] right Perform eigenvalue decomposition:

[0113]

[0114] in A diagonal matrix of eigenvalues. This is the eigenvector matrix. The eigenvector corresponding to the largest eigenvalue is selected as the principal direction.

[0115]

[0116] The main direction of the dynamic point set corresponding to the interpolation 3D bounding box is used as the interpolation main direction, and the main direction of the dynamic point set corresponding to the key 3D bounding box is used as the key main direction.

[0117] This method can effectively suppress the influence of edge noise in scenarios with uneven point cloud distribution, and can also maintain the stability of orientation estimation when dynamic objects are partially occluded, resulting in the loss of some point sets.

[0118] S3: Calculate the difference between the interpolation principal direction and the key principal direction, and adjust the direction of the interpolation 3D box according to the difference to obtain the interpolation 3D box with optimized direction;

[0119] Specifically, for an interpolation principal direction in a dynamic object sequence, the key principal direction corresponding to the key 3D box that is closer to its corresponding interpolation 3D box is selected and the difference is calculated. This difference determines the difference in the direction of the interpolation 3D box relative to the corresponding dynamic object point set and the direction angle of the manually labeled key 3D box relative to the corresponding dynamic object point set. The difference is added to the original angle of the interpolation 3D box to obtain the interpolation 3D box with optimized direction.

[0120] S4: Calculate the density distribution characteristics and geometric distance distribution characteristics of the dynamic object point sets corresponding to the interpolated 3D bounding box and the key 3D bounding box after the optimization direction, and construct a composite loss function;

[0121] Specifically, the methods for extracting the density distribution features and geometric distance distribution features of the corresponding dynamic object point sets from the optimized interpolated 3D bounding boxes and key 3D bounding boxes are consistent, where:

[0122] Density distribution feature extraction:

[0123] Let the spatial domain of the interpolated 3D bounding box or the key 3D bounding box be... Divide it evenly along the x, y, z axes into A cube grid is used to create a set of grid indices. For each grid cell Its density eigenvalue is defined as:

[0124]

[0125] in For a dynamic set of object points, For smoothing coefficients, This is the indicator function. The standardized density distribution vector is obtained through L2 normalization:

[0126]

[0127] Introduced during density distribution feature extraction The smoothing term enhances robustness to sparse point clouds.

[0128] Geometric distance distribution feature extraction:

[0129] For each point Calculate the signed distances from it to the six faces of the 3D frame. Take the minimum absolute distance as the geometric feature:

[0130]

[0131] Divide the distance space into M intervals. Construct a distance distribution histogram:

[0132]

[0133] After bilinear interpolation smoothing, the geometric distance feature vector is obtained:

[0134]

[0135] In the process of extracting geometric distance distribution features, the fine-grained geometric constraints of the object surface distribution are captured by differentiable histogram statistics, which has higher optimization sensitivity compared with traditional IOU metric.

[0136] Construction of the composite loss function:

[0137] Define the density distribution difference term:

[0138]

[0139] Geometric distance distribution difference term:

[0140]

[0141] Introduce a rotational smoothing regularization term:

[0142]

[0143] The final composite loss function is:

[0144]

[0145] in For weight parameters, , These represent the rotation and translation changes between adjacent keyframes, respectively.

[0146] This composite function uses KL divergence to measure the difference in geometric distribution, enhancing its sensitivity to the topological structure of the object surface. It also introduces a Lie group regularization term to ensure the smoothness of the pose transformation and avoid abrupt changes during the optimization process.

[0147] S5: The iterative algorithm of dual-mode dynamic momentum term is used to iteratively optimize the center point position of the interpolated 3D box after the optimization direction. The composite loss function is fused in each iteration. The optimization stops when the value of the composite loss function is lower than the threshold or the number of consecutive unimproved times reaches the maximum allowable value, and the optimal center point position of the interpolated 3D box is obtained.

[0148] Specifically, the modeling of the dual-mode dynamic momentum term can be defined by defining the update amount at step t in the iterative optimization process as:

[0149]

[0150] in Update amount for the center point of the 3D bounding box. For dynamic learning rate, This represents the gradient of the composite loss function. A two-mode dynamic momentum term is introduced. Its expression is:

[0151]

[0152] This includes the baseline momentum term. and environmental momentum Through dynamic weights Dynamic equilibrium reference momentum term and environmental momentum term.

[0153] Reference momentum term

[0154] Reference momentum term The preset attenuation coefficient follows an exponential decay law:

[0155]

[0156] in This is the decay rate hyperparameter, used to control the strength of historical gradient information retention.

[0157] Environmental momentum term

[0158] Environmental momentum term Dynamic calculation based on the distribution characteristics of point clouds in the xy plane:

[0159] F1: Calculate the variance of the dynamic point set P on the x and y axes.

[0160]

[0161] in , The mean of the point cloud coordinates.

[0162] F2: Generating the environmental momentum term through nonlinear mapping.

[0163]

[0164] in This is the sensitivity coefficient, when the point cloud distribution is scattered. When it is large, Approaching 0 suppresses excessive updates; conversely, approaching 2 accelerates convergence. This achieves scene-adaptive optimization step size control.

[0165] Dynamic weights

[0166] Weight The number of iterations is jointly determined by the point cloud distribution characteristics.

[0167]

[0168] in , These are learnable parameters that change with the number of iterations. The algorithm gradually strengthens the dominant role of the baseline momentum to avoid excessive oscillations in the later stages of iteration.

[0169] when The iteration stops when the value is less than 0.01 for 5 consecutive times or when the maximum number of iterations reaches 50.

[0170] S6: Update the center point position of the interpolated 3D bounding box after optimization based on the optimal center point position to complete the label optimization of the interpolated 3D bounding box.

[0171] Specifically, each iteration uses The center point of the interpolated 3D bounding box after the optimization direction is modified is obtained. When the condition is met and the iteration stops, the final optimized 3D bounding box is obtained.

[0172] The final optimized interpolation 3D box is as follows: Figure 4 and Figure 5 As shown, where Figure 4 The figure shows a sequence of dynamic objects in the global coordinate system. As can be seen from the figure, the scheme described in this application achieves a smoothly optimized 3D bounding box without manual modification. Figure 5 The interpolated 3D bounding box optimized for frame 34, and Figure 3 The comparison shows that the center point position and rotation angle of the optimized interpolated 3D frame have been greatly improved, and it fits the point cloud inside the frame perfectly.

[0173] Figure 6 This paper compares the optimization results of the interpolated 3D bounding box with the existing solution (application number: CN202411062156.2). The first column represents the center point error between the optimized 3D bounding box and the reference ground truth bounding box, and the second column represents the rotation error between the optimized 3D bounding box and the reference ground truth bounding box. The results show that the center point error between the optimized 3D bounding box and the reference ground truth bounding box of the solution described in this application is within 0.1 meters on average, and the rotation error is within 0.5 degrees on average. It meets the annotation requirements of autonomous driving scenarios without additional adjustments.

[0174] Corresponding to the aforementioned embodiments of the automatic optimization method for 3D bounding box labels in 4D dynamic object interactive annotation, this application also provides embodiments of an automatic optimization device for 3D bounding box labels in 4D dynamic object interactive annotation.

[0175] Figure 7 This is a diagram illustrating an automatic optimization device for 3D bounding box labels in 4D dynamic object interactive annotation, according to an exemplary embodiment. (Reference) Figure 7 The device includes:

[0176] Data acquisition module 1 is used to acquire all key 3D boxes, all interpolated 3D boxes, the set of dynamic object points corresponding to the key 3D boxes, and the set of dynamic object points corresponding to the interpolated 3D boxes in the 4D dynamic object sequence under the global coordinate system.

[0177] The main direction calculation module 2 is used to calculate the main direction of the dynamic object point set corresponding to the interpolated 3D frame and the key 3D frame respectively, so as to obtain the interpolated main direction and the key main direction.

[0178] The orientation optimization module 3 is used to calculate the difference between the interpolation main orientation and the key main orientation, and adjust the orientation of the interpolation 3D box according to the difference to obtain the interpolation 3D box with optimized orientation.

[0179] The composite loss function construction module 4 is used to calculate the density distribution characteristics and geometric distance distribution characteristics of the dynamic object point set corresponding to the interpolated 3D box and the key 3D box after the optimization direction, respectively, and construct the composite loss function.

[0180] The optimal center point solution module 5 is used to iteratively optimize the center point position of the interpolated 3D box after optimization direction using an iterative algorithm of dual-mode dynamic momentum term. Each iteration integrates the composite loss function. When the composite loss function value is lower than the threshold or the number of consecutive unimproved times reaches the maximum allowable value, the optimization stops and the optimal center point position of the interpolated 3D box is obtained.

[0181] The center point optimization module 6 is used to update the center point position of the interpolated 3D box after optimization based on the optimal center point position of the interpolated 3D box to complete the optimization of the interpolated 3D box label.

[0182] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0183] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0184] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the automatic optimization method for 3D bounding box labels in 4D dynamic object interactive annotation as described above. Figure 8 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is the automatic optimization device for 3D bounding box labels in 4D dynamic object interactive annotation provided by an embodiment of the present invention. Except for... Figure 8 In addition to the processor and memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0185] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the automatic optimization method for 3D bounding box labels in 4D dynamic object interactive annotation as described above. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of a wind turbine, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any device with data processing capabilities and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the device with data processing capabilities, and can also be used to temporarily store data that has been output or will be output.

[0186] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0187] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for automatically optimizing 3D bounding box labels in a 4D dynamic object sequence, characterized in that, include: Obtain all key 3D bounding boxes, all interpolated 3D bounding boxes, the set of dynamic object points corresponding to the key 3D bounding boxes, and the set of dynamic object points corresponding to the interpolated 3D bounding boxes in the 4D dynamic object sequence under the global coordinate system; Calculate the principal directions of the dynamic object point sets corresponding to the interpolated 3D bounding box and the key 3D bounding box respectively to obtain the interpolation principal direction and the key principal direction; Calculate the difference between the interpolation principal direction and the key principal direction, and adjust the orientation of the interpolated 3D box according to the difference to obtain the interpolated 3D box with optimized orientation; Calculate the density distribution characteristics and geometric distance distribution characteristics of the dynamic object point sets corresponding to the interpolated 3D bounding box and the key 3D bounding box after the optimization direction, and construct a composite loss function; An iterative algorithm using a dual-mode dynamic momentum term is employed to iteratively optimize the center point position of the interpolated 3D bounding box after optimization. Each iteration incorporates the composite loss function. Optimization stops when the composite loss function value falls below a threshold or the number of consecutive unimproved iterations reaches the maximum allowable value, thus obtaining the optimal center point position of the interpolated 3D bounding box. Dynamic object information exists for each frame of the point cloud in the global coordinate system. The dual-mode dynamic momentum term includes a reference momentum term and an environmental momentum term. The reference momentum term is the basic attenuation coefficient. The environmental momentum term is dynamically calculated based on the distribution characteristics of the point cloud in the x and y directions; finally, dynamic weights are introduced for both, and the summation yields the dual-mode dynamic momentum term. The interpolation 3D box label optimization is completed by updating the center point position of the interpolated 3D box after optimizing the direction based on the optimal center point position.

2. The method according to claim 1, characterized in that, Obtain all key 3D bounding boxes, all interpolated 3D bounding boxes, the set of dynamic object points corresponding to the key 3D bounding boxes, and the set of dynamic object points corresponding to the interpolated 3D bounding boxes in the 4D dynamic object sequence in the global coordinate system, including: The point cloud in a time series is transformed to the global coordinate system through pose transformation to obtain the global point cloud; For a given dynamic object, the start, end, and inflection points of its trajectory in the global point cloud are selected as keyframes, and key 3D bounding boxes are manually annotated. Based on manually annotated key 3D bounding boxes, interpolation is performed on other frames in the dynamic object sequence to obtain all interpolated 3D bounding boxes; Enlarge the size of the key 3D bounding box, and the set of points contained therein is used as the potential dynamic object point set corresponding to the key 3D bounding box; enlarge the size of the interpolation 3D bounding box, and the set of points contained therein is used as the potential dynamic object point set corresponding to the interpolation 3D bounding box. Clustering is performed on the potential dynamic object point set corresponding to the key 3D frame and the potential dynamic object point set corresponding to the interpolation 3D frame to obtain multiple cluster point sets. The cluster point set with the most points is used as the dynamic object point set corresponding to the key 3D frame and the dynamic object point set corresponding to the interpolation 3D frame.

3. The method according to claim 1, characterized in that, The principal directions of the dynamic object point sets corresponding to the interpolated 3D bounding box and the key 3D bounding box are calculated respectively to obtain the interpolation principal direction and the key principal direction, including: Construct an exponential decay weight of the distance from the center of the dynamic object point set to the center of the corresponding interpolated 3D box and the key 3D box, and a proportional weight of the height distribution of the point set, to form a two-factor fusion weight matrix; The effective center positions of the point set are calculated based on the two-factor fusion weight matrix to obtain the weighted center points of the dynamic object point set corresponding to the interpolated 3D frame and the weighted center points of the dynamic object point set corresponding to the key 3D frame. Based on the weighted center points of the dynamic object point set corresponding to the interpolated 3D frame and the weighted center points of the dynamic object point set corresponding to the key 3D frame, a weighted covariance matrix is ​​constructed and eigenvalue decomposition is performed. For each eigenvalue decomposition result, the eigenvector with the largest eigenvalue is taken as the principal direction, where the principal direction of the dynamic object point set corresponding to the interpolated 3D frame is taken as the interpolation principal direction, and the principal direction of the dynamic object point set corresponding to the key 3D frame is taken as the key principal direction.

4. The method according to claim 1, characterized in that, Calculate the density distribution characteristics and geometric distance distribution characteristics of the dynamic object point sets corresponding to the interpolated 3D bounding box and the key 3D bounding box after the optimization direction, respectively, and construct a composite loss function, including: Within the interpolated 3D frame and the key 3D frame after the optimization direction, the space is divided into grids, and the number of points in each grid is counted as the density distribution within the 3D frame. After normalizing the density distribution, a vectorization operation is performed to obtain the density distribution characteristics of the dynamic object point set corresponding to the interpolated 3D frame and the key 3D frame. Calculate the distance from each point in the dynamic object point set to the six faces of the corresponding interpolation 3D box or key 3D box, take the minimum distance as the feature value of the point, assign the point to a predefined distance interval based on the minimum distance from the point to the 3D box face, count the number of points in each interval, form a distance distribution histogram, normalize the histogram, and obtain the final geometric distance distribution features. The difference between the density distribution feature and the geometric distance distribution feature is calculated, a regularization term is added, and the final composite loss function is the weighted sum of these three terms.

5. An automatic optimization device for 3D bounding box labels in a 4D dynamic object sequence, characterized in that, include: The data acquisition module is used to acquire all key 3D boxes, all interpolated 3D boxes, the set of dynamic object points corresponding to the key 3D boxes, and the set of dynamic object points corresponding to the interpolated 3D boxes in the 4D dynamic object sequence under the global coordinate system. The main direction calculation module is used to calculate the main direction of the dynamic object point set corresponding to the interpolated 3D frame and the key 3D frame respectively, so as to obtain the interpolated main direction and the key main direction. The orientation optimization module is used to calculate the difference between the interpolation principal orientation and the key principal orientation, and adjust the orientation of the interpolated 3D box according to the difference to obtain the interpolated 3D box with optimized orientation. The composite loss function construction module is used to calculate the density distribution characteristics and geometric distance distribution characteristics of the dynamic object point set corresponding to the interpolated 3D box and the key 3D box after the optimization direction, respectively, and construct the composite loss function. The optimal center point solution module is used to iteratively optimize the center point position of the interpolated 3D bounding box after optimization direction using an iterative algorithm with a dual-mode dynamic momentum term. Each iteration incorporates the composite loss function. Optimization stops when the composite loss function value is below a threshold or the number of consecutive unimproved iterations reaches the maximum allowable value, thus obtaining the optimal center point position of the interpolated 3D bounding box. Dynamic object information exists for each frame of the point cloud in the global coordinate system. The dual-mode dynamic momentum term includes a reference momentum term and an environmental momentum term. The reference momentum term is the basic attenuation coefficient. The environmental momentum term is dynamically calculated based on the distribution characteristics of the point cloud in the x and y directions; finally, dynamic weights are introduced for both, and the summation yields the dual-mode dynamic momentum term. The center point optimization module is used to update the center point position of the interpolated 3D box after optimization based on the optimal center point position of the interpolated 3D box to complete the optimization of the interpolated 3D box label.

6. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-4.

7. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-4.

Images