Automatic obstacle annotation method, apparatus, electronic device, storage medium, and program

Abstract

The present invention provides an automatic obstacle annotation method, apparatus, electronic device, storage medium, and program. [Solution] The method includes optimizing target parameters in a projection relationship based on reprojection errors. The projection relationship is used to project a target obstacle from a reference frame to a frame to be optimized, and the projection relationship satisfies the constraint that the position of the target obstacle in different frames is unified in an obstacle coordinate system constructed based on the target obstacle. The method also includes determining the target pose of the target obstacle based on the optimized target parameters.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and particularly to technical fields such as autonomous driving, neural networks, and deep learning.

Background Art

[0002] With the progress of artificial intelligence technology, autonomous vehicles are becoming an important part of future transportation. The core of autonomous driving technology is to achieve the autonomous driving of vehicles, which requires not only that the vehicle can understand the surrounding environment, but also the ability to process various driving situations.

[0003] In this process, environmental perception technology plays a crucial role. In particular, the accurate detection and identification of obstacles have a decisive impact on ensuring the safety of autonomous vehicles and improving navigation efficiency. Here, obstacle detection depends on a large amount of annotation data.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides an automatic annotation method, apparatus, electronic device, storage medium, and program for obstacles.

Means for Solving the Problems

[0005] According to one aspect of the present disclosure, an automatic annotation method for obstacles is provided, and the method includes: Optimizing target parameters in a projection relationship based on a reprojection error, where the projection relationship is used to project a target obstacle from a reference frame to an optimization target frame, and the projection relationship satisfies a constraint condition that positions of the target obstacle in different frames in an obstacle coordinate system constructed based on the target obstacle are unified; and Determining a target pose of the target obstacle based on the optimized target parameters.

[0006] According to another aspect of this disclosure, an automatic obstacle annotation device is provided, the device is: An optimization module for optimizing target parameters in a projection relationship based on reprojection errors, wherein the projection relationship is used to project a target obstacle from a reference frame to a frame to be optimized, and the projection relationship satisfies the constraint that the position of the target obstacle in different frames is unified in an obstacle coordinate system constructed based on the target obstacle. It comprises a decision module for determining the target pose of a target obstacle based on optimized target parameters.

[0007] According to another aspect of this disclosure, an electronic device is provided, which is At least one processor, The system comprises at least one processor and memory that is communicated with, The memory stores instructions that are executable by the at least one processor, and when such instructions are executed by the at least one processor, the at least one processor performs any one of the methods in the embodiments of the present disclosure.

[0008] Another aspect of the present disclosure provides a non-temporary computer-readable storage medium storing computer instructions for causing a computer to perform any one of the methods of the embodiments of the present disclosure.

[0009] According to another aspect of the present disclosure, a program, when executed by a processor, provides a program for causing any of the methods in the embodiments of the present disclosure to be performed.

[0010] According to another aspect of this disclosure, an autonomous vehicle including the aforementioned electronic device is provided.

[0011] It should be understood that the content contained herein is not intended to describe any key points or important features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Further details of other features of this disclosure are provided in the specification below. [Brief explanation of the drawing]

[0012] [Figure 1] This is a flowchart of an automatic obstacle annotation method according to one embodiment of the present disclosure. [Figure 2] This is a flowchart for determining the initial pixel point cloud in the reference frame of a target obstacle according to one embodiment of the present disclosure. [Figure 3] This is a flowchart for determining the reprojection error according to one embodiment of the present disclosure. [Figure 4] This is a flowchart of an automatic obstacle annotation method according to one embodiment of the present disclosure. [Figure 5] This is a block diagram of a BA service optimization process according to one embodiment of the present disclosure. [Figure 6] This is a schematic diagram showing the configuration of an automatic obstacle annotation device according to one embodiment of the present disclosure. [Figure 7] This is a block diagram of an electronic device for realizing an automatic obstacle annotation method according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0013] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the accompanying drawings. These drawings include various details of the embodiments of the present disclosure to aid understanding, and should be considered as illustrative only. Accordingly, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, well-known descriptions of functions and structures are omitted in the following description for clarity and brevity.

[0014] The terms “First” and “Second” in this disclosure are used to distinguish similar subjects and are not used to describe a particular order or priority. Furthermore, the terms “includes” and “having,” and any variations thereof, are intended to be non-exclusive inclusions, meaning, for example, a set of steps or units. A method, system, product, or apparatus is not limited to the expressly described steps or units and may include other steps or units not expressly described or specific to these processes, methods, products, or apparatus.

[0015] In the field of autonomous driving, highly accurate environmental sensing capabilities are key to ensuring safety and efficient navigation. At the heart of environmental perception is obstacle detection, and accurate identification and positioning of obstacles are crucial for path planning and obstacle avoidance strategies. In this process, a large amount of annotation data is required for obstacle detection and identification.

[0016] In related technologies, a 3D detection model is provided that automatically annotates point clouds and BEV (Bird's Eye View) data to obtain a 3D position box of the target to be detected. The 3D position box provided by this 3D detection model includes the pose position of the detection target (e.g., center of gravity, orientation, supplementation, etc.) and also contains information such as category.

[0017] In particular, 3D detection models must rely on a large amount of annotation data to achieve target detection. This data includes two major classes: dynamic targets and static targets. The quality of the annotation method greatly affects the accuracy of target detection by the 3D detection model and its annotation capabilities. Especially in the field of autonomous driving, the quality of the annotation method affects the actual user experience of autonomous driving. Annotation capabilities greatly influence the autonomous driving system's ability to sense surrounding obstacles, and the annotation viewing distance determines the upper limit of the autonomous driving system.

[0018] However, there are several drawbacks in the method of automatically annotating obstacles in related technologies. First, the conventional radar scanning range is limited. For example, for long-distance moving targets, medium- and long-distance or low static targets, the scanned point cloud is very sparse. As a result, whether it is manual annotation or model-based automatic annotation methods, there are limitations in the annotation ability, and ultimately, the upper limit of the 3D detection model's ability is restricted. Second, some automatic annotation methods need to first complete the annotation on the point cloud and then match it to the corresponding image to realize the annotation for the image. However, due to problems such as calibration of internal and external parameters and synchronization of timestamps between the point cloud and the image, the consistency between the image and the point cloud is poor, which affects the accuracy of automatic annotation. Due to the limited annotation accuracy, the annotation ability of the 3D detection model is also restricted.

[0019] Therefore, the embodiments of the present disclosure provide a method for automatically annotating obstacles. This method can support the automatic detection of moving and static targets and improve the annotation viewing distance. As shown in FIG. 1, it mainly includes the following content.

[0020] In S101, based on the reprojection error, optimize the target parameters in the projection relationship. The projection relationship is used to project the target obstacle from the reference frame to the optimization target frame. The projection relationship satisfies the constraint condition that the positions of the target obstacle in different frames are unified in the obstacle coordinate system constructed based on the target obstacle.

[0021] Both the reference frame and the optimization target frame in the embodiments of the present disclosure are two-dimensional images, which may also be images collected by a camera sensor. In the field of autonomous driving, it can be a bird's-eye view obtained by fusing images collected by multiple surround-view cameras of the vehicle.

[0022] Here, the target obstacle may be a static target or a moving target.

[0023] Here, reprojection error is an index used in computer vision to measure the difference between the projected position of a 3D point in a 2D image and the actually observed feature point position. This is a core concept in beam alignment algorithms, and by minimizing reprojection error, the optimized 3D point can be made closer to the actual situation.

[0024] Embodiments of this disclosure construct an obstacle coordinate system so that the same target obstacle in different frame images has its position unified after being projected from the image coordinate system of the different frame images to the obstacle coordinate system. For example, in the field of autonomous driving, obstacle D is transformed from frame A to the obstacle coordinate system to obtain position coordinate P1, and obstacle D is transformed from frame B to the obstacle coordinate system to obtain position coordinate P2. Since the obstacle coordinate system is constructed with obstacle D itself as the center, P1 is equal to P2.

[0025] During implementation, a relationship is constructed between the image coordinate system, camera coordinate system, and obstacle coordinate system using the world coordinate system as a mediator, thereby modeling the projection relationship between the reference frame and the frame to be optimized. By optimizing this projection relationship, the constructed projection relationship is made to approximate the actual situation, ultimately yielding core parameters (i.e., target parameters) for determining the obstacle's pose.

[0026] In embodiments of this disclosure, target parameters in a projection relationship are optimized based on reprojection errors, so that the projection relationship constructed based on the optimized target parameters can accurately describe the projection relationship between a reference frame and a frame to be optimized.

[0027] In S102, the target pose of the target obstacle is determined based on the optimized target parameters.

[0028] Here, the target pose of the target obstacle may include the center position, angle, dimensions, etc., of the 3D position box of the target obstacle in 3D space.

[0029] In the embodiments of this disclosure, by optimizing the target parameters in the projection relationship based on the reprojection error, the projected result can be brought closer to the true situation, improving projection accuracy and thereby obtaining target parameters that match the actual situation. Based on the optimized target parameters, the target pose of the target obstacle is determined, and automatic annotation of the target obstacle's pose is realized. Throughout the entire process, target obstacle pose estimation can be completed using a 2D image acquired synchronously with the point cloud, eliminating the need for internal and external parameter calibration between the point cloud and the image, and ensuring consistency between the target pose results obtained between the point cloud and the image. Furthermore, this method can achieve target obstacle pose annotation with only a reference frame and an optimization target frame, and can be applied even to targets with very sparse point clouds, thereby improving the annotation viewing distance between static and dynamic targets.

[0030] In the embodiments of this disclosure, as described above, the reference frame and the frame to be optimized may be a bird's-eye view in autonomous driving. For example, environmental images around the vehicle can be collected using a 6-channel surround view camera or a 7-channel surround view camera and merged into a single bird's-eye view.

[0031] A bird's-eye view transforms the environmental information surrounding a vehicle from a conventional front-view or side-view image to a top-down perspective. Such a perspective is particularly useful for understanding the spatial arrangement around the vehicle, detecting obstacles, and planning driving routes.

[0032] In the field of autonomous driving, a reference frame refers to a single BEV image selected as a reference within a certain time period. An optimization target frame refers to a different BEV image that needs to be compared with the reference frame. The reference frame and optimization target frame are usually obtained from a sequence of consecutive processing target frames and represent environmental conditions that change over time.

[0033] In the embodiments of this disclosure, automatic annotation of a bird's-eye view is completed by using the bird's-eye view as a reference frame and an optimization target frame in order to meet the data requirements in autonomous driving technology.

[0034] Of course, it can be understood that even with point cloud images collected by radar sensors, the problem of sparse point clouds making target detection impossible can be overcome by improving the annotation capabilities of the point cloud data using bird's-eye views collected in sync with the point cloud. Furthermore, using the method according to the embodiments of this disclosure, it is also possible to perform pose detection on target obstacles that are not detected in the point cloud and interpolate the target pose in frames where target obstacles are not detected.

[0035] In the embodiments of this disclosure, the quality of the reference frame affects the efficiency of iterative optimization. During implementation, images that satisfy pre-set conditions for the significance of target features can be selected from the frame sequence to be processed, based on the target features of the target obstacle, and used as reference frames. Target features refer to features used to identify the target obstacle and that can describe the unique characteristics of the target obstacle. Here, the significance of the target feature can be used to indicate the degree to which the target feature can accurately describe the target obstacle.

[0036] Each frame image in the frame sequence to be processed is a two-dimensional image. Through a two-dimensional image detection model, the frame sequence to be processed can be detected, and two-dimensional position box information of the target obstacle can be obtained. This two-dimensional position box information includes information such as the two-dimensional position box (i.e., center point and dimensions), category, and detection confidence level of the target obstacle in the corresponding frame image.

[0037] Selecting images that satisfy pre-set conditions can be implemented by selecting the image with the highest detection confidence for the target obstacle. Alternatively, an image with image quality higher than a quality threshold and confidence higher than a confidence threshold can be selected to satisfy the conditions. Another method is to select the image with the largest dimensions of the 2D position box to satisfy the conditions. In implementation, it can be understood that the image should be one that can stably and accurately describe the characteristics of the target obstacle.

[0038] In the embodiments of this disclosure, by selecting images that satisfy pre-set conditions for the significance of target features based on the target features of the target obstacle and using them as reference frames, a good data basis can be provided for iterative optimization of projection relationships, thereby improving the speed and accuracy of iterative optimization.

[0039] For both static and dynamic targets, reference frames can be selected based on the significance of the target features. In implementation, since the dynamic target's pose may change, the frame closest to the frame to be optimized and containing the target obstacle can be preferentially selected as the reference frame. For this closest frame, the significance requirement for the target obstacle features can be appropriately lowered.

[0040] In embodiments of this disclosure, the automatic annotation of dynamic targets is completed by utilizing the strong relationship between the nearest frame and the frame to be optimized, in order to improve the accuracy of the automatic annotation of dynamic targets.

[0041] During implementation, if the target obstacle is a dynamic target, the frame image where the 3D detection model failed to detect the target obstacle can be acquired and used as the frame to be optimized. The 3D detection model is then used to perform target detection on the point cloud or 2D image to obtain the target pose of the target obstacle. Subsequently, an appropriate reference frame can be selected according to the frame to be optimized.

[0042] For example, when the target obstacle is a dynamic target, improving the annotation viewing distance can result in a relatively sparse point cloud, or less information containing the target obstacle in the image, leading to insufficient training data for existing 3D detection models. This limits the performance of the 3D detection model and ultimately results in some frames failing to detect the dynamic target. In this case, the embodiments of this disclosure determine the frame images in which the target obstacle detection failure occurred as the frame to be optimized, and based on the detection results of the 3D detection model, the automatic obstacle annotation method according to the embodiments of this disclosure is performed on the frames with detection failures. By continuing to process and optimize the frames in which the failure detection occurred, the annotation capability for dynamic targets at long viewing distances can be improved.

[0043] For a static goal, you can select a reference frame and then determine the frame to optimize, which can be implemented as follows:

[0044] In step A1, the optimization direction is obtained.

[0045] In step A2, based on the reference frame and the optimization direction, the frame to be optimized is selected from the frame sequence to be processed.

[0046] Here, the optimization direction is marked to indicate the direction in which optimization should proceed forward or backward from the reference frame in the frame sequence being processed, in order to facilitate the completion of automatic annotation.

[0047] During implementation, starting from a reference frame, the system can compare the significance of the target obstacle's features in frames prior to and after the reference frame, and select the direction with the better significance as the optimization direction. For example, if the significance of the target feature in a frame prior to the reference frame is better than its significance in a frame after the reference frame, then forward from the reference frame is selected as the optimization direction. Otherwise, backward from the reference frame is selected as the optimization direction.

[0048] Additionally, it is possible to select and optimize frames from a reference frame according to the default optimization direction, and if the reprojection error does not converge easily after multiple iterative optimizations, the opposite direction to the default optimization direction can be selected as the optimization direction.

[0049] After determining the optimization direction, the frame to be optimized is selected from the sequence of frames to be processed based on the reference frame and the optimization direction. For example, if the 2D image detection model detects a target obstacle for a total of m frames after the reference frame, one or more frames can be selected from these m frames to be optimized. Regardless of whether one frame or multiple frames are selected as the frame to be optimized, the method of optimizing the target parameters based on the reprojection error for each frame remains the same.

[0050] In embodiments of this disclosure, when the target obstacle is a static target, the optimization direction can be obtained and the frame to be optimized can be selected based on the reference frame, thereby focusing on processing frames of interest in a particular optimization direction, and thus completing the automatic annotation of the target obstacle.

[0051] In some embodiments, the frame sequence to be processed can be long, and if too many frames need to be processed at once, the computational burden increases. Therefore, to improve the overall efficiency of automatic annotation, the number of frames to be optimized each time can be limited by setting the sequence length.

[0052] For example, suppose a 2D image detection model detects that any frame from frame n to frame m (where m is greater than n, and both m and n are positive integers) in a 2D BEV image sequence contains a target obstacle. From these, a frame with relatively good feature significance is selected and used as a reference frame. If (mn) is large and all (mn) frames are images that require processing, a lot of computational resources will be consumed. Therefore, by setting the sequence length to allow for rational use of computational resources, the number of frames processed each time a target obstacle is detected can be kept within a reasonable range.

[0053] After determining the reference frame and the frame to be optimized, the true values ​​of the projection points in the reprojection error can be automatically determined to facilitate the optimization of the target parameters in the projection relationship. These are the actual pixel point clouds of the target obstacle in the frame to be optimized.

[0054] In embodiments of this disclosure, the true projection point values ​​required for the reprojection error of the frame to be optimized are determined based on a pixel-level trajectory tracking method.

[0055] For each pixel in the initial pixel point cluster of the target obstacle's reference frame, a co-tracker (tracking algorithm) can be used to track these pixel points. This allows us to obtain the trajectory points of these pixel points in other frame images and acquire trajectory information for each pixel point. These trajectory points can be used as the true projection points on the corresponding frame to be optimized. For example, if pixel point trajectory tracking yields the trajectory point cluster C1 in frame A, which is the frame to be optimized for the target obstacle, C1 can be used as the true projection points in frame A and then used to determine the reprojection error.

[0056] In the embodiments of this disclosure, pixel-level trajectory tracking technology can automatically determine the true projection point values ​​required for reprojection error, thereby improving the optimization efficiency and accuracy of the target parameters.

[0057] During implementation, there may be multiple pixel points on the same target obstacle, but to improve the optimization efficiency of the target parameters, each pixel point does not need to participate in the target parameter optimization process. Therefore, during implementation, it is sufficient to sample the pixel points of the target obstacle in the reference frame and obtain the initial pixel points of the target obstacle in the reference frame. This allows some pixel points to be omitted during the subsequent optimization process, thereby improving optimization efficiency. In the embodiments of this disclosure, considering the need to optimize using the pixel points of the target obstacle, the initial pixel point set in the reference frame of the target obstacle can be continuously determined using a 2D detection box acquired by a 2D image detection model. The implementation method is shown in Figure 2.

[0058] In S201, the 2D image detection model obtains a 2D position box of the target obstacle detected from the reference frame.

[0059] In S202, points are uniformly scattered based on a 2D position box to obtain an initial pixel point cloud in the reference frame of the target obstacle.

[0060] In the embodiments of this disclosure, the initial pixel point cloud can be determined by reusing the detection results of target obstacles by a two-dimensional image detection model, thereby improving optimization efficiency.

[0061] In some embodiments, pixel points within the 2D position box do not necessarily belong to the target obstacle; therefore, in order to further improve the optimization accuracy of the target parameters, step S202 can be implemented as follows in the embodiments of this disclosure.

[0062] In S2021, obtain the mask image of the target obstacle in the reference frame.

[0063] In S2022, points are uniformly scattered based on the 2D position box and mask image in the reference frame of the target obstacle to obtain an initial pixel point cloud.

[0064] Here, the target obstacle is segmented from the reference frame to obtain a mask image of the target obstacle.

[0065] During implementation, the Segment Anything Model (SAM) can be used to process the reference frame and obtain a mask image of the target obstacle. By processing the reference frame using the Segment Anything Model (SAM), a flexible, efficient, and accurate high-quality segmented mask can be generated, thereby improving the quality of the initial pixel points and ultimately increasing the annotation accuracy of the target obstacle.

[0066] When uniformly dispersing points based on a 2D position box and a mask image in the reference frame of the target obstacle, not all pixel points within the 2D position box are located on the target obstacle. Therefore, by further limiting the range of the scattered points using the mask image, it is ensured that each pixel point in the initial pixel point set belongs to the target obstacle. Here, the 2D position box can provide the approximate location and range of the initial pixel points of the target obstacle, thereby improving dispersal efficiency. The mask image can provide the fine boundary of the initial pixel points of the target obstacle. By using both together, the confidence that the resulting initial pixel point set belongs to the target obstacle is increased, thereby ensuring that all tracked pixel points belong to the target obstacle, improving the accuracy of the initial pixel points, and consequently improving the optimization accuracy of the target parameters and the annotation accuracy of the target obstacle.

[0067] In some embodiments, the embodiment for determining the reprojection error may include the following, as shown in Figure 3.

[0068] In S301, the initial pixel point cloud of the target obstacle in the reference frame is mapped to the frame to be optimized based on the projection relationship, and the projected point cloud of the target obstacle in the frame to be optimized is obtained.

[0069] In the embodiments of this disclosure, the target obstacle is projected onto the frame to be optimized and can be implemented as a step as shown in Figure 3.

[0070] In S3011, based on the pixel point depth parameter, a back projection operation is performed on the initial pixel point group in the reference frame of the target obstacle to obtain the first three-dimensional spatial position of the target obstacle in the camera coordinate system, and the pixel point depth parameter is used to represent the depth value of each initial pixel point in the reference frame.

[0071] Here, the process of the back projection operation is as shown in equation (1).

number

[0072] In equation (1), JPEG2026099787000003.jpg1126 H is the first 3D spatial position of the obtained target obstacle in the camera coordinate system, H is the uniformization operation, and K is an intrinsic parameter of the camera. JPEG2026099787000004.jpg1126 This is the initial pixel point cloud in the reference frame of the target obstacle, JPEG2026099787000005.jpg1126 This is the pixel point depth parameter.

[0073] Here, the pixel point depth parameter can be determined by iterative optimization, so we just need to provide it with an initial value.

[0074] During implementation, to improve the optimization efficiency of the target parameters, a first preset value for the pixel depth parameter can be obtained, and then the first preset value is randomly perturbed to obtain an initial value for the pixel depth parameter.

[0075] In some possible embodiments, a first preset value for the pixel depth parameter can be obtained based on prior knowledge, and then random perturbation can be achieved by adding random noise to the first preset value. To obtain the initial value of the pixel depth parameter, Gaussian noise can be added to the first preset value, or uniformly distributed noise within a certain range can be added to the first preset value.

[0076] In some other possible embodiments, the depth value of the target obstacle in any frame image detected from the 3D detection model may be used as the first preset value. By preferentially selecting the depth value in an image of a frame close to the reference frame, the depth parameter of the initial pixel point can be brought closer to the actual situation in the reference frame, thereby increasing the convergence speed.

[0077] In the embodiments of this disclosure, by randomly perturbing the initial value of the pixel depth parameter, a variety of initial values ​​can be generated, allowing for better adaptation to optimization target frames with different initial values ​​of the pixel depth parameter. This avoids the problem of potentially falling into a specific local mode when fixed initial values ​​are used. Furthermore, since randomly perturbed initial values ​​allow the optimization process to be performed with different initial states, the generalization ability of the automatic obstacle annotation method according to the embodiments of this disclosure can be enhanced.

[0078] In S3012, the first 3D spatial position is transformed into the obstacle coordinate system based on the pose transformation parameters from the camera coordinate system to the obstacle coordinate system, and the second 3D spatial position is obtained.

[0079] That is, as shown in equation (2), the first three-dimensional spatial position of the target obstacle in the camera coordinate system is transformed from the camera coordinate system of the reference frame to the obstacle coordinate system using the pose transformation parameter.

number

[0080] In equation (2), JPEG2026099787000007.jpg1126 This is the second three-dimensional spatial position in the obstacle coordinate system, JPEG2026099787000008.jpg1126 is the first three-dimensional spatial position of the target obstacle in the camera coordinate system, JPEG2026099787000009.jpg1126 This is a pose transformation parameter from the camera coordinate system to the obstacle coordinate system.

[0081] In the embodiments of this disclosure, in the case of a static target, the target obstacle is stationary, JPEG2026099787000010.jpg1126 Although it is a constant, in the case of a dynamic target, the target obstacle is moving, JPEG2026099787000011.jpg1126 Each frame to be optimized has a corresponding value.

[0082] If the target obstacle is a static target, it can be understood that the target parameters that need to be optimized in the projection relationship may include the pixel point depth parameter.

[0083] Since the pose transformation parameter is constant, the optimization process does not need to recalculate or adjust this parameter. By focusing only on the pixel point depth parameter in the optimization process, the complexity and time required for calculations can be reduced, and the optimization efficiency of static targets can be significantly improved.

[0084] Similarly, if the target obstacle is a dynamic target, the pose transformation parameters are not fixed because the target obstacle is moving. Therefore, the target parameters that need to be optimized in the projection relationship include the pixel point depth parameter and the pose transformation parameter. By optimizing the pixel point depth parameter and the pose transformation parameter for dynamic targets, it is possible to accurately obtain the necessary and important parameters for determining the pose of the target obstacle and improve the annotation viewing distance of dynamic targets.

[0085] Since the pose transformation parameters also need to be determined through iterative optimization for the dynamic goal, it is also necessary to provide a single initial value.

[0086] During implementation, the second preset value of the pose transformation parameter is obtained, and then the second preset value is randomly perturbed to obtain the initial value of the pose transformation parameter.

[0087] The second preset value is the initial setting for the pose transformation parameters and can be obtained through some prior knowledge, experience, or preliminary analysis under a specific task scenario. Then, the second preset value is randomly perturbed to obtain the initial value for the pose transformation parameters.

[0088] In some embodiments, the initial settings for the pose transformation parameters can be determined by equation (3).

number

[0089] In equation (3), JPEG2026099787000013.jpg1126 This is the initial setting for the pose transformation relationship from the obstacle coordinate system to the camera coordinate system at the reference frame time, and is the inverse matrix of the pose transformation parameters from the camera coordinate system to the obstacle coordinate system, JPEG2026099787000014.jpg1126 This is first used as the second preset value for the pose transformation parameters that need to be optimized. After optimization, the pose transformation parameters can be obtained by calculating its inverse matrix. JPEG2026099787000015.jpg727 This is the positional transformation relationship between the world coordinate system and the camera coordinate system at the reference frame time. JPEG2026099787000016.jpg813 The obstacle coordinate system at the reference frame time is the pose relative to the world coordinate system.

[0090] In the embodiments of this disclosure, the initial values ​​of the pose transformation parameters obtained by randomly perturbing a second preset value can be adapted to different frames to be optimized, avoiding the problem of potentially falling into a specific local mode when fixed initial values ​​are used. The randomly perturbed initial values ​​allow the optimization process to be performed in different initial states, thereby enhancing the generalization capability of the automatic obstacle annotation method according to the embodiments of this disclosure.

[0091] In S3013, the second 3D spatial position is projected onto the frame to be optimized to obtain a projected point cloud.

[0092] Based on the above, in the obstacle coordinate system constructed with the target obstacle, the position of the target obstacle in different frames is unified. Using the camera intrinsic parameter K, a projection operation is performed in any i-th frame (i.e., any frame to be optimized), and the projected point cloud of the target obstacle in the i-th frame (i.e., the frame to be optimized) is obtained. The specific implementation method can be achieved by equation (4).

number

[0093] In equation (4), JPEG2026099787000018.jpg1126 is the obtained projected point cloud, and K is an intrinsic parameter of the camera. JPEG2026099787000019.jpg1144 is the second three-dimensional spatial position of the target obstacle in the obstacle coordinate system, JPEG2026099787000020.jpg1032 This is the pose transformation parameter from the camera coordinate system to the obstacle coordinate system for the frame to be optimized.

[0094] In the embodiments of this disclosure, back projection and projection operations can be used to more accurately obtain the projected point cloud in the frame to be optimized for the reference frame.

[0095] In S302, the reprojection error is determined based on the projected point cloud and the true projected point values ​​of the frame to be optimized.

[0096] The method for obtaining the true values ​​of the projection points is as described above and will not be repeated here. The reprojection error is given by equation (5).

number

[0097] In equation (5), JPEG2026099787000022.jpg1126 This represents the reprojection error, JPEG2026099787000023.jpg1126 This is a loss function that combines the advantages of mean squared error (MSE) and absolute error (MAE), JPEG2026099787000024.jpg1126 This is the projected point cloud in the frame to be optimized for the target obstacle. JPEG2026099787000025.jpg1126 This is the true value of the projection point in the target frame for optimization of the target obstacle.

[0098] In the embodiments of this disclosure, the accuracy and reliability of the projection relationship constructed based on the current target parameters can be evaluated by comparing the difference between the projected point cloud and the true values ​​of the projected points. When the reprojection error is small, it indicates that the projection effect better reflects the actual situation, which helps to improve the optimization accuracy of the target parameters and thereby improve the accuracy of automatically annotating the pose of the target obstacle.

[0099] In the embodiments of this disclosure, the depth parameter of the pixel point JPEG2026099787000026.jpg1126 Considering the possibility that the loss may not converge easily, a depth regularization term can be added to ensure normal convergence of the loss.

[0100] Here, a depth regularization term is used to reduce the dispersion of depth at each point of the target obstacle. This solves the problem of inaccurate depth estimation at each point due to the large dispersion of depth at each point of the target obstacle. The depth regularization term is as shown in equation (6).

number

[0101] In equation (6), JPEG2026099787000028.jpg1126 This is the loss of the depth regularization term, JPEG2026099787000029.jpg1126 This is the depth parameter of the current pixel point ( JPEG2026099787000030.jpg1126 This is the depth value of the i-th pixel point among them. JPEG2026099787000031.jpg1130 This is the average of the current depth values ​​of all initial pixel points on the target obstacle.

[0102] For example, after the nth round of optimization is completed, the loss of the depth regularization term is determined using the depth value after the nth round of optimization, and the optimization process for the (n+1)th round target parameter is performed. Here, n is a positive integer.

[0103] Subsequently, the total projection loss can be determined based on the reprojection error and the depth regularization term of the projection points, and the target parameters in the projection relationship are optimized based on the total projection loss. The total projection loss is given by equation (7).

number

[0104] In equation (7), JPEG2026099787000033.jpg1126 This is the total projection loss, JPEG2026099787000034.jpg1126 This represents the reprojection error, JPEG2026099787000035.jpg1126 This is a weighting coefficient corresponding to the reprojection error, JPEG2026099787000036.jpg1126 This is the loss of the depth regularization term, JPEG2026099787000037.jpg1126 This is the weighting coefficient corresponding to the depth normalization term.

[0105] In the embodiments of this disclosure, the reprojection error can measure the positional difference between the projected pixel point and the actual point (true value), and the depth regularization term takes into account the rationality of the pixel point depth information. By combining the two to determine the total projection loss, the quality of the projection can be evaluated more comprehensively and the convergence speed can be increased.

[0106] In summary, to simplify implementation, the obstacle coordinate system for the target obstacle can be constructed based on the world coordinate system.

[0107] The world coordinate system is a global, fixed coordinate system used to describe the position and orientation of all objects in the entire environment. The obstacle coordinate system, on the other hand, is a local coordinate system centered on the target obstacle and is used to describe the pose changes of the target obstacle itself. The obstacle coordinate system is constructed based on the world coordinate system; it can be constructed by moving the origin of the world coordinate system to the position of the obstacle in the world coordinate system.

[0108] For static targets, the pose transformation parameters from the target obstacle's camera coordinate system to the obstacle's coordinate system for each frame of the optimization target are fixed and invariant. That is, the target obstacle is usually invariant in the world coordinate system.

[0109] On the other hand, dynamic targets generally change in the world coordinate system due to their movement. Therefore, a transition term can be superimposed on the initial values ​​of the pose transformation parameters. This transition term is used to represent the relative changes in the target obstacle between the reference frame and the frame being optimized, and optimizing this transition term enables the optimization of the pose transformation parameters.

[0110] In other words, the initial value of the pose transformation parameter establishes a relationship between the dynamic target and the world coordinate system, but this relationship changes as the dynamic target moves. Therefore, in the embodiments of this disclosure, such changes can be expressed and optimized in the transition term. When optimizing the pose transformation parameter, only the parameter in that transition term needs to be optimized. For the i-th frame, i.e., the frame to be optimized, the relationship between the obstacle coordinate system and the camera coordinate system is as shown in equation (8).

number

[0111] In equation (8), JPEG2026099787000039.jpg1126 This is the inverse matrix of the pose transformation parameters corresponding to the i-th frame, which is the frame to be optimized. JPEG2026099787000040.jpg1126 This is the pose transformation of the target obstacle from the world coordinate system to the camera coordinate system at the i-th frame time. JPEG2026099787000041.jpg1126 This is the pose transformation (i.e., transition term) of the obstacle coordinate system to the world coordinate system at the i-th frame time, and can be determined by the pose of the i-th frame. Finally, by combining this with the result of the reference frame time from equation (3), the inverse matrix superimposed with the position transformation parameter of the transition term can be initialized as shown in equation (9).

number

[0112] Here, JPEG2026099787000043.jpg1626 This is used to represent rotation and is calculated by equation (3). JPEG2026099787000044.jpg1126 It can be decided based on this, JPEG2026099787000045.jpg1126 This represents the velocity component of the target obstacle in the x-direction. JPEG2026099787000046.jpg1126 This represents the velocity component of the target obstacle in the y-direction, JPEG2026099787000047.jpg1126 represents the velocity component of the target obstacle in the z direction, and (0 0 0 1) represents the homogenization operation. JPEG2026099787000048.jpg1126 This represents the time difference between the frame being optimized and the reference frame.

[0113] It can be understood that after optimizing the corresponding pose transformation relation in equation (9), the optimized position transformation parameters can be obtained by solving its inverse matrix.

[0114] In the embodiments of this disclosure, the obstacle coordinate system is constructed based on the world coordinate system, making it easier to describe and track the movement of dynamic targets. By superimposing transition terms on the initial values ​​of the pose transformation parameters, changes in the movement of dynamic targets between different frames can be described flexibly and simply, improving the efficiency of automatic annotation.

[0115] In the embodiments of this disclosure, a two-dimensional image detection model can obtain detection results obtained by performing target detection on a frame sequence to be processed for static targets, and based on the detection results, candidate targets belonging to the target category are selected and designated as target obstacles.

[0116] Two-dimensional image detection models are typically based on deep learning techniques such as convolutional neural networks (CNNs). CNNs have the ability to automatically extract image features and, through a combination of multi-layer convolutional, pooled, and fully connected layers, can perform feature learning and classification for various targets in input images. For implementation, the YOLO model can be used as a two-dimensional image detection model.

[0117] For each frame image in the frame sequence to be processed, the 2D image detection model performs independent processing, outputs the position of each candidate object detected within the image, and displays it in a 2D position box. Here, the information in the 2D position box can include parameters such as the center point, dimensions, category, detection confidence level, and tracking ID (Identification). Subsequently, based on the detection results, candidate objects belonging to the target category can be selected as target obstacles. For example, obstacles that are too small in dimensions can be filtered out, and among the remaining candidate objects, those that cannot be accurately detected by the conventional 3D detection model, or static targets in newly added categories, will be designated as target obstacles in the embodiments of this disclosure. Specifically, this includes stone pillars and street lamp posts.

[0118] In embodiments of this disclosure, a two-dimensional image detection model can perform target detection on a frame sequence to be processed, obtain detection results, and efficiently and accurately acquire static target categories that require annotation. By optimizing target parameters in projection relationships, automatic detection of static targets in these categories is facilitated. This method can be combined with a three-dimensional detection model to effectively improve the efficiency of target obstacle detection. In the field of autonomous driving, it is possible to improve the annotation viewing distance, increase the number of static target types that can be automatically annotated, and overcome the drawback that 3D detection models cannot automatically annotate sparse or low obstacles.

[0119] In some embodiments, when the target obstacle is a dynamic target, the target pose of the target obstacle is determined based on the pose transformation parameters in the optimized target parameters, the camera intrinsic parameters, and the 3D position box of the target obstacle in the reference frame. This process is shown in equation (10).

number

[0120] In equation (10), JPEG2026099787000050.jpg1126 This is the target pose for the frame to be optimized. JPEG2026099787000051.jpg1126 This is a 3D box detected from a reference frame based on a 3D detection model. JPEG2026099787000052.jpg1126 These are optimized pose transformation parameters, JPEG2026099787000053.jpg1126 This was obtained through optimization. JPEG2026099787000054.jpg1126 It can be obtained by the reverse operation.

[0121] In the embodiments of this disclosure, the 3D position box of the reference frame can be used to estimate the target pose of the target obstacle in the optimization target frame, and the annotation viewing distance and annotation effect of the dynamic target can be improved when the viewing distance of the optimization target frame is far and the point cloud is sparse.

[0122] Therefore, if the target obstacle is a static target, the initial pixel points of the target obstacle in the reference frame are transformed into the camera coordinate system based on the optimized target parameters, a first point cloud of the target obstacle in the camera coordinate system is obtained, the first point cloud is processed using a plane estimation method, and the target pose of the target obstacle in the reference frame is obtained.

[0123] Here, determining the target pose of a target obstacle using a plane estimation method can be implemented as follows. First, a denoising operation is performed on the first point cloud in the camera coordinate system of the target obstacle to remove noise and obtain a first intermediate point cloud. The denoising operation removes points far from the cloud, optimizing the quality of the first point cloud. Next, the interior points, plane, and normal vector of the first intermediate point cloud can be estimated using the Random Sampling Consistency (RANSAC) algorithm. This allows for the estimation of the center point, angle, and dimensions of the static target, thereby obtaining the target pose of the static target in the reference frame.

[0124] In embodiments of this disclosure, for static targets, the target pose in the reference frame of the target obstacle can be determined by plane estimation in order to achieve automatic annotation of static targets.

[0125] To further improve the detection efficiency of target obstacles in other frames, the target pose of the target obstacle in other frames can also be obtained based on a global back projection strategy. Specifically, this can be implemented as follows:

[0126] Using a global back projection strategy, the target pose in the reference frame of the target obstacle is transformed into the pose in the target frame of the target obstacle. The target frame is an image frame other than the frame to be optimized, for which the pose of the target obstacle needs to be estimated.

[0127] Specifically, the beam adjustment method based on the aforementioned target optimizes the position of all points of the target obstacle in 3D space, and the dimensions, orientation, and position of the target obstacle are obtained using plane estimation. Furthermore, the pose of the target obstacle in each target frame is obtained through pose transformations such as the transformation from the world coordinate system to the camera coordinate system.

[0128] In the target frame, the dimensions and position of the target obstacle can be obtained directly from the world coordinate system, and its orientation can be determined by the camera's pose.

[0129] In the embodiments of this disclosure, a global back projection strategy can be used to improve viewing distance for long-range static targets.

[0130] In some embodiments, after obtaining the target pose in each frame image of the target obstacle, the trajectory of the target obstacle can be tracked based on the target pose in different frame images of the target obstacle, and trajectory information of the target obstacle can be obtained.

[0131] The automatic obstacle annotation method according to the embodiments of this disclosure can be used as an aid to a 3D detection model. Therefore, in implementation, the target pose can be obtained using the 3D detection model, and for target obstacles that cannot be detected by the 3D detection model, the automatic annotation can be completed using the method according to the embodiments of this disclosure. In either method, the target pose of the target obstacle can be obtained. The results of the two methods are combined to obtain the target pose of the target obstacle at all time points. Subsequently, the trajectory of the target obstacle is tracked in the world coordinate system based on target tracking technology and Kalman filter technology, and the frame images before and after the presence of the target obstacle are associated to obtain trajectory information of the target obstacle.

[0132] The Kalman filter can predict the position and velocity of a target obstacle and correct it based on observed data. This is very effective in handling shielding or noise interference, and can improve tracking accuracy.

[0133] In target tracking, the Kalman filter is used to model the motion trajectory of a target obstacle. By modeling the motion of the target obstacle, the Kalman filter can predict the position of the target obstacle at the next time step in real time.

[0134] In short, the Kalman filter is a crucial tool in the field of goal tracking, significantly improving the performance and robustness of goal tracking by providing accurate state prediction and correction.

[0135] In embodiments of this disclosure, tracking the trajectory of a target obstacle allows for in-depth analysis of the trajectory at different frame times, providing a data basis for subsequent use.

[0136] In the embodiments of this disclosure, after obtaining trajectory information of a target obstacle, the 2D detection results can be associated with the 3D detection results, and this is mainly used to correct the categories of the 3D detection model. For example, the 3D position boxes of some frames in the trajectory information are detected and obtained based on the 3D detection model, and there may be errors in their categories. In implementation, for the frame to be corrected in the trajectory information, the 3D position box of the target obstacle in the frame to be corrected is determined based on the 2D position box of the target obstacle in the frame to be corrected. Here, the 2D position box is obtained by detecting the target obstacle using a 2D image detection model, and the classification result of the 3D position box of the frame to be corrected is corrected based on the classification result of the 2D position box.

[0137] Here, the 3D position box of the frame to be corrected is the result detected by the 3D detection model. If the classification accuracy of the 3D detection model is lower than that of the 2D image detection model, there may be errors in the category. Therefore, for the target obstacle in the same frame to be corrected, the 2D position box obtained by the 2D image detection model and the 3D position box obtained by the 3D detection model are compared to obtain the intersection over union (IoU) of the 2D and 3D position boxes. If the intersection over union satisfies the pre-set conditions, it is confirmed that the 3D position box corresponding to the 2D position box in the same frame to be corrected belongs to the same target obstacle.

[0138] Subsequently, by associating the classification categories of the 3D position box with the classification categories of the 2D position box, category information for consistent trajectory generation can be obtained. That is, if the detected category of the 3D position box does not match the detected category of the 2D position box, the classification result of the 3D position box is corrected based on the classification result of the 2D position box. This can be implemented as follows: The categories of the 2D position box in different frame images of the target obstacle are obtained, and the category with a classification confidence level greater than the confidence threshold is selected as the final category of the target obstacle. If there are multiple categories with a classification confidence level greater than the confidence threshold, the category with the highest average classification confidence level is preferentially selected as the final category.

[0139] If the category of the 3D position box does not match the final category, the category of the 3D position box can be modified to match the final category.

[0140] In the embodiments of this disclosure, the classification result of the 3D position box in the frame to be corrected is modified based on the classification result of the 2D position box. If the 2D detection result and the 3D detection result do not match, the 2D detection result is used to modify the 3D detection result, thereby improving the type accuracy of target obstacle detection.

[0141] In the embodiments of this disclosure, filtering of false detections can also be achieved by determining the confidence level of the trajectory information. This confidence level is determined based on at least one of the following: the confidence level of the 2D position box of the target obstacle recognized by the 2D image detection model, the occlusion rate of the target obstacle, and the cross union, where the cross union is the cross union between the 2D position box and the 3D position box of the target obstacle. If the confidence level falls below a target threshold, the trajectory information is determined to be a false detection trajectory.

[0142] The confidence level of the 2D position box recognized by the 2D image detection model of the target obstacle is an evaluation index of the accuracy of the detected 2D position box by the 2D image detection model.

[0143] The occlusion rate of a target obstacle represents the degree to which the target obstacle is obscured by other objects in each frame. In implementation, the occlusion status in the current frame can be determined in conjunction with the target obstacle's historical trajectory.

[0144] The intersection union is the intersection union of the 2D position boxes obtained by detecting the target obstacle using a 2D image detection model and the 3D boxes obtained by detecting the target obstacle using a 3D detection model.

[0145] During implementation, the reliability of the target obstacle trajectory information can be calculated using a weighted average. However, the weight coefficients for each piece of information can be adjusted according to the actual needs.

[0146] If the confidence level of the trajectory information is greater than the set threshold, the trajectory is considered valid; otherwise, it is considered a false positive. False positives are filtered out, while false positives are retained.

[0147] In the embodiments of this disclosure, the reliability of trajectory information is determined based on at least one of the reliability of the 2D position box, the occlusion rate, and the cross union. By fusing the above-mentioned information, the reliability of target detection can be evaluated from multiple angles, thus further increasing the reliability of the resulting trajectory. Considering these factors together, it is possible to more accurately position actual target obstacles and avoid treating falsely detected target obstacles as actual target obstacles.

[0148] As described above, using the field of autonomous driving as an example, the automatic obstacle annotation method provided in the embodiments of this disclosure can improve the viewing distance of obstacle annotation, as shown in Figure 4.

[0149] In S401, a point cloud frame or BEV frame is acquired.

[0150] In S402, the acquired point cloud frame or BEV frame is input into the corresponding 3D detection model.

[0151] Here, the point cloud frame employs a 3D detection model corresponding to point clouds, such as DSVT (Dynamic Sparse Voxel Transformer). The 2D BEV employs a 3D detection model corresponding to 2D BEV, such as PETR (Position Embedding Transformation and Refinement).

[0152] In S403, the 3D detection model outputs an initial 3D box bbox_3D of the target obstacle, which includes information such as the obstacle's 3D center point, orientation, dimensions, and category.

[0153] In S404, the BEV frame is input to the 2D image detection model.

[0154] In S405, a 2D image detection model outputs a 2D box (bbox_2d) of a candidate obstacle, containing information such as the 2D center point, dimensions, category, and confidence level of the obstacle.

[0155] In S406, obstacles with 2D box dimensions smaller than a predetermined size are filtered and excluded from the candidate obstacles, static targets of the target category are selected from the remaining candidate obstacles to be used as target obstacles, a BA service is requested to optimize the target parameters in the projection relationship, and the conversion from bbox_2d to bbox_3D is realized.

[0156] In other words, the optimization process of projection relationships described in the aforementioned formula allows us to obtain the target pose of the target obstacle in the reference frame, and the global back projection strategy allows us to obtain the target pose of the target obstacle in other frames.

[0157] In S407, after obtaining the bbox_3D for all frames, the target tracking module can use a Kalman filter and target tracking techniques to correlate target obstacles in different frames and obtain trajectory information for the target obstacles.

[0158] In S408, the category of the target obstacle in bbox_3D is corrected based on the association between the 2D detection results and the 3D detection results, and category information for consistent trajectory generation is obtained.

[0159] Furthermore, in addition to enabling automatic annotation and improved viewing distance for static targets, the embodiments of this disclosure can also improve the annotation viewing distance for dynamic targets.

[0160] For example, in S409, frames with missed detections of dynamic targets by the 3D detection model can be identified for a point cloud frame or BEV frame, and these frames can be designated as frames for optimization. For example, it is possible to determine whether or not there are missed detections based on the continuous detection status of the 3D position boxes of the same target obstacle.

[0161] For missing point cloud frames, the viewing distance of the dynamic target can be improved by acquiring BEV frames collected in synchronization with the point cloud frames. For missing BEV frames, the viewing distance of the dynamic target can be improved based on the BEV frames. Specifically, for missing frames, the BEV frame of the most recent frame in which a pose was detected by the 3D detection model can be selected as a reference frame, and a BA service can be requested to optimize based on the trajectory information of the initial pixel points of the reference frame and the 2D position box.

[0162] In S410, a BA service is requested, the projection relationship is optimized based on the reference frame, and optimized pose transformation parameters and pixel point depth parameters are obtained. Subsequently, the target pose in the optimized frame of the dynamic target is obtained based on the position transformation parameters.

[0163] In S411, filtering of falsely detected trajectory information is achieved based on the false detection filtering module.

[0164] In S412, the annotation results are output.

[0165] As described above, the BA service optimization process includes a pre-processing module, an optimization module, and an output module, as shown in Figure 5.

[0166] The preprocessing module preprocesses the received requests and obtains mainly eight types of request information, such as the following:

[0167] (1) Dynamic / Status Identifier: Used to indicate whether the request is for a dynamic or static target, and this identifier affects the OCBA optimization process, i.e., whether the optimized target parameters include the pause transformation parameter. (2) Obstacle list: Represented in the form of obstacle trajectory IDs, this list has a one-to-one correspondence with subsequent information as a key, and during implementation, annotation of multiple target obstacles can be requested, and this list stores each obstacle that needs to be automatically annotated. (3) Starting frame (i.e., reference frame): Indicates the frame number from which optimization of each obstacle begins. In implementation, select one frame with prominent features and use it as the reference frame for the target obstacle. (4) Two-dimensional box of the starting frame: Shows the two-dimensional box of the target obstacle at the starting frame. (5) Optimization direction: Indicates whether the target obstacle optimizes forward or backward. (6) Number of frames to optimize: Indicates the number of frames in which the target obstacle participates in the optimization process. (7) Pose information: Includes pose transformation matrices from camera coordinate system to world coordinate system, transformation matrices from camera to radar, etc. (8) Setting of OCBA-related parameters: for example, sequence length, depth noise variance, matrix variance, etc.

[0168] Here, as mentioned above, the sequence length is used to reduce resource usage, and the sequence length is less than or equal to the number of frames to be optimized. Depth noise variance is used to randomly perturb the initial values ​​of the pixel point depth. Depth noise variance may include the mean and variance of a Gaussian distribution to facilitate generating noise by randomly sampling from a Gaussian distribution to randomly perturb the pixel point depth parameter. Matrix variance is used to randomly perturb the pause transform parameter.

[0169] After obtaining a reference frame in the sequence to be processed, the target obstacle is divided using the SAM model, a mask image of the target obstacle is obtained, and points are uniformly scattered based on the 2D blocks of the target obstacle and the mask image to obtain the initial pixel points of the target obstacle.

[0170] Next, based on the initial pixel points of the target obstacle, a co-tracker is used to track these points, thereby obtaining a series of trajectory information for the initial pixel points in each frame to be optimized. The trajectories of these tracked points are then used as the true pixel point values ​​in the OCBA optimization frame to optimize the relevant target parameters.

[0171] The optimization module includes DataLoader, OCBA Optimizer, and Visualizer. DataLoader provides data for each optimization, and after optimization is complete, the effects are evaluated using visualization.

[0172] DataLoader determines the image position of 2D pixel points in the reference frame. JPEG2026099787000055.jpg1126 , 2D pixel point true value of the frame to be optimized JPEG2026099787000056.jpg1126 Camera internal parameter K, initial value of pixel point depth parameter JPEG2026099787000057.jpg1126 Initial values ​​of the position transformation parameters from the camera coordinate system to the obstacle coordinate system. JPEG2026099787000058.jpg1126 To provide.

[0173] The Optimizer's optimization process is an iterative process of optimization. First, the initial pixel points... JPEG2026099787000059.jpg1126 , camera internal parameter K and pixel point depth parameter JPEG2026099787000060.jpg1126 The back projection operation is performed using the back projection result JPEG2026099787000061.jpg1126 This is obtained as the corresponding 3D spatial position point in the camera coordinate system for the initial 2D pixel point in the reference frame of the target obstacle.

[0174] next, JPEG2026099787000062.jpg1126 Reusing the 3D spatial position point, the camera coordinate system of the reference frame is used to determine the point in the obstacle coordinate system. JPEG2026099787000063.jpg1126 Convert to this. Since the points of obstacles in the obstacle coordinate system are unified, directly JPEG2026099787000064.jpg1126 This allows us to transform points from the object coordinate system to points in an arbitrary i-frame camera coordinate system. Simultaneously, we use the internal parameter K to perform a projection operation in an arbitrary i-th frame and obtain the 2D pixel coordinates of obstacles in the i-th frame. OCBA allows us to determine the 2D pixel position of each pixel point in the initial frame in other frames. JPEG2026099787000065.jpg1126 To obtain.

[0175] Furthermore, this process is optimized using reprojection errors, JPEG2026099787000066.jpg1126 Considering that the loss is difficult to converge, a depth regularization term is added to ensure normal convergence of the loss.

[0176] Ultimately, the entire OCBA optimization process was realized through lossless backtransfer and gradient descent updates, resulting in an optimized process. JPEG2026099787000067.jpg1126 and JPEG2026099787000068.jpg1126 To obtain.

[0177] The Visualizer in Figure 5 allows for a more intuitive display of the annotation results presented after optimization. For example, it can display the automatically annotated bbox_3d in a 2D image of a low stone pillar.

[0178] In the output module, optimized for static targets JPEG2026099787000069.jpg1126 , and, JPEG2026099787000070.jpg1126 , JPEG2026099787000071.jpg1126 , JPEG2026099787000072.jpg1126 Through a set of parameters such as the one shown, the result of the back projection JPEG2026099787000073.jpg1126 This can be obtained, and then the target pose of the static target obstacle can be obtained based on plane estimation. In implementation, the corresponding pixel point depth parameters of all 2D pixel points of the target obstacle JPEG2026099787000074.jpg1126 The points can be obtained, and the 3D spatial position of the target obstacle in the camera coordinate system can be obtained by the backprojection operation of equation (1). These points are denoised using a denoising algorithm such as DBSACN, and points far from the point cloud are removed. Furthermore, the Random Sampling Consistency (RANSAC) algorithm is used to estimate the interior points, planes, and normal vectors of these point clouds, enabling the estimation of the center point, angle, and dimensions of the static target. For other frames, since the global position of the stationary obstacle is unified, the current target obstacle position can be backprojected to other frames using a global backprojection strategy, that is, the obstacle in the current frame can be transformed into the global coordinate system (i.e., world coordinate system), and then transformed back into each frame, thereby effectively enabling the annotation of stationary obstacles.

[0179] Optimized for dynamic goals JPEG2026099787000075.jpg1126 3D boxes detected from the reference frame based on a 3D detection model JPEG2026099787000076.jpg1126 , and, JPEG2026099787000077.jpg1126 , JPEG2026099787000078.jpg1126 , JPEG2026099787000079.jpg1126 , JPEG2026099787000080.jpg1126 Based on a set of parameters such as these, the target pose of the dynamic target for the frame to be optimized is obtained.

[0180] In summary, the solution of this disclosure complements the limitations of existing radar scanning ranges and the shortcomings of existing automatic annotation methods, achieving static target detection independent of 3D detection models and improved dynamic target annotation viewing distance. It can support the annotation of various static targets, such as low stone pillars and lampposts, and improves the dynamic target annotation viewing distance to greater distances. This annotation data can be used to drive the vehicle-side BEV obstacle detection model, significantly improving model annotation capabilities and enabling timely obstacle avoidance by the main vehicle.

[0181] Based on a similar technical concept, this disclosure also provides an automatic obstacle annotation device 600, as shown in Figure 6, An optimization module 601 for optimizing target parameters in a projection relationship based on reprojection errors, wherein the projection relationship is used to project a target obstacle from a reference frame to a frame to be optimized, and the projection relationship satisfies the constraint that the positions of the target obstacle in different frames are unified in an obstacle coordinate system constructed based on the target obstacle. The system includes a decision module 602 for determining the target pose of a target obstacle based on optimized target parameters.

[0182] In some embodiments, the optimization module is A projection unit for mapping the initial pixel point cloud of the target obstacle in the reference frame to the frame to be optimized based on projection relationships, and obtaining the projected point cloud of the target obstacle in the frame to be optimized, It includes an error determination unit for determining the reprojection error based on the projected point cloud and the true projected point values ​​of the frame to be optimized.

[0183] In some embodiments, the optimization module is A loss determination unit for determining the total projection loss based on the reprojection error and the depth regularization term of the projection point, Includes an optimization unit for optimizing target parameters in projection relationships based on total projection loss.

[0184] In some embodiments, a depth regularization term is used to reduce the degree of depth discreteness of each pixel point of the target obstacle.

[0185] In some embodiments, for the frame to be optimized, the true projection point values ​​required for the reprojection error are determined based on a pixel-level trajectory tracking device.

[0186] In some embodiments, the projection unit is A backprojection subunit for performing a backprojection operation on an initial pixel point cluster in a reference frame of a target obstacle based on a pixel point depth parameter to obtain a first three-dimensional spatial position of the target obstacle in the camera coordinate system, wherein the pixel point depth parameter is used to represent the depth value of each initial pixel point in the reference frame, and A transformation subunit for transforming a first 3D spatial position into an obstacle coordinate system and obtaining a second 3D spatial position based on pose transformation parameters from the camera coordinate system to the obstacle coordinate system, It includes a projection subunit for projecting a second 3D spatial position onto the frame to be optimized and obtaining a projected point cloud.

[0187] In some embodiments, when the target obstacle is a static target, the target parameters include pixel point depth parameters.

[0188] In some embodiments, when the target obstacle is a dynamic target, the target parameters include pixel point depth parameters and pose transformation parameters.

[0189] In some embodiments, the optimization module is A first acquisition unit for obtaining a first preset value of the pixel point depth parameter, It includes a first perturbation unit for randomly perturbing a first preset value to obtain an initial value for the pixel point depth parameter.

[0190] In some embodiments, the optimization module is A second acquisition unit for obtaining the second preset value of the pose transformation parameter, It includes a second perturbation unit for randomly perturbing a second preset value to obtain an initial value for the pose transformation parameter.

[0191] In some embodiments, the obstacle coordinate system is constructed based on the world coordinate system. When the target obstacle is a dynamic target, the pause transformation parameter is calculated by superimposing a transition term onto the initial value of the pause transformation parameter. This transition term is used to represent the relative change of the target obstacle between the reference frame and the frame being optimized. Optimizing the transition term is then used to optimize the pause transformation parameter.

[0192] In some embodiments, the projection unit is specifically, The 2D image detection model obtains a 2D position box of the target obstacle detected from the reference frame, It is used to uniformly scatter points based on a 2D position box and obtain an initial pixel point cloud in the reference frame of the target obstacle.

[0193] In some embodiments, the projection unit is specifically, Obtaining a mask image of the target obstacle in the reference frame, This method is used to obtain an initial pixel point cloud by uniformly scattering points based on a 2D position box and a mask image in the reference frame of the target obstacle.

[0194] In some embodiments, the projection unit is specifically, This method uses the Segment Anything Model (SAM) to segment target obstacles from a reference frame and obtain a mask image of the target obstacles.

[0195] In some embodiments, the projection unit is specifically, This method is used to select images from a sequence of frames to be processed that satisfy pre-defined conditions regarding the prominence of target features, based on the target features of the target obstacle, and to use them as reference frames.

[0196] In some embodiments, the projection unit is specifically, If the target obstacle is a static target, obtain the optimal direction, It is used to select frames to be optimized from a sequence of frames to be processed, based on the reference frame and the optimization direction.

[0197] In some embodiments, the projection unit is specifically, When the target obstacle is a dynamic target, the 3D detection model acquires frame images where it fails to detect the target obstacle and uses them as frames for optimization. The 3D detection model is then used to detect the target in a point cloud or 2D image to obtain the target pose of the target obstacle.

[0198] In some embodiments, the projection unit is specifically, When the target obstacle is a dynamic target, this is used to preferentially select the frame that is closest to the frame being optimized and also contains the target obstacle as the reference frame.

[0199] In some embodiments, the projection unit is specifically, The process involves obtaining detection results obtained by a 2D image detection model performing target detection on a frame sequence to be processed, and It is used to select candidate objects belonging to the target category based on the detection results and designate them as target obstacles.

[0200] In some embodiments, the reference frame and the frame to be optimized are bird's-eye views during autonomous driving.

[0201] In some embodiments, the automatic obstacle annotation device further includes: The system includes a trajectory tracking module that tracks the trajectory of a target obstacle based on the target pose in different frame images of the target obstacle, and obtains trajectory information of the target obstacle.

[0202] In some embodiments, the automatic obstacle annotation device further includes: The process involves determining the 3D position box of the target obstacle in the frame to be corrected based on the 2D position box of the target obstacle in the frame to be corrected, wherein the 2D position box is obtained by detecting the target obstacle using a 2D image detection model. The system includes a correction module used to modify the classification result of the 3D position box of the frame to be corrected based on the classification result of the 2D position box.

[0203] In some embodiments, the automatic obstacle annotation device further includes: The reliability of the trajectory information is determined based on at least one piece of information: the reliability of the 2D position box of the target obstacle identified by the 2D image detection model, the occlusion rate of the target obstacle, and the cross union, where the cross union is the cross union of the 2D position box and the 3D position box of the target obstacle. The system includes a filtering module that determines if the confidence level falls below a target threshold, and determines that the trajectory information is a falsely detected trajectory.

[0204] In some embodiments, the decision module specifically, If the target obstacle is a dynamic target, the target pose of the target obstacle in the optimized frame is determined based on the pose transformation parameters, camera intrinsic parameters, and the 3D position box of the target obstacle in the reference frame.

[0205] In some embodiments, the decision module specifically, If the target obstacle is a static target, the initial pixel points of the target obstacle in the reference frame are transformed into the camera coordinate system based on optimized target parameters, and a first point cloud of the target obstacle in the camera coordinate system is obtained. This is used to process the first point cloud using a plane estimation device and obtain the target pose of the target obstacle in the reference frame.

[0206] In some embodiments, the decision module further: The global back projection strategy is used to transform the target pose in the reference frame of the target obstacle into the pose in the target frame of the target obstacle. The target frame is an image frame other than the frame to be optimized, for which the pose of the target obstacle needs to be estimated.

[0207] The technical solutions disclosed herein ensure that the acquisition, storage, and application of users' personal information comply with applicable laws and regulations and do not violate public order or morality.

[0208] According to embodiments of the present disclosure, the present disclosure further provides electronic devices, non-temporary computer-readable storage media, and program products.

[0209] Figure 7 shows a schematic block diagram showing how an electronic device 700 according to an embodiment of the present disclosure can be implemented. The electronic device refers to various types of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other compatible computers. The electronic device further refers to various types of mobile devices, such as personal digital assistants, cellular phones, intelligent phones, wearable devices, and other similar computer devices. The components, their connections, and functions described in this disclosure are illustrative and do not limit the implementation of anything described or specified in this disclosure.

[0210] As shown in Figure 7, device 700 includes a computing unit 701 that can perform various appropriate operations and processes based on computer program instructions stored in read-only memory (ROM) 702 or computer program instructions loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 can further store various programs and data necessary for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are connected to each other via bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0211] The multiple components in device 700 are connected to I / O interface 705, and the multiple components include an input unit 706 such as a keyboard and a mouse, an output unit 707 such as various displays and speakers, a storage unit 708 such as a magnetic disk and an optical disk, and a communication unit 709 such as a network card, a modem, and a wireless communication transceiver. The communication unit 709 permits the device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various carrier networks.

[0212] The computing unit 701 may be various general-purpose and / or dedicated processing components having processing and computing capabilities. Some examples of the computing unit 701 include a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, a computing unit that executes various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc., but are not limited thereto. The computing unit 701 executes each method described above, for example, the automatic annotation method for obstacles. For example, in some embodiments, the automatic annotation method for obstacles can be realized as a computer software program tangibly included in a machine-readable medium such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed into the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the aforementioned automatic annotation method for obstacles can be executed. Additionally, in other embodiments, the computing unit 701 can be configured to execute the automatic annotation method for obstacles in any other suitable manner (for example, firmware).

[0213] In various embodiments of the systems or techniques described in this disclosure, they can be implemented by digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. Each of these embodiments may include being executed by one or more computer programs that are executed and / or interpreted in a programmable system including at least one programmable processor, where the programmable processor can be a dedicated or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and can transfer the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0214] The program code for executing the methods of this disclosure can be created in any combination of one or more programming languages. By providing this program code to a processor or controller of a general-purpose computer, a dedicated computer, or other programming data processing device, when the program code is executed by the processor or controller, it can execute the functions / operations defined in the flowchart and / or block diagram. The program code may be executed entirely in the machine, partially in the machine, partially executed in the machine as an independent software encapsulation and partially executed in a remote machine, or entirely executed in a remote machine or server.

[0215] In this disclosure, machine-readable media may be tangible media containing or storing programs used by or in conjunction with instruction execution systems, devices, or equipment. Machine-readable media may be machine-readable signal media or machine-readable storage media. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any suitable combination of the contents described above. Further specific examples of machine-readable storage media include one or more wired electrical connections, portable computer disk cartridges, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any combination of the contents described above.

[0216] To provide user interaction, a computer may implement the systems and technologies described herein, which may include an annotation device for annotating information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor), a keyboard and pointing device for the user to provide input to the computer (e.g., a mouse or trackball). Other types of devices may also be used to provide user interaction; for example, the feedback provided to the user may be any form of sensor feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be accepted in any form (e.g., acoustic input, voice input, haptic input).

[0217] The systems and technologies described herein can be implemented in computing systems that include background components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include front-end components (e.g., user computers having a graphics user interface or network browser, through which users can interact with embodiments of the systems and technologies described herein), or in any combination of such background components, middleware components, or front-end components. Components of the system can be connected to one another via digital data communication in any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0218] A computer system can include a client and a server. Typically, the client and server are geographically separated and interact via a communication network. The client-server relationship is created by a computer program that operates on the corresponding computer. The server may be a cloud server, a server in a distributed system, or a server incorporating blockchain technology, etc.

[0219] It should be understood that steps can be newly ranked, added, or deleted using the various forms of flows shown above. For example, each step described in this disclosure may be executed in parallel, sequentially, or in a different order. This disclosure is not limited to this, as long as the technical solutions disclosed herein can achieve the desired results.

[0220] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations, and substitutions are possible due to design considerations and other factors. Any changes, equivalent substitutions, and improvements within the gist and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An automatic annotation method for obstacles, The method involves optimizing target parameters in a projection relationship based on reprojection errors, wherein the projection relationship is used to project a target obstacle from a reference frame to a frame to be optimized, and the projection relationship satisfies the constraint that the positions of the target obstacle in different frames are unified in an obstacle coordinate system constructed based on the target obstacle. This includes determining the target pose of the target obstacle based on optimized target parameters, A method for automatically annotating obstacles.

2. The automatic annotation method for obstacles further includes determining the reprojection error, Determining the aforementioned reprojection error means that Based on the projection relationship, the initial pixel point cloud of the target obstacle in the reference frame is mapped into the optimization target frame, and the projected point cloud of the obstacle in the optimization target frame is obtained. This includes determining the reprojection error based on the projected point cloud and the true projected point values ​​of the frame to be optimized, The method for automatically annotating obstacles according to claim 1.

3. Optimizing the target parameters in the projection relationship based on the aforementioned reprojection error is The total projection loss is determined based on the aforementioned reprojection error and the depth regularization term of the projection point. This includes optimizing the target parameter in the projection relationship based on the total projection loss, The method for automatically annotating obstacles according to claim 1.

4. The depth regularization term is used to reduce the degree of depth discreteness of each pixel point of the target obstacle. The method for automatically annotating obstacles according to claim 3.

5. For the frame to be optimized, the true projection point value required for the reprojection error is determined based on a pixel-level trajectory tracking method. The method for automatically annotating obstacles according to claim 1.

6. Based on the projection relationship, mapping the initial pixel point cloud of the target obstacle in the reference frame into the optimization target frame and obtaining the projected point cloud of the obstacle in the optimization target frame is: Based on a pixel point depth parameter, a back projection operation is performed on the initial pixel point cloud of the target obstacle in the reference frame to obtain a first three-dimensional spatial position of the target obstacle in the camera coordinate system, wherein the pixel point depth parameter is used to represent the depth value of each initial pixel point in the reference frame. Based on the pose transformation parameters from the camera coordinate system to the obstacle coordinate system, the first three-dimensional spatial position is transformed into the obstacle coordinate system to obtain a second three-dimensional spatial position. The process includes projecting the second three-dimensional spatial position onto the frame to be optimized and obtaining the projected point cloud, The method for automatically annotating obstacles according to claim 2.

7. If the target obstacle is a static target, the target parameters include the pixel point depth parameters. The method for automatically annotating obstacles according to claim 6.

8. If the target obstacle is a dynamic target, the target parameters include the pixel point depth parameter and the pose transformation parameter. The method for automatically annotating obstacles according to claim 6.

9. The automatic annotation method for obstacles further includes determining the initial value of the pixel point depth parameter, Determining the initial value of the aforementioned pixel point depth parameter is: Obtaining a first preset value for the pixel point depth parameter, This includes randomly perturbing the first preset value to obtain an initial value for the pixel point depth parameter, The method for automatically annotating obstacles according to claim 6.

10. The automatic annotation method for obstacles further includes determining the initial value of the pose transformation parameter, Determining the initial values ​​of the aforementioned pause transformation parameters is: Obtaining the second preset value for the pose transformation parameter, This includes randomly perturbing the second preset value to obtain an initial value for the pose transformation parameter, The method for automatically annotating obstacles according to claim 6.

11. The aforementioned obstacle coordinate system is constructed based on the world coordinate system. If the target obstacle is a dynamic target, the pose transformation parameter is configured by superimposing a transition term onto the initial value of the pose transformation parameter, the transition term is used to represent the relative change of the target obstacle between the reference frame and the frame to be optimized, and the pose transformation parameter is optimized by optimizing the transition term. The method for automatically annotating obstacles according to claim 6.

12. The automatic annotation method for obstacles further includes determining the initial pixel point cloud of the target obstacle in the reference frame, Determining the initial pixel point cloud of the target obstacle in the reference frame is: The 2D image detection model obtains a 2D position box of the target obstacle detected from the reference frame, This includes uniformly scattering points based on the two-dimensional position box to obtain an initial pixel point cloud of the target obstacle in the reference frame, The method for automatically annotating obstacles according to claim 2.

13. Distributing points uniformly based on the two-dimensional position box to obtain an initial pixel point cloud of the target obstacle in the reference frame is: To obtain the mask image of the target obstacle in the reference frame, This includes uniformly scattering points based on the two-dimensional position box of the target obstacle in the reference frame and the mask image to obtain the initial pixel point cloud, The method for automatically annotating obstacles according to claim 12.

14. Obtaining the mask image of the target obstacle in the reference frame is, This includes using a Segment Anything Model (SAM) to divide the target obstacle from the reference frame and obtain a mask image of the target obstacle. The method for automatically annotating obstacles according to claim 13.

15. The automatic annotation method for obstacles further includes determining the reference frame, Determining the aforementioned reference frame means This includes selecting an image from the frame sequence to be processed that satisfies a predetermined condition regarding the prominence of the target features based on the target features of the target obstacle, and using that image as the reference frame. The method for automatically annotating obstacles according to claim 1.

16. The automatic annotation method for obstacles further includes determining the frame to be optimized, Determining the frame to be optimized means If the aforementioned target obstacle is a static target, the optimization direction is obtained, This includes selecting the frame to be optimized from the frame sequence to be processed based on the reference frame and the optimization direction, The method for automatically annotating obstacles according to claim 1.

17. The automatic annotation method for obstacles further includes determining the frame to be optimized, Determining the frame to be optimized means If the target obstacle is a dynamic target, the process involves obtaining a frame image in which the 3D detection model fails to detect the target obstacle and using that as the frame to be optimized, wherein the 3D detection model is used to detect the target in a point cloud or 2D image in order to obtain the target pose of the target obstacle. The method for automatically annotating obstacles according to claim 1.

18. If the target obstacle is a dynamic target, the frame closest to the frame to be optimized and containing the target obstacle is preferentially selected as the reference frame. The method for automatically annotating obstacles according to claim 17.

19. The automatic annotation method for obstacles further includes determining the target obstacle, Determining the aforementioned target obstacle is, The process involves obtaining detection results obtained by a 2D image detection model performing target detection on a frame sequence to be processed, and This includes selecting candidate objects belonging to the target category based on the detection results and designating them as target obstacles. The method for automatically annotating obstacles according to claim 1.

20. The aforementioned reference frame and the frame to be optimized are bird's-eye views during autonomous driving. The method for automatically annotating obstacles according to claim 1.

21. The aforementioned automatic annotation method for obstacles further includes: This includes tracking the trajectory of the target obstacle based on the target pose in different frame images of the target obstacle and obtaining trajectory information of the target obstacle. The method for automatically annotating obstacles according to claim 1.

22. The aforementioned automatic annotation method for obstacles further includes: The process involves determining the three-dimensional position box of the target obstacle in the frame to be corrected based on the two-dimensional position box of the target obstacle in the frame to be corrected, wherein the two-dimensional position box is obtained by detecting the target obstacle using a two-dimensional image detection model. This includes modifying the classification result of the three-dimensional position box of the frame to be corrected based on the classification result of the two-dimensional position box, The method for automatically annotating obstacles according to claim 21.

23. The aforementioned automatic annotation method for obstacles further includes: The reliability of the trajectory information is determined based on at least one piece of information: the reliability of the two-dimensional position box of the target obstacle identified by the two-dimensional image detection model, the occlusion rate of the target obstacle, and the cross union, wherein the cross union is the cross union of the two-dimensional position box and the three-dimensional position box of the target obstacle. If the confidence level falls below a target threshold, it is determined that the trajectory information is a falsely detected trajectory. The method for automatically annotating obstacles according to claim 21.

24. Determining the target pose of the target obstacle based on the optimized target parameters is: If the target obstacle is a dynamic target, the process includes determining the target pose of the target obstacle in the optimized frame based on the pose transformation parameters, camera intrinsic parameters, and the three-dimensional position box of the target obstacle in the reference frame. The method for automatically annotating obstacles according to claim 1.

25. Determining the target pose of the target obstacle based on the optimized target parameters is: If the target obstacle is a static target, the initial pixel points of the target obstacle in the reference frame are transformed into the camera coordinate system based on the optimized target parameters, and a first point group of the target obstacle in the camera coordinate system is obtained. This includes processing the first point cloud using a plane estimation method to obtain the target pose of the target obstacle in the reference frame, The method for automatically annotating obstacles according to claim 1.

26. The aforementioned automatic annotation method for obstacles further includes: This includes using a global back projection strategy to convert the target pose of the target obstacle in the reference frame to the pose of the target obstacle in the target frame, The target frame is an image frame other than the frame to be optimized, for which it is necessary to estimate the pose of the target obstacle. The method for automatically annotating obstacles according to claim 25.

27. An automatic obstacle annotation device, An optimization module for optimizing target parameters in a projection relationship based on reprojection errors, wherein the projection relationship is used to project a target obstacle from a reference frame to a frame to be optimized, and the projection relationship satisfies the constraint that the positions of the target obstacle in different frames are unified in an obstacle coordinate system constructed based on the target obstacle. A decision module for determining the target pose of the target obstacle based on optimized target parameters, An automatic obstacle annotation device.

28. At least one processor, The system comprises at least one processor and a memory that is communicated with by it, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor performs the method according to any one of claims 1 to 26. Electronic devices.

29. A non-temporary computer-readable storage medium storing computer instructions that cause a computer to perform the method described in any one of claims 1 to 26.

30. A program for a computer, when executed by a processor, that enables the implementation of the method according to any one of claims 1 to 26.

Images