Obstacle automatic labeling method, device and equipment and storage medium

Abstract

The present disclosure provides an obstacle automatic labeling method and device, equipment and storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of automatic driving, neural network, deep learning and the like. The specific implementation scheme is: based on the re-projection error, the target parameters in the projection relationship are optimized; the projection relationship is used for projecting the target obstacle from the reference frame to the frame to be optimized; the projection relationship satisfies the following constraint condition: in the obstacle coordinate system established by the target obstacle, the positions of the target obstacle in different frames are unified; the target pose of the target obstacle is determined based on the optimized target parameters.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of automatic driving, neural networks, deep learning, etc. BACKGROUND

[0002] With the continuous progress of artificial intelligence technology, autonomous vehicles are gradually becoming an important part of future transportation. The core of autonomous driving technology is to achieve autonomous driving of vehicles, which not only requires vehicles to understand their surroundings, but also needs to have the ability to handle various driving situations.

[0003] In this process, environmental perception technology plays a crucial role, especially the accurate detection and identification of obstacles, which has a decisive impact on ensuring the safety of autonomous vehicles and improving their navigation efficiency. Among them, obstacle detection relies on a large amount of labeled data. SUMMARY

[0004] The present disclosure provides an automatic labeling method, device, equipment and storage medium for obstacles.

[0005] According to an aspect of the present disclosure, an automatic labeling method for obstacles is provided, comprising:

[0006] Based on the re-projection error, the target parameters in the projection relationship are optimized; the projection relationship is used to project the target obstacle from the reference frame to the frame to be optimized; the projection relationship satisfies the following constraint condition: in the obstacle coordinate system established by the target obstacle, the positions of the target obstacle in different frames are uniform;

[0007] Based on the optimized target parameters, the target pose of the target obstacle is determined.

[0008] According to another aspect of the present disclosure, an automatic labeling device for obstacles is provided, comprising:

[0009] The optimization module is configured to optimize the target parameters in the projection relationship based on the re-projection error; the projection relationship is used to project the target obstacle from the reference frame to the frame to be optimized; the projection relationship satisfies the following constraint condition: in the obstacle coordinate system established by the target obstacle, the positions of the target obstacle in different frames are uniform;

[0010] The determination module is configured to determine the target pose of the target obstacle based on the optimized target parameters.

[0011] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0012] at least one processor; and

[0013] a memory in communication with the at least one processor; wherein,

[0014] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any of the embodiments of the present disclosure.

[0015] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause the computer to perform the method according to any of the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to any of the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, an autonomous vehicle is provided, comprising the electronic device as described above.

[0018] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0020] Figure 1 is a flowchart of an automatic labeling method of an obstacle according to an embodiment of the present disclosure;

[0021] Figure 2 is a flowchart of determining an initial pixel point set of a target obstacle in a reference frame according to an embodiment of the present disclosure;

[0022] Figure 3 is a flowchart of determining a re-projection error according to an embodiment of the present disclosure;

[0023] Figure 4 is a flowchart of an automatic labeling method of an obstacle according to an embodiment of the present disclosure;

[0024] Figure 5 is a BA service optimization process framework diagram according to an embodiment of the present disclosure;

[0025] Figure 6 is a structural diagram of an automatic labeling device of an obstacle according to an embodiment of the present disclosure;

[0026] Figure 7 is a block diagram of an electronic device for implementing an automatic labeling method of an obstacle according to an embodiment of the present disclosure. Detailed Implementation

[0027] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0028] The terms “first,” “second,” etc., used in this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0029] In the field of autonomous driving, highly accurate environmental perception is crucial for ensuring safe and efficient navigation. The core of environmental perception is obstacle detection; accurate obstacle identification and localization are essential for path planning and obstacle avoidance strategies. Obstacle detection and identification require a large amount of labeled data.

[0030] Related technologies provide 3D detection models that can automatically annotate point clouds or BEV (Bird's Eye View) images to obtain 3D bounding boxes for detected targets. The 3D bounding boxes provided by these detection models include the target's pose (such as location center, orientation, and size) as well as category information.

[0031] In particular, 3D detection models rely heavily on labeled data for object detection. This data includes both dynamic and static targets. The quality of the labeling method significantly impacts the accuracy and labeling capability of the 3D detection model. Especially in the field of autonomous driving, the quality of the labeling method affects the actual driving experience. Labeling capability largely determines the autonomous driving system's ability to perceive surrounding obstacles, while the labeled line-of-sight distance determines the upper limit of the autonomous driving system.

[0032] However, the automatic labeling method of obstacles in the related art has some defects: first, the conventional radar scanning range is limited, such as for long-distance dynamic targets, medium and long-distance or low static targets, the scanned point cloud is very sparse; this limits the labeling ability of both manual labeling and automatic labeling methods based on models; ultimately, it limits the upper limit of the ability of the three-dimensional detection model; second, some automatic labeling methods need to first complete labeling on the point cloud, and then match to the corresponding image to realize labeling of the image; however, due to the problems of internal and external parameter calibration, timestamp synchronization, etc. between the point cloud and the image, the consistency between the image and the point cloud is poor, thereby affecting the accuracy of automatic labeling; the limited labeling accuracy also limits the labeling ability of the three-dimensional detection model.

[0033] Therefore, the embodiments of the present disclosure provide an automatic labeling method of obstacles. The method supports automatic detection of dynamic targets and static targets, and can improve the labeling range. As shown in Figure 1 the main contents include the following:

[0034] S101, based on a re-projection error, optimizing a target parameter in a projection relationship; the projection relationship is used to project a target obstacle from a reference frame to a frame to be optimized; the projection relationship satisfies the following constraint condition: in an obstacle coordinate system established by the target obstacle, the positions of the target obstacle in different frames are uniform.

[0035] The reference frame and the frame to be optimized in the embodiments of the present disclosure are both two-dimensional images, which can be images collected by a camera sensor. In the field of automatic driving, they can be bird's eye view images obtained by fusing images collected by multiple surround-view cameras of a vehicle.

[0036] The target obstacle can be a static target or a dynamic target.

[0037] The re-projection error is an index used to measure the difference between the projected position of a three-dimensional point in a two-dimensional image and the actual observed feature point position in computer vision. It is a core concept in bundle adjustment algorithm, and by minimizing the re-projection error, the optimized three-dimensional point can approach the actual situation.

[0038] The embodiments of the present disclosure establish an obstacle coordinate system, so that the same target obstacle in different frame images is uniform in position after being projected from the image coordinate system of different frame images to the obstacle coordinate system. For example, in the field of automatic driving, obstacle D is converted from A frame image to obstacle coordinate system to obtain position coordinate P1, and obstacle D is converted from B frame image to obstacle coordinate system to obtain position coordinate P2. Because the obstacle coordinate system is established with the obstacle D itself as the center, P1 is equal to P2.

[0039] In implementation, the correlation between the image coordinate system, the camera coordinate system and the obstacle coordinate system is established in the world coordinate system, so as to model the projection relationship between the reference frame and the frame to be optimized. By optimizing the target parameters in the projection relationship, the established projection relationship is approximated to the actual situation, and finally the core parameters (i.e. target parameters) for determining the pose of the target obstacle are obtained.

[0040] In the embodiments of the present disclosure, the target parameters in the projection relationship are optimized based on the re-projection error, so that the projection relationship established based on the optimized target parameters can accurately describe the projection relationship between the reference frame and the frame to be optimized.

[0041] In S102, the target pose of the target obstacle is determined based on the optimized target parameters.

[0042] The target pose of the target obstacle can include the center position, angle and size of the three-dimensional position box of the target obstacle in the three-dimensional space.

[0043] In the embodiments of the present disclosure, the target parameters in the projection relationship are optimized based on the re-projection error, so that the projection relationship established based on the optimized target parameters can accurately describe the projection relationship between the reference frame and the frame to be optimized.

[0044] In the embodiments of the present disclosure, as described above, the reference frame and the frame to be optimized can be the bird's eye view in autonomous driving. For example, the surrounding environment images of the vehicle can be collected by using a surround six camera or a surround seven camera, and then fused into a bird's eye view.

[0045] The bird's eye view converts the environment information around the vehicle from the traditional front view or side view image to the perspective from the top. This perspective is particularly useful for understanding the spatial layout around the vehicle, detecting obstacles and planning the driving path.

[0046] In the field of autonomous driving, the reference frame refers to a frame of BEV image selected as a reference within a certain time. The frame to be optimized refers to another frame of BEV image which needs to be compared with the reference frame. The reference frame and the frame to be optimized are usually obtained from a sequence of continuous frames to be processed, representing the environment state changing over time.

[0047] In the embodiments of the present disclosure, by taking the bird's eye view as the reference frame and the frame to be optimized, the data requirement in the automatic driving technology is met, and the automatic labeling of the bird's eye view is completed.

[0048] Of course, it can be understood that for the point cloud image collected by the radar sensor, the bird's eye view collected synchronously with the point cloud can also be used to improve the labeling capability of the point cloud data, so as to overcome the problem that the point cloud is sparse and the target cannot be detected. Moreover, the method provided in the embodiments of the present disclosure can be used to detect the pose of the target obstacle missed in the point cloud, so as to supplement the target pose of the target obstacle in the missed frame.

[0049] In the embodiments of the present disclosure, the quality of the reference frame affects the iteration efficiency of optimization. In implementation, from the sequence of frames to be processed, an image whose object feature of the target obstacle satisfies a preset condition in terms of saliency can be selected as the reference frame. The object feature can refer to a feature that can describe the unique characteristics of the target obstacle, and the object feature is used to identify the target obstacle. The saliency of the object feature can be used to represent the degree to which the object feature can accurately describe the target obstacle.

[0050] Each image in the sequence of frames to be processed is a two-dimensional image. A two-dimensional image detection model can be used to detect the sequence of frames to be processed to obtain two-dimensional position box information of the target obstacle. The two-dimensional position box information includes the two-dimensional position box (i.e., the center point and the size) of the target obstacle in the corresponding frame image, the category, the detection confidence, and the like.

[0051] The image whose saliency satisfies the preset condition can be implemented as an image whose detection confidence of the target obstacle is the highest. Alternatively, an image whose quality is higher than a quality threshold and whose detection confidence is higher than a confidence threshold can be selected as the image satisfying the condition. An image whose size of the two-dimensional position box is the largest can also be selected as the image satisfying the condition. It can be understood that in implementation, as long as the image can stably and accurately describe the characteristics of the target obstacle.

[0052] In the embodiments of the present disclosure, based on the object feature of the target obstacle, an image whose saliency of the object feature satisfies a preset condition is selected as the reference frame, so that a high-quality image can be selected as the reference frame, thereby providing a good data basis for the iterative optimization of the projection relationship and improving the speed and accuracy of the iterative optimization.

[0053] For both static targets and dynamic targets, the reference frame can be selected based on the saliency of the object feature. In implementation, for dynamic targets, since the pose of the dynamic target can change, a frame closest to the frame to be optimized and containing the target obstacle can be preferentially selected as the reference frame. The requirement for the saliency of the characteristics of the target obstacle in the closest frame can be appropriately lowered.

[0054] In the embodiments of the present disclosure, the automatic labeling of dynamic targets is completed by virtue of the strong correlation in pose between the most recent frame and the frame to be optimized, so as to improve the accuracy of automatic labeling of dynamic targets.

[0055] In implementation, in the case that the target obstacle is a dynamic target, the frame image in which the three-dimensional detection model misses the target obstacle can be acquired as the frame to be optimized; the three-dimensional detection model is used for target detection on point cloud or two-dimensional image to obtain the target pose of the target obstacle. Then, a suitable reference frame can be selected according to the frame to be optimized.

[0056] For example, in the case that the target obstacle is a dynamic target, due to the improvement of the labeling distance, the point cloud presents a relatively sparse state or the image contains less information of the target obstacle, and the existing three-dimensional detection model lacks corresponding training data, so that the performance of the three-dimensional detection model is limited, and finally the dynamic target is missed in some frames. At this time, the frame image in which the target obstacle is missed in the embodiments of the present disclosure is determined as the frame to be optimized, so that on the basis of the detection result of the three-dimensional detection model, the automatic labeling method of the target obstacle provided in the embodiments of the present disclosure is implemented for the missed frame to continue processing and optimizing the missed frame, so as to improve the labeling capability for dynamic targets at long distance.

[0057] For static targets, the reference frame can be selected first, and then the frame to be optimized is determined, which can be implemented as:

[0058] Step A1, acquiring an optimization direction;

[0059] Step A2, selecting the frame to be optimized from the sequence of frames to be processed based on the reference frame and the optimization direction.

[0060] The optimization direction indicates whether to optimize forward or backward from the reference frame in the sequence of frames to be processed to complete automatic labeling.

[0061] In implementation, the frames before the reference frame and the frames after the reference frame can be compared in terms of the saliency of the object features of the target obstacle, and the direction with better saliency is selected as the optimization direction. For example, if the saliency of the object features in the frames before the reference frame is better than that in the frames after the reference frame, the direction from the reference frame forward is selected as the optimization direction. Otherwise, the direction from the reference frame backward is selected as the optimization direction.

[0062] In addition, the frame to be optimized can also be selected and optimized in the default optimization direction from the reference frame, and after multiple iterations of optimization, if the re-projection error is difficult to converge, the opposite direction of the default optimization direction can be selected as the optimization direction.

[0063] After the optimization direction is determined, a frame to be optimized is selected from the sequence of frames to be processed based on the reference frame and the optimization direction. For example, from the reference frame, a total of m frames are detected to include the target obstacle by the two-dimensional image detection model, and then one or more frames can be selected from the m frames as the frame to be optimized. Whether one or more frames to be optimized are selected, the implementation of optimizing the target parameter based on the re-projection error is the same for each frame.

[0064] In the embodiments of the present disclosure, in the case that the target obstacle is a static target, by obtaining the optimization direction and selecting the frame to be optimized based on the reference frame, the frames that need to be focused on in a specific optimization direction can be processed in a targeted manner, so as to complete the automatic labeling of the target obstacle.

[0065] In some embodiments, since the sequence of frames to be processed can be long, the number of frames to be processed each time is too large, and the calculation burden is also larger. Therefore, the time length can be set to limit the number of frames to be optimized each time, so as to improve the overall efficiency of automatic labeling.

[0066] For example, by the two-dimensional image detection model, it is detected that the nth frame to the mth frame in the sequence of two-dimensional BEV images all include the target obstacle, m is greater than n, and m and n are both positive integers. A frame with better feature saliency is selected as the reference frame. If (m-n) is large, (m-n) frames are all treated as frame images to be processed, which will consume more computing resources. Therefore, the time length can be set so that the number of frames to be processed for the target obstacle each time is within a reasonable range, so as to reasonably utilize the computing resources.

[0067] After the reference frame and the frame to be optimized are determined, in order to facilitate the optimization of the target parameter in the projection relationship, the true value of the projection point in the re-projection error can also be automatically determined. That is, the actual pixel point set of the target obstacle in the frame to be optimized.

[0068] In the embodiments of the present disclosure, for the frame to be optimized, the true value of the projection point required by the re-projection error is determined based on a pixel-level trajectory tracking method.

[0069] For each pixel point in the initial pixel point set of the target obstacle in the reference frame, a co-tracker (a kind of tracking algorithm) can be used to track these pixel points respectively, so as to obtain the trajectory points of the pixel points in other frame images, thereby obtaining the trajectory information of each pixel point. These trajectory points are used as the true value of the projection point on the corresponding frame to be optimized. For example, by pixel point trajectory tracking, a trajectory point set C1 of the target obstacle in the frame to be optimized A is obtained, and C1 is used as the true value of the projection point of frame A for subsequent determination of the re-projection error.

[0070] The pixel-level trajectory tracking technology can automatically determine the true value of the projection point required by the re-projection error, and can improve the optimization efficiency and accuracy of the target parameter.

[0071] In implementation, there can be many pixel points on the same target obstacle, but in order to improve the optimization efficiency of the target parameter, it is not necessary for each pixel point to participate in the optimization process of the target parameter. Therefore, in implementation, the pixel points of the target obstacle in the reference frame are sampled to obtain the initial pixel points of the target obstacle in the reference frame, so that a part of the pixel points can be omitted in the subsequent optimization process to improve the optimization efficiency. In the embodiment of the present disclosure, since the pixel points of the target obstacle need to be used for optimization, the two-dimensional detection box obtained by the two-dimensional image detection model can be used to determine the initial pixel point set of the target obstacle in the reference frame, and the implementation manner is as shown in the following figure: Figure 2

[0072] S201, obtaining a two-dimensional position box of a target obstacle detected by a two-dimensional image detection model from a reference frame.

[0073] S202, uniformly scattering points based on the two-dimensional position box to obtain an initial pixel point set of the target obstacle in the reference frame.

[0074] In the embodiment of the present disclosure, the detection result of the target obstacle obtained by the two-dimensional image detection model is reused to determine the initial pixel point set, which can improve the optimization efficiency.

[0075] In some embodiments, the pixel points in the two-dimensional position box do not necessarily belong to the target obstacle, and therefore, in order to further improve the optimization accuracy of the target parameter, in the embodiment of the present disclosure, step S202 can be implemented as:

[0076] S2021, obtaining a mask image of the target obstacle in the reference frame.

[0077] S2022, uniformly scattering points based on the two-dimensional position box of the target obstacle in the reference frame and the mask image to obtain the initial pixel point set.

[0078] Wherein, the target obstacle is segmented from the reference frame to obtain the mask image of the target obstacle.

[0079] In implementation, the reference frame can be processed by SAM (Segment Anything Model) to obtain the mask (MASK) image of the target obstacle. The reference frame is processed by the segment anything model, which can flexibly, efficiently and accurately generate high-quality segmentation masks to improve the quality of the initial pixel points and ultimately improve the labeling accuracy of the target obstacle.

[0080] ​In the case of uniform point scattering based on the two-dimensional position box of the target obstacle in the reference frame and the mask map, since not all the pixel points in the two-dimensional position box are on the target obstacle, the mask map is used to further limit the range of point scattering, thereby ensuring that each pixel point on the initial pixel point set belongs to the target obstacle. The two-dimensional position box can provide the approximate position and range of the initial pixel points of the target obstacle, thereby improving the point scattering efficiency. The mask map can provide the fine boundary of the initial pixel points of the target obstacle, and the two are used in combination to obtain the initial pixel point set with higher confidence of belonging to the target obstacle, so as to ensure that the tracked pixel points belong to the target obstacle, improve the accuracy of the initial pixel points, and thereby improve the optimization accuracy of the target parameters and the labeling accuracy of the target obstacle.

[0081] In some embodiments, the implementation of determining the re-projection error can include the following steps as shown in Figure 3

[0082] S301, mapping the initial pixel point set of the target obstacle in the reference frame to the to-be-optimized frame based on the projection relationship, to obtain a projection point set of the target obstacle in the to-be-optimized frame.

[0083] In the embodiments of the present disclosure, projecting the target obstacle into the to-be-optimized frame can be implemented as the steps shown in Figure 3

[0084] S3011, performing a back projection operation on the initial pixel point set of the target obstacle in the reference frame based on the pixel point depth parameter, to obtain a first three-dimensional space position of the target obstacle in the camera coordinate system; the pixel point depth parameter is used to represent the depth value of each initial pixel point in the reference frame.

[0085] The process of the back projection operation is shown in formula (1):

[0086]

[0087] In formula (1), pts 3d_camera to obtain the first three-dimensional space position of the target obstacle in the camera coordinate system; H is a homogenization operation; K is a camera intrinsic parameter; is the initial pixel point set of the target obstacle in the reference frame; pts depth is the pixel point depth parameter.

[0088] The pixel point depth parameter can be determined by iterative optimization, so an initial value is provided.

[0089] In implementation, in order to improve the optimization efficiency of the target parameters, a first preset value of the pixel point depth parameter can be obtained; and the first preset value is randomly disturbed to obtain the initial value of the pixel point depth parameter. ​​

[0090] In some possible implementation manners, the first preset value of the pixel depth parameter can be acquired according to prior knowledge, and then random noise is added to the first preset value to realize random disturbance. For example, Gaussian noise is added on the basis of the first preset value, or noise with uniform distribution is added to the first preset value within a certain range, so as to obtain the initial value of the pixel depth parameter.

[0091] In other possible implementation manners, the depth value of the target obstacle in any frame image detected from the three-dimensional detection model can also be used as the first preset value. The depth value in the frame image close to the reference frame can be preferentially selected, so that the initial pixel depth parameter is closer to the actual situation of the reference frame, thereby improving the convergence speed.

[0092] In the embodiments of the present disclosure, the initial value of the pixel depth parameter is randomly disturbed, so that diversified initial values can be generated, and the initial value of the pixel depth parameter can better adapt to different to-be-optimized frames. The problem that the optimization process may fall into a specific local mode when the initial value is fixed can be avoided. Moreover, the initial value after random disturbance can enable the optimization process to be performed in different initial states, and the generalization capability of the automatic obstacle labeling method provided by the embodiments of the present disclosure can be improved.

[0093] S3012, convert the first three-dimensional space position into the obstacle coordinate system based on the pose transformation parameter between the camera coordinate system and the obstacle coordinate system, to obtain a second three-dimensional space position.

[0094] That is, the first three-dimensional space position of the target obstacle in the camera coordinate system is converted from the camera coordinate system of the reference frame to the obstacle coordinate system by using the pose transformation parameter, and the process is as shown in formula (2):

[0095]

[0096] In formula (2), pts 3d_object is the second three-dimensional space position in the obstacle coordinate system; pts 3d_camera is the first three-dimensional space position of the target obstacle in the camera coordinate system; is the pose transformation parameter between the camera coordinate system and the obstacle coordinate system.

[0097] In the embodiments of the present disclosure, for a static target, because the target obstacle is static, pts is a constant; for a dynamic target, because the target obstacle is moving, pts has a respective value in each to-be-optimized frame.

[0098] It can also be understood that, in the case of a static target obstacle, the target parameters that need to be optimized in the projection relationship include the pixel point depth parameter.

[0099] Since the pose transformation parameter is a constant, it does not need to be recalculated or adjusted during the optimization process, which can make the optimization process only focus on the pixel point depth parameter, reduce the complexity of the calculation and the required time, and significantly improve the optimization efficiency of the static target.

[0100] Similarly, in the case of a dynamic target obstacle, since the target obstacle is moving, the pose transformation parameter is not fixed. Accordingly, the target parameters that need to be optimized in the projection relationship include the pixel point depth parameter and the pose transformation parameter. For a dynamic target, by optimizing the pixel point depth parameter and the pose transformation parameter, the key parameters required to determine the pose of the target obstacle can be accurately obtained to improve the labeling range of the dynamic target.

[0101] Since the pose transformation parameter also needs to be determined by iterative optimization for a dynamic target, an initial value needs to be provided for it.

[0102] In implementation, a second preset value of the pose transformation parameter can be obtained; the second preset value is randomly disturbed to obtain an initial value of the pose transformation parameter.

[0103] The second preset value is an initial setting of the pose transformation parameter, which can be obtained through some prior knowledge, experience, or preliminary analysis in a specific task scenario. Then the second preset value is randomly disturbed to obtain the initial value of the pose transformation parameter.

[0104] In some embodiments, the initial setting of the pose transformation parameter can be determined by formula (3):

[0105]

[0106] In formula (3), is the initial setting of the pose transformation relationship from the obstacle coordinate system to the camera coordinate system at the reference frame time, which is the inverse matrix of the pose transformation parameter between the camera coordinate system and the obstacle coordinate system, so It can be used as the second preset value of the pose transformation parameter that needs to be optimized, and after optimization, the inverse matrix thereof can be calculated to obtain the pose transformation parameter; is the position transformation relationship between the world coordinate system and the camera coordinate system at the reference frame time; is the pose of the obstacle coordinate system relative to the world coordinate system at the reference frame time.

[0107] In the embodiments of the present disclosure, the initial value of the pose transformation parameter obtained by randomly perturbing the second preset value can adapt to different to-be-optimized frames, and can avoid the problem that a specific local mode may be fallen into due to a fixed initial value. The initial value after random perturbation can enable the optimization process to be performed in different initial states, and can improve the generalization ability of the automatic obstacle labeling method provided by the embodiments of the present disclosure.

[0108] In S3013, the second three-dimensional space position is projected into the to-be-optimized frame to obtain a set of projection points.

[0109] Based on the foregoing content, in the obstacle coordinate system established with the target obstacle, the positions of the target obstacle in different frames are uniform. The projection operation in any i-th frame (i.e., any to-be-optimized frame) is realized by using the camera intrinsic parameter K to obtain the set of projection points of the target obstacle in the i-th frame (i.e., the to-be-optimized frame). The specific implementation can be realized by formula (4):

[0110]

[0111] In formula (4), is the set of projection points; K is the camera intrinsic parameter; pts 3d_object is the second three-dimensional space position of the target obstacle in the obstacle coordinate system; is the pose transformation parameter between the camera coordinate system of the to-be-optimized frame and the obstacle coordinate system.

[0112] In the embodiments of the present disclosure, by means of the back projection and projection operation, the set of projection points of the reference frame in the to-be-optimized frame can be more accurately obtained.

[0113] In S302, the re-projection error is determined based on the set of projection points and the ground truth of the projection points of the to-be-optimized frame.

[0114] The ground truth of the projection points has been described in the foregoing content, and will not be described here again. The re-projection error can be as shown in formula (5):

[0115]

[0116] In formula (5), loss proj represents the re-projection error; Huber is a loss function combining the advantages of mean square error (MSE) and absolute error (MAE); is the set of projection points of the target obstacle in the to-be-optimized frame; pts 2d_gt is the ground truth of the projection points of the target obstacle in the to-be-optimized frame.

[0117] In the embodiments of the present disclosure, by comparing the difference between the set of projection points and the true value of the projection points, the accuracy and reliability of the projection relationship established based on the current target parameter can be evaluated. In the case of small reprojection error, it indicates that the projection effect better reflects the true situation, which helps to improve the optimization accuracy of the target parameter, thereby improving the accuracy of automatically labeling the pose of the target obstacle.

[0118] In the embodiments of the present disclosure, considering the depth parameter pts depth may not be convergent, so a depth regularization term can be added to ensure normal convergence of the loss.

[0119] Wherein, the depth regularization term is used to reduce the dispersion degree of the depth of each point of the target obstacle. In this way, the problem of inaccurate depth estimation of each point caused by the large dispersion degree of the depth of each point of the target obstacle is solved. The depth regularization term is shown in formula (6):

[0120]

[0121] In formula (6), loss norm is the loss of the depth regularization term; pts depth_i is the depth value of the i th pixel point in the current depth parameter (pts depth ) of the pixel point; and mean(pts depth ) is the mean of the current depth values of all initial pixel points on the target obstacle.

[0122] For example, after the n th round of optimization is completed, the depth value after the n th round of optimization is used to determine the loss of the depth regularization term to perform the n+1 th round of target parameter optimization process, wherein n is a positive integer.

[0123] Then, the total projection loss can be determined based on the reprojection error and the depth regularization term of the projection point, and the target parameter in the projection relationship can be optimized based on the total projection loss. The total projection loss is shown in formula (7):

[0124] loss=w1*loss proj +w2*loss norm (7)

[0125] In formula (7), loss is the total projection loss; loss proj represents the reprojection error; w1 is the weight coefficient corresponding to the reprojection error; loss norm is the loss of the depth regularization term; and w2 is the weight coefficient corresponding to the depth regularization term.

[0126] In the embodiments of the present disclosure, the reprojection error can measure the position difference between the projected pixel point and the actual point (ground truth), and the depth regularization term considers the rationality of the pixel depth information. By combining the two to determine the total projection loss, the quality of the projection can be more comprehensively evaluated, and the convergence speed can be improved.

[0127] In summary, in order to facilitate implementation, the obstacle coordinate system of the target obstacle can be established based on the world coordinate system.

[0128] The world coordinate system is a global fixed coordinate system used to describe the positions and postures of all objects in the entire environment. The obstacle coordinate system is a local coordinate system centered on the target obstacle, used to describe the pose change of the target obstacle itself. The obstacle coordinate system is established based on the world coordinate system, which can be established by moving the origin of the world coordinate system to the position of the obstacle in the world coordinate system.

[0129] For a static target, the pose transformation parameters of the target obstacle from the camera coordinate system to the obstacle coordinate system of each frame to be optimized are fixed and unchanged. That is, under normal circumstances, the target obstacle is unchanged in the world coordinate system.

[0130] However, for a dynamic target, it usually changes in the world coordinate system due to its moving characteristics. Therefore, a transition term can be superimposed on the initial value of the pose transformation parameters; the transition term is used to express the relative change of the target obstacle between the reference frame and the frame to be optimized, and the optimization of the pose transformation parameters is realized by optimizing the transition term.

[0131] That is, the initial value of the pose transformation parameter establishes a relationship between the dynamic target and the world coordinate system, but this relationship will change with the movement of the dynamic target. Therefore, in the embodiments of the present disclosure, this change can be expressed in the transition term for optimization. When optimizing the pose transformation parameter, only the parameters in the transition term need to be optimized. For the i-th frame, i.e., the frame to be optimized, the relationship between the obstacle coordinate system and the camera coordinate system is shown in formula (8):

[0132]

[0133] In formula (8), is the inverse matrix of the pose transformation parameter corresponding to the i-th frame, i.e., the frame to be optimized; is the pose transformation of the target obstacle in the world coordinate system to the camera coordinate system at the i-th frame; is the pose transformation of the obstacle coordinate system relative to the world coordinate system at the i-th frame (i.e., the transition term), which can be determined by the pose of the i-th frame. Finally, combined with the result of the reference frame in expression (3), the inverse matrix of the position transformation parameter superimposed with the transition term can be initialized as expression (9):

[0134]

[0135] wherein, For representing rotation, can be calculated based on expression (3) determined;

[0136] V x represents the velocity component of the target obstacle in the x direction; V y represents the velocity component of the target obstacle in the y direction; V y represents the velocity component of the target obstacle in the z direction; (0 0 0 1) represents a homogenization operation; and Δt represents the time difference between the frame to be optimized and the reference frame.

[0137] It can be understood that after the pose transformation relationship corresponding to the optimization expression (9) is solved, the inverse matrix thereof is solved, and the optimized position transformation parameter can be obtained.

[0138] In the embodiments of the present disclosure, the obstacle coordinate system is established based on the world coordinate system, which can more conveniently describe and track the motion of dynamic targets. By superimposing the transfer term on the initial value of the pose transformation parameter, the motion change of the dynamic target between different frames can be flexibly and simply described, and the efficiency of automatic labeling is improved.

[0139] In the embodiments of the present disclosure, for a static target, the detection result obtained by a two-dimensional image detection model performing target detection on the frame sequence to be processed can be acquired; and based on the detection result, a candidate object belonging to the target category is selected as the target obstacle.

[0140] The two-dimensional image detection model is usually based on deep learning technology, such as a convolutional neural network (CNN). The CNN has the ability to automatically extract image features, and through the combination of multiple convolutional layers, pooling layers and fully connected layers, various targets in the input image can be learned and classified. In implementation, a Yolo model can be used as the two-dimensional image detection model.

[0141] For each frame image in the frame sequence to be processed, the two-dimensional image detection model will perform independent processing, and output the position of each detected candidate object in the image in the form of a two-dimensional position box. The information of the two-dimensional position box can include parameters such as a center point, a size, a category, a detection confidence, a tracking ID (Identification, identification), etc. Then, based on the detection result, a candidate object belonging to the target category is selected as the target obstacle. For example, some small-sized obstacles are filtered out, and the remaining candidate objects that cannot be accurately detected by a conventional three-dimensional detection model or new categories of static targets are selected as the target obstacles in the embodiments of the present disclosure. Specifically, low stone piers, lamp poles, etc.

[0142] In the embodiments of the present disclosure, the detection result of the two-dimensional image detection model on the target detection of the to-be-processed frame sequence can efficiently and accurately obtain the static target class that needs to be labeled. Therefore, the target parameters in the projection relationship can be optimized to realize the automatic detection of the static target. In combination with the three-dimensional detection model, the detection efficiency of the target obstacle can be effectively improved. In the field of automatic driving, the labeling range can be improved, the types of static targets that can be automatically labeled can be increased, and the defect that the three-dimensional detection model cannot automatically label sparse or low obstacles can be overcome.

[0143] In some embodiments, in the case that the target obstacle is a dynamic target, the target pose of the target obstacle is determined based on the pose transformation parameter in the optimized target parameter, the camera intrinsic parameter, and the three-dimensional position box of the target obstacle in the reference frame. The process is shown in formula (10):

[0144]

[0145] In formula (10), obj i is the target pose of the to-be-optimized frame; obj init is the 3D box detected from the reference frame based on the three-dimensional detection model; is the optimized pose transformation parameter; is the target pose of the target obstacle obtained by optimization; is obtained by inverse operation.

[0146] In the embodiments of the present disclosure, the target pose of the target obstacle in the to-be-optimized frame can be estimated by referring to the three-dimensional position box of the reference frame. In the case that the to-be-optimized frame has a long range and the point cloud is sparse, the labeling range and labeling effect of the dynamic target can be improved.

[0147] Correspondingly, in the case that the target obstacle is a static target, the initial pixel point of the target obstacle in the reference frame is converted into the camera coordinate system based on the optimized target parameter to obtain a first point group of the target obstacle in the camera coordinate system; and a plane estimation method is used to process the first point group to obtain the target pose of the target obstacle in the reference frame.

[0148] In the embodiments of the present disclosure, the plane estimation method is used to determine the target pose of the target obstacle, which can be implemented by first performing a denoising operation on the first point group of the target obstacle in the camera coordinate system to remove noise in the first point group to obtain a first intermediate point set. The denoising operation can remove outliers and optimize the quality of the first point group. Then, for the first intermediate point set, the inner points, the plane and the normal vector of the first intermediate point group can be estimated by using the random sample consensus (RANSAC) algorithm, so as to estimate the center point, the angle and the size of the static target to obtain the target pose of the static target in the reference frame.

[0149] In the embodiments of the present disclosure, for a static target, the target pose of the target obstacle in the reference frame can be determined through plane estimation to realize automatic labeling of the static target.

[0150] To further improve the detection efficiency of the target obstacle in other frames, the target pose of the target obstacle in other frames can also be obtained based on a global back projection strategy. Specifically, the method can be implemented as follows:

[0151] The target pose of the target obstacle in the reference frame is converted into the pose of the target obstacle in the target frame by using the global back projection strategy.

[0152] The target frame is a frame to be optimized or an image frame other than the frame to be optimized, which needs to estimate the pose of the target obstacle.

[0153] That is, the positions of all points of the target obstacle in the 3D space are obtained by optimizing the aforementioned object-based beam adjustment method, the size, orientation and position of the target obstacle are obtained by using plane estimation, and the pose of the target obstacle in each target frame is obtained by pose transformation, such as transformation from the world coordinate system to the camera coordinate system.

[0154] In the target frame, the size and position of the target obstacle can be directly obtained from the world coordinate system, and the orientation can be determined by the camera pose.

[0155] In the embodiments of the present disclosure, the range of vision can be improved by using the global back projection strategy for a static target at a long distance.

[0156] In some embodiments, after obtaining the target pose of the target obstacle in each frame image, the target obstacle can be tracked based on the target pose of the target obstacle in different frame images to obtain the trajectory information of the target obstacle.

[0157] The automatic labeling method of the obstacle provided in the embodiments of the present disclosure can be used as an aid for a three-dimensional detection model. Therefore, in implementation, the target pose can be obtained by using the three-dimensional detection model, and for the target obstacle that cannot be detected by the three-dimensional detection model, the automatic labeling can be completed by using the method provided in the embodiments of the present disclosure. Regardless of the method, the target pose of the target obstacle will be obtained. The results of the two methods are summarized to obtain the target pose of the target obstacle at all times. Then, the target obstacle is tracked based on the target tracking technology and the Kalman filtering technology in the world coordinate system to associate the front and back frame images in which the target obstacle is located, so as to obtain the trajectory information of the target obstacle.

[0158] The Kalman filter can predict the position and speed of the target obstacle and correct it according to the observation data. This is very effective for handling occlusion or noise interference, and can improve the accuracy of tracking.

[0159] In target tracking, Kalman filtering is used to model the motion trajectory of target obstacles. By modeling the motion of target obstacles, Kalman filtering can predict the position of target obstacles at the next time instant in real time.

[0160] In summary, Kalman filtering is a very important tool in the field of target tracking, which significantly improves the performance and robustness of target tracking by providing accurate state prediction and correction.

[0161] In the embodiments of the present disclosure, by tracking the trajectory of the target obstacle, the trajectory of the target obstacle at different frame instants can be analyzed in depth, providing data basis for subsequent use.

[0162] In the embodiments of the present disclosure, after obtaining the trajectory information of the target obstacle, the 2D detection result can be associated with the three-dimensional detection result, mainly used to correct the category of the three-dimensional detection model. For example, the three-dimensional position box of some frames in the trajectory information is detected based on the three-dimensional detection model, and the category thereof may have errors. In implementation, for the to-be-corrected frame of the trajectory information, based on the two-dimensional position box of the target obstacle in the to-be-corrected frame, the three-dimensional position box of the target obstacle in the to-be-corrected frame is determined; wherein the two-dimensional position box is obtained by detecting the target obstacle by using the two-dimensional image detection model; and based on the classification result of the two-dimensional position box, the classification result of the three-dimensional position box of the to-be-corrected frame is corrected.

[0163] The three-dimensional position box of the to-be-corrected frame is the result detected by the three-dimensional detection model. If the classification accuracy of the three-dimensional detection model is lower than that of the two-dimensional image detection model, the category thereof may have errors. Therefore, for the target obstacle in the same to-be-corrected frame, the two-dimensional position box obtained by the two-dimensional image detection model can be compared with the three-dimensional position box obtained by the three-dimensional detection model to obtain the Intersection over Union (IoU) of the two-dimensional position box and the three-dimensional position box. If the IoU meets a preset condition, it is confirmed that the three-dimensional position box corresponding to the two-dimensional position box in the same to-be-corrected frame belongs to the same target obstacle.

[0164] Then, the classification categories of the two-dimensional position box and the three-dimensional position box can be associated to obtain consistent category information of the trajectory. That is, in the case where the detection category of the three-dimensional position box and the detection category of the two-dimensional position box are inconsistent, the classification result of the three-dimensional position box is corrected based on the classification result of the two-dimensional position box. It can be implemented that the categories of the two-dimensional position box of the target obstacle in different frame images are obtained, and the category with a classification confidence greater than a confidence threshold is selected as the final category of the target obstacle. If there are multiple categories with a classification confidence greater than a confidence threshold, the category with the highest average classification confidence is preferentially selected as the final category.

[0165] If the category of the three-dimensional position box is inconsistent with the final category, the category of the three-dimensional position box can be corrected to the final category.

[0166] In the embodiments of the present disclosure, based on the classification result of the two-dimensional position box, the classification result of the three-dimensional position box in the frame to be corrected is corrected. When the two-dimensional and three-dimensional detection results are inconsistent, the two-dimensional detection result can be used to correct the three-dimensional detection result, and the type accuracy of target obstacle detection is improved.

[0167] In the embodiments of the present disclosure, the confidence of the trajectory information can also be determined to filter the false detection; the confidence is determined based on at least one of the following information: the confidence of the two-dimensional position box of the target obstacle recognized by the two-dimensional image detection model, the occlusion rate of the target obstacle, and the intersection over union; the intersection over union is the intersection over union between the two-dimensional position box and the three-dimensional position box of the target obstacle; in the case where the confidence is lower than a target threshold, the trajectory information is determined as a false detection trajectory.

[0168] The confidence of the two-dimensional position box of the target obstacle recognized by the two-dimensional image detection model is an evaluation index of the accuracy of the two-dimensional image detection model for the detected two-dimensional position box.

[0169] The occlusion rate of the target obstacle represents the degree of occlusion of the target obstacle by other objects in each frame. In implementation, the occlusion in the current frame can be judged in combination with the historical trajectory of the target obstacle.

[0170] The intersection over union is the intersection over union between the two-dimensional position box of the target obstacle detected by the two-dimensional image detection model and the three-dimensional position box detected by the three-dimensional detection model.

[0171] In implementation, the confidence of the trajectory information of the target obstacle can be calculated by using a weighted average. The weight coefficients of each kind of information can be adjusted according to actual needs.

[0172] When the confidence of the trajectory information is greater than a set threshold, the trajectory is considered as a valid trajectory, otherwise, it is likely to be a false detection trajectory. The false detection trajectory is filtered, and otherwise, it is retained.

[0173] In the embodiments of the present disclosure, the confidence of the trajectory information is determined based on at least one of the two-dimensional position box confidence, the occlusion rate, and the intersection over union. When the above multiple information is fused, the reliability of target detection can be evaluated from multiple angles, and thus the confidence of the trajectory is higher. Considering these factors comprehensively can more accurately locate the real target obstacle and avoid treating the false detection target obstacle as the real target obstacle.

[0174] In summary, taking the field of autonomous driving as an example, the automatic obstacle labeling method provided in the embodiments of the present disclosure can improve the visibility of obstacle labeling, such asFigure 4 as shown:

[0175] S401, acquire a point cloud frame or a BEV frame.

[0176] S402, input the acquired point cloud frame or BEV frame into a corresponding three-dimensional detection model.

[0177] wherein the point cloud frame adopts a three-dimensional detection model corresponding to the point cloud, such as DSVT (Dynamic Sparse Voxel Transformer); and the two-dimensional BEV adopts a three-dimensional detection model corresponding to the two-dimensional BEV, such as PETR (Position Embedding Transformation and Refinement).

[0178] S403, output an initial 3D box bbox_3d of a target obstacle by the three-dimensional detection model, including 3D center point, orientation, size and category of the obstacle and the like.

[0179] S404, input the BEV frame into a two-dimensional image detection model.

[0180] S405, output a 2D box (bbox_2d) of a candidate obstacle by the two-dimensional image detection model, including 2D center point, size, category and confidence of the obstacle and the like.

[0181] S406, filter out the obstacle with a 2D box size smaller than a preset size from the candidate obstacle, and select a static target of a target category from the remaining candidate obstacle as a target obstacle, so as to request the BA service to optimize the target parameter in the projection relationship, and realize a conversion from bbox_2d to bbox_3d.

[0182] That is, the target pose of the target obstacle in the reference frame is obtained through the optimization process of the projection relationship described by the foregoing formula, and the target pose of the target obstacle in other frames can be obtained through a global back projection strategy.

[0183] S407, after obtaining the bbox_3d of all frames, the target tracking module can be used to correlate the target obstacles in different frames through Kalman filtering and target tracking technology, so as to obtain the trajectory information of the target obstacle.

[0184] S408, based on the correlation of the 2D and 3D detection results, modify the category of the target obstacle in the bbox_3d, and obtain the category information consistent with the trajectory generation.

[0185] In addition, not only static targets can realize automatic labeling and range enhancement, but also the disclosed embodiments can enhance the labeling range of dynamic targets.

[0186] As in S409, it can be determined that the three-dimensional detection model misses the detection frame of the dynamic target in the point cloud frame or the BEV frame, and the missed detection frame is taken as the frame to be optimized. As can be determined whether it is missed based on the continuous detection of the three-dimensional position box of the same target obstacle.

[0187] If it is missed for the point cloud frame, the BEV frame collected synchronously with the point cloud frame can be obtained to realize the range enhancement of the dynamic target. If it is missed for the BEV frame, the range enhancement of the dynamic target can be realized on the basis of the BEV frame. Specifically, for the missed frame, the BEV frame with the latest frame detected by the three-dimensional detection model is selected as the reference frame, and then the BA service is called for optimization based on the trajectory information of the initial pixel points of the reference frame and the two-dimensional position box.

[0188] S410, requesting the BA service to optimize the projection relationship on the basis of the reference frame, obtaining the optimized pose transformation parameter and pixel point depth parameter, and then obtaining the target pose of the dynamic target in the frame to be optimized based on the position transformation parameter.

[0189] S411, based on the false detection filtering module, filtering the false detection trajectory information.

[0190] S412, outputting the labeling result.

[0191] In summary, the BA service optimization process is as shown in Figure 5 , which includes a preprocessing module, an optimization module, and an output module.

[0192] In the preprocessing module, the received request is preprocessed to obtain the following request information, mainly including 8 categories:

[0193] (1) dynamic and static identification, used to mark whether the request is for a dynamic target or a static target, which affects the optimization process of OCBA, that is, whether the target parameter to be optimized includes the pose transformation parameter;

[0194] (2) obstacle list, embodied in the form of obstacle trajectory ID, which is one-to-one corresponding to the subsequent information as a key; in implementation, multiple target obstacles can be requested to be labeled, and each obstacle to be automatically labeled is stored in the list.

[0195] (3) starting frame (also known as reference frame), marking the frame from which each obstacle starts to be optimized; in implementation, a frame with significant features is selected as the reference frame of the target obstacle;

[0196] (4) 2D box of the starting frame, marking the two-dimensional position box of the target obstacle in the starting frame;

[0197] (5) optimization direction, marking whether the target obstacle is optimized forward or backward;

[0198] (6) the number of frames to be optimized, indicating how many frames of the target obstacle are involved in the optimization process;

[0199] (7) pose information, including the pose transformation matrix of the camera to the world coordinate system, the transformation matrix of the camera to the radar, etc.

[0200] (8) OCBA related parameter settings, such as the length of the time sequence, the depth noise variance, the matrix variance, etc.

[0201] As described above, the length of the time sequence is used to reduce the occupation of resources, and the length of the time sequence is less than or equal to the number of frames to be optimized. The depth noise variance is used to randomly disturb the initial value of the pixel point depth. The depth noise variance can include the mean and variance of the Gaussian distribution, so as to randomly sample noise from the Gaussian distribution to randomly disturb the pixel point depth parameter. The matrix variance is used to randomly disturb the pose transformation parameter.

[0202] After obtaining the reference frame in the sequence to be processed, the SAM model is used to segment the target obstacle, so as to obtain the MASK graph of the target obstacle. Then, based on the 2D frame and the MASK graph of the target obstacle, uniform point scattering is performed to obtain the initial pixel point of the target obstacle.

[0203] Then, based on the initial pixel point of the target obstacle, the co-tracker is used to track these points, so as to obtain a series of trajectory information of the initial pixel point in each frame to be optimized. The trajectory of the point on the tracking is used as the true value of the pixel point in the frame to be optimized for optimizing the related target parameters.

[0204] The optimization module includes DataLoader, Optimizer and Visualizer. DataLoader provides data for each optimization, and after optimization, Visualizer is used for effect verification.

[0205] DataLoader provides the pixel position pts of the 2D pixel point of the reference frame 2d_init , the true value pts of the 2D pixel point of the frame to be optimized 2d_gt , the initial value pts of the pixel point depth parameter of the camera intrinsic parameter K depth , and the initial value T of the pose transformation parameter between the camera coordinate system and the obstacle coordinate system c2o .

[0206] The optimization process of Optimizer is a process of continuous iteration and optimization. First, the initial pixel point pts is used for back projection operation 2d_init , the camera intrinsic parameter K and the pixel point depth parameter pts depth to obtain the back projection result pts 3d_camera, which is the 3D space position point corresponding to the initial 2D pixel point of the target obstacle in the reference frame in the camera coordinate system.

[0207] Then, the 3D space position point is converted from the camera coordinate system of the reference frame to the point pts in the obstacle coordinate system. 3d_object Since the points of the obstacle in the obstacle coordinate system are uniform, the point pts can be directly converted from the object coordinate system to the point in the camera coordinate system of any i-th frame. At the same time, the projection operation in any i-th frame is realized by using the intrinsic parameter K to obtain the 2D pixel coordinates of the obstacle in the i-th frame. Through the OCBA, the 2D pixel positions of the initial frame in other frames are obtained

[0208] Further, the re-projection error is used to optimize the process, and a depth regularization term is added to ensure the normal convergence of the loss. depth

[0209] Finally, the entire OCBA optimization process is realized through loss backpropagation and gradient descent update, and the optimized pts depth and T c2o are obtained.

[0210] Figure 5 The Visualizer in the output module can more intuitively display the labeling results after optimization, for example, the automatically labeled bbox_3d can be displayed in the two-dimensional image of the low stone pier.

[0211] In the output module, for the static target, the back-projection result pts 3d_camera can be obtained by using the optimized pts depth , pts 2d_init , K, T c2o and other parameter sets, and then the target pose of the static target obstacle is obtained based on plane estimation. In implementation, the corresponding pixel depth parameter pts depth ​The 3D space position of the point cloud of the target obstacle in the camera coordinate system can be obtained through the back projection operation of formula (1). The points are denoised and outliers are removed by using a denoising algorithm such as DBSACN. The inliers, planes and normal vectors of the point cloud are estimated by using a random sample consensus (RANSAC) algorithm, so as to estimate the center point, angle and size of the static target. For other frames, since the global position of the static obstacle is uniform, the current target obstacle position can be back projected to other frames by using a global back projection strategy, that is, the obstacle in the current frame is converted to the global coordinate system (i.e., the world coordinate system), and then converted to each frame, so as to effectively realize the labeling of the static obstacle.

[0212] For a dynamic target, the target pose of the dynamic target in the to-be-optimized frame is obtained based on the optimized 3D box obj init , and pts 2d_init , K, T c2o , obj init and other parameter sets detected from the reference frame based on the optimized three-dimensional detection model.

[0213] In summary, the solution proposed in the present disclosure makes up for the defects of the limited scanning range of the existing radar and the existing automatic labeling scheme, realizes the improvement of the static target detection and dynamic target labeling range without relying on a three-dimensional detection model, can support the labeling of various static targets such as low stone piers and lamp poles, and improves the labeling range of dynamic targets to a farther distance. The labeling data can be used to drive the BEV obstacle detection model at the vehicle end, greatly improves the model labeling capability, and realizes timely obstacle avoidance of the host vehicle.

[0214] Based on the same technical concept, the present disclosure also provides an automatic obstacle labeling device 600, as shown in Figure 6 , comprising:

[0215] An optimization module 601 is configured to optimize target parameters in a projection relationship based on a re-projection error, the projection relationship being used to project a target obstacle from a reference frame to a to-be-optimized frame, and the projection relationship satisfying the following constraint condition: in an obstacle coordinate system established by the target obstacle, the positions of the target obstacle in different frames are uniform.

[0216] A determination module 602 is configured to determine a target pose of the target obstacle based on the optimized target parameters.

[0217] In some embodiments, the optimization module comprises:

[0218] A projection unit is configured to map an initial pixel point set of the target obstacle in the reference frame to the to-be-optimized frame based on the projection relationship, to obtain a projection point set of the target obstacle in the to-be-optimized frame.

[0219] an error determination unit configured to determine a reprojection error based on the set of projection points and ground truth projection points of the frame to be optimized.

[0220] In some embodiments, the optimization module comprises:

[0221] a loss determination unit configured to determine a total projection loss based on the reprojection error and a depth regularization term of the projection points;

[0222] an optimization unit configured to optimize the target parameter in the projection relationship based on the total projection loss.

[0223] In some embodiments, the depth regularization term is configured to reduce a discrete degree of the depth of each pixel point of the target obstacle.

[0224] In some embodiments, the ground truth projection points required by the reprojection error for the frame to be optimized are determined based on a pixel-level trajectory tracking method.

[0225] In some embodiments, the projection unit comprises:

[0226] a back-projection sub-unit configured to perform a back-projection operation on the initial set of pixel points of the target obstacle in the reference frame based on pixel depth parameters to obtain a first three-dimensional spatial position of the target obstacle in a camera coordinate system, wherein the pixel depth parameters are configured to represent depth values of each initial pixel point in the reference frame;

[0227] a conversion sub-unit configured to convert the first three-dimensional spatial position to a second three-dimensional spatial position in an obstacle coordinate system based on a pose transformation parameter between the camera coordinate system and the obstacle coordinate system;

[0228] a projection sub-unit configured to project the second three-dimensional spatial position into the frame to be optimized to obtain the set of projection points.

[0229] In some embodiments, in a case where the target obstacle is a static target, the target parameter comprises the pixel depth parameters.

[0230] In some embodiments, in a case where the target obstacle is a dynamic target, the target parameter comprises the pixel depth parameters and the pose transformation parameter.

[0231] In some embodiments, the optimization module comprises:

[0232] a first acquisition unit configured to acquire a first preset value of the pixel depth parameters;

[0233] a first perturbation unit configured to randomly perturb the first preset value to obtain an initial value of the pixel depth parameters.

[0234] In some embodiments, the optimization module comprises:

[0235] The second acquisition unit is configured to acquire a second preset value of the pose transformation parameter.

[0236] The second disturbance unit is configured to disturb the second preset value randomly to obtain an initial value of the pose transformation parameter.

[0237] In some embodiments, the obstacle coordinate system is established based on the world coordinate system.

[0238] In a case where the target obstacle is a dynamic target, the pose transformation parameter has a transition term superimposed on the initial value of the pose transformation parameter; the transition term is used to express the relative change of the target obstacle between the reference frame and the frame to be optimized, and the optimization of the pose transformation parameter is achieved by optimizing the transition term.

[0239] In some embodiments, the projection unit is specifically configured to:

[0240] acquire a two-dimensional position box of the target obstacle detected by the two-dimensional image detection model from the reference frame;

[0241] perform uniform point scattering based on the two-dimensional position box to obtain an initial pixel point set of the target obstacle in the reference frame.

[0242] In some embodiments, the projection unit is specifically configured to:

[0243] acquire a mask image of the target obstacle in the reference frame;

[0244] perform uniform point scattering based on the two-dimensional position box and the mask image of the target obstacle in the reference frame to obtain the initial pixel point set.

[0245] In some embodiments, the projection unit is specifically configured to:

[0246] segment the target obstacle from the reference frame by using a segmentation-all model to obtain a mask image of the target obstacle.

[0247] In some embodiments, the projection unit is specifically configured to:

[0248] from the sequence of frames to be processed, based on the object features of the target obstacle, screen images whose object feature saliency satisfies a preset condition as the reference frame.

[0249] In some embodiments, the projection unit is specifically configured to:

[0250] in a case where the target obstacle is a static target, acquire an optimization direction;

[0251] based on the reference frame and the optimization direction, select a frame to be optimized from the sequence of frames to be processed.

[0252] In some embodiments, the projection unit is specifically configured to:

[0253] In a case where the target obstacle is a dynamic target, a frame image in which the three-dimensional detection model misses detection of the target obstacle is obtained as an optimization frame; the three-dimensional detection model is used for target detection on a point cloud or a two-dimensional image to obtain a target pose of the target obstacle.

[0254] In some embodiments, the projection unit is specifically configured to:

[0255] In a case where the target obstacle is a dynamic target, a frame closest to the optimization frame and containing the target obstacle is preferentially selected as a reference frame.

[0256] In some embodiments, the projection unit is specifically configured to:

[0257] Obtaining a detection result obtained by a two-dimensional image detection model performing target detection on the frame sequence;

[0258] Based on the detection result, a candidate object belonging to a target category is selected as a target obstacle.

[0259] In some embodiments, the reference frame and the optimization frame are bird's eye views in autonomous driving.

[0260] In some embodiments, further comprising a trajectory tracking module configured to:

[0261] Based on the target pose of the target obstacle in different frame images, the trajectory tracking module is configured to track a trajectory of the target obstacle to obtain trajectory information of the target obstacle.

[0262] In some embodiments, further comprising a correction module configured to:

[0263] For the trajectory information to be corrected frame, the correction module is configured to determine a three-dimensional position box of the target obstacle in the to-be-corrected frame based on a two-dimensional position box of the target obstacle in the to-be-corrected frame; the two-dimensional position box is obtained by detecting the target obstacle by the two-dimensional image detection model.

[0264] Based on the classification result of the two-dimensional position box, the correction module is configured to correct a classification result of the three-dimensional position box of the to-be-corrected frame.

[0265] In some embodiments, further comprising a filtering module configured to:

[0266] The filtering module is configured to determine a confidence of the trajectory information; the confidence is determined based on at least one of the following information: a confidence of a two-dimensional position box of the target obstacle recognized by the two-dimensional image detection model, an occlusion rate of the target obstacle, and an intersection over union; the intersection over union is an intersection over union between the two-dimensional position box and a three-dimensional position box of the target obstacle.

[0267] In a case where the confidence is lower than a target threshold, the filtering module is configured to determine that the trajectory information is a false detection trajectory.

[0268] In some embodiments, the determining module is specifically configured to:

[0269] In a case where the target obstacle is a dynamic target, based on the pose transformation parameter in the optimized target parameter, the camera intrinsic parameter, and the three-dimensional position box of the target obstacle in the reference frame, a target pose of the target obstacle in the frame to be optimized is determined.

[0270] In some embodiments, the determining module is specifically configured to:

[0271] In a case where the target obstacle is a static target, an initial pixel point of the target obstacle in the reference frame is converted into the camera coordinate system based on the optimized target parameter, to obtain a first point group of the target obstacle in the camera coordinate system;

[0272] The first point group is processed by using a plane estimation device, to obtain a target pose of the target obstacle in the reference frame.

[0273] In some embodiments, the determining module is further configured to:

[0274] The target pose of the target obstacle in the reference frame is converted into a pose of the target obstacle in the target frame by using a global back projection strategy;

[0275] The target frame is the frame to be optimized or an image frame other than the frame to be optimized, which needs to estimate the pose of the target obstacle.

[0276] The specific functions and examples of the modules and sub-modules of the device of the embodiments of the present disclosure are described in the above method embodiments, and will not be described here.

[0277] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0278] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0279] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0280] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0281] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0282] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the automatic obstacle labeling method. For example, in some embodiments, the automatic obstacle labeling method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the automatic obstacle labeling method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the automatic obstacle labeling method by any other suitable means (e.g., by means of firmware).

[0283] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0284] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0285] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0286] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0287] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0288] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0289] It should be understood that various forms of flow shown above can be used, with steps reordered, added, or removed. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without limitation herein, so long as the desired results of the technology disclosed in the present disclosure are achieved.

[0290] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the principles of the disclosure. Accordingly, modifications, equivalent alternatives and improvements should be included within the scope of the disclosure.

Claims

1. An automatic obstacle labeling method, comprising: optimizing a target parameter in a projection relationship based on a re-projection error; the projection relationship is used to project a target obstacle from a reference frame to a frame to be optimized; the projection relationship satisfies the following constraint condition: in an obstacle coordinate system established based on the target obstacle, the positions of the target obstacle in different frames are uniform; determining a target pose of the target obstacle based on the optimized target parameter; determining the re-projection error, comprising: based on a pixel point depth parameter, performing a back projection operation on an initial pixel point set of the target obstacle in the reference frame to obtain a first three-dimensional space position of the target obstacle in a camera coordinate system; the pixel point depth parameter is used to represent the depth value of each initial pixel point in the reference frame; based on a pose transformation parameter between the camera coordinate system and the obstacle coordinate system, converting the first three-dimensional space position to the obstacle coordinate system to obtain a second three-dimensional space position; projecting the second three-dimensional space position into the frame to be optimized to obtain the set of projection points; based on the set of projection points and the true value of the projection points of the frame to be optimized, determining the re-projection error.

2. The method of claim 1, wherein, the optimization of the target parameter in the projection relationship based on the re-projection error, comprising: determining a total projection loss based on the re-projection error and a depth regularization term of the projection points; based on the total projection loss, optimizing the target parameter in the projection relationship.

3. The method of claim 2, wherein the depth regularization term is used to reduce the dispersion degree of the depth of each pixel point of the target obstacle.

4. The method of claim 1, wherein, For the frame to be optimized, the true value of the projection points required by the re-projection error is determined based on a pixel-level trajectory tracking method.

5. The method of claim 1, wherein, In the case that the target obstacle is a static target, the target parameter includes the pixel point depth parameter.

6. The method of claim 1, wherein, In the case that the target obstacle is a dynamic target, the target parameter includes the pixel point depth parameter and the pose transformation parameter.

7. The method of claim 1, wherein, determining the initial value of the pixel point depth parameter, comprising: obtaining a first preset value of the pixel point depth parameter; randomly perturbing the first preset value to obtain the initial value of the pixel point depth parameter.

8. The method of claim 1, wherein, determining the initial value of the pose transformation parameter, comprising: obtaining a second preset value of the pose transformation parameter; randomly perturbing the second preset value to obtain the initial value of the pose transformation parameter.

9. The method of any one of claims 1, 6-8, wherein, The obstacle coordinate system is established based on a world coordinate system; In the case that the target obstacle is a dynamic target, the pose transformation parameter has a transition term superimposed on the initial value of the pose transformation parameter; the transition term is used to express the relative change of the target obstacle between the reference frame and the frame to be optimized, and the optimization of the pose transformation parameter is realized by optimizing the transition term.

10. The method of claim 1, wherein, determining the initial pixel point set of the target obstacle in the reference frame, comprising: obtaining a two-dimensional position box of the target obstacle detected by a two-dimensional image detection model from the reference frame; based on the two-dimensional position box, uniformly scattering points to obtain the initial pixel point set of the target obstacle in the reference frame.

11. The method of claim 10, wherein, The uniform scattering based on the two-dimensional position box obtains an initial pixel point set of the target obstacle in the reference frame, and includes: Obtain a mask image of the target obstacle in the reference frame; Uniformly scatter based on the two-dimensional position box of the target obstacle in the reference frame and the mask image to obtain the initial pixel point set.

12. The method of claim 11, wherein, The acquisition of the mask image of the target obstacle in the reference frame includes: Segmenting the target obstacle from the reference frame by using a segmentation-all model to obtain the mask image of the target obstacle.

13. The method of claim 1, wherein, Determine the reference frame, including: From the sequence of frames to be processed, based on the object features of the target obstacle, the images whose object feature saliency meets the preset condition are selected as the reference frame.

14. The method of claim 1, wherein, Determine the frame to be optimized, including: In the case that the target obstacle is a static target, an optimization direction is obtained; Based on the reference frame and the optimization direction, the frame to be optimized is selected from the sequence of frames to be processed.

15. The method of claim 1, wherein, Determine the frame to be optimized, including: In the case that the target obstacle is a dynamic target, a frame image in which the target obstacle is missed by a three-dimensional detection model is selected as the frame to be optimized; the three-dimensional detection model is used for target detection on point cloud or two-dimensional image to obtain the target pose of the target obstacle.

16. The method of claim 15, wherein, In the case that the target obstacle is a dynamic target, a frame closest to the frame to be optimized and containing the target obstacle is selected as the reference frame.

17. The method of claim 1, wherein, Determine the target obstacle, including: Obtain the detection result of the two-dimensional image detection model for target detection on the sequence of frames to be processed; Based on the detection result, the candidate objects belonging to the target category are selected as the target obstacle.

18. The method of claim 1, wherein, The reference frame and the frame to be optimized are bird's eye view in automatic driving.

19. The method of claim 1, further comprising: Based on the target pose of the target obstacle in different frame images, the target obstacle is tracked to obtain the trajectory information of the target obstacle.

20. The method of claim 19, further comprising: For the trajectory information to be corrected frame, based on the two-dimensional position box of the target obstacle in the frame to be corrected, the three-dimensional position box of the target obstacle in the frame to be corrected is determined; wherein the two-dimensional position box is obtained by detecting the target obstacle by a two-dimensional image detection model; Based on the classification result of the two-dimensional position box, the classification result of the three-dimensional position box of the frame to be corrected is corrected.

21. The method of claim 19, further comprising: Determine the confidence of the trajectory information; The confidence is determined based on at least one of the following information: the confidence of the two-dimensional position box of the target obstacle recognized by the two-dimensional image detection model, the occlusion rate of the target obstacle, and the intersection over union; the intersection over union is the intersection over union between the two-dimensional position box and the three-dimensional position box of the target obstacle; In the case that the confidence is lower than the target threshold, the trajectory information is determined as a false detection trajectory.

22. The method of claim 1, wherein, The determination of the target pose of the target obstacle based on the optimized target parameter includes: In a case where the target obstacle is a dynamic target, a target pose of the target obstacle in the frame to be optimized is determined based on the pose transformation parameter in the optimized target parameter, the camera intrinsic parameter, and a three-dimensional position box of the target obstacle in the reference frame.

23. The method of claim 1, wherein, The target pose of the target obstacle is determined based on the optimized target parameter, and the target pose of the target obstacle comprises: In a case where the target obstacle is a static target, an initial pixel point of the target obstacle in the reference frame is converted into a camera coordinate system based on the optimized target parameter, to obtain a first point group of the target obstacle in the camera coordinate system. A plane estimation method is used to process the first point group, to obtain a target pose of the target obstacle in the reference frame.

24. The method of claim 23, further comprising: The target pose of the target obstacle in the reference frame is converted into a pose of the target obstacle in a target frame by using a global back-projection strategy. The target frame is the frame to be optimized or an image frame other than the frame to be optimized, and the pose of the target obstacle in the target frame needs to be estimated.

25. An automatic obstacle labeling apparatus, comprising: An optimization module configured to optimize a target parameter in a projection relationship based on a re-projection error. The projection relationship is used to project a target obstacle from a reference frame to a frame to be optimized. The projection relationship satisfies the following constraint condition: in an obstacle coordinate system established based on the target obstacle, positions of the target obstacle in different frames are uniform. A determination module configured to determine a target pose of the target obstacle based on the optimized target parameter. The optimization module comprises a projection unit, and the projection unit comprises: A back-projection sub-unit configured to perform a back-projection operation on an initial pixel point set of the target obstacle in the reference frame based on a pixel point depth parameter, to obtain a first three-dimensional space position of the target obstacle in a camera coordinate system; the pixel point depth parameter is used to represent a depth value of each initial pixel point in the reference frame. A conversion sub-unit configured to convert the first three-dimensional space position into a second three-dimensional space position in the obstacle coordinate system based on a pose transformation parameter between the camera coordinate system and the obstacle coordinate system. A projection sub-unit configured to project the second three-dimensional space position into the frame to be optimized, to obtain the set of projection points. An error determination unit configured to determine the re-projection error based on the set of projection points and a ground truth of the projection points of the frame to be optimized.

26. The apparatus of claim 25, wherein, The optimization module comprises: A loss determination unit configured to determine a total projection loss based on the re-projection error and a depth regularization term of the projection points. An optimization unit configured to optimize the target parameter in the projection relationship based on the total projection loss.

27. The apparatus of claim 26, wherein the depth regularization term is used to reduce a discrete degree of depths of pixel points of the target obstacle.

28. The apparatus of claim 25, wherein, For the frame to be optimized, the ground truth of the projection points required by the re-projection error is determined based on a pixel-level trajectory tracking method.

29. The apparatus of claim 25, wherein, In a case where the target obstacle is a static target, the target parameter comprises the pixel point depth parameter.

30. The apparatus of claim 25, wherein, In a case where the target obstacle is a dynamic target, the target parameter comprises the pixel point depth parameter and the pose transformation parameter.

31. The apparatus of claim 25, wherein, The optimization module comprises: A first acquisition unit configured to acquire a first preset value of the pixel point depth parameter; A first perturbation unit configured to randomly perturb the first preset value to obtain an initial value of the pixel point depth parameter.

32. The apparatus of claim 25, wherein, The optimization module comprises: A second acquisition unit configured to acquire a second preset value of the pose transformation parameter; A second perturbation unit configured to randomly perturb the second preset value to obtain an initial value of the pose transformation parameter.

33. The apparatus of any one of claims 25, 30-32, wherein, The obstacle coordinate system is established based on a world coordinate system. In a case where the target obstacle is a dynamic target, the pose transformation parameter has a transition term superimposed on the initial value of the pose transformation parameter; the transition term is used to express the relative change of the target obstacle between the reference frame and the frame to be optimized, and the optimization of the pose transformation parameter is realized by optimizing the transition term.

34. The apparatus of claim 25, wherein, The projection unit is specifically configured to: acquire a two-dimensional position box of the target obstacle detected by a two-dimensional image detection model from the reference frame; perform uniform point scattering based on the two-dimensional position box to obtain an initial pixel point set of the target obstacle in the reference frame.

35. The apparatus of claim 34, wherein, The projection unit is specifically configured to: acquire a mask image of the target obstacle in the reference frame; perform uniform point scattering based on the two-dimensional position box of the target obstacle in the reference frame and the mask image to obtain the initial pixel point set.

36. The apparatus of claim 35, wherein, The projection unit is specifically configured to: segment the target obstacle from the reference frame by using a segmentation-all model to obtain a mask image of the target obstacle.

37. The apparatus of claim 25, wherein, The projection unit is specifically configured to: from a sequence of frames to be processed, based on the object features of the target obstacle, screen an image whose object feature saliency satisfies a preset condition as the reference frame.

38. The apparatus of claim 25, wherein, The projection unit is specifically configured to: in a case where the target obstacle is a static target, acquire an optimization direction; based on the reference frame and the optimization direction, select the frame to be optimized from a sequence of frames to be processed.

39. The apparatus of claim 25, wherein, The projection unit is specifically configured to: in a case where the target obstacle is a dynamic target, acquire a frame image in which the target obstacle is missed by a three-dimensional detection model as the frame to be optimized; the three-dimensional detection model is used for target detection on point cloud or two-dimensional image to obtain a target pose of the target obstacle.

40. The apparatus of claim 39, wherein, The projection unit is specifically configured to: in a case where the target obstacle is a dynamic target, select a frame closest to the frame to be optimized and containing the target obstacle as the reference frame.

41. The apparatus of claim 25, wherein, The optimization module is specifically configured to: acquire a detection result obtained by a two-dimensional image detection model performing target detection on a sequence of frames to be processed; based on the detection result, screen a candidate object belonging to a target category as the target obstacle.

42. The apparatus of claim 25, wherein, The reference frame and the frame to be optimized are bird's-eye views in autonomous driving. 43.The apparatus of claim 25, further comprising a trajectory tracking module configured to: track a trajectory of the target obstacle based on target poses of the target obstacle in different frame images to obtain trajectory information of the target obstacle. 44.The apparatus of claim 43, further comprising a correction module configured to: obtain the two-dimensional position box by detecting the target obstacle using a two-dimensional image detection model; and correct a classification result of the three-dimensional position box of the frame to be corrected based on a classification result of the two-dimensional position box. 45.The apparatus of claim 43, further comprising a filtering module configured to: determine a confidence of the trajectory information, the confidence being determined based on at least one of a confidence of a two-dimensional position box of the target obstacle identified by the two-dimensional image detection model, an occlusion rate of the target obstacle, and an intersection over union between the two-dimensional position box and the three-dimensional position box of the target obstacle; and determine the trajectory information as a false trajectory if the confidence is lower than a target threshold. For the trajectory information to be corrected frame, based on the two-dimensional position box of the target obstacle in the to-be-corrected frame, the three-dimensional position box of the target obstacle in the to-be-corrected frame is determined; wherein, The determination module is specifically configured to: determine a target pose of the target obstacle in the frame to be optimized based on the pose transformation parameter in the optimized target parameter, the camera intrinsic parameter, and the three-dimensional position box of the target obstacle in the reference frame if the target obstacle is a dynamic target. The determination module is specifically configured to: convert an initial pixel point of the target obstacle in the reference frame to a camera coordinate system based on the optimized target parameter to obtain a first point cloud of the target obstacle in the camera coordinate system if the target obstacle is a static target; and process the first point cloud using a plane estimation apparatus to obtain the target pose of the target obstacle in the reference frame. 48.The apparatus of claim 47, wherein the determination module is further configured to: convert the target pose of the target obstacle in the reference frame to a pose of the target obstacle in a target frame using a global back-projection strategy; and the target frame is the frame to be optimized or an image frame other than the frame to be optimized, for which the pose of the target obstacle needs to be estimated. 49.An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-24. The computer instructions are used to enable the computer to perform the method of any one of claims 1-24.

46. The device of claim 25, wherein, 51.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-24. ​ 47. The apparatus of claim 25, wherein, ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 50. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, ​ ​

Images