Three-dimensional target detection frame labeling method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure disclose a three-dimensional target detection box labeling method and device, electronic equipment and storage medium, comprising: determining each two-dimensional target detection box according to the to-be-detected image, determining the camera pose and the reconstructed environment information according to the combined detection data, for each two-dimensional target detection box, determining each vanishing point set according to the camera intrinsic matrix, the camera rotation matrix and each preset sampling heading angle, for each vanishing point set, determining each corner point set in combination with the two-dimensional target detection box and each upper edge frame sampling point, for each corner point set, determining each three-dimensional to-be-processed detection box in combination with the camera intrinsic matrix, the camera pose and the three-dimensional ground, for each two-dimensional target detection box, determining the three-dimensional target detection box in combination with each post-frame two-dimensional detection box, each three-dimensional to-be-processed detection box and each post-frame three-dimensional detection box, and labeling in the to-be-detected image, so as to realize the combination of the two-dimensional target detection box and the detection data, improve the accuracy and speed of labeling.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision, and more particularly to a method, apparatus, electronic device, and storage medium for annotating bounding boxes for three-dimensional object detection. Background Technology

[0002] In the field of autonomous driving, 3D target recognition is a very important task. Typically, the annotation of 3D target detection boxes in 3D target detection tasks requires many prerequisites, including precise measurement equipment and relatively tedious manual annotation, in order to obtain good annotation results.

[0003] However, when using sophisticated measuring equipment, external parameters need to be calibrated and time synchronization is required. If deep neural networks are used for annotation in the pure visual annotation method of two-dimensional (2D) target detection, the algorithm has poor robustness. If the surrounding environment is reconstructed and instance segmentation is performed, the computational power requirement is too high and the processing efficiency is low. Summary of the Invention

[0004] To address or at least partially address the aforementioned technical problems, this disclosure provides a method, apparatus, electronic device, and storage medium for annotating three-dimensional target detection boxes. By combining detection data with two-dimensional target detection boxes, the accuracy and speed of three-dimensional target detection box annotation are improved, and the workload of manual annotation is reduced.

[0005] In a first aspect, embodiments of this disclosure provide a method for annotating three-dimensional target detection bounding boxes, the method comprising:

[0006] Based on the image to be detected, determine each two-dimensional target detection box, the category corresponding to each two-dimensional target detection box, and the confidence level. Based on the image to be detected and the detection data corresponding to the image to be detected, determine the camera pose and reconstruct environmental information. The detection data includes at least one of inertial measurement data, wheel speed measurement data, and global positioning data.

[0007] For each two-dimensional target detection box, the set of vanishing points corresponding to each preset sampling heading angle is determined based on the camera intrinsic parameter matrix, the camera rotation matrix, and each preset sampling heading angle.

[0008] For each set of vanishing points, based on the set of vanishing points, the two-dimensional target detection box, and the preset sampling points of each upper edge box, a set of corner points corresponding to the sampling points of each upper edge detection box is determined;

[0009] For each set of corner points, based on the set of corner points, the camera intrinsic parameter matrix, the camera pose, and the 3D ground in the reconstructed environment information, the 3D detection box corresponding to the set of corner points, as well as the size and pose information of the 3D detection box, are determined.

[0010] For each two-dimensional target detection box, a three-dimensional target detection box is determined from each three-dimensional target detection box based on the two-dimensional target detection box, each subsequent two-dimensional detection box in the next frame of the image to be detected, each three-dimensional target detection box corresponding to the two-dimensional target detection box, and each subsequent three-dimensional detection box corresponding to each subsequent two-dimensional detection box. Then, based on the category and confidence level of the two-dimensional target detection box, the three-dimensional target detection box is labeled in the image to be detected according to its size and pose information.

[0011] Secondly, embodiments of this disclosure also provide a three-dimensional target detection bounding box annotation device, the device comprising:

[0012] A two-dimensional target detection box construction module is used to determine each two-dimensional target detection box, the category corresponding to each two-dimensional target detection box, and the confidence level based on the image to be detected, and to determine the camera pose and reconstruct environmental information based on the image to be detected and the detection data corresponding to the image to be detected; wherein, the detection data includes at least one of inertial measurement data, wheel speed measurement data, and global positioning data;

[0013] The vanishing point set determination module is used to determine the vanishing point set corresponding to each preset sampling heading angle for each two-dimensional target detection box, based on the camera intrinsic parameter matrix, camera rotation matrix, and each preset sampling heading angle.

[0014] The corner point set determination module is used to determine, for each vanishing point set, the corner point set corresponding to the sampling points of each upper edge detection box based on the vanishing point set, the two-dimensional target detection box, and the preset sampling points of each upper edge box;

[0015] The 3D detection bounding box determination module is used to determine, for each set of corner points, the 3D detection bounding box corresponding to the set of corner points, as well as the size and pose information of the 3D detection bounding box, based on the set of corner points, the camera intrinsic parameter matrix, the camera pose, and the 3D ground in the reconstructed environment information.

[0016] The 3D target detection box annotation module is used to determine a 3D target detection box from each 3D target detection box based on the 2D target detection box, each subsequent 2D detection box in the next frame of the image to be detected, each 3D target detection box corresponding to the 2D target detection box, and each subsequent 3D detection box corresponding to each subsequent 2D detection box. The module then annotates the 3D target detection box in the image to be detected according to the category and confidence level of the 2D target detection box, and based on the size and pose information of the 3D target detection box.

[0017] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional target detection bounding box annotation method as described above.

[0018] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional target detection bounding box annotation method as described above.

[0019] This disclosure provides a method for annotating 3D target detection bounding boxes. Using an image to be detected, it determines each 2D target detection bounding box, its category, and confidence level. Based on the image and detection data, it determines the camera pose and reconstructs environmental information, avoiding the use of LiDAR and millimeter-wave radar data. For each 2D target detection bounding box, it determines a set of vanishing points corresponding to each preset sampling heading angle based on the camera intrinsic parameter matrix, camera rotation matrix, and preset sampling heading angles. Then, for each set of vanishing points, it determines multiple corner point sets based on the vanishing point set, the 2D target detection bounding box, and preset upper edge box sampling points. For each corner point set, it further... Based on the corner point set, camera intrinsic parameter matrix, camera pose, and 3D ground in the reconstructed environment information, multiple 3D detection boxes to be processed, along with their size and pose information, are determined. Furthermore, for each 2D target detection box, a 3D target detection box is determined from each 3D target detection box based on the 2D target detection box, the 2D detection boxes in each subsequent frame, the 3D detection boxes to be processed, and the 3D detection boxes in each subsequent frame. The 3D target detection boxes are then labeled in the image to be detected. This approach combines detection data with 2D target detection boxes to improve the accuracy and speed of 3D target detection box labeling, while reducing computational requirements and the workload of manual labeling. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 This is a flowchart of a three-dimensional target detection bounding box annotation method according to an embodiment of this disclosure;

[0022] Figure 2 This is a schematic diagram illustrating one method of determining a set of corner points in an embodiment of this disclosure;

[0023] Figure 3 This is a schematic diagram illustrating the calculation of the three-dimensional coordinates of a top corner point in an embodiment of this disclosure;

[0024] Figure 4 This is a schematic diagram of a multiple pair of three-dimensional detection box matching pairs in an embodiment of this disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of a three-dimensional target detection box annotation device according to an embodiment of this disclosure;

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0027] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0028] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0029] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0030] Figure 1 This is a flowchart illustrating a 3D object detection bounding box annotation method according to an embodiment of this disclosure. The method can be executed by a 3D object detection bounding box annotation device, which can be implemented in software and / or hardware, and can be configured in an electronic device. Figure 1 As shown, the method may specifically include the following steps:

[0031] S110. Based on the image to be detected, determine the detection boxes of each two-dimensional target, the category corresponding to each two-dimensional target detection box, and the confidence level. Based on the image to be detected and the detection data corresponding to the image to be detected, determine the camera pose and reconstruct the environmental information.

[0032] The detection image is the image for which 3D bounding box annotation is performed. The 2D target detection box is the 2D detection box obtained after 2D target recognition of the image. The category is the object category corresponding to the 2D detection box after 2D target recognition of the image, such as vehicle, pedestrian, etc. The confidence level is the reliability of 2D target recognition of the image. Detection data includes at least one of inertial measurement data, wheel speed meter data, and global positioning data. Inertial measurement data is data measured by an Inertial Measurement Unit (IMU). Wheel speed meter data is data collected by wheel speed sensors. Global positioning data is data measured by the Global Positioning System (GPS). Camera pose is the position and orientation of the camera. Reconstructed environmental information is the surrounding environment information of the image to be detected, such as ground information. Camera pose and reconstructed environmental information can be obtained through SLAM (Simultaneous Localization and Mapping).

[0033] Specifically, for 2D object recognition of the image to be detected, algorithms such as Faster-RCNN (Faster Region-Convolutional Neural Network), YOLO (YouOnly Look Once), SSD (Single Shot MultiBox Detector), and CenterNet can be used to identify 2D object detection boxes, categories, and confidence scores. Furthermore, based on the image to be detected and its corresponding detection data, SLAM mapping can be used to recover camera pose and reconstruct environmental information. Feature point-based or direct SLAM algorithms such as ORB-SLAM3 (a SLAM system that solves pure vision or visual-inertial navigation hybrid maps), VINS (Visual Inertial Navigation System), and DM-VIO (Delayed Marginalization Visual-Inertial Odometry) can be used to solve for camera pose and sparsely, semi-densely, or densely reconstruct the surrounding environment to obtain reconstructed environmental information.

[0034] S120. For each two-dimensional target detection box, determine the set of vanishing points corresponding to each preset sampling heading angle based on the camera intrinsic parameter matrix, camera rotation matrix, and each preset sampling heading angle.

[0035] The camera intrinsic parameter matrix reflects the camera's inherent properties. The camera rotation matrix reflects the camera's rotation direction in 3D space. The preset sampling heading angle is a pre-defined heading angle for subsequent vanishing point set calculations, for example, in the positive and negative Y... ° Each interval y within the interval ° Taking a preset sampling heading angle, a total of [number] angles can be obtained. A preset sampling heading angle. A vanishing point is the point in an image where a parallel line disappears at a distance. The vanishing point set is the set of three vanishing points corresponding to the two-dimensional object detection box.

[0036] Specifically, for each two-dimensional target detection box, based on the camera intrinsic parameter matrix, camera rotation matrix, and each preset sampling heading angle, three vanishing points corresponding to the two-dimensional target detection box and each preset sampling heading angle can be determined, i.e., a set of vanishing points.

[0037] Based on the above example, the set of vanishing points corresponding to each preset sampling heading angle can be determined using the following method:

[0038] For each preset sampling heading angle, the set of vanishing points corresponding to the preset sampling heading angle is determined by the following formula:

[0039]

[0040]

[0041]

[0042] Where K is the camera intrinsic parameter matrix, R is the camera rotation matrix, yaw is the preset sampling heading angle, and VP is the camera rotation matrix. x VP y VP z These are the vanishing points in the vanishing point set. The above formula can be used to calculate the three vanishing points at the current preset sampling heading angle.

[0043] If any vanishing point in the set of vanishing points obtained in the first step contains 0 in any of its three dimensions, the result is unusable. Therefore, the set of vanishing points is recalculated using non-zero preset roll and pitch angles. Specifically, this can be done as follows:

[0044] If at least one vanishing point in the vanishing point set contains 0 in any of its three dimensions, then the set of vanishing points corresponding to the preset sampled heading angle is determined using the following formula:

[0045]

[0046]

[0047]

[0048] Where roll is the preset roll angle and pitch is the preset pitch angle.

[0049] S130. For each set of vanishing points, determine the set of corner points corresponding to the sampling points of each upper edge detection box based on the set of vanishing points, the two-dimensional target detection box, and the preset sampling points of each upper edge box.

[0050] The upper edge bounding box sampling points are obtained by sampling points at preset intervals on the upper edge of the 2D target detection box. The corner point set is the set of eight corner points corresponding to the subsequent 3D target detection box, used to construct the 3D target detection box.

[0051] It is understandable that for each set of vanishing points, a set of corner points with the same number of preset upper bounding box sampling points can be obtained.

[0052] For example, such as Figure 2 The diagram shown illustrates the determination of the corner point set. First, the two-dimensional target detection box is ABCD, consisting of a set of three vanishing points (VPs). x VP y VP z And one upper bounding box sampling point (p1), which can be calculated to obtain Then, through calculation, we can obtain Finally, through calculation, we obtain as well as in, Let represent the line formed by two points, and × indicates finding the intersection of the two lines. Therefore, for the vanishing point set (VP... x VP y VP z ), the set of corner points (p1, p2, p3, p4, p5, p6, p7 and p8) corresponding to the sampling point (p1) of the upper edge detection box.

[0053] S140. For each set of corner points, based on the set of corner points, the camera intrinsic parameter matrix, the camera pose, and the 3D ground in the reconstructed environment information, determine the 3D detection box corresponding to the set of corner points, as well as the size and pose information of the 3D detection box.

[0054] The 3D bounding box is obtained by reconstructing the 3D shape of the corner point set. Its dimensions include length, width, and height. Its pose information includes six degrees of freedom.

[0055] Specifically, for each set of corner points, the four bottom corner points can be projected onto the 3D ground in the reconstructed environment information. Their spatial positions are then obtained using camera height and other parameters from the camera pose. Using the intersection of two lines, the spatial positions of the four top corner points can be calculated. Figure 3 As shown, the three-dimensional detection box to be processed can be determined based on the position of each corner point in space. This mainly includes the pose (position and orientation) of the three-dimensional detection box to be processed, as well as the length, height and width information, i.e., pose information and size information.

[0056] Based on the above example, the 3D bounding box corresponding to the set of corner points, as well as the size and pose information of the 3D bounding box, can be determined in the following way:

[0057] For each set of corner points, the three-dimensional coordinates of each bottom corner point in the set are determined using the following formula:

[0058]

[0059]

[0060] in, Let K be the three-dimensional coordinates of the i-th bottom corner point. -1 p i Let be the back projection ray corresponding to the i-th bottom corner point in 3D space, K be the camera intrinsic parameter matrix, [n,m] represent the 3D ground in the reconstructed environment information, n be the normal vector of the 3D ground in the camera coordinate system, and m be the distance from the camera center to the 3D ground;

[0061] Based on the camera pose and the three-dimensional coordinates of each bottom corner point, determine the three-dimensional coordinates of each top corner point in the corner point set;

[0062] Based on the three-dimensional coordinates of each top corner point and each bottom corner point in the corner point set, determine the three-dimensional detection box corresponding to the corner point set, as well as the size and pose information of the three-dimensional detection box.

[0063] The bottom corner points are the four corner points located at the bottom of the corner point set. The top corner points are the four corner points located at the top of the corner point set.

[0064] Specifically, the 3D coordinates of each bottom corner point in the corner point set can be calculated using formulas. Furthermore, based on the camera's optical center position and height in the camera pose, combined with the 3D ground, the 3D coordinates of each top corner point corresponding to each bottom corner point can be determined using similar triangles. Then, based on the 3D coordinates of each top and bottom corner point, a 3D bounding box can be constructed, and its size and pose information can be calculated.

[0065] Based on the above example, after determining the 3D detection boxes corresponding to the corner point set, as well as the size and pose information of the 3D detection boxes, preliminary screening can be performed based on the error of each 3D detection box. Specifically, this can be done as follows:

[0066] For each 3D detection bounding box, sample each edge of the 3D detection bounding box to determine the sampling points of each edge;

[0067] The distance error is determined based on each edge sampling point and the corresponding two-dimensional target detection box in the three-dimensional detection box.

[0068] The shape error is determined based on the current aspect ratio, current height-to-width ratio, preset maximum aspect ratio, preset minimum aspect ratio, preset maximum height-to-width ratio, and preset minimum height-to-width ratio of the 3D detection box to be processed.

[0069] The total error of the 3D detection box to be processed is determined based on the distance error and the shape error.

[0070] For each 2D target detection box, the 3D target detection boxes are filtered based on the total error of each corresponding 3D target detection box, and the corresponding 3D target detection boxes are updated.

[0071] In this context, edge sampling points are multiple sampling points obtained by sampling on each edge of the 3D target detection box. Distance error is the average of the nearest distances between each edge sampling point and the edges extracted using the edge extraction algorithm within the 2D target detection box. Current aspect ratio is the length-to-width ratio of the 3D target detection box's dimensions, and current height-to-width ratio is the height-to-width ratio. Preset maximum aspect ratio, preset minimum aspect ratio, preset maximum height-to-width ratio, and preset minimum height-to-width ratio are thresholds pre-set based on the aspect ratios of the real object. Shape error is a numerical value used to represent whether the shape of the 3D target detection box closely approximates the real object. Total error is an integration of distance error and shape error, such as summation or weighted summation.

[0072] Specifically, for each 3D detection bounding box, each edge of the bounding box can be sampled one by one to obtain multiple edge sampling points on each edge. Then, the edge pixels extracted using the Canny edge detection algorithm for the 2D target detection box are compared with the Chamfer distance of each edge sampling point to obtain the distance error. The distance error can be determined using the following formula:

[0073]

[0074] Where I is the set of edge pixels extracted by the Canny edge algorithm in the 2D object detection box, T is the set of edge sampling points obtained by each edge sampling of the 3D object detection box, d(T, I) is the distance from each edge sampling point on the 3D object detection box to the nearest pixel among the edge pixels extracted by the Canny edge algorithm in the 2D object detection box, and E dist This is the distance error, used to indicate whether the edge of the 3D detection box matches the edge of the real object.

[0075] Therefore, the shape error can be determined using the following formula:

[0076] E shape =max(s l -σ lm ,0)+max(σ ls -s l ,0)+max(s h -σ hm,0)+max(σ hs -s h ,0)

[0077] Among them, s l s h These represent the current aspect ratio and current height aspect ratio of the 3D detection bounding box, respectively, σ ls σ lm To preset the maximum aspect ratio and the minimum aspect ratio, σ hs σ hm E is used to preset the maximum aspect ratio and the minimum aspect ratio. shape This refers to shape error.

[0078] Furthermore, the total error of the 3D detection box can be determined based on the distance error and shape error using the following formula:

[0079] E total =w dist E dist +w shape E shape

[0080] Among them, E total E represents the total error. dist and E shape These are distance error and shape error, respectively. dist and w shape These are the weighting coefficients for distance error and shape error, respectively, which can be determined in advance according to requirements.

[0081] Finally, the multiple 3D detection boxes corresponding to each 2D target detection box are filtered according to the total error. For example, the top n 3D detection boxes with the lowest total error are selected to update the 3D detection boxes corresponding to the 2D target detection box.

[0082] S150. For each two-dimensional target detection box, based on the two-dimensional target detection box, the two-dimensional detection boxes in the next frame of the image to be detected, the three-dimensional target detection boxes corresponding to the two-dimensional target detection box, and the three-dimensional target detection boxes corresponding to the two-dimensional target detection boxes, the three-dimensional target detection boxes are determined from each three-dimensional target detection box. Based on the category and confidence level of the two-dimensional target detection box, the three-dimensional target detection box is labeled in the image to be detected according to the size and pose information of the three-dimensional target detection box.

[0083] Among them, the 2D detection box in the next frame is a 2D detection box obtained by 2D target recognition in the next frame of the image to be detected. The 3D detection box in the next frame is a 3D detection box obtained by 3D reconstruction of the 2D detection box in the next frame. The reconstruction process is similar to that of the 3D detection box to be processed, and will not be described in detail here. The 3D target detection box is the 3D detection box determined from each 3D detection box to be processed, which is then used for subsequent annotation in the image to be detected.

[0084] Specifically, for each 2D target detection box, matching 2D detection boxes in subsequent frames of the image to be detected can be determined. Then, 3D detection box matching is performed on each 3D target detection box corresponding to the 2D target detection box and each subsequent 3D detection box corresponding to the matching 2D target detection box, resulting in multiple successfully matched 3D detection box pairs. These successfully matched 3D detection box pairs are then analyzed, and the 3D target detection box in the best-matching pair is determined as the 3D target detection box. The 3D target detection boxes are then labeled in the image to be detected according to their size and pose information to complete the 3D annotation process.

[0085] Based on the above example, the following steps can be used to determine the 3D target detection box from each 3D detection box to be processed:

[0086] Step 1: Extract feature points from the two-dimensional target detection box to obtain the first feature point set. Extract feature points from each subsequent frame's two-dimensional detection box to obtain the second feature point set. Based on the first feature point set, each second feature point set, and the intersection-union ratio between the two-dimensional target detection box and each subsequent frame's two-dimensional detection box, determine the matching two-dimensional detection box corresponding to the two-dimensional target detection box.

[0087] The first feature point set is the set of feature points extracted from the 2D object detection bounding box. The second feature point set is the set of feature points extracted from the 2D detection bounding boxes in subsequent frames. The intersection-union ratio (IUU) between the 2D object detection bounding box and each subsequent frame's 2D detection bounding box is the ratio of the area intersection to the area union of the 2D object detection bounding box and each subsequent frame's 2D detection bounding box. The matching 2D detection bounding box is the one in each subsequent frame's 2D detection bounding box that matches the 2D object detection bounding box.

[0088] Specifically, feature points are extracted from the two-dimensional target detection box to obtain a first feature point set. Feature points are extracted from each subsequent frame's two-dimensional detection box to obtain a second feature point set. Feature point descriptors and optical flow tracing are used for matching. The intersection-union ratio between the two-dimensional target detection box and each subsequent frame's two-dimensional detection box is combined to perform matching between the two-dimensional target detection box and each subsequent frame's two-dimensional detection box, resulting in a matching two-dimensional detection box corresponding to the two-dimensional target detection box.

[0089] Step 2: Determine the matching pairs of three-dimensional detection boxes based on the first matching heading angle, first length, first width, and first height of each three-dimensional detection box corresponding to the two-dimensional target detection box, the second matching heading angle, second length, second width, and second height of each subsequent frame three-dimensional detection box corresponding to the matched two-dimensional detection box, the matching heading angle threshold, the matching length threshold, the matching width threshold, and the matching height threshold.

[0090] In this system, the first heading angle to be matched is the heading angle of the 3D detection box to be processed. The first length, first width, and first height are the length, width, and height of the 3D detection box to be processed. The second heading angle to be matched is the heading angle of the 3D detection box in the subsequent frame. The second length, second width, and second height are the length, width, and height of the 3D detection box in the subsequent frame. The matching heading angle threshold, matching length threshold, matching width threshold, and matching height threshold are pre-determined matching thresholds based on requirements. The 3D detection box matching pair includes a first 3D matching box and a second 3D matching box; the first 3D matching box is a 3D detection box to be processed in the image to be detected, and the second 3D matching box is a 3D detection box in the subsequent frame that matches the first 3D matching box.

[0091] Specifically, the matching of the 3D detection box to be processed with the 3D detection boxes of each subsequent frame is performed using the following two formulas:

[0092]

[0093]

[0094] in, Let be the heading angle of a 3D target bounding box corresponding to the j-th 2D target bounding box in the i-th frame (the image to be detected). The heading angle of a subsequent 3D detection box in the (i+1)th frame (the next frame image) corresponding to the matching 2D detection box of the j-th 2D target detection box can be calculated from the 6-DOF pose of the 3D detection box to be processed. Let the length, width, and height of a 3D bounding box corresponding to the j-th 2D target bounding box in the i-th frame be defined. Let be the length, width, and height of the 3D detection box in the (i+1)th frame, corresponding to the matching 2D detection box of the j-th 2D target detection box, and Δyaw be the matching heading angle threshold. For matching length threshold, matching width threshold, and matching height threshold.

[0095] If the above two formulas are satisfied, the matching is considered successful, and multiple pairs of 3D detection boxes can be obtained.

[0096] For example, such as Figure 4 The diagram illustrates multiple pairs of 3D detection box matching pairs. It can be seen that the two large hollow boxes represent the mutually matching 2D target detection boxes and the matching 2D detection boxes in the i-th and i+1-th frames, respectively. The small solid boxes within the 2D target detection boxes represent the corresponding 3D detection boxes, and the small solid boxes within the matching 2D detection boxes represent the corresponding 3D detection boxes in subsequent frames. The connecting lines represent successfully matched 3D detection box pairs. As shown in the figure, there are 6 3D detection box matching pairs.

[0097] Step 3: Based on the pixel positions of each first feature point in the first feature point set, the pixel positions of each second feature point in the second feature point set, the depth information of each first feature point, the depth information of each second feature point, the camera pose of the image to be detected, and the camera pose of the next frame image, determine the spatial position of each first feature point and the spatial position of each second feature point.

[0098] Here, pixel position refers to the location of the feature point in two-dimensional space. Depth information is the measurement information pointing in the two-dimensional space direction. Feature point spatial position refers to the location of the feature point in three-dimensional space.

[0099] Specifically, the 3D position of the two-dimensional target detection box and each feature point in the matching two-dimensional detection box can be recovered using the following formula:

[0100]

[0101] in, Let be the pixel coordinates of the l-th feature point in the i-th frame, that is, the pixel position of the l-th first feature point in the set of first feature points. These are the pixel coordinates of the l-th feature point in the (i+1)-th frame, which are the pixel positions of the l-th second feature point in the set of second feature points. This refers to the depth information of the l-th feature point in the i-th frame, which is the depth information of the l-th first feature point in the set of first feature points. This refers to the depth information of the l-th feature point in the (i+1)-th frame, which is the depth information of the l-th second feature point in the set of second feature points. Let be the camera pose of the i-th frame (the camera pose of the image to be detected). Let L be the camera pose of the (i+1)th frame (the camera pose of the next frame), and L be the total number of matched feature points. Let l be the spatial location of the l-th feature point in the i-th frame, that is, the spatial location of the l-th first feature point in the set of first feature points. Correspondingly, E represents the spatial location of the l-th feature point in the (i+1)-th frame, which is the spatial location of the l-th second feature point in the set of second feature points. p To solve the objective, let E p The minimum can be obtained by solving the objective function. as well as

[0102] Furthermore, the spatial positions of the feature points of each first feature point and the feature points of each second feature point can be obtained.

[0103] Step 4: For each pair of 3D detection box matching pairs, based on the center position of the first 3D matching box in the 3D detection box matching pair, the spatial position of the feature points of each first feature point, and the preset distance threshold, determine the first feature point to be used from each first feature point, and determine the second feature point to be used from each second feature point based on each first feature point.

[0104] Here, the center position refers to the 3D position of the geometric center of the 3D matching box. The preset distance threshold is a pre-set threshold used to filter feature points with motion consistency. The first feature point to be used is the first feature point that satisfies motion consistency, and the second feature point to be used is the second feature point that satisfies motion consistency.

[0105] Specifically, for each pair of 3D detection bounding boxes, the distance between the center position of the first 3D matching box and the spatial position of each first feature point is used to determine whether it exceeds a preset distance threshold. If it is less than the threshold, the first feature point is considered to be associated with the first 3D matching box and is determined as the first feature point to be used; otherwise, it is considered to be a background point. Then, feature points corresponding to each first feature point to be used are determined from each second feature point as second feature points to be used.

[0106] For example, the following formula is used to determine whether the distance between the center position of the first 3D matching box in the 3D detection box matching pair and the spatial position of each first feature point exceeds a preset distance threshold:

[0107]

[0108] Where s is a preset distance threshold, Let $\mathbf{j}$ be the center position of the first 3D matching box in the $i$-th 2D target detection box in the $i$-th frame (the center position of the first 3D matching box in the 3D detection box matching pair). The spatial position of the l-th first feature point extracted from the j-th two-dimensional target detection box in the i-th frame (the spatial position of the l-th first feature point in the set of first feature points).

[0109] Step 5: Determine the matching value based on the number of first feature points to be used, the number of second feature points to be used, the spatial position of each first feature point to be used, the spatial position of each second feature point to be used, the center position of the first 3D matching box, and the center position of the second 3D matching box.

[0110] The matching value is a numerical value used to measure the matching effect of each pair of 3D detection boxes.

[0111] Specifically, based on the number of first feature points to be used and the number of second feature points to be used, and combined with the motion consistency between the first feature points to be used and the first 3D matching box and the second feature points to be used and the second 3D matching box, i.e. the motion consistency between two consecutive frames, the matching value of the pair of 3D detection boxes to be processed can be jointly calculated.

[0112] For example, the matching value is determined using the following formula:

[0113]

[0114] Among them, S jm The sum of the number of first-use feature points in the first 3D matching box of the j-th 2D target detection box in the i-th frame and the number of second-use feature points in the second 3D matching box in the (i+1)-th frame is given.

[0115] The center position of the first 3D matching box that matches the j-th 2D target detection box in frame i and the center position of the second 3D matching box in frame i+1. The spatial positions of the l-th first feature point of the first 3D matching box that matches the j-th 2D target detection box in frame i, and the spatial positions of the l-th second feature point of the second 3D matching box in frame i+1, where L is the total number of matched feature points, w jm and w dir These are the weights corresponding to the number of feature points and the weights corresponding to motion consistency, respectively, which can be set in advance according to actual needs.

[0116] Step 6: Based on the matching values ​​of each 3D detection box pair, determine the 3D detection box pair with the highest matching value as the target matching pair, and take the first 3D matching box in the target matching pair as the 3D target detection box.

[0117] Among them, the target matching pair is the pair with the highest matching value among the matching pairs of each 3D detection box to be processed.

[0118] Specifically, the matching values ​​of each 3D target detection box matching pair corresponding to the same 2D target detection box are compared, and the 3D target detection box matching pair with the highest matching value is determined as the target matching pair corresponding to the 2D target detection box. The first 3D matching box in the target matching pair is taken as the 3D target detection box corresponding to the 2D target detection box, so as to complete the construction of the 3D target detection box.

[0119] Building upon the above example, after determining the 3D target detection boxes from each 3D detection box to be processed, and before annotating the 3D target detection boxes in the image to be detected according to the category and confidence level of the 2D target detection boxes, and based on the size and pose information of the 3D target detection boxes, further optimization and constraints can be applied to the 3D target detection boxes. This can be achieved through the following steps:

[0120] Step 1: For each 3D target detection box in each frame of the image, reproject the 3D target detection box onto the next frame of the image to obtain a 2D projected detection box. Based on the pose of the 3D target detection box, the pose of the corresponding 3D detection box in the next frame, and the center distance between the 2D projected detection box and the 2D detection box in the next frame, construct the first constraint condition for the 3D target detection box.

[0121] In this context, the 2D projection detection box is a 2D box in the subsequent frame image that is reprojected from the 3D target detection box. The subsequent frame's 3D detection box is a second 3D detection box that corresponds to the 3D target detection box in the target matching pair; the subsequent frame's 2D detection box is a 2D detection box in the subsequent frame image that corresponds to the same target as the subsequent frame's 3D detection box.

[0122] Specifically, the first constraint is as follows:

[0123]

[0124] Among them, T i Let T be the pose of the j-th 3D object detection box in the i-th frame. i+1 Let be the pose of the 3D detection box in the j-th subsequent frame in the (i+1)-th frame. The distance between the center of the 2D projection detection box obtained by projecting the 3D target detection box in frame i to frame i+1 and the center of the 2D detection box obtained by matching detection in frame i+1 is given by the following method. The residual term constituting this error is I, where I is the number of camera frames. C r As the primary constraint objective, with C r Minimize as the objective, optimize to obtain T i .

[0125] This first constraint is not used when the 2D object detection boxes overlap or are located at the image edge, thus avoiding the use of deformed 2D object detection boxes in the optimization.

[0126] Step 2: Based on the size information of each 3D target detection box corresponding to the same target in each frame image, construct the second constraint condition of the 3D target detection box.

[0127] Specifically, residuals are constructed from the differences in length, width, and height of the three-dimensional target detection boxes corresponding to the same target in each frame of the image, thereby optimizing the size of the three-dimensional target detection boxes.

[0128] The second constraint is as follows:

[0129]

[0130] In the formula, Let the length, height, and width of the j-th 3D object detection bounding box be denoted as . Let C be the length, width, and height of the j-th 3D object detection bounding box in the i-th frame, where I is the number of camera frames and J is the number of matched 3D object detection bounding boxes. dd As the second constraint objective, with C dd Minimize as the objective, optimize to obtain

[0131] Step 3: For each 3D target detection box in each frame of the image, construct the third constraint condition of the 3D target detection box based on the position of the 3D target detection box, the position of the 3D target detection box in the next frame, the spatial position of each first feature point to be used associated with the 3D target detection box, and the spatial position of each second feature point to be used in the 3D target detection box in the next frame.

[0132] Specifically, residuals are constructed by comparing the first and second feature points to be used with the motion consistency of the 3D target detection box in consecutive frames, thereby optimizing the position of the 3D target detection box. The third constraint is as follows:

[0133]

[0134] in, Let j be the position of the j-th 3D object detection box, i.e., the geometric center position. This represents the position of the 3D detection box in the following frame corresponding to the j-th 3D object detection box, which is the geometric center position of the 3D detection box corresponding to the 3D object detection box in the (i+1)-th frame. Let l be the spatial location of the first feature point to be used corresponding to the j-th 3D object detection box. Let L be the spatial location of the l-th second feature point to be used corresponding to the j-th 3D object detection box (the second feature point to be used is in the (i+1)-th frame), where I is the number of camera frames and L is the number of matched feature points in the 3D object detection box, i.e., the number of first feature points to be used. pd As the third constraint objective, with C pdMinimize as the objective, optimize to obtain

[0135] Step 4: Optimize each 3D object detection box in each frame of the image according to the first constraint, the second constraint, and the third constraint. Update each 3D object detection box according to the optimized 3D object detection box and update the size and pose information of each 3D object detection box according to the size and pose information of the optimized 3D object detection box.

[0136] Specifically, by combining the above constraints—namely, the first, second, and third constraints—and configuring different weight coefficients, optimized 3D object detection boxes, i.e., size and pose information, can be obtained. Then, the optimized 3D object detection boxes replace the original 3D object detection boxes, and the optimized 3D object detection boxes replace the original 3D object detection boxes with their size and pose information.

[0137] Based on the above example, the 3D target detection box can be labeled in the following way:

[0138] If the category corresponding to the 2D object detection box is different from the category corresponding to the matching 2D object detection box, then the maximum confidence value is determined based on the confidence value of the 2D object detection box and the confidence value of the matching 2D object detection box, and the category corresponding to the maximum confidence value is determined as the category of the 3D object detection box.

[0139] The 3D object detection boxes and their categories are labeled in the image to be detected based on their size and pose information.

[0140] Specifically, if the category identified by the matching 2D object detection box in a subsequent frame image differs from that of the matching 2D object detection box, then the confidence scores of the two are compared, and the category corresponding to the maximum confidence score is taken as the category of the object corresponding to the 2D object detection box. By performing an inverse projection transformation on the 3D object detection box, the pose and size information of the corresponding 3D object detection box on the image to be detected can be obtained, and its category can be labeled, thus completing the automatic labeling of the 3D object detection box.

[0141] The 3D target detection bounding box annotation method provided in this embodiment determines each 2D target detection bounding box, its category, and confidence level based on the image to be detected. It also determines the camera pose and reconstructs environmental information based on the image and detection data, avoiding the use of LiDAR and millimeter-wave radar data. For each 2D target detection bounding box, it determines a set of vanishing points corresponding to each preset sampling heading angle based on the camera intrinsic parameter matrix, camera rotation matrix, and preset sampling heading angles. Furthermore, for each set of vanishing points, it determines multiple corner point sets based on the vanishing point set, the 2D target detection bounding box, and preset upper edge box sampling points. Based on the corner point set, camera intrinsic parameter matrix, camera pose, and the 3D ground in the reconstructed environment information, multiple 3D detection boxes to be processed, along with their size and pose information, are determined. Furthermore, for each 2D target detection box, a 3D target detection box is determined from each 3D target detection box based on the 2D target detection box, the 2D detection boxes in each subsequent frame, the 3D detection boxes to be processed, and the 3D detection boxes in each subsequent frame. The 3D target detection boxes are then labeled in the image to be detected. This approach combines detection data with 2D target detection boxes to improve the accuracy and speed of 3D target detection box labeling, while reducing the workload of manual labeling.

[0142] Figure 5 This is a schematic diagram of the structure of a three-dimensional target detection bounding box annotation device according to an embodiment of this disclosure. Figure 5 As shown: The device includes: a two-dimensional target detection box construction module 510, a vanishing point set determination module 520, a corner point set determination module 530, a three-dimensional target detection box determination module 540, and a three-dimensional target detection box annotation module 550.

[0143] The system includes a two-dimensional target detection box construction module 510, which determines each two-dimensional target detection box, its corresponding category, and confidence level based on the image to be detected, and determines the camera pose and reconstructed environmental information based on the image to be detected and the detection data corresponding to the image to be detected; wherein the detection data includes at least one of inertial measurement data, wheel speed measurement data, and global positioning data; a vanishing point set determination module 520, which determines, for each two-dimensional target detection box, a vanishing point set corresponding to each preset sampling heading angle based on the camera intrinsic parameter matrix, the camera rotation matrix, and each preset sampling heading angle; and a corner point set determination module 530, which determines, for each vanishing point set, a corner point set corresponding to each upper edge detection box sampling point based on the vanishing point set, the two-dimensional target detection box, and preset upper edge detection box sampling points; and a three-dimensional target detection box... The bounding box determination module 540 is used to determine, for each set of corner points, a 3D bounding box to be processed corresponding to the set of corner points, as well as the size and pose information of the 3D bounding box to be processed. The 3D target detection box annotation module 550 is used to determine, for each 2D target detection box, a 3D target detection box from each 3D bounding box to be processed, based on the 2D target detection box, each subsequent 2D detection box in the next frame of the image to be detected, each 3D bounding box to be processed corresponding to the 2D target detection box, and each subsequent 3D detection box to be processed corresponding to each subsequent 2D detection box. The 3D target detection box is then annotated in the image to be detected according to the category and confidence level of the 2D target detection box, and according to the size and pose information of the 3D target detection box.

[0144] Based on the above example, optionally, after determining the three-dimensional detection boxes to be processed corresponding to the set of corner points, as well as the size and pose information of the three-dimensional detection boxes to be processed, the method further includes: a preliminary screening module, used to sample each edge of each three-dimensional detection box to be processed, and determine each edge sampling point; determine the distance error based on each edge sampling point and the two-dimensional target detection box corresponding to the three-dimensional detection box to be processed; determine the shape error based on the current aspect ratio, current height aspect ratio, preset maximum aspect ratio, preset minimum aspect ratio, preset maximum height aspect ratio, and preset minimum height aspect ratio of the three-dimensional detection box to be processed; determine the total error of the three-dimensional detection box to be processed based on the distance error and the shape error; and for each two-dimensional target detection box, screen each three-dimensional detection box to be processed based on the total error of each three-dimensional detection box to be processed corresponding to the two-dimensional target detection box, and update each three-dimensional detection box to be processed corresponding to the two-dimensional target detection box.

[0145] Based on the above example, optionally, the vanishing point set determination module 520 is further configured to determine the vanishing point set corresponding to the preset sampled heading angle for each preset sampled heading angle using the following formula:

[0146]

[0147]

[0148]

[0149] Where K is the camera intrinsic parameter matrix, R is the camera rotation matrix, yaw is the preset sampling heading angle, and VP is the camera rotation matrix. x VP y VP z Each vanishing point is a vanishing point in the vanishing point set;

[0150] If at least one vanishing point in the vanishing point set contains 0 in any of its three dimensions, then the set of vanishing points corresponding to the preset sampled heading angle is determined using the following formula:

[0151]

[0152]

[0153]

[0154] Where roll is the preset roll angle and pitch is the preset pitch angle.

[0155] Based on the above example, optionally, the three-dimensional detection box determination module 540 is further configured to determine the three-dimensional coordinates of each bottom corner point in the corner point set using the following formula:

[0156]

[0157]

[0158] in, Let K be the three-dimensional coordinates of the i-th bottom corner point. -1 p i Let be the back projection ray corresponding to the i-th bottom corner point in 3D space, K be the camera intrinsic parameter matrix, [n,m] represent the 3D ground in the reconstructed environment information, n be the normal vector of the 3D ground in the camera coordinate system, and m be the distance from the camera center to the 3D ground;

[0159] Based on the camera pose and the 3D coordinates of each bottom corner point, determine the 3D coordinates of each top corner point in the corner point set; based on the 3D coordinates of each top corner point and each bottom corner point in the corner point set, determine the 3D detection box to be processed corresponding to the corner point set, as well as the size and pose information of the 3D detection box to be processed.

[0160] Based on the above example, optionally, the 3D target detection box annotation module 550 is further configured to extract feature points from the 2D target detection box to obtain a first feature point set, extract feature points from each subsequent frame 2D detection box to obtain each second feature point set, and determine the matching 2D detection box corresponding to the 2D target detection box based on the first feature point set, each second feature point set, and the intersection-union ratio between the 2D target detection box and each subsequent frame 2D detection box; and determine the matching 2D detection box corresponding to the 2D target detection box based on the first matching heading angle, first length, first width, and first... The following parameters are used to determine the matching pairs of three-dimensional detection boxes: height, second heading angle to be matched, second length, second width, and second height of each subsequent frame's three-dimensional detection box corresponding to the matching two-dimensional detection box, matching heading angle threshold, matching length threshold, matching width threshold, and matching height threshold. The three-dimensional detection box matching pairs include a first three-dimensional matching box and a second three-dimensional matching box. The first three-dimensional matching box is a three-dimensional detection box in the image to be detected, and the second three-dimensional matching box is a three-dimensional detection box in the subsequent frame image that matches the first three-dimensional matching box. Based on the first feature point... The pixel positions of each first feature point in the set, the pixel positions of each second feature point in the second feature point set, the depth information of each first feature point, the depth information of each second feature point, the camera pose of the image to be detected, and the camera pose of the next frame image are used to determine the spatial positions of the feature points of each first feature point and the feature points of each second feature point. For each pair of 3D detection box matching pairs, based on the center position of the first 3D matching box in the 3D detection box matching pair, the spatial positions of the feature points of each first feature point, and a preset distance threshold, a first target feature point is determined from each first feature point. Feature points are identified, and second feature points are determined from each second feature point based on each first feature point to be used. Matching values ​​are determined based on the number of first feature points to be used, the number of second feature points to be used, the spatial positions of each first feature point to be used, the spatial positions of each second feature point to be used, the center position of the first 3D matching box, and the center position of the second 3D matching box. Based on the matching values ​​of each 3D detection box matching pair, the 3D detection box matching pair with the highest matching value is determined as the target matching pair, and the first 3D matching box in the target matching pair is used as the 3D target detection box.

[0161] Based on the above example, optionally, after determining the 3D target detection boxes from each 3D detection box to be processed, and before labeling the 3D target detection boxes in the image to be detected according to the category and confidence level corresponding to the 2D target detection boxes, and according to the size and pose information of the 3D target detection boxes, the method further includes: a constraint optimization module, used to reproject each 3D target detection box in each frame image onto the next frame image to obtain a 2D projected detection box, and construct a first constraint condition for the 3D target detection box based on the pose of the 3D target detection box, the pose of the 3D detection box in the next frame corresponding to the 3D target detection box, and the center distance between the 2D projected detection box and the 2D detection box in the next frame; wherein, the 3D detection box in the next frame is the second 3D detection box in the target matching pair that has a corresponding relationship with the 3D target detection box; the 2D detection box in the next frame is the next frame Two-dimensional detection boxes in the image that correspond to the same target as the three-dimensional detection boxes in the subsequent frame; based on the size information of each three-dimensional target detection box corresponding to the same target in each frame image, construct the second constraint condition of the three-dimensional target detection box; for each three-dimensional target detection box in each frame image, construct the third constraint condition of the three-dimensional target detection box based on the position of the three-dimensional target detection box, the position of the three-dimensional detection box in the subsequent frame, the spatial position of each first feature point to be used associated with the three-dimensional target detection box, and the spatial position of each second feature point to be used in the three-dimensional detection box in the subsequent frame; optimize each three-dimensional target detection box in each frame image based on the first constraint condition, the second constraint condition, and the third constraint condition, and update each three-dimensional target detection box based on the optimized three-dimensional target detection box, and update the size information and pose information of each three-dimensional target detection box based on the size information and pose information of the optimized three-dimensional target detection box.

[0162] Based on the above example, optionally, the 3D target detection box annotation module 550 is further configured to: if the category corresponding to the 2D target detection box is different from the category corresponding to the matching 2D detection box, determine the maximum confidence value based on the confidence value corresponding to the 2D target detection box and the confidence value corresponding to the matching 2D detection box, and determine the category corresponding to the maximum confidence value as the category of the 3D target detection box; and annotate the 3D target detection box and its category in the image to be detected according to the size information and pose information of the 3D target detection box.

[0163] The three-dimensional target detection bounding box annotation device provided in this embodiment can execute the steps in the three-dimensional target detection bounding box annotation method provided in this embodiment, and has the execution steps and beneficial effects, which will not be repeated here.

[0164] Figure 6This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 6 It shows a schematic diagram of a structure suitable for implementing the electronic device 600 in the embodiments of this disclosure. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0165] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes to implement the methods of embodiments as described in this disclosure, based on a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of electronic device 600. The processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0166] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the three-dimensional target detection bounding box annotation method as described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0167] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0168] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0169] Based on the image to be detected, determine each two-dimensional target detection box, the category corresponding to each two-dimensional target detection box, and the confidence level. Based on the image to be detected and the detection data corresponding to the image to be detected, determine the camera pose and reconstruct environmental information. The detection data includes at least one of inertial measurement data, wheel speed measurement data, and global positioning data.

[0170] For each two-dimensional target detection box, the set of vanishing points corresponding to each preset sampling heading angle is determined based on the camera intrinsic parameter matrix, the camera rotation matrix, and each preset sampling heading angle.

[0171] For each set of vanishing points, based on the set of vanishing points, the two-dimensional target detection box, and the preset sampling points of each upper edge box, a set of corner points corresponding to the sampling points of each upper edge detection box is determined;

[0172] For each set of corner points, based on the set of corner points, the camera intrinsic parameter matrix, the camera pose, and the 3D ground in the reconstructed environment information, the 3D detection box corresponding to the set of corner points, as well as the size and pose information of the 3D detection box, are determined.

[0173] For each two-dimensional target detection box, a three-dimensional target detection box is determined from each three-dimensional target detection box based on the two-dimensional target detection box, each subsequent two-dimensional detection box in the next frame of the image to be detected, each three-dimensional target detection box corresponding to the two-dimensional target detection box, and each subsequent three-dimensional detection box corresponding to each subsequent two-dimensional detection box. Then, based on the category and confidence level of the two-dimensional target detection box, the three-dimensional target detection box is labeled in the image to be detected according to its size and pose information.

[0174] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0175] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0176] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A method for annotating bounding boxes for three-dimensional target detection, characterized in that, The method includes: Based on the image to be detected, determine each two-dimensional target detection box, the category corresponding to each two-dimensional target detection box, and the confidence level. Based on the image to be detected and the detection data corresponding to the image to be detected, determine the camera pose and reconstruct the environmental information. The detection data includes at least one of inertial measurement data, wheel speed measurement data, and global positioning data. For each two-dimensional target detection box, based on the camera intrinsic parameter matrix, camera rotation matrix, and each preset sampling heading angle, a set of vanishing points corresponding to each preset sampling heading angle is determined; wherein, the set of vanishing points is a set consisting of three vanishing points corresponding to the two-dimensional target detection box, and the vanishing point is the point where the parallel line in the image to be detected disappears at a distance; For each set of vanishing points, based on the set of vanishing points, the two-dimensional target detection box, and the preset sampling points of each upper edge box, a set of corner points corresponding to the sampling points of each upper edge detection box is determined; For each set of corner points, based on the set of corner points, the camera intrinsic parameter matrix, the camera pose, and the 3D ground in the reconstructed environment information, the 3D detection box corresponding to the set of corner points, as well as the size and pose information of the 3D detection box, are determined. For each two-dimensional target detection box, a three-dimensional target detection box is determined from each three-dimensional target detection box based on the two-dimensional target detection box, each subsequent two-dimensional detection box in the next frame of the image to be detected, each three-dimensional target detection box corresponding to the two-dimensional target detection box, and each subsequent three-dimensional detection box corresponding to each subsequent two-dimensional detection box. Then, based on the category and confidence level of the two-dimensional target detection box, the three-dimensional target detection box is labeled in the image to be detected according to its size and pose information.

2. The method according to claim 1, characterized in that, After determining the 3D detection bounding box corresponding to the set of corner points, and the size and pose information of the 3D detection bounding box, the method further includes: For each 3D detection box to be processed, each edge of the 3D detection box is sampled to determine the sampling points of each edge; The distance error is determined based on each edge sampling point and the two-dimensional target detection box corresponding to the three-dimensional detection box to be processed; The shape error is determined based on the current aspect ratio, current height-to-width ratio, preset maximum aspect ratio, preset minimum aspect ratio, preset maximum height-to-width ratio, and preset minimum height-to-width ratio of the three-dimensional detection box to be processed. The total error of the three-dimensional detection box to be processed is determined based on the distance error and the shape error. For each two-dimensional target detection box, the three-dimensional target detection boxes are filtered according to the total error of each three-dimensional target detection box corresponding to the two-dimensional target detection box, and the three-dimensional target detection boxes corresponding to the two-dimensional target detection box are updated.

3. The method according to claim 1, characterized in that, The step of determining the set of vanishing points corresponding to each preset sampling heading angle based on the camera intrinsic parameter matrix, camera rotation matrix, and each preset sampling heading angle includes: For each preset sampling heading angle, the set of vanishing points corresponding to the preset sampling heading angle is determined by the following formula: ; ; ; Where K is the camera intrinsic parameter matrix, R is the camera rotation matrix, and yaw is the preset sampling heading angle. Each vanishing point is a set of vanishing points. If at least one vanishing point in the vanishing point set contains 0 in any of its three dimensions, then the set of vanishing points corresponding to the preset sampled heading angle is determined using the following formula: ; ; ; Where roll is the preset roll angle and pitch is the preset pitch angle.

4. The method according to claim 1, characterized in that, For each set of corner points, based on the set of corner points, the camera intrinsic parameter matrix, the camera pose, and the 3D ground in the reconstructed environment information, the 3D bounding box corresponding to the set of corner points, as well as the size and pose information of the 3D bounding box, are determined, including: For each set of corner points, the three-dimensional coordinates of each bottom corner point in the set are determined using the following formula: ; ; in, Let i be the three-dimensional coordinates of the i-th bottom corner point. Let be the back projection ray corresponding to the i-th bottom corner point in three-dimensional space. For the camera intrinsic parameter matrix, This represents the three-dimensional ground in the reconstructed environmental information. It is the normal vector of the three-dimensional ground in the camera coordinate system. The distance from the camera center to the three-dimensional ground surface; Based on the camera pose and the three-dimensional coordinates of each bottom corner point, determine the three-dimensional coordinates of each top corner point in the corner point set; Based on the three-dimensional coordinates of each top corner point and each bottom corner point in the corner point set, determine the three-dimensional detection box to be processed corresponding to the corner point set, as well as the size and pose information of the three-dimensional detection box to be processed.

5. The method according to claim 1, characterized in that, The step of determining a 3D target detection box from each 3D target detection box based on the 2D target detection box, each subsequent 2D detection box in the next frame of the image to be detected, each 3D target detection box corresponding to the 2D target detection box, and each subsequent 3D detection box corresponding to each subsequent 2D detection box includes: Feature points are extracted from the two-dimensional target detection box to obtain a first feature point set. Feature points are extracted from each subsequent frame two-dimensional detection box to obtain a second feature point set. Based on the first feature point set, each second feature point set, and the intersection-union ratio between the two-dimensional target detection box and each subsequent frame two-dimensional detection box, a matching two-dimensional detection box corresponding to the two-dimensional target detection box is determined. Based on the first matching heading angle, first length, first width, and first height of each three-dimensional detection box corresponding to the two-dimensional target detection box, the second matching heading angle, second length, second width, and second height of each subsequent frame three-dimensional detection box corresponding to the matched two-dimensional detection box, a matching heading angle threshold, a matching length threshold, a matching width threshold, and a matching height threshold, a three-dimensional detection box matching pair is determined; wherein, the three-dimensional detection box matching pair includes a first three-dimensional matching box and a second three-dimensional matching box; the first three-dimensional matching box is a three-dimensional detection box in the image to be detected, and the second three-dimensional matching box is a three-dimensional detection box in the subsequent frame image that matches the first three-dimensional matching box; Based on the pixel position of each first feature point in the first feature point set, the pixel position of each second feature point in the second feature point set, the depth information of each first feature point, the depth information of each second feature point, the camera pose of the image to be detected, and the camera pose of the next frame image, the feature point spatial position of each first feature point and the feature point spatial position of each second feature point are determined. For each pair of three-dimensional detection boxes to be processed, based on the center position of the first three-dimensional matching box in the pair, the spatial position of the feature points of each first feature point, and the preset distance threshold, a first feature point to be used is determined from each first feature point, and a second feature point to be used is determined from each second feature point based on the first feature point to be used. The matching value is determined based on the number of the first feature points to be used, the number of the second feature points to be used, the spatial position of each first feature point to be used, the spatial position of each second feature point to be used, the center position of the first 3D matching box, and the center position of the second 3D matching box. Based on the matching value of each 3D detection box pair, the 3D detection box pair with the highest matching value is determined as the target matching pair, and the first 3D matching box in the target matching pair is taken as the 3D target detection box.

6. The method according to claim 5, characterized in that, After determining the 3D target detection boxes from each 3D target detection box, and before labeling the 3D target detection boxes in the image to be detected according to the category and confidence level of the 2D target detection boxes, and according to the size and pose information of the 3D target detection boxes, the method further includes: For each 3D target detection box in each frame image, the 3D target detection box is reprojected into the next frame image to obtain a 2D projected detection box. Based on the pose of the 3D target detection box, the pose of the corresponding 3D detection box in the next frame, and the center distance between the 2D projected detection box and the 2D detection box in the next frame, a first constraint condition for the 3D target detection box is constructed. The 3D detection box in the next frame is the second 3D detection box in the target matching pair that corresponds to the 3D target detection box. The 2D detection box in the next frame is the 2D detection box in the next frame image that corresponds to the same target as the 3D detection box in the next frame. Based on the size information of each 3D target detection box corresponding to the same target in each frame image, a second constraint condition for the 3D target detection box is constructed. For each 3D target detection box in each frame of the image, a third constraint condition for the 3D target detection box is constructed based on the position of the 3D target detection box, the position of the 3D detection box in the next frame, the spatial position of each first feature point to be used associated with the 3D target detection box, and the spatial position of each second feature point to be used in the 3D detection box in the next frame. Based on the first constraint, the second constraint, and the third constraint, each 3D target detection box in each frame image is optimized, and each 3D target detection box is updated based on the optimized 3D target detection box. The size information and pose information of each 3D target detection box are also updated based on the size information and pose information of the optimized 3D target detection box.

7. The method according to claim 1, characterized in that, The step of labeling the three-dimensional target detection box in the image to be detected according to the category and confidence level corresponding to the two-dimensional target detection box, and according to the size and pose information of the three-dimensional target detection box, includes: If the category corresponding to the two-dimensional target detection box is different from the category corresponding to the matching two-dimensional detection box, then the maximum confidence value is determined based on the confidence value corresponding to the two-dimensional target detection box and the confidence value corresponding to the matching two-dimensional detection box, and the category corresponding to the maximum confidence value is determined as the category of the three-dimensional target detection box. The three-dimensional target detection boxes and their categories are labeled in the image to be detected according to their size and pose information.

8. A three-dimensional target detection bounding box annotation device, characterized in that, include: A two-dimensional target detection box construction module is used to determine each two-dimensional target detection box, the category corresponding to each two-dimensional target detection box, and the confidence level based on the image to be detected, and to determine the camera pose and reconstruct environmental information based on the image to be detected and the detection data corresponding to the image to be detected; wherein, the detection data includes at least one of inertial measurement data, wheel speed measurement data, and global positioning data; The vanishing point set determination module is used to determine the vanishing point set corresponding to each preset sampling heading angle for each two-dimensional target detection box, based on the camera intrinsic parameter matrix, the camera rotation matrix, and each preset sampling heading angle; wherein, the vanishing point set is a set consisting of three vanishing points corresponding to the two-dimensional target detection box, and the vanishing point is the point where the parallel line in the image to be detected disappears at a distance; The corner point set determination module is used to determine, for each vanishing point set, the corner point set corresponding to the sampling points of each upper edge detection box based on the vanishing point set, the two-dimensional target detection box, and the preset sampling points of each upper edge box; The 3D detection bounding box determination module is used to determine, for each set of corner points, the 3D detection bounding box corresponding to the set of corner points, as well as the size and pose information of the 3D detection bounding box, based on the set of corner points, the camera intrinsic parameter matrix, the camera pose, and the 3D ground in the reconstructed environment information. The 3D target detection box annotation module is used to determine a 3D target detection box from each 3D target detection box based on the 2D target detection box, each subsequent 2D detection box in the next frame of the image to be detected, each 3D target detection box corresponding to the 2D target detection box, and each subsequent 3D detection box corresponding to each subsequent 2D detection box. The module then annotates the 3D target detection box in the image to be detected according to the category and confidence level of the 2D target detection box, and based on the size and pose information of the 3D target detection box.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional target detection bounding box annotation method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the three-dimensional target detection bounding box annotation method as described in any one of claims 1-7.

Images