Three-dimensional target detection method and system based on two-dimensional detection result prompt

Abstract

The invention discloses a three-dimensional target detection method and system based on two-dimensional detection result prompt, in the method, a two-dimensional target detector and a three-dimensional target detector are respectively constructed, and the three-dimensional target detector comprises an image encoder, a prompt encoder module, a feature fusion module, a three-dimensional space decoder and a direction fine adjustment module. Respectively converting the two-dimensional detection frame and the target category of each target in the monocular image into features corresponding to the two-dimensional detection frame and the target category through a prompt encoder module; a feature fusion module fuses the features corresponding to the two-dimensional detection frame and the target category and the features of the monocular image into fused features; and finally, a direction fine adjustment module is used as a post-processing module, and the direction of the output three-dimensional detection frame is continuously adjusted according to the two-dimensional detection frame. Compared with an existing three-dimensional target detection method, the method and the system have the advantage that the detection precision is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional target detection, and in particular to a three-dimensional target detection method and system based on two-dimensional detection result prompts. Background Art

[0002] 3D object detection technology is a key development direction for smart cities, intelligent driving, and smart manufacturing. The goal of 3D object detection is to recover the 6-degree-of-freedom pose and 3D bounding box dimensions of all objects of interest in a scene. Currently, many 3D object detection systems use either binocular cameras or augment their sensors with lidar and radar. Common sensor-based 3D object detection methods suffer from poor performance due to increasing scene complexity, with missed and false detections common. For example, ambulances, tricycles, occluded vehicles or pedestrians, and smaller vehicles or pedestrians are often missed during detection.

[0003] The common sensor-based two-dimensional target detection method still performs very stably in more complex scenarios due to its advantages such as easy acquisition of training data, large data volume, and more mature methods. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention proposes a 3D object detection method and system based on 2D detection result prompts, which can fully utilize 2D object detection results to improve the accuracy of 3D object detection. The specific technical solution is as follows:

[0005] A three-dimensional target detection method based on two-dimensional detection result prompts includes the following steps:

[0006] Step 1: Obtain an image dataset, where each monocular image in the image dataset is annotated with a 3D detection box and target category for each target; project the 3D detection box of each target onto the image coordinate system based on the known intrinsic and extrinsic parameters of the image sensor that captured the monocular image to obtain a 2D detection box for each target; construct a 2D target detection training set using the monocular image and the 2D detection box and target category of each target in the image;

[0007] Step 2: Construct a two-dimensional object detector and train it using the two-dimensional object detection training set;

[0008] Step 3: Construct a 3D object detector and train it using the image dataset and the output of the trained 2D object detector; the 3D object detector includes an image encoder, a cue encoder module, a feature fusion module, a 3D space decoder, and a direction fine-tuning module;

[0009] The image encoder converts the input monocular image into features of the monocular image; the prompt encoder module converts the two-dimensional detection box and target category of each target in the monocular image into features corresponding to the two-dimensional detection box and target category respectively; the feature fusion module fuses the features corresponding to the two-dimensional detection box and target category, and the features of the monocular image into fused features; the projection module converts the fused features into a voxel feature map; the three-dimensional space decoder converts the voxel feature map into a three-dimensional detection box and target category of each target in the input monocular image;

[0010] Step 4: Use the trained two-dimensional target detector and three-dimensional target detector to perform three-dimensional target detection, and the direction fine-tuning module continuously adjusts the direction of the three-dimensional detection frame according to the two-dimensional detection frame output by the two-dimensional target detector, so that it is as close as possible to the direction of the two-dimensional detection frame, thereby updating the output three-dimensional detection frame.

[0011] A three-dimensional target detection system based on two-dimensional detection result prompts, the system includes an image acquisition module, a two-dimensional target detection module and a three-dimensional target detection module;

[0012] The image acquisition module is used to construct an image dataset, in which each monocular image in the image dataset is annotated with a three-dimensional detection frame and target category of each target; and based on the known internal and external parameters of the image sensor that captured the monocular image, the three-dimensional detection frame of each target is projected into the image coordinate system to obtain a two-dimensional detection frame of each target; and a two-dimensional target detection training set is constructed using the monocular image and the two-dimensional detection frame and target category of each target in the image;

[0013] The two-dimensional target detection module has a built-in trained two-dimensional target detector for detecting targets in an input monocular image and outputting a two-dimensional detection frame and target category for each target in the monocular image;

[0014] The 3D object detection module has a built-in trained 3D object detector for performing object detection on an input monocular image and outputting a 3D detection frame and an object category for each object in the monocular image. During the object detection process, the direction of the 3D detection frame of each object in the output monocular image is closest to the direction of the 2D detection frame based on the 2D detection frame and object category of each object output by the 2D object detection module.

[0015] The three-dimensional target detector includes an image encoder, a prompt encoder module, a feature fusion module, a projection module, a three-dimensional space decoder and a direction fine-tuning module; the image encoder is used to convert the input monocular image into the features of the monocular image; the prompt encoder module is used to convert the two-dimensional detection box and target category of each target in the monocular image into features corresponding to the two-dimensional detection box and the target category respectively; the feature fusion module is used to fuse the features corresponding to the two-dimensional detection box and the target category, and the features of the monocular image into fused features; the projection module is used to convert the fused features into a voxel feature map; the three-dimensional space decoder is used to convert the voxel feature map into a three-dimensional detection box and target category of each target in the input monocular image; the direction fine-tuning module is used to continuously adjust the direction of the three-dimensional detection frame according to the two-dimensional detection frame, so that it is closest to the direction of the two-dimensional detection frame, and update the three-dimensional detection frame.

[0016] An electronic device includes one or more processors and a storage device for storing one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements a three-dimensional target detection method based on two-dimensional detection result prompts.

[0017] A computer-readable storage medium stores a program, which, when executed by a processor, implements a three-dimensional target detection method based on two-dimensional detection result prompts.

[0018] The beneficial effects of the present invention are as follows:

[0019] The present invention can make full use of the results of two-dimensional target detection, use them as prompt features, and fuse them with the features of the monocular image to train the decoder of the three-dimensional target detector; and design a direction fine-tuning module as a post-processing module, which also continuously adjusts the direction of the output three-dimensional detection frame according to the two-dimensional detection frame, so that the accuracy of the three-dimensional target detection method of the present invention is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of a three-dimensional target detection method based on two-dimensional detection result prompts according to an embodiment of the present invention.

[0021] Figure 2 Schematic diagram of a prompt encoder module according to an embodiment of the present invention.

[0022] Figure 3 Schematic diagram of feature fusion performed by a feature fusion module according to an embodiment of the present invention.

[0023] Figure 4 FIG. 1 is a schematic diagram of a direction fine-tuning module according to an embodiment of the present invention performing direction adjustment on a three-dimensional detection frame.

[0024] Figure 5 3 is a comparison chart of the detection results of the method according to the embodiment of the present invention and the BEVHeight method. DETAILED DESCRIPTION

[0025] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0026] On the one hand, one embodiment of the present invention provides a three-dimensional detection method based on two-dimensional detection result prompts, such as Figure 1 As shown, the following steps are included:

[0027] Step 1: Obtain an image dataset. Each monocular image in the image dataset is annotated with the 3D detection frame and target category of each target. Based on the known internal and external parameters of the image sensor that captured the monocular image, project the 3D detection frame of each target into the image coordinate system to obtain the 2D detection frame of each target. Use the monocular image and the 2D detection frame and target category of each target in the image to construct a 2D target detection training set.

[0028] Step 2: Construct a two-dimensional target detector and train it using the two-dimensional target detection training set; the two-dimensional target detector includes a visual encoder and a two-dimensional decoder, the visual encoder is used to convert the input monocular image into the features of the monocular image; the two-dimensional decoder includes a two-dimensional target detection head and a post-processing module, which is used to convert the features of the monocular image into a two-dimensional detection box and target category for each target in the monocular image.

[0029] Step 3: Build a 3D object detector and train it using the image dataset and the output results of the trained 2D object detector; the 3D object detector includes an image encoder, a hint encoder module, a feature fusion module, a projection module, a 3D space decoder and a direction fine-tuning module.

[0030] (1) Image encoder

[0031] The image encoder converts the input monocular image into monocular image features.

[0032] (2) Prompt encoder module

[0033] The prompt encoder module converts the two-dimensional detection box and target category of each target in the monocular image into features corresponding to the two-dimensional detection box and target category respectively. Figure 2 As shown, the prompt encoder module includes a normalization layer, a fully connected layer, and a splicing layer. The normalization layer is used to normalize the coordinates of the upper left corner and lower right corner of the input two-dimensional detection box to obtain a 2×2 matrix A, A∈R 2×2The fully connected layer is used to multiply the matrix A with a random Gaussian initialized matrix B, and add a learnable bias matrix C to obtain the matrix D, D = A × B + C, where B ∈ R 2×M , C∈R 2×M , M represents the number of columns of the preset matrix B; the splicing layer is used to repeat the category label corresponding to the input two-dimensional detection frame M times to obtain an M-dimensional vector, and splice the M-dimensional vector with the matrix D in the row direction to obtain the spliced ​​feature F prompt , F prompt ∈R 3×M .

[0034] (3) Feature fusion module

[0035] The feature fusion module fuses the features corresponding to the two-dimensional detection box and the target category, as well as the features of the monocular image into fusion features. Figure 3 As shown, the feature fusion module includes a Transformer-based self-attention layer, a Transformer-based cross-attention layer, a fully connected layer, and a normalization layer; the Transformer-based self-attention layer is used to transform the feature F output by the prompt encoder module prompt Perform self-attention operation and then input it into the normalization layer, which normalizes it to obtain feature f1;

[0036] The Transformer cross attention layer is used to perform a cross attention operation on the feature f1 and the feature F(X) output by the three-dimensional object detector, and then inputs the result into the fully connected layer and the normalization layer to obtain the feature f2; at this time, the query based on the Transformer cross attention layer is f1, and the key and value are both F(X);

[0037] The Transformer-based cross-attention layer is also used to perform a cross-attention operation on the feature f2 and the feature F(X) output by the three-dimensional object detector, and then input it into the normalization layer to obtain the fused feature f as the output of the feature fusion module; at this time, the query based on the Transformer cross-attention layer is F(X), and the key and value are both f2.

[0038] (4) Projection module

[0039] The projection module converts the fusion feature f into a voxel feature map. The projection module includes a voxel space construction unit, a feature conversion unit, a depth estimation unit, a feature fusion unit, and a space projection unit.

[0040] The voxel space construction unit is used to construct a three-dimensional voxel space, where the X-axis range is (0, X max ), the Y-axis range is (−Y range / 2, Y range / 2), the Z-axis range is (Z min , Z max ), each voxel has a size of V x 、V y 、V z , so that the dimension of the three-dimensional voxel space is N x × N y × N z .

[0041] The feature conversion unit is a 1×1×C in ×C mid The convolution kernel performs convolution operation on the input feature map to generate an intermediate feature map (size H×W×C mid ), and split it into the first sub-feature map (size H×W×C f ) and the second sub-feature map (size H×W×C d ).

[0042] The depth estimation unit applies the Softmax activation function to the second sub-feature map to estimate the depth probability distribution, where the depth range (0, D max ) is divided into C d discrete intervals;

[0043] The feature fusion unit is used to expand the first sub-feature map to H×W×1×C f And expand the second sub-feature map to H×W×C d ×1, and then perform bitwise multiplication to generate a fused feature map (size H×W×C d ×C f ).

[0044] The spatial projection unit projects the fusion feature map into the 3D voxel space based on the internal and external parameters of the sensor, and passes it through the second convolution kernel (size 1×1×C f ×C out ) to extract features and output the final voxel feature map (size N x ×N y ×C out ).

[0045] (5) Decoder for three-dimensional space

[0046] The 3D decoder converts the fused features into a 3D detection box and target category for each target in the input monocular image and outputs them.

[0047] Step 4: Use the trained 2D object detector and 3D object detector to perform 3D object detection. The direction fine-tuning module continuously adjusts the direction of the 3D detection frame according to the 2D detection frame output by the 2D object detector, making it as close as possible to the direction of the 2D detection frame, thereby updating the output 3D detection frame.

[0048] Among them, Figure 4 As shown, the direction fine-tuning module specifically performs the following operations:

[0049] Based on the known internal and external parameters of the image sensor that captures the monocular image, the 3D detection frame of each target in the monocular image output by the 3D space decoder is projected into the image coordinate system to obtain the projected detection frame, and the intersection-and-union ratio (IoU) of the projected detection frame and the corresponding 2D detection frame is calculated. The direction of the 3D detection frame is continuously changed, and the projection and IoU calculation operations are repeated. The direction of the 3D detection frame corresponding to the maximum IoU is selected as the final direction to update the 3D detection frame.

[0050] Corresponding to the aforementioned embodiment of three-dimensional object detection based on two-dimensional detection result prompts, the present invention also provides an embodiment of a three-dimensional object detection system based on two-dimensional detection result prompts, the system comprising: the system comprises an image acquisition module, a two-dimensional object detection module and a three-dimensional object detection module, wherein:

[0051] The image acquisition module is used to construct an image dataset. Each monocular image in the image dataset is annotated with the 3D detection box and target category of each target. Based on the known internal and external parameters of the image sensor that captured the monocular image, the 3D detection box of each target is projected into the image coordinate system to obtain the 2D detection box of each target. The 2D target detection training set is constructed using the monocular image and the 2D detection box and target category of each target in the image.

[0052] The 2D object detection module has a built-in trained 2D object detector, which is used to detect objects in the input monocular image and output the 2D detection box and object category of each object in the monocular image.

[0053] The 3D object detection module has a built-in trained 3D object detector, which is used to detect objects in the input monocular image and output the 3D detection box and object category of each object in the monocular image. During the object detection process, the direction of the 3D detection box of each object in the output monocular image is made closest to the direction of the 2D detection box based on the 2D detection box and object category prompts of each object output by the 2D object detection module.

[0054] The three-dimensional target detector includes an image encoder, a prompt encoder module, a feature fusion module, a projection module, a three-dimensional space decoder and a direction fine-tuning module; the image encoder is used to convert the input monocular image into the features of the monocular image; the prompt encoder module is used to convert the two-dimensional detection frame and target category of each target in the monocular image into features corresponding to the two-dimensional detection frame and the target category respectively; the feature fusion module is used to fuse the features corresponding to the two-dimensional detection frame and the target category, as well as the features of the monocular image into fused features; the projection module is used to convert the fused features into a voxel feature map; the three-dimensional space decoder is used to convert the voxel feature map into the three-dimensional detection frame and target category of each target in the input monocular image; the direction fine-tuning module is used to continuously adjust the direction of the three-dimensional detection frame according to the two-dimensional detection frame, so that it is closest to the direction of the two-dimensional detection frame, and update the three-dimensional detection frame.

[0055] The above three-dimensional object detection method is further explained below with reference to a specific example.

[0056] This embodiment uses the Dair-V2x roadside image dataset to construct a training dataset, including 5,000 training image data and corresponding 3D target labeling results, and 2,000 test image data and corresponding 3D target labeling results. The 3D target labeling result is recorded as Label_3D, which specifically includes the center point coordinates X1_3D, Y1_3D, and Z1_3D of the target in the 3D coordinate system, the length, width, and height L, H, and W of the target, the heading angle Yaw of the target, and the target category Cls.

[0057] First, the two-dimensional object detector Detector_2D is trained.

[0058] The three-dimensional target labeling result Label_3D in the training dataset is converted into a two-dimensional target labeling result Label_2D using the internal and external parameters of the roadside camera provided by the Dair-V2x roadside image dataset. Specifically, it includes the horizontal and vertical coordinates X1_2D, Y1_2D, X2_2D, Y2_2D of the upper left and lower right corners of the target's two-dimensional detection box in the image coordinate system, and the category Cls, where the categories specifically include vehicles, cyclists, and pedestrians.

[0059] In this embodiment, ResNet101 is used as the encoder for the two-dimensional object detector. The decoder for two-dimensional object detection includes an Anchor Free-based two-dimensional object detection head and a post-processing module. The two-dimensional object detection head outputs the target center coordinates Center_x and Center_y, offsets Offset_x and Offset_y, and the target category Cls. The post-processing module is used to set target filtering with confidence thresholds and non-maximum suppression. During the training phase, the target center coordinates Center_x and Center_y and category Cls are regressed using the Focal Loss function, while the offsets Offset_x and Offset_y are regressed using the L1 Loss function. The two-dimensional target detector Detector_2D is trained using 5000 training image data Image_train and the corresponding two-dimensional target annotation results Label_2D to obtain the trained Detector_2D; and the trained Detector_2D is used to infer the 5000 training image data Image_train and the 2000 test image data Image_test to obtain the two-dimensional detection boxes Box2D_train and Box2D_test for each target respectively.

[0060] Then, the 3D object detector is trained. The training dataset consists of 5,000 training images (Image_train) and the corresponding 3D object annotations (Label_3D). The test dataset consists of 2,000 test images (Image_test). The shapes of Image_train and Image_test are 864*1536*3.

[0061] For the prompt encoder module, the input is Box2D_train and Box2D_test. First, the prompt encoder module normalizes the coordinates X1_2D, Y1_2D, X2_2D, and Y2_2D of Box2D_train or Box2D_test to obtain Box2D_norm. At this time, the shape of Box2D_norm is N*2*2, where N represents the number of targets detected by Detector_2D. Then, Box2D_norm is multiplied by a random matrix B, where the elements of B are initialized with Gaussian distribution and the shape is N*2*512. Then, a learnable C (training order) is added to the result. The segment is continuously updated, and the inference stage remains unchanged), the shape size is N*2*512, so that Box2D_features=Box2D_norm*B+C is obtained, where the shape size of Box2D_features is N*2*512; the category C of Box2D_train or Box2D_test is expanded into a matrix C_expand with the same first and third dimensions as Box2D_features, that is, the shape size is N*1*512, and concatenated with Box2D_features to obtain Box2D_features_C, with a shape size of N*3*512, which is the output of the prompt encoder module.

[0062] ResNet101 is used as the encoder of the three-dimensional object detector. The encoder extracts features from 5000 training image data Image_train or Image_test to obtain the image features Image_features2, which have a shape size of 54*96*512.

[0063] The feature fusion module first expands the features Image_features2 of the image output by the encoder to the same number as Box2D_features_C output by the prompt encoder module, that is, the shape size is N*54*96*512; then self-attention operation (Self-attention) is performed on Box2D_features_C, and then it is normalized to obtain F; then cross-attention operation (Cross-attention) is performed on F and normalized to obtain G, where the Key and Value in Cross-attention are both Image_features2, and Query is F; then G is fully connected (MLP) and normalized to obtain H; finally, cross-attention is performed on H and normalized to obtain Image_features2_prompted, where the Key and Value in Cross-attention are Image_features2 and Box2D_features_C, respectively, and Query is H. Image_features2_prompted is the fusion feature of Image_features2 and Box2D_features_C output by the prompt encoder module. As the output of the feature fusion module, its shape size is N*54*96*512.

[0064] Initialize the three-dimensional voxel space, where the value ranges of the X, Y, and Z axes are [0 m, 102.4 m], [-51.2 m, 51.2 m], and [-5 m, 3 m] respectively, and the size of each voxel is [0.1 m, 0.1 m, 8 m]. Therefore, the shape and size of the three-dimensional voxel space Voxel is 1024 * 1024 * 1. Then, 5. Through a convolutional operator with a shape and size of 1 * 1 * 512 * 286, convert Image_features2_prompted into Image_features2_all with a shape and size of 54 * 96 * 286, and split it into Image_features2_f and Image_features2_d with shape and sizes of 54 * 96 * 80 and 54 * 96 * 206 respectively. Then, connect Image_features2_d to the Softmax activation function for depth estimation, that is, split 0 m to 104 m into 206 parts, and each part represents its respective depth. Image_features2_f is the image feature. Then, expand Image_features2_f and Image_features2_d into 54 * 96 * 1 * 80 and 54 * 96 * 206 * 1 respectively, and multiply them bit by bit to get Image_features2_df with a shape and size of 54 * 96 * 206 * 80. Finally, project Image_features2_df into the three-dimensional voxel space Voxel through the internal and external parameters of the roadside acquisition device, and then perform feature extraction through a convolutional operator with a shape and size of 1 * 1 * 80 * 256 to obtain Image_features2_prompted_voxel with a shape and size of 1024 * 1024 * 256.

[0065] Finally, use the three-dimensional detection head and post-processing module of CenterPoint as the decoder in the three-dimensional space to convert Image_features2_prompted_voxel into the three-dimensional detection box and target category of the target, that is, {Box3D[j], where j is the index of the three-dimensional detection target, 0 < j <= N_2D}, including the three-dimensional coordinates X1_3D, Y1_3D, Z1_3D of the target center point, length, width, and height L, H, W, yaw direction, and category Cls.

[0066] In the training stage, the x and y coordinates X1_3D, Y1_3D of the target center point and the category Cls are calculated by regression using the Focal Loss function, and the offset amounts Offset_x, Offset_y, the z coordinate of the target center point, length, width, and height L, H, W, and yaw direction are calculated by regression using the L1 Loss function.

[0067] The 3D object detector Detector_3D is trained using 5000 training image data Image_train and the corresponding 3D object annotation results Label_3D to obtain the trained Detector_3D.

[0068] Use the trained Detector_3D to infer the 2000 test image data Image_test to obtain Box3D_test as the output of the three-dimensional object detector Detector_3D.

[0069] During the testing phase, the Box2D_test output by the two-dimensional target detector Detector_2D and the Box3D_test output by the three-dimensional target detector Detector_3D are input into the direction fine-tuning module. The direction fine-tuning module projects the Box3D_test into the same two-dimensional space as the Box2D_test using the internal and external parameters of the road-side camera provided by the Dair-V2x dataset, and calculates the overlap ratio IOU between the two. The direction Yaw in the Box3D_test is continuously adjusted, and the projection and IOU calculation are repeated until the IOU reaches the maximum value. The corresponding direction Yaw is obtained and updated to the Box3D_test. The updated Box3D_test is the output of the direction fine-tuning module.

[0070] To verify the advantages of the method and system of the present invention, the authoritative 3D object detection metric mAP is used to compare with the more popular monocular vision-based 3D object detection methods on the Dair-V2x dataset. The specific results are shown in Table 1. All methods are trained and tested using the same training and test sets as the Dair-V2x dataset.

[0071] The sources of the existing methods in Table 1 are as follows:

[0072] The source of ImvoxelNet is: Rukhovich D, Vorontsova A, Konushin A. Imvoxelnet: Image to voxelsprojection for monocular and multi-view general-purpose 3d object detection[C] / / Proceedings of the IEEE / CVF winter conference on applications of computer vision. 2022: 2397-2406. The source of MV2D is: Wang Z, Huang Z, Fu J, et al. Object as query: Lifting any 2d object detector to 3d detection[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023: 3791-3800. The source of BEVFormer is: Li Z, Wang W, Li H, et al. Bevformer: learning bird's-eye-view representation from lidar-camera via spatiotemporal transformers[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. The source of BEVFormer v2 is: Yang C, Chen Y, Tian H, et al. Bevformer v2: Adapting modern image backbones to bird's-eye-view recognition via perspective supervision[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2023: 17830-17839. The source of SparseBEV is: Liu H, Teng Y, Lu T, et al. Sparsebev: High-performance sparse 3d object detection from multi-camera videos[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2023: 18580-18590. The source of BEVDepth is: Li Y, Ge Z, Yu G, et al. Bevdepth: Acquisition ofreliable depth for multi-view 3d object detection[C] / / Proceedings of the AAAIconference on artificial intelligence. 2023, 37(2): 1477-1485. The source of BEVHeight is: Yang L, Yu K, Tang T, et al. Bevheight: A robust framework for vision-based roadside 3d object detection[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023:21611-21620.

[0073] Table 1 Detection results of the method according to the embodiment of the present invention and several existing 3D object detection measurement methods

[0074]

[0075] It can be seen from Table 1 that compared with the existing methods, the method of the embodiment of the present invention has the highest accuracy in three-dimensional target detection. Figure 5 shown.

[0076] Figure 5 Figures (a) through (d) are the detection results of BEVHeight, and Figures (e) through (h) are the detection results of the method of this embodiment. Comparing Figures (a) and (e), we can see that the method of this embodiment achieves more accurate direction prediction than BEVHeight. Comparing Figures (b) and (f), we can see that the method of this embodiment has a higher recall rate for small objects. Comparing Figures (c) and (g), and (d) and (h), we can see that the method of this embodiment has a higher recall rate for occluded objects.

[0077] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art will still be able to modify the technical solutions described in the foregoing examples or substitute equivalents for some of the technical features therein. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention.

Claims

1. A three-dimensional target detection method based on two-dimensional detection result prompts, characterized in that: The steps include: Step 1: Obtain an image dataset, where each monocular image in the image dataset is annotated with a 3D detection box and target category for each target; project the 3D detection box of each target onto the image coordinate system based on the known intrinsic and extrinsic parameters of the image sensor that captured the monocular image to obtain a 2D detection box for each target; construct a 2D target detection training set using the monocular image and the 2D detection box and target category of each target in the image; Step 2: Construct a two-dimensional object detector and train it using the two-dimensional object detection training set; Step 3: Construct a 3D object detector and train it using the image dataset and the output of the trained 2D object detector; The three-dimensional object detector includes an image encoder, a hint encoder module, a feature fusion module, a three-dimensional space decoder and a direction fine-tuning module; The image encoder converts the input monocular image into features of the monocular image; the prompt encoder module converts the two-dimensional detection box and target category of each target in the monocular image into features corresponding to the two-dimensional detection box and target category respectively; the feature fusion module fuses the features corresponding to the two-dimensional detection box and target category, and the features of the monocular image into fused features; the projection module converts the fused features into a voxel feature map; the three-dimensional space decoder converts the voxel feature map into a three-dimensional detection box and target category of each target in the input monocular image; Step 4: Use the trained two-dimensional target detector and three-dimensional target detector to perform three-dimensional target detection, and the direction fine-tuning module continuously adjusts the direction of the three-dimensional detection frame according to the two-dimensional detection frame output by the two-dimensional target detector, so that it is as close as possible to the direction of the two-dimensional detection frame, thereby updating the output three-dimensional detection frame.

2. The three-dimensional target detection method based on two-dimensional detection result prompts according to claim 1, characterized in that: The direction fine-tuning module specifically performs the following operations: Projecting a three-dimensional detection frame of each target in the monocular image output by the three-dimensional space decoder into an image coordinate system based on known intrinsic and extrinsic parameters of the image sensor that captured the monocular image to obtain a projected detection frame, and calculating an intersection-over-union ratio between the projected detection frame and the corresponding two-dimensional detection frame; The direction of the 3D detection frame is continuously changed, and the projection and intersection-over-union (IoU) calculation operations are repeated. The direction of the 3D detection frame corresponding to the maximum IoU is selected as the final direction, and the 3D detection frame is updated as the output of the direction fine-tuning module.

3. The three-dimensional target detection method based on two-dimensional detection result prompts according to claim 1, characterized in that: The two-dimensional target detector includes a visual encoder and a two-dimensional decoder. The visual encoder is used to convert the input monocular image into the features of the monocular image; the two-dimensional decoder includes a two-dimensional target detection head and a post-processing module, which is used to convert the features of the monocular image into a two-dimensional detection box and target category for each target in the monocular image.

4. The three-dimensional target detection method based on two-dimensional detection result prompts according to claim 1, characterized in that: The prompt encoder module includes a normalization layer, a fully connected layer, and a splicing layer. The normalization layer is used to normalize the coordinates of the upper left corner and lower right corner of the input two-dimensional detection box to obtain a 2×2 matrix A, A∈R 2×2 ; The fully connected layer is used to multiply the matrix A with a matrix B initialized by a random Gaussian, and add a learnable bias matrix C to obtain the matrix D, D = A × B + C, where B ∈ R 2×M , C∈R 2×M , M represents the number of columns of the preset matrix B; The splicing layer is used to repeat the category label corresponding to the input two-dimensional detection frame M times to obtain an M-dimensional vector, and splice the M-dimensional vector with the matrix D in the row direction to obtain the spliced ​​feature F prompt , F prompt ∈R 3×M .

5. The three-dimensional target detection method based on two-dimensional detection result prompts according to claim 4, characterized in that: The feature fusion module includes a Transformer-based self-attention layer, a Transformer-based cross-attention layer, a fully connected layer, and a normalization layer. The Transformer-based self-attention layer is used to transform the feature F output by the prompt encoder module. prompt Perform a self-attention operation and then input it into the normalization layer, which normalizes it to obtain feature f1; The Transformer-based cross attention layer is used to perform a cross attention operation on feature f1 and the feature F(X) output by the three-dimensional object detector, and then input the result into a fully connected layer and a normalization layer to obtain feature f2; at this time, the query based on the Transformer-based cross attention layer is f1, and the key and value are both F(X); The Transformer-based cross-attention layer is also used to perform a cross-attention operation on the feature f2 and the feature F(X) output by the three-dimensional target detector, and then input the normalization layer to obtain the feature f as the output of the feature fusion module; at this time, the query based on the Transformer cross-attention layer is F(X), and the key and value are both f2.

6. The three-dimensional target detection method based on two-dimensional detection result prompts according to claim 1, characterized in that: The loss functions for training the two-dimensional target detector use Focal Loss and L1 Loss, respectively, to optimize the target category, the coordinates of the center point of the target's two-dimensional detection frame in the image coordinate system, and the length and width of the target's two-dimensional detection frame.

7. The three-dimensional target detection method based on two-dimensional detection result prompts according to claim 6, characterized in that: The loss functions for training the three-dimensional target detector use Focal Loss and L1 Loss, respectively, to optimize the target category, the coordinates of the center point of the target's three-dimensional detection frame in the camera coordinate system, the length, width, and height of the target's three-dimensional detection frame, and the target's heading angle.

8. A three-dimensional target detection system based on two-dimensional detection result prompts, characterized in that: The system includes an image acquisition module, a two-dimensional target detection module and a three-dimensional target detection module; The image acquisition module is used to construct an image dataset, in which each monocular image in the image dataset is annotated with a three-dimensional detection frame and target category of each target; and based on the known internal and external parameters of the image sensor that captured the monocular image, the three-dimensional detection frame of each target is projected into the image coordinate system to obtain a two-dimensional detection frame of each target; and a two-dimensional target detection training set is constructed using the monocular image and the two-dimensional detection frame and target category of each target in the image; The two-dimensional target detection module has a built-in trained two-dimensional target detector for detecting targets in an input monocular image and outputting a two-dimensional detection frame and target category for each target in the monocular image; The 3D object detection module has a built-in trained 3D object detector for performing object detection on an input monocular image and outputting a 3D detection frame and object category for each object in the monocular image; During the target detection process, the direction of the three-dimensional detection frame of each target in the output monocular image is closest to the direction of the two-dimensional detection frame according to the two-dimensional detection frame of each target and the target category prompt output by the two-dimensional target detection module; The three-dimensional target detector includes an image encoder, a prompt encoder module, a feature fusion module, a projection module, a three-dimensional space decoder and a direction fine-tuning module; the image encoder is used to convert the input monocular image into the features of the monocular image; the prompt encoder module is used to convert the two-dimensional detection box and target category of each target in the monocular image into features corresponding to the two-dimensional detection box and the target category respectively; the feature fusion module is used to fuse the features corresponding to the two-dimensional detection box and the target category, and the features of the monocular image into fused features; the projection module is used to convert the fused features into a voxel feature map; the three-dimensional space decoder is used to convert the voxel feature map into a three-dimensional detection box and target category of each target in the input monocular image; the direction fine-tuning module is used to continuously adjust the direction of the three-dimensional detection frame according to the two-dimensional detection frame, so that it is closest to the direction of the two-dimensional detection frame, and update the three-dimensional detection frame.

9. An electronic device, characterized in that: It includes one or more processors and a storage device for storing one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements the three-dimensional target detection method based on two-dimensional detection result prompts as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the three-dimensional target detection method based on two-dimensional detection result prompts as described in any one of claims 1 to 7 is implemented.

Images