Target detection and model training method, model, device, vehicle and storage medium

Abstract

The embodiment of the invention relates to a target detection and model training method, a model, a device, a vehicle and a storage medium. The target detection method comprises the following steps: acquiring at least one target image acquired by at least one camera arranged on a vehicle at the same moment; inputting the at least one target image into the target detection model to obtain a corresponding 2D target detection result and a 3D target detection result; wherein the target detection model comprises a backbone network used for obtaining image features of multiple scales of at least one target image; the target bottleneck network is used for fusing the image features of multiple scales into a fused feature; the 2D detection network is used for performing target detection on the fusion features to obtain a 2D target detection result; and the 3D detection network is used for performing target detection on the fused features to obtain a 3D target detection result. The method can be applied to the vehicle, 2D target detection and 3D target detection are achieved, and meanwhile the requirement for the computing power of the vehicle is lowered.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of automobile technology, and in particular to a target detection method, a target detection model training method, a target detection model, a device, a vehicle, and a storage medium. Background Art

[0002] In the field of intelligent assisted driving, real-time and accurate target detection is the basis for ensuring that the vehicle can perceive the surrounding environment. Among them, there are two common detection methods, one is 2D detection method and the other is 3D detection method; the existing more mature target detection is 2D detection method, but the 2D detection method assumes that the ground on which the vehicle is located is flat. When the ground is uneven, the accuracy of the detection results obtained by the 2D detection method is poor. 3D detection is based on the 3D detection method in the bird's eye view (Bird Eye View, BEV) space. The 3D detection method can predict the position, speed, size, posture and other information of the target in the 3D space, but the 3D detection method has weak long-distance perception ability, resulting in poor accuracy of the detection results obtained at long distances.

[0003] Therefore, in order to ensure the accuracy of target detection, the best solution at present is to deploy the 3D detection model (for 3D detection) and the 2D detection model (for 2D detection) in the BEV space in the device at the same time, and then obtain and utilize the 2D detection results and 3D detection results. However, deploying the 2D detection model and the 3D detection model at the same time requires a lot of computing resources and high computing power requirements for the device, while the computing resources of the vehicle are limited, which makes this solution impossible to apply in the vehicle. Therefore, in the case where there is no need to deploy the existing 2D detection model and 3D detection model at the same time, or in the case of limited computing resources, how to use a single model to obtain 2D detection results and 3D detection results has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a target detection method, a target detection model training method, a target detection model, an apparatus, a vehicle and a storage medium, which can be applied to vehicles to achieve 2D target detection and 3D target detection while reducing the computing power requirements for the vehicle.

[0005] In a first aspect, an embodiment of the present application provides a target detection method, comprising: obtaining, through at least one camera arranged on a vehicle, at least one target image captured by at least one camera arranged on the vehicle at the same time; inputting at least one target image into a target detection model to obtain corresponding 2D target detection results and 3D target detection results; wherein the target detection model comprises: a backbone network, a target bottleneck network, a 2D detection network and a 3D detection network; the backbone network is used to obtain image features of multiple scales of at least one target image; the target bottleneck network is used to fuse image features of multiple scales into a fused feature; the 2D detection network is used to perform target detection on the fused feature to obtain a 2D target detection result; the 3D detection network is used to perform target detection on the fused feature to obtain a 3D target detection result.

[0006] In the second aspect, an embodiment of the present application provides a target detection model, including: a backbone network, a target bottleneck network, a 2D detection network and a 3D detection network; the backbone network is used to obtain image features of multiple scales of at least one target image; the target bottleneck network is used to fuse image features of multiple scales into a fused feature; the 2D detection network is used to perform target detection on the fused feature to obtain a 2D target detection result; the 3D detection network is used to perform target detection on the fused feature to obtain a 3D target detection result.

[0007] In a third aspect, an embodiment of the present application provides a target detection model training method, including: obtaining a training set, the training set including multiple 2D sample data and multiple BEV sample data, each 2D sample data including: an image, multiple 2D detection boxes, and the types of each target included in the image; each BEV sample data including: multiple frames of images with aligned timestamps, multiple 3D detection boxes, the types of each target included in the image, the orientation angle of each target relative to the vehicle, and the speed of each target; constructing an initial target detection model and multiple detection tasks; based on the multiple detection tasks, training the initial target detection model through the training set to obtain a target detection model.

[0008] In a fourth aspect, an embodiment of the present application provides a target detection device, comprising: an acquisition module, used to acquire at least one target image captured by at least one camera set on a vehicle at the same time; an input module, used to input at least one target image into a target detection model to obtain corresponding 2D target detection results and 3D target detection results; wherein the target detection model comprises: a backbone network, a target bottleneck network, a 2D detection network and a 3D detection network; the backbone network is used to obtain image features of multiple scales of at least one target image; the target bottleneck network is used to fuse image features of multiple scales into a fused feature; the 2D detection network is used to perform target detection on the fused feature to obtain a 2D target detection result; the 3D detection network is used to perform target detection on the fused feature to obtain a 3D target detection result.

[0009] In a fifth aspect, an embodiment of the present application provides a target detection model training device, including: an acquisition module, used to acquire a training set, the training set includes multiple 2D sample data and multiple BEV sample data, each 2D sample data includes: an image, multiple 2D detection boxes, and the types of each target included in the image; each BEV sample data includes: multiple frames of images with aligned timestamps, multiple 3D detection boxes, the types of each target included in the image, the orientation angle of each target relative to the vehicle, and the speed of each target; a construction module, used to construct an initial target detection model and multiple detection tasks; a training module, used to train the initial target detection model through the training set based on multiple detection tasks to obtain a target detection model.

[0010] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a processor, wherein the processor is used to execute a computer program stored in a memory, and when the computer program is executed by the processor, the steps of the method provided in the first aspect or the third aspect are implemented.

[0011] In a seventh aspect, an embodiment of the present application provides a vehicle, comprising: the target detection device of the fourth aspect, or the target detection model training device of the fifth aspect, or the electronic device of the sixth aspect.

[0012] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method provided in the first aspect or the third aspect are implemented.

[0013] In the technical solution provided by the embodiment of the present application, at least one target image collected by at least one camera set on the vehicle at the same time is obtained; at least one target image is input into the target detection model to obtain corresponding 2D target detection results and 3D target detection results; wherein the target detection model includes: a backbone network, a target bottleneck network, a 2D detection network and a 3D detection network; the backbone network is used to obtain image features of multiple scales of at least one target image; the target bottleneck network is used to fuse image features of multiple scales into a fused feature; the 2D detection network is used to perform target detection on the fused feature to obtain a 2D target detection result; the 3D detection network is used to perform target detection on the fused feature to obtain a 3D target detection result. Because the target detection model is based on multiple 2D target detection tasks and multiple 3D target detection tasks, and is trained through multiple 2D sample data and multiple BEV sample data, by inputting at least one target image into the target detection model, both 2D target detection results and 3D target detection results can be obtained, which is convenient for selecting the required target detection results according to subsequent needs; and, in the target detection model, the 2D detection network and the 3D detection network share the backbone network and the target bottleneck network. Compared with deploying the 2D target detection model and the 3D target detection model in the vehicle, the two models are used to perform feature extraction, feature fusion, target detection and other processing on the input target image respectively. The target detection model of this solution requires fewer computing resources and can be used in vehicles, with low requirements on the computing power of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0016] Figure 1 A schematic diagram of a target detection method provided in an embodiment of the present application;

[0017] Figure 2 A schematic diagram of the orientation angle provided in the embodiment of the present application;

[0018] Figure 3 A flowchart of another target detection method provided in an embodiment of the present application;

[0019] Figure 4 A flowchart of another target detection method provided in an embodiment of the present application;

[0020] Figure 5 A schematic diagram of the structure of a target detection model provided in an embodiment of the present application;

[0021] Figure 6 A flowchart of another target detection method provided in an embodiment of the present application;

[0022] Figure 7 A schematic diagram of the structure of another target detection model provided in an embodiment of the present application;

[0023] Figure 8 A flowchart of another target detection method provided in an embodiment of the present application;

[0024] Fig. 9 A schematic diagram of a multi-dimensional matrix output by a 2D bottleneck subnetwork provided in an embodiment of the present application;

[0025] Fig.10 A schematic diagram of the structure of a target detection model including a feature selector provided in an embodiment of the present application;

[0026] Fig.11 A flowchart of a target detection model training method provided in an embodiment of the present application;

[0027] Fig.12 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application;

[0028] Fig.13 A schematic diagram of the hardware structure of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0031] The technical solution of the present application is explained in detail below through several specific embodiments.

[0032] Figure 1 A flow chart of a target detection method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the process includes the following steps 101 and 102.

[0033] 101. Acquire at least one target image captured by at least one camera disposed on a vehicle at the same time.

[0034] It can be understood that at least one target image captured at the same time is obtained based on the image captured synchronously by at least one camera, such as by synchronously capturing the environmental image outside the vehicle through six cameras arranged in the vehicle, namely the front, left front, left rear, right front, right rear, and rear. The specific implementation method is not limited here.

[0035] 102. Input at least one target image into a target detection model to obtain corresponding 2D target detection results and 3D target detection results.

[0036] Among them, the target detection model includes: a backbone network (Backbone), a target bottleneck network (Neck), a 2D detection network and a 3D detection network; the backbone network is used to obtain image features of multiple scales of at least one target image; the target bottleneck network is used to fuse image features of multiple scales into a fused feature; the 2D detection network is used to perform target detection on the fused feature to obtain a 2D target detection result; the 3D detection network is used to perform target detection on the fused feature to obtain a 3D target detection result.

[0037] It can be understood that the backbone network is trained based on multiple detection tasks through a training set, and the multiple detection tasks include: multiple 2D target detection tasks and multiple 3D target detection tasks, and the training set includes multiple 2D sample data and multiple bird's-eye view BEV sample data; the target bottleneck network is trained based on multiple 2D target detection tasks and / or multiple 3D target detection tasks through multiple image features output by the backbone network; the 2D detection network is trained based on multiple 2D target detection tasks through multiple fused features output by the target bottleneck network; the 3D detection network is trained based on multiple 3D target detection tasks through multiple fused features output by the target bottleneck network.

[0038] It can be understood that for a target image input into the target detection model, the image features of multiple scales of the target image are extracted through the backbone network and transmitted to the target bottleneck network. The target bottleneck network is used to fuse the image features of multiple scales to obtain a fused feature (that is, multiple features are fused into one feature), and the one fused feature is input into the 2D detection network and the 3D detection network respectively. The 2D detection network performs target detection based on the fused feature to obtain a 2D target detection result, and the 3D detection network performs target detection based on the fused feature of at least one target image to obtain a 3D target detection result. For the one target image, the 2D target detection result includes: the 2D position information of each target included in the one target image (such as: the center point coordinates (x, y) of each target), the 2D size information of each target (such as: the length and width of the detection box where each target is located), and the type information of each target (such as: pedestrians, cars, cones, etc.); for at least one target image, the 3D target detection result includes: the 3D position information of each target included in the at least one target image in the BEV space (such as: the center point coordinates (x, y, z) of each target), the 3D size information of each target (such as: the length, width and height of the 3D detection box where each target is located), the orientation angle of each target relative to the vehicle (such as Figure 2 As shown, the angle formed by the longitudinal axis of the target 201 and the longitudinal axis of the vehicle 200, angle 1 is the orientation angle of the target), the speed information of each target (including: lateral speed and longitudinal speed), and the type information of each target.

[0039] It can be understood that in target detection, objects vary in size, and there may be extremely small and extremely large objects. In order to ensure that the target is not missed during detection, it is necessary to extract image features of different scales from the image, that is, image features of multiple scales. Multiple scales refer to sampling of images at different granularities (such as sampling with different pixel densities). Different features of the image can be obtained at different scales. Sampling with smaller granularity and denser density can obtain more detailed features in the image, and sampling with larger granularity and sparser density can obtain global features in the image. Therefore, through multiple scales, more comprehensive features of the image can be extracted, including both global overall information and local detailed information included in the image.

[0040] It can be understood that the backbone network is mainly used for feature extraction of images. It is a network that has been pre-trained in large data sets (such as ImageNet, COCO, etc.), and is a convolutional neural network that already has pre-trained parameters. Therefore, in the process of training to obtain the target detection model, it is only necessary to fine-tune the parameters of the backbone network according to different detection tasks. The mainstream backbone networks have structures such as ResNet and DarkNet. The specific network structure of the backbone network is not limited in the embodiments of the present application, and the specific network structure is determined according to actual needs.

[0041] It can be understood that the target bottleneck network between the detection network and the backbone network is used to fuse the image features of multiple scales output by the backbone network. The mainstream bottleneck networks include: SSD, STDN, FPN, PANet and other structures. The specific network structure of the target bottleneck network is not limited in the embodiment of the present application, and the specific network structure is determined according to actual needs.

[0042] Optionally, the target bottleneck network includes: a 2D bottleneck subnetwork and a BEV bottleneck subnetwork, or the target bottleneck network is a single bottleneck network; when the target bottleneck network includes a 2D bottleneck network and a BEV bottleneck network, the 2D bottleneck network is obtained by training multiple image features based on multiple 2D target detection tasks, and the BEV bottleneck network is obtained by training multiple image features based on multiple 3D target detection tasks; when the target bottleneck network is a single bottleneck network, the single bottleneck network is obtained by training multiple image features based on multiple detection tasks.

[0043] It can be understood that when the target bottleneck network is a single bottleneck network, the 2D detection network and the 3D detection network share the single bottleneck network, that is, the output of the single bottleneck network serves as both the input of the 2D detection network and the input of the 3D detection network; when the target bottleneck network includes a 2D bottleneck subnetwork and a BEV bottleneck subnetwork, the output of the 2D bottleneck subnetwork is the input of the 2D detection network, and the output of the BEV bottleneck subnetwork is the input of the 3D detection network.

[0044] It can be understood that the choice of the target bottleneck network can be selected according to actual needs. If the requirement for detection accuracy is not high, a single bottleneck network can be selected, which also saves computing resources; if the requirement for detection accuracy is high, a 2D bottleneck subnetwork and a BEV bottleneck subnetwork can be selected.

[0045] It can be understood that the obtained 2D target detection results and 3D target detection results can be selected according to actual needs. For example, for targets at close distances (such as less than or equal to 60 meters), the 3D target detection results shall prevail, and for targets at close distances (such as greater than 60 meters), the 2D target detection results shall prevail; the 2D target detection results and the 3D target detection results can also be made to complement each other. When there are targets with inconsistent predictions in the 2D target detection results and the 3D target detection results, the 2D target detection results or the 3D target detection results shall prevail according to actual needs.

[0046] In an embodiment of the present application, at least one target image with the same timestamp is obtained through at least one camera set on the vehicle; at least one target image is input into the target detection model to obtain corresponding 2D target detection results and 3D target detection results; wherein the target detection model includes: a backbone network, a target bottleneck network, a 2D detection network and a 3D detection network; the backbone network is used to obtain image features of multiple scales of at least one target image; the target bottleneck network is used to fuse image features of multiple scales into a fused feature; the 2D detection network is used to perform target detection on the fused feature to obtain a 2D target detection result; the 3D detection network is used to perform target detection on the fused feature to obtain a 3D target detection result. Because the target detection model is based on multiple 2D target detection tasks and multiple 3D target detection tasks, and is trained through multiple 2D sample data and multiple BEV sample data, by inputting at least one target image into the target detection model, both 2D target detection results and 3D target detection results can be obtained, which is convenient for selecting the required target detection results according to subsequent needs; and, in the target detection model, the 2D detection network and the 3D detection network share the backbone network and the target bottleneck network. Compared with deploying the 2D target detection model and the 3D target detection model in the vehicle, the two models are used to perform feature extraction, feature fusion, target detection and other processing on the input target image respectively. The target detection model of this solution requires fewer computing resources and can be used in vehicles, with low requirements on the computing power of the vehicle.

[0047] In some embodiments of the present application, Figure 1 ,like Figure 3 As shown, the above step 101 can be specifically implemented through the following steps 101a and 101b.

[0048] 101a. Synchronously collect at least one image using at least one camera disposed on the vehicle.

[0049] 101b. Preprocess each of the at least one image to obtain at least one target image.

[0050] The preprocessing includes at least one of the following: scaling, cropping, and normalization.

[0051] It is understandable that because the target detection model requires the input of an image of a fixed size, the image needs to be scaled, but directly scaling the image may cause image deformation, so the image is generally cropped first.

[0052] It can be understood that normalization is to adjust the size of the eigenvalue to a fixed range. The goal of normalization is to determine a mapping relationship, mapping the original data to the interval [a, b]. In practical applications, [a, b] is generally: [0, 1] or [-1, 1]. If normalization is not performed, when the eigenvalue is relatively large, the gradient value will also be large, and when the eigenvalue is small, the gradient value will also be small. During the back propagation of the model, the gradient value update is the same as the learning rate. When the learning rate is small, the small gradient value will lead to slow update. When the learning rate is large, the large gradient value will cause the model to be difficult to converge. Therefore, in order to make the model training converge smoothly, the image is normalized and the eigenvalues ​​of different dimensions are adjusted to a similar range. A unified learning rate can be used to accelerate model training. For example, normalization can be to divide the pixel value in the image by a preset value to ensure that the pixel value of the image is limited to a preset range; normalization can also be to subtract the mean from each pixel value in the image and divide it by the variance (this method can remove the average brightness value in the image).

[0053] In the embodiment of the present application, at least one image is synchronously captured by at least one camera installed on the vehicle; each of the at least one image is preprocessed to obtain at least one target image, and the preprocessing includes at least one of the following: scaling processing, cropping processing, and normalization processing. In this way, it is ensured that the image input to the target detection model meets the requirements and is normalized, which can not only make the model converge quickly during the target detection model training stage; but also make data of different dimensions comparable, making subsequent predictions more accurate.

[0054] In some embodiments of the present application, the target bottleneck network is a single bottleneck network, the 2D detection network includes: multiple 2D detection heads (2D Head), and the 3D detection network includes: a BEV conversion sub-network and multiple 3D detection heads (3D Head); combined Figure 3 ,like Figure 4 As shown, the above step 102 can be specifically implemented through the following steps 102a to 102e.

[0055] 102a. Perform feature extraction on at least one target image through a backbone network to obtain a plurality of scale features corresponding to each target image in the at least one target image.

[0056] It can be understood that for each target image in the at least one target image, the backbone network is input and the output is a plurality of scale features corresponding to the target image.

[0057] 102b. Perform feature fusion on multiple scale features through a single bottleneck network to obtain a first fused feature corresponding to each target image in at least one target image.

[0058] 102c. Detect the first fusion feature through multiple 2D detection heads to obtain a 2D target detection result.

[0059] The 2D target detection result includes multiple detection results output by multiple 2D detection heads.

[0060] Among them, different 2D detection heads correspond to different 2D object detection tasks.

[0061] It can be understood that in the target detection model, the Head is used to output classification results and positioning results based on the input. The 2D detection network includes different 2D detection heads, and each 2D detection head corresponds to a different target detection task, such as: the Head used to detect whether the target image includes a pedestrian's Head, the Head used to detect whether the target image includes a vehicle's Head, etc.

[0062] 102d. Based on the internal and external parameters of at least one camera, the first fusion feature is mapped to the target BEV space through the BEV conversion sub-network to obtain a first BEV feature.

[0063] It can be understood that the intrinsic parameters of the camera refer to the optical characteristic parameters of the camera, including: focal length, principal point coordinates and distortion parameters; the intrinsic parameters are used to convert coordinates from the camera coordinate system to the pixel coordinate system; the extrinsic parameters of the camera refer to the positional relationship between at least one camera, including: rotation matrix and translation vector, and the role of the extrinsic parameters is to convert coordinates from the world coordinate system to the camera coordinate system.

[0064] It can be understood that at least one first fusion feature corresponding to at least one target image is input into the BEV conversion subnetwork, and the first BEV feature in the target BEV space is output.

[0065] 102e. Detect the first BEV feature through multiple 3D detection heads to obtain a 3D target detection result.

[0066] The 3D target detection result includes multiple detection results output by multiple 3D detection heads.

[0067] Among them, different 3D detection heads correspond to different 3D object detection tasks.

[0068] It is understandable that Figure 5 Figure 1 shows the network structure of the target detection model. The 2D detection network and the 3D detection network share the backbone network and a single bottleneck network.

[0069] It should be noted that the execution order of the above step 102c and step 102d does not distinguish between the first and the last, and they can be executed synchronously or asynchronously.

[0070] In an embodiment of the present application, the target bottleneck network is a single bottleneck network, the 2D detection network includes: multiple 2D detection heads, and the 3D detection network includes: a BEV conversion subnetwork and multiple 3D detection heads; at least one target image is input into the target detection model to obtain corresponding 2D target detection results and 3D target detection results, including extracting features from at least one target image through a backbone network to obtain multiple scale features corresponding to each target image in at least one target image; performing feature fusion on multiple scale features through a single bottleneck network to obtain a first fused feature corresponding to each target image in at least one target image; and performing feature fusion on the first fused feature through multiple 2D detection heads. The fused features are used for detection to obtain a 2D target detection result, which includes multiple detection results output by multiple 2D detection heads, and different 2D detection heads correspond to different 2D target detection tasks; based on the internal and external parameters of at least one camera, at least one first fused feature corresponding to at least one target image is mapped to the target BEV space through the BEV conversion subnetwork to obtain a first BEV feature; the first BEV feature is detected through multiple 3D detection heads to obtain a 3D target detection result, which includes multiple detection results output by multiple 3D detection heads, and different 3D detection heads correspond to different 3D target detection tasks. In this way, the 2D detection network and the 3D detection network share a backbone network and a single bottleneck network. For the image input into the target detection model, only one feature extraction is required through the backbone network, and feature fusion is required through a single bottleneck network, and the fused features can be input into the 2D detection network and the 3D detection network to obtain 2D target detection results and 3D target detection results. While realizing 2D target detection and 3D target detection, the amount of calculation is reduced, thereby reducing the data processing pressure of the vehicle.

[0071] In some embodiments of the present application, the target bottleneck network includes: a 2D bottleneck network and a BEV bottleneck network; the 2D detection network includes: multiple 2D detection heads, and the 3D detection network includes: a BEV conversion sub-network and multiple 3D detection heads; combined Figure 3 ,like Figure 6 As shown, the above step 102 can be specifically implemented through the following steps 102f to 102e.

[0072] 102f. Perform feature extraction on at least one target image through the backbone network to obtain multiple scale features corresponding to each target image in the at least one target image.

[0073] 102g. Perform feature fusion on multiple scale features through a 2D bottleneck sub-network to obtain a second fused feature corresponding to each target image in at least one target image.

[0074] 102h. Detect the second fusion feature through multiple 2D detection heads to obtain a 2D target detection result.

[0075] 102i. Perform feature fusion on multiple scale features through the BEV bottleneck sub-network to obtain a third fused feature corresponding to each target image in at least one target image.

[0076] 102j. Based on the internal and external parameters of at least one camera, at least one third fusion feature corresponding to at least one target image is mapped to the target BEV space through a BEV conversion subnetwork to obtain a second BEV feature.

[0077] 102k. Detect the second BEV feature through multiple 3D detection heads to obtain a 3D target detection result.

[0078] It is understandable that Figure 7 As shown in Figure 1, it is the network structure of the target detection model. The target bottleneck network includes the 2D bottleneck sub-network and the BEV bottleneck sub-network.

[0079] It can be understood that the 2D bottleneck subnetwork is trained based on multiple 2D target detection tasks; the BEV bottleneck subnetwork is trained based on multiple 3D target detection tasks; for different target detection tasks, the 2D bottleneck subnetwork and the BEV bottleneck subnetwork have different focuses during learning, and the fusion features obtained for different detection tasks are more accurate, which makes the final target detection results more accurate.

[0080] In the embodiment of the present application, the target bottleneck network includes: a 2D bottleneck subnetwork and a BEV bottleneck subnetwork; the 2D detection network includes: multiple 2D detection heads, and the 3D detection network includes: a BEV conversion subnetwork and multiple 3D detection heads; at least one target image is input into the target detection model to obtain corresponding 2D target detection results and 3D target detection results, including: extracting features from at least one target image through the backbone network to obtain multiple scale features corresponding to each target image in at least one target image; performing feature fusion on multiple scale features through the 2D bottleneck subnetwork to obtain at least one target image. The second fusion feature corresponding to each target image in the image; the second fusion feature is detected by multiple 2D detection heads to obtain 2D target detection results; the multiple scale features are feature-fused by the BEV bottleneck sub-network to obtain the third fusion feature corresponding to each target image in at least one target image; based on the internal and external parameters of at least one camera, at least one third fusion feature corresponding to at least one target image is mapped to the target BEV space by the BEV conversion sub-network to obtain the second BEV feature; the second BEV feature is detected by multiple 3D detection heads to obtain 3D target detection results. In this way, the 2D detection network corresponds to the 2D bottleneck sub-network, and the 3D detection network corresponds to the BEV bottleneck sub-network. During the training process, the 2D bottleneck sub-network and the BEV bottleneck sub-network are trained based on different target detection tasks. Therefore, the learning focuses are different, and the fusion features obtained for different detection tasks are more accurate, which makes the final target detection results more accurate.

[0081] In some embodiments of the present application, Figure 6 ,like Figure 8 As shown, before the above step 102h, the target detection method provided in the embodiment of the present application further includes the following step 103, and the above step 102h can be specifically implemented by the following step 1021.

[0082] 103. Select at least one second fusion feature corresponding to each 2D detection head from at least one second fusion feature corresponding to at least one target image according to the feature selection relationship.

[0083] The feature selection relationship includes a plurality of corresponding relationships, and each corresponding relationship is used to indicate at least one second fusion feature included in a 2D detection head.

[0084] It can be understood that in actual applications, for a 2D target detection task, it may not be necessary to detect whether the target is included in the entire image of the image captured by at least one camera set in the vehicle. Therefore, when detecting through the 2D detection head, it is only necessary to detect the fusion features corresponding to some target images in at least one target image. For example, for the cone barrel detection task, it is necessary to detect whether the target image includes the cone barrel and the position and size of the cone barrel. However, in actual applications, only the cone barrel that appears in front of the vehicle is concerned, and the cone barrel that appears in other positions of the vehicle can be ignored. Therefore, when the 2D target detection task corresponding to the 2D detection head is the cone barrel task, only the fusion features corresponding to the target image captured by the camera set in the front direction of the vehicle need to be input into the 2D detection head, and the fusion features corresponding to the target images captured by the other cameras do not need to be input into the 2D detection head.

[0085] It can be understood that the output of the 2D bottleneck subnetwork is a four-dimensional matrix (B, C, H, W), where B corresponds to the number of cameras, batch 0 corresponds to the first camera, batch 1 corresponds to the second camera, and so on; C and H represent the height and width of the matrix respectively, and W represents the number of channels.

[0086] For example, Fig. 9 As shown in the figure, batch 0 represents the image features corresponding to the target image captured by the first camera. Similarly, the second camera corresponds to batch 1, and so on.

[0087] Exemplarily, each corresponding relationship corresponds to a feature selector, a 2D detection head corresponds to a feature selector, a plurality of second fused features are input into a feature selector, and the feature selector outputs at least one fused feature corresponding to the 2D detection head. Fig.10 As shown in the figure, taking 6 target images as an example, batch=6 is used to indicate the image features corresponding to the 6 target images collected by 6 cameras; Bev batch=1 is used to indicate that the image features corresponding to the 6 target images collected by 6 cameras are mapped to the bird's-eye view features in the BEV space. The fusion features corresponding to the target images collected by the forward camera are finally input to the 2D detection head cone barrel (indicating the cone barrel detection task corresponding to the 2D detection head).

[0088] 1021. Detect at least one second fusion feature corresponding to each 2D detection head through each 2D detection head among the multiple 2D detection heads to obtain a 2D target detection result.

[0089] It can be understood that each 2D detection head outputs a corresponding 2D target detection result based on at least one second fusion feature input.

[0090] It can be understood that the target detection task corresponding to each 2D detection head is a multi-classification task and a positioning task, such as: the 2D target detection task corresponding to the 2D detection head is: vehicles (cars, trucks, tricycles, buses) and the positions of vehicles; for a target image, if the target image includes a vehicle, the information included in the output of the 2D detection head is: (confidence, the probability that the target is a car, truck, tricycle, bus, the center point coordinates of the target, the length and width of the target); wherein, the confidence is used to indicate the probability that the target is included in the detection frame, and generally, if it is greater than 0.5, it is considered that the detection frame includes the target, and the sum of the probability that the target is a car, the probability of a truck, the probability of a tricycle, and the probability of a bus is 1.

[0091] In an embodiment of the present application, before the second fusion feature is detected by multiple 2D detection heads to obtain the 2D target detection result, the method also includes: selecting at least one second fusion feature corresponding to each 2D detection head from at least one second fusion feature corresponding to at least one target image according to a feature selection relationship, the feature selection relationship includes multiple corresponding relationships, each corresponding relationship is used to indicate at least one second fusion feature included in a 2D detection head; detecting the second fusion feature by multiple 2D detection heads to obtain the 2D target detection result, including: detecting at least one second fusion feature corresponding to each 2D detection head by each of the multiple 2D detection heads to obtain the 2D target detection result. In this way, the fusion features input to the 2D detection head can be reduced through the feature selection relationship, so the data that the 2D detection head needs to process is reduced, further reducing the requirements for the vehicle's computing power and reducing the vehicle's data processing pressure.

[0092] In the present application embodiment, Fig.11 As shown, a target detection module training method is provided, which also includes the following steps 1101 to 1103.

[0093] 1101. Obtain a training set.

[0094] The training set includes multiple 2D sample data and multiple BEV sample data. Each 2D sample data includes: an image, multiple 2D detection frames, and the types of each target included in the image; each BEV sample data includes: multiple frames of images with aligned timestamps, multiple 3D detection frames, the types of each target included in the image, the orientation angle of each target relative to the vehicle, and the speed of each target.

[0095] It can be understood that the training set can be the acquired labeled 2D sample data and BEV sample data, or it can be constructed by labeling tools based on the detection task.

[0096] 1102. Build an initial target detection model and multiple detection tasks.

[0097] 1103. Based on multiple detection tasks, an initial target detection model is trained by using a training set to obtain a target detection model.

[0098] It can be understood that multiple detection tasks are performed in series, sample data is forward propagated to obtain detection results, and gradients are calculated by back propagation to update the weight parameters of the model. Specifically, the backbone network and single bottleneck network shared by the 2D detection network and the 3D detection network, in the process of training the model based on multiple detection tasks and through the training set, the gradients between different detection tasks are first accumulated, and the parameters are updated after all tasks have completed a forward propagation and reverse calculation, to ensure that the backbone network and the single bottleneck network shared by the 2D detection network and the 3D detection network can fully learn the information of different tasks; for the 2D detection network and the 3D detection network, the parameters can be updated after each task completes a forward propagation and reverse calculation.

[0099] In the embodiment of the present application, before inputting multiple target images into the target detection model, the method further includes: obtaining a training set, the training set includes multiple 2D sample data and multiple BEV sample data, each 2D sample data includes: an image, multiple 2D detection frames, and the types of each target included in the image; each BEV sample data includes: multiple frames of images with aligned timestamps, multiple 3D detection frames, the types of each target included in the image, the orientation angle of each target relative to the vehicle, and the speed of each target; constructing an initial target detection model and multiple detection tasks; based on multiple detection tasks, the initial target detection model is trained through the training set to obtain a target detection model. In this way, it is ensured that the trained target detection model can simultaneously realize 2D target detection and 3D target detection.

[0100] In some embodiments of the present application, the above step 1101 can be specifically implemented through the following steps 1101a to 1101d.

[0101] 1101a. Acquire multiple images.

[0102] 1101b. Perform 2D annotation on each of the multiple images to obtain multiple 2D sample data.

[0103] The 2D annotation includes: annotating the type of each target included in the image, and annotating the 2D detection box of each target.

[0104] 1101c. Acquire multiple groups of images and radar point cloud data corresponding to each group of images.

[0105] The multiple images included in each group of images have the same timestamp as the corresponding radar point cloud data.

[0106] 1101d. Based on each group of images and the radar point cloud data corresponding to each group of images, 3D annotation is performed in the 3D point cloud space to obtain multiple BEV sample data.

[0107] The 3D annotation includes: annotating the type of each target included in the image, annotating the 3D detection frame of each target, annotating the orientation angle of each target relative to the vehicle, and annotating the speed of each target.

[0108] It can be understood that for multiple images and radar point cloud data collected under actual road conditions, the images are annotated through the target tool.

[0109] In the embodiment of the present application, obtaining a training set includes: obtaining multiple images; performing 2D annotation on each of the multiple images to obtain multiple 2D sample data, wherein the 2D annotation includes: annotating the type of each target included in the image, and annotating the 2D detection frame of each target; obtaining multiple groups of images and radar point cloud data corresponding to each group of images, wherein the timestamps of the multiple images included in each group of images and the corresponding radar point cloud data are the same; based on each group of images and the radar point cloud data corresponding to each group of images, performing 3D annotation in 3D point cloud space to obtain multiple BEV sample data, wherein the 3D annotation includes: annotating the type of each target included in the image, annotating the 3D detection frame of each target, annotating the orientation angle of each target relative to the vehicle, and annotating the speed of each target. Annotating the images collected under actual road conditions ensures that the training set used in the training process is more suitable for vehicle application scenarios.

[0110] The present application provides a target detection model, including: a backbone network, a target bottleneck network, a 2D detection network and a 3D detection network; the backbone network is used to obtain image features of multiple scales of at least one target image; the target bottleneck network is used to fuse image features of multiple scales into a fused feature; the 2D detection network is used to perform target detection on the fused feature to obtain a 2D target detection result; the 3D detection network is used to perform target detection on the fused feature to obtain a 3D target detection result.

[0111] It can be understood that the backbone network is trained based on multiple detection tasks through a training set, and the multiple detection tasks include: multiple 2D target detection tasks and multiple 3D target detection tasks, and the training set includes multiple 2D sample data and multiple bird's-eye view BEV sample data; the target bottleneck network is trained based on multiple 2D target detection tasks and / or multiple 3D target detection tasks through multiple image features output by the backbone network; the 2D detection network is trained based on multiple 2D target detection tasks through multiple fused features output by the target bottleneck network; the 3D detection network is trained based on multiple 3D target detection tasks through multiple fused features output by the target bottleneck network.

[0112] It should be noted that the target detection model is used in the above-mentioned target detection method, and can obtain corresponding 2D detection results and 3D detection results based on at least one input target image. The specific implementation process refers to the description of the above-mentioned target detection method, which will not be repeated here.

[0113] like Fig.12 As shown, a target detection device is provided for the present application, including: an acquisition module 1201, used to acquire at least one target image collected by at least one camera set on a vehicle at the same time; an input module 1202, used to input at least one target image into a target detection model to obtain corresponding 2D target detection results and 3D target detection results; wherein the target detection model includes: a backbone network, a target bottleneck network, a 2D detection network and a 3D detection network; the backbone network is used to obtain image features of multiple scales of at least one target image; the target bottleneck network is used to fuse image features of multiple scales into a fused feature; the 2D detection network is used to perform target detection on the fused feature to obtain a 2D target detection result; the 3D detection network is used to perform target detection on the fused feature to obtain a 3D target detection result.

[0114] In some embodiments of the present application, the acquisition module 1201 is specifically used to synchronously capture at least one image through at least one camera set on the vehicle; preprocess each frame of the at least one image to obtain at least one target image, and the preprocessing includes at least one of the following: scaling processing, cropping processing, and normalization processing.

[0115] In some embodiments of the present application, the target bottleneck network includes: a 2D bottleneck subnetwork and a BEV bottleneck subnetwork, or the target bottleneck network is a single bottleneck network; when the target bottleneck network includes a 2D bottleneck subnetwork and a BEV bottleneck subnetwork, the 2D bottleneck subnetwork is obtained by training multiple image features based on multiple 2D target detection tasks, and the BEV bottleneck network is obtained by training multiple image features based on multiple 3D target detection tasks; when the target bottleneck network is a single bottleneck network, the single bottleneck network is obtained by training multiple image features based on multiple detection tasks.

[0116] In some embodiments of the present application, the target bottleneck network is a single bottleneck network, the 2D detection network includes: multiple 2D detection heads, and the 3D detection network includes: a BEV conversion subnetwork and multiple 3D detection heads; the input module 1202 is specifically used to extract features from at least one target image through the backbone network to obtain multiple scale features corresponding to each target image in at least one target image; perform feature fusion on multiple scale features through the single bottleneck network to obtain a first fused feature corresponding to each target image in at least one target image; detect the first fused feature through multiple 2D detection heads to obtain a 2D Target detection results, 2D target detection results include multiple detection results output by multiple 2D detection heads, and different 2D detection heads correspond to different 2D target detection tasks; based on the internal and external parameters of at least one camera, at least one first fusion feature corresponding to at least one target image is mapped to the target BEV space through the BEV conversion subnetwork to obtain a first BEV feature; through multiple 3D detection heads, the first BEV feature is detected to obtain a 3D target detection result, and the 3D target detection result includes multiple detection results output by multiple 3D detection heads, and different 3D detection heads correspond to different 3D target detection tasks.

[0117] In some embodiments of the present application, the target bottleneck network includes: a 2D bottleneck subnetwork and a BEV bottleneck subnetwork; the 2D detection network includes: multiple 2D detection heads, and the 3D detection network includes: a BEV conversion subnetwork and multiple 3D detection heads; the input module 1202 is specifically used to extract features from at least one target image through the backbone network to obtain multiple scale features corresponding to each target image in the at least one target image; perform feature fusion on multiple scale features through the 2D bottleneck subnetwork to obtain a second fused feature corresponding to each target image in the at least one target image; perform detection on the second fused feature through multiple 2D detection heads to obtain a 2D target detection result; perform feature fusion on multiple scale features through the BEV bottleneck subnetwork to obtain a third fused feature corresponding to each target image in the at least one target image; based on the internal and external parameters of at least one camera, map at least one third fused feature corresponding to the at least one target image to the target BEV space through the BEV conversion subnetwork to obtain a second BEV feature; perform detection on the second BEV feature through multiple 3D detection heads to obtain a 3D target detection result.

[0118] In some embodiments of the present application, the target detection device also includes a selection module for selecting, according to a feature selection relationship, at least one second fusion feature corresponding to each 2D detection head from at least one second fusion feature corresponding to at least one target image before detecting the second fusion feature through multiple 2D detection heads to obtain a 2D target detection result, wherein the feature selection relationship includes multiple corresponding relationships, and each corresponding relationship is used to indicate at least one second fusion feature included in a 2D detection head; the input module 1202 is specifically used to detect at least one second fusion feature corresponding to each 2D detection head through each 2D detection head in the multiple 2D detection heads to obtain a 2D target detection result.

[0119] The target detection device provided in the embodiment of the present application can be used to execute the steps of the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0120] It should be noted that the above-mentioned target detection device can be the electronic device in the above-mentioned method embodiment of the present application, or it can be a functional module and / or functional entity in the electronic device that can realize the function of the device embodiment, and the embodiment of the present application is not limited.

[0121] In the embodiments of the present application, each module can implement the target detection method provided in the above method embodiments and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0122] The beneficial effects of various implementation methods in this embodiment can be specifically referred to the beneficial effects of the corresponding implementation methods in the above target detection method embodiment. To avoid repetition, they will not be repeated here.

[0123] In an embodiment of the present application, a target detection model training device is provided, including: an acquisition module for acquiring a training set, the training set includes multiple 2D sample data and multiple BEV sample data, each 2D sample data includes: an image, multiple 2D detection frames, and the types of each target included in the image; each BEV sample data includes: multiple frames of images with aligned timestamps, multiple 3D detection frames, the types of each target included in the image, the orientation angle of each target relative to the vehicle, and the speed of each target; a construction module for constructing an initial target detection model and multiple detection tasks; a training module for training the initial target detection model through the training set based on multiple detection tasks to obtain a target detection model. The target detection model training device provided in the embodiment of the present application can be used to execute the steps of the above-mentioned target detection model training method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0124] The present application also provides an electronic device, such as Fig.13As shown, the electronic device may include: a processor 1301, a memory 1302, and a program or instruction stored in the memory 1302 and executable on the processor 1301. When the program or instruction is executed by the processor 1301, the various processes of the target detection method provided in the above method embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0125] An embodiment of the present application provides a vehicle, which includes the above-mentioned target detection device or electronic device and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0126] An embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the target detection method provided by the above method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0127] An embodiment of the present application also provides a computer program product, wherein the computer program product includes a computer program or instructions. When the computer program product runs on a processor, the processor executes the computer program or instructions to implement the various processes of the target detection method provided by the above method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0128] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned target detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0129] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0130] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, servers and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0131] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0133] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0134] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A target detection method, characterized in that: include: Acquire at least one target image captured by at least one camera disposed on the vehicle at the same time; Inputting the at least one target image into a target detection model to obtain corresponding 2D target detection results and 3D target detection results; Wherein, the target detection model includes: Backbone network, target bottleneck network, 2D detection network and 3D detection network; the backbone network is used to obtain image features of multiple scales of at least one target image; the target bottleneck network is used to fuse image features of multiple scales into a fused feature; the 2D detection network is used to perform target detection on the fused feature to obtain a 2D target detection result; the 3D detection network is used to perform target detection on the fused feature to obtain a 3D target detection result.

2. The method according to claim 1, characterized in that The step of acquiring at least one target image captured by at least one camera disposed on the vehicle at the same time comprises: Synchronously collecting at least one image through at least one camera disposed on the vehicle; Preprocessing is performed on each of the at least one image to obtain the at least one target image, wherein the preprocessing includes at least one of the following: scaling processing, cropping processing, and normalization processing.

3. The method according to claim 1, characterized in that The target bottleneck network includes: a 2D bottleneck subnetwork and a BEV bottleneck subnetwork, or the target bottleneck network is a single bottleneck network; In the case where the target bottleneck network includes the 2D bottleneck subnetwork and the BEV bottleneck subnetwork, the 2D bottleneck subnetwork is obtained by training with multiple image features based on the multiple 2D target detection tasks, and the BEV bottleneck subnetwork is obtained by training with the multiple image features based on the multiple 3D target detection tasks; In the case where the target bottleneck network is the single bottleneck network, the single bottleneck network is obtained by training the multiple image features based on the multiple detection tasks.

4. The method according to claim 3, characterized in that The target bottleneck network is the single bottleneck network, the 2D detection network includes: a plurality of 2D detection heads, and the 3D detection network includes: a BEV conversion subnetwork and a plurality of 3D detection heads; the at least one target image is input into the target detection model to obtain the corresponding 2D target detection result and 3D target detection result, including: Performing feature extraction on the at least one target image through the backbone network to obtain a plurality of scale features corresponding to each target image in the at least one target image; By means of the single bottleneck network, the plurality of scale features are subjected to feature fusion to obtain a first fusion feature corresponding to each target image in the at least one target image; The first fusion feature is detected by the multiple 2D detection heads to obtain the 2D object detection result, where the 2D object detection result includes multiple detection results output by the multiple 2D detection heads, and different 2D detection heads correspond to different 2D object detection tasks; Based on the internal and external parameters of the at least one camera, mapping at least one first fusion feature corresponding to the at least one target image to a target BEV space through the BEV conversion subnetwork to obtain a first BEV feature; The first BEV feature is detected by the multiple 3D detection heads to obtain the 3D target detection result, which includes multiple detection results output by multiple 3D detection heads. Different 3D detection heads correspond to different 3D target detection tasks.

5. The method according to claim 3, characterized in that: The target bottleneck network includes: a 2D bottleneck subnetwork and a BEV bottleneck subnetwork; the 2D detection network includes: a plurality of 2D detection heads, and the 3D detection network includes: a BEV conversion subnetwork and a plurality of 3D detection heads; the inputting the at least one target image into the target detection model to obtain the corresponding 2D target detection result and 3D target detection result includes: Performing feature extraction on the at least one target image through the backbone network to obtain a plurality of scale features corresponding to each target image in the at least one target image; By means of the 2D bottleneck sub-network, the plurality of scale features are subjected to feature fusion to obtain a second fusion feature corresponding to each target image in the at least one target image; Detecting the second fused features through the multiple 2D detection heads to obtain the 2D target detection result; By means of the BEV bottleneck sub-network, the plurality of scale features are subjected to feature fusion to obtain a third fusion feature corresponding to each target image in the at least one target image; Based on the internal and external parameters of the at least one camera, mapping at least one third fused feature corresponding to the at least one target image to the target BEV space through the BEV conversion subnetwork to obtain a second BEV feature; The second BEV feature is detected by the multiple 3D detection heads to obtain the 3D target detection result.

6. The method according to claim 5, characterized in that Before detecting the second fusion feature by the multiple 2D detection heads to obtain the 2D target detection result, the method further includes: Selecting, according to a feature selection relationship, at least one second fused feature corresponding to each 2D detection head from at least one second fused feature corresponding to the at least one target image, wherein the feature selection relationship includes a plurality of corresponding relationships, each corresponding relationship being used to indicate at least one second fused feature included in a 2D detection head; The detecting the second fusion feature by the multiple 2D detection heads to obtain the 2D target detection result includes: By using each 2D detection head among the multiple 2D detection heads, at least one second fusion feature corresponding to each 2D detection head is detected to obtain the 2D target detection result.

7. A target detection model, characterized in that: include: Backbone network, target bottleneck network, 2D detection network and 3D detection network; the backbone network is used to obtain image features of multiple scales of at least one target image; the target bottleneck network is used to fuse image features of multiple scales into a fused feature; the 2D detection network is used to perform target detection on the fused feature to obtain a 2D target detection result; the 3D detection network is used to perform target detection on the fused feature to obtain a 3D target detection result.

8. A target detection model training method, characterized in that: include: Acquire a training set, wherein the training set includes multiple 2D sample data and multiple BEV sample data, each 2D sample data includes: an image, multiple 2D detection frames, and the type of each target included in the image; each BEV sample data includes: multiple frames of images with aligned timestamps, multiple 3D detection frames, the type of each target included in the image, the orientation angle of each target relative to the vehicle, and the speed of each target; Constructing an initial target detection model and the multiple detection tasks; Based on the multiple detection tasks, the initial target detection model is trained by using the training set to obtain a target detection model.

9. The method according to claim 8, characterized in that The obtaining of the training set comprises: Get multiple images; Performing 2D annotation on each of the multiple images to obtain the multiple pieces of 2D sample data, wherein the 2D annotation includes: annotating the type of each target included in the image and annotating the 2D detection frame of each target; Acquire multiple groups of images and radar point cloud data corresponding to each group of images, wherein the multiple images included in each group of images and the corresponding radar point cloud data have the same timestamp; Based on each group of images and the radar point cloud data corresponding to each group of images, 3D annotation is performed in the 3D point cloud space to obtain the multiple BEV sample data, and the 3D annotation includes: annotating the type of each target included in the image, annotating the 3D detection box of each target, annotating the orientation angle of each target relative to the vehicle, and annotating the speed of each target.

10. A target detection device, characterized in that: include: An acquisition module, which acquires at least one target image captured by at least one camera arranged on the vehicle at the same time; An input module, inputting the at least one target image into a target detection model to obtain corresponding 2D target detection results and 3D target detection results; Wherein, the target detection model includes: Backbone network, target bottleneck network, 2D detection network and 3D detection network; the backbone network is used to obtain image features of multiple scales of at least one target image; the target bottleneck network is used to fuse image features of multiple scales into a fused feature; the 2D detection network is used to perform target detection on the fused feature to obtain a 2D target detection result; the 3D detection network is used to perform target detection on the fused feature to obtain a 3D target detection result.

11. A target detection model training device, characterized in that: include: An acquisition module is used to acquire a training set, wherein the training set includes multiple 2D sample data and multiple BEV sample data, each 2D sample data includes: an image, multiple 2D detection frames, and the type of each target included in the image; each BEV sample data includes: multiple frames of images with aligned timestamps, multiple 3D detection frames, the type of each target included in the image, the orientation angle of each target relative to the vehicle, and the speed of each target; A construction module, used to construct an initial target detection model and the multiple detection tasks; The training module is used to train the initial target detection model through a training set based on the multiple detection tasks to obtain a target detection model.

12. A vehicle, characterized in that: include: The target detection device of claim 10, or the target detection model training device of claim 11.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6, 8 and 9 are implemented.

Images