Camera-based target detection methods, devices, equipment, and storage media

Abstract

This application provides a camera-based target detection method, apparatus, device, and storage medium, relating to the field of data processing technology. The method includes: inputting a target image captured by a camera deployed on a vehicle into an image feature extraction network for feature extraction, obtaining an image feature vector of the target image, and transforming the image feature vector to obtain an environmental feature vector; inputting the image feature vector and the camera's intrinsic parameters into a depth estimation network for depth estimation, obtaining depth estimation information for each foreground target in the target image; wherein the depth estimation network is obtained in advance through direct supervision based on sample images labeled with real feature information of foreground targets; and obtaining a three-dimensional target detection result for the target image based on the depth estimation information and the environmental feature vector. This application improves the robustness and accuracy of depth estimation, thereby also improving the overall robustness of the algorithm and the accuracy of the target detection result.

Description

Technical Field

[0001] This application relates to data processing technology, and more particularly to a camera-based target detection method, apparatus, device, and storage medium. Background Technology

[0002] In the field of autonomous driving, camera-based 3D object detection algorithms have received great attention from the industry due to their advantages of low cost and wide applicability, and have therefore developed very rapidly.

[0003] However, while cameras can provide rich texture and color information, they lose depth information when acquiring image data. How to make accurate depth estimates based on camera-captured images has always been a key focus and challenge in camera-based 3D object detection algorithms. In recent years, researchers have proposed relatively reliable depth estimation networks such as BEVDet and BEVDepth based on the LSS framework for camera 3D object detection, performing depth estimation pixel-by-pixel. However, this pixel-by-pixel depth estimation is inherently very difficult, and the results are affected by various factors such as camera intrinsic and extrinsic parameters, vehicle pose, etc. In practical applications, the robustness of these models remains limited, and the detection accuracy is also relatively low.

[0004] Therefore, there is an urgent need for a new camera-based object detection method to improve the robustness of depth estimation and the accuracy of object detection results. Summary of the Invention

[0005] This application provides a camera-based target detection method, apparatus, device, and storage medium to address issues such as how to improve the robustness of depth estimation and the accuracy of target detection results.

[0006] In a first aspect, this application provides a camera-based target detection method, the method comprising:

[0007] The target image captured by the camera deployed on the vehicle is input into the image feature extraction network for feature extraction to obtain the image feature vector of the target image. The image feature vector is then transformed to obtain the environmental feature vector. The image feature vector represents the feature data of the target image in the camera coordinate system, and the environmental feature vector represents the feature data of the target image in the global coordinate system.

[0008] The image feature vector and the camera intrinsic parameters of the camera are input into a depth estimation network for depth estimation to obtain depth estimation information of each foreground target in the target image; wherein, the depth estimation network is obtained in advance by direct supervision based on sample images labeled with real feature information of foreground targets;

[0009] Based on the depth estimation information and the environmental feature vector, the three-dimensional target detection result of the target image is obtained.

[0010] Optionally, obtaining the 3D target detection result of the target image based on the depth estimation information and the environmental feature vector includes:

[0011] Based on the depth estimation information and the environmental feature vector, a bird's-eye view feature vector is obtained; wherein, the bird's-eye view feature vector represents the feature data based on the bird's-eye view plane after the depth estimation information and the environmental feature vector are fused.

[0012] The bird's-eye view feature vector is processed to obtain the three-dimensional target detection result of the target image.

[0013] Optionally, obtaining the bird's-eye view feature vector based on the depth estimation information and the environmental feature vector includes:

[0014] According to preset weighting coefficients, the depth estimation information is weighted and superimposed onto the environmental feature vector to obtain a fused feature vector; wherein, the fused feature vector represents the feature data obtained after fusing the depth estimation information and the environmental feature vector;

[0015] The fused feature vector is input into an efficient voxel pool for processing to obtain the bird's-eye view feature vector.

[0016] Optionally, processing the bird's-eye view feature vector to obtain the three-dimensional target detection result of the target image includes:

[0017] The bird's-eye view feature vector is input into the target detection head for target detection to obtain the three-dimensional target detection result of the target image; wherein, the target detection head is a pre-trained model used to determine the three-dimensional target detection result of the target image.

[0018] Optionally, before inputting the image feature vector and the camera intrinsic parameters of the camera into a depth estimation network to perform depth estimation and obtain depth estimation information of each foreground object in the target image, the method further includes:

[0019] Obtain a sample image labeled with real feature information of a foreground target; wherein, the real feature information includes the two-dimensional detection box, category, angle, and depth of the foreground target, and the size information of the foreground target is labeled on the two-dimensional detection box;

[0020] The depth estimation network is directly supervised using the sample images to obtain a trained depth estimation network.

[0021] Optionally, the step of transforming the image feature vector to obtain the environmental feature vector includes:

[0022] The image feature vector is converted into the environment feature vector according to a preset transformation formula from the camera coordinate system to the global coordinate system.

[0023] Optionally, the three-dimensional target detection result includes one or more of the following: the three-dimensional detection bounding box of the target object, the category of the target object, the position of the target object, the size of the target object, the pose information of the target object, and the number of target objects.

[0024] Secondly, this application provides a camera-based target detection device, the device comprising:

[0025] An image processing unit is used to input the target image captured by a camera deployed on the vehicle into an image feature extraction network for feature extraction, obtain the image feature vector of the target image, and perform transformation processing on the image feature vector to obtain an environmental feature vector; wherein, the image feature vector represents the feature data of the target image in the camera coordinate system, and the environmental feature vector represents the feature data of the target image in the global coordinate system;

[0026] The depth estimation unit is used to input the image feature vector and the camera intrinsic parameters of the camera into the depth estimation network to perform depth estimation and obtain the depth estimation information of each foreground target in the target image; wherein, the depth estimation network is obtained in advance by direct supervision based on sample images labeled with real feature information of foreground targets;

[0027] The target detection unit is used to obtain the three-dimensional target detection result of the target image based on the depth estimation information and the environmental feature vector.

[0028] Thirdly, this application provides an electronic device, the electronic device comprising: a processor, and a memory communicatively connected to the processor;

[0029] The memory stores computer-executed instructions;

[0030] The processor executes computer execution instructions stored in the memory to implement the method as described in the preceding one.

[0031] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as described in any of the preceding claims.

[0032] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, is used to implement the method as described in any of the foregoing.

[0033] The camera-based target detection method, apparatus, device, and storage medium provided in this application include: inputting a target image acquired by a camera deployed on a vehicle into an image feature extraction network for feature extraction, obtaining an image feature vector of the target image, and performing transformation processing on the image feature vector to obtain an environmental feature vector; wherein, the image feature vector represents the feature data of the target image in the camera coordinate system, and the environmental feature vector represents the feature data of the target image in the global coordinate system; inputting the image feature vector and the camera intrinsic parameters of the camera into a depth estimation network for depth estimation, obtaining depth estimation information of each foreground target in the target image; wherein, the depth estimation network is obtained in advance through direct supervision based on sample images labeled with real feature information of foreground targets; and obtaining a three-dimensional target detection result of the target image based on the depth estimation information and the environmental feature vector. In this application, when estimating the depth information of the foreground target in the target image through the depth estimation network, the camera parameters only use the camera intrinsic parameters, thereby shielding the influence of other factors such as camera extrinsic parameters and vehicle body posture on the depth estimation. In addition, the depth estimation network of this application is obtained by direct supervision using sample images labeled with real feature information of the foreground target, which improves the robustness and accuracy of depth estimation, and thus also improves the robustness of the overall algorithm and the accuracy of the target detection results. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0035] Figure 1 A schematic flowchart of a camera-based target detection method provided in an embodiment of this application;

[0036] Figure 2 An example architecture diagram of a camera-based target detection scheme provided in an embodiment of this application;

[0037] Figure 3 A schematic diagram of a camera-based target detection device provided in an embodiment of this application;

[0038] Figure 4 A schematic diagram of another camera-based target detection device provided in this application embodiment;

[0039] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0040] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0042] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] In recent years, camera 3D object detection algorithms based on the LSS framework have developed rapidly. Within the LSS framework, the camera's depth estimation module has the greatest impact on accuracy. LSS proposes a pixel-wise depth distribution estimation representation to implicitly learn pixel-wise depth information and construct a view frustum feature space. The view frustum is then projected onto the Bird-Eye-View (BEV) plane to obtain BEV features before object detection. Object detection methods based on the LSS architecture can obtain relatively dense BEV features; however, because each pixel predicts a series of depth probability values, the computational cost is relatively high, making it difficult to meet the high-efficiency requirements of autonomous driving.

[0044] Based on this, Huang et al. proposed the BEVDet 3D object detection framework using the LSS framework. Subsequent studies found that the depth estimation error in BEVDet remained relatively large, significantly impacting the final object detection results. Therefore, they proposed direct supervision of pixel depth to improve depth estimation accuracy. The later proposed BEVDepth constructed a depth sub-network, using depth information from the point cloud to supervise the sub-network and learn the true depth values. However, in the BEVDepth scheme, due to the similarity of background point image features and the need to estimate different depth values, and considering the locality of convolution, it is not an easy problem to converge without incorporating positional information. Furthermore, foreground points are easily affected by various factors such as camera intrinsic and extrinsic parameters, vehicle pose, etc. In real-world scenarios, they are easily affected by road smoothness, vehicle tilt, etc., resulting in limited robustness of the model in practical applications, and consequently, unsatisfactory object detection accuracy.

[0045] To address the aforementioned issues, this application provides a camera-based target detection method. To improve the accuracy and robustness of target depth estimation, it attempts to transform the pixel-level depth estimation problem into target-level depth estimation. This allows for the use of not only structured information but also greater reliance on the size information of the foreground target for depth prediction. Relying on target size information eliminates the influence of camera extrinsic parameters and vehicle pose, significantly improving the robustness of depth estimation in scenarios where the target is fully visible.

[0046] Specifically, in this application, only the camera intrinsic parameters are used as input in the camera parameters when performing depth estimation, thereby shielding the influence of camera extrinsic parameters and vehicle body pose on depth estimation. In addition, this application also directly supervises the depth estimation network through the real feature information of the foreground target in the image, which makes the depth estimation information more accurate and the model more robust. Furthermore, the depth of the foreground target is weighted and superimposed on the feature map, thereby generating a BEV feature map containing more robust depth estimation information, which improves the robustness and accuracy of the overall detection algorithm.

[0047] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0048] Figure 1This is a flowchart illustrating a camera-based target detection method provided in an embodiment of this application. The executing entity in this embodiment can be a camera-based target detection device, which can be located on an electronic device. The electronic device can be a mobile terminal, such as a mobile phone, tablet, or computer, or it can be located on a vehicle such as a car; this application does not impose any limitations. This embodiment uses a camera-based target detection device as an example for detailed explanation.

[0049] like Figure 1 As shown, the camera-based target detection method provided in this embodiment includes:

[0050] S101. Input the target image captured by the camera deployed on the vehicle into the image feature extraction network for feature extraction, obtain the image feature vector of the target image, and perform transformation processing on the image feature vector to obtain the environmental feature vector; wherein, the image feature vector represents the feature data of the target image in the camera coordinate system, and the environmental feature vector represents the feature data of the target image in the global coordinate system.

[0051] For example, cameras can be installed at multiple different locations on the vehicle, depending on the actual detection range requirements, to capture environmental images of the vehicle's surroundings as target images for target detection in this application. This application does not limit the number or distribution of cameras. For instance, at least six cameras can be installed on the vehicle to acquire images of the entire area around it.

[0052] For example, the camera-based target detection device of this application inputs the target images captured simultaneously by the camera deployed on the vehicle into an image feature extraction network for feature extraction, obtaining an image feature vector for each target image, and then transforming the image feature vector to obtain an environment feature vector. The image feature vector includes feature data of the target image in the camera coordinate system, and the environment feature vector includes feature data of the target image in the global coordinate system.

[0053] The image feature extraction network can be a pre-trained neural network used to extract features from the target image; this application does not limit its specific structure. For example, it can use a standalone backbone network structure, or it can be a backbone network plus a multi-scale feature fusion network structure. The backbone network can be a convolutional neural network such as ResNet or VGG, and the multi-scale feature fusion network can be a convolutional neural network such as FPN; this application does not impose any restrictions. After the target image is input into the image feature extraction network, the network performs feature extraction and outputs the corresponding image feature vector of the target image. Then, the camera-based target detection device of this application further transforms the image feature vector output by the image feature extraction network to obtain the feature vector of the target image in the global coordinate system.

[0054] In one example, transforming image feature vectors to obtain environmental feature vectors can include:

[0055] Based on the preset transformation formula from the camera coordinate system to the global coordinate system, the image feature vector is converted into the environment feature vector.

[0056] For example, the preset transformation formula from the camera coordinate system to the global coordinate system can be pre-designed, and this application does not limit it; it can be any method that can be implemented using existing technology. After obtaining the image feature vector of the target image, the image feature vector can be converted into an environment feature vector by combining it with the preset transformation formula from the camera coordinate system to the global coordinate system.

[0057] S102. Input the image feature vector and the camera intrinsic parameters of the camera into the depth estimation network to perform depth estimation and obtain the depth estimation information of each foreground target in the target image; wherein, the depth estimation network is obtained in advance by direct supervision based on sample images labeled with real feature information of foreground targets.

[0058] For example, after obtaining the image feature vector, the target detection device of this application further inputs the image feature vector of the target image and the camera intrinsic parameters of the camera that acquired the target image into a pre-trained depth estimation network to perform depth estimation, so as to obtain the depth estimation information of each foreground target in the target image.

[0059] Camera intrinsic parameters are parameters related to the camera's own characteristics. For a specific camera, camera intrinsic parameters are generally not easily changed. Camera intrinsic parameters may include transformation parameters from the camera coordinate system to the pixel coordinate system. Camera intrinsic parameters can be represented by a matrix K, which may include parameters such as focal length, pixel size, and distortion coefficients; this application does not impose any limitations.

[0060] Furthermore, the depth estimation network is also obtained through direct supervision based on sample images labeled with real feature information of the foreground object. The real feature information of the foreground object can include its two-dimensional bounding box, category, angle, and depth, etc., and the bounding box can also be labeled with the size information of the foreground object, such as length, width, and height.

[0061] In some examples, the depth estimation network is not pre-trained. Before inputting the image feature vector and the camera's intrinsic parameters into the depth estimation network to perform depth estimation and obtain the depth estimation information of each foreground object in the target image, the depth estimation network needs to be trained. Therefore, the solution of this application may also include:

[0062] S10. Obtain a sample image labeled with real feature information of the foreground target; wherein, the real feature information includes the two-dimensional detection box, category, angle and depth of the foreground target, and the size information of the foreground target is labeled on the two-dimensional detection box.

[0063] S20. Directly supervise the depth estimation network using sample images to obtain a trained depth estimation network.

[0064] For example, during model training, sample images labeled with the true feature information of the foreground object are first acquired. These sample images are then used to directly supervise the depth estimation network to obtain a trained depth estimation network. The true feature information can include the foreground object's 2D bounding box (bbox2d), class, angle (theta), and depth. The 2D bounding box can be labeled with the specific dimensions of the foreground object, such as length, width, and height. This dimension information can be proportional, enlarged, or reduced, depending on the actual needs, and can be used to describe the size and position of the foreground object. The foreground object's class refers to its type, such as vehicle, pedestrian, or obstacle. The angle refers to the angle of the foreground object relative to the corresponding camera. The depth refers to the depth information of the foreground object relative to the camera. The true feature information of the foreground object on the sample image can be manually labeled or labeled after detection by the corresponding neural network model. If labeled after detection by the neural network model, it can be manually verified to ensure the accuracy of the true feature information of the foreground object on the sample image.

[0065] In this application, on the one hand, when estimating the depth information of the foreground target in the target image through the depth estimation network, the camera parameters only use the camera intrinsic parameters, thereby shielding the influence of other factors such as camera extrinsic parameters and vehicle body posture on the depth estimation; on the other hand, the depth estimation network is directly supervised by using sample images labeled with real feature information of the foreground target, which also improves the robustness and accuracy of the depth estimation, and thus improves the overall robustness of the algorithm and the accuracy of the target detection results.

[0066] S103. Based on the depth estimation information and environmental feature vectors, obtain the three-dimensional target detection results of the target image.

[0067] For example, after obtaining depth estimation information and environmental feature vectors, the camera-based target detection device of this application can obtain the three-dimensional target detection result of the target image based on the depth estimation information and environmental feature vectors.

[0068] The specific content of the 3D target detection results can be set according to the detection requirements of the actual application scenario, and this application does not impose any restrictions. In one example, the 3D target detection results may include one or more of the following: the 3D detection bounding box of the target object, the category of the target object, the position of the target object, the size of the target object, the pose information of the target object, and the number of target objects.

[0069] For example, a 3D bounding box can be used to circle the target object and label it with specific information such as its category, location, size, pose, and number. The target object's category can be pedestrians, vehicles, obstacles, etc.; its location can be represented by its distance and orientation from the vehicle; its size can be represented by its length, width, and height; its pose information can include its rotation angle and orientation; and if there are multiple target objects, the total number of objects detected can also be output. In practical applications, one or more of the above target object information can be output as the 3D target detection result, depending on the specific detection requirements.

[0070] This application does not impose restrictions on how to obtain the 3D target detection result of the target image based on depth estimation information and environmental feature vectors. In one example, obtaining the 3D target detection result of the target image based on depth estimation information and environmental feature vectors may include:

[0071] S1031. Obtain the bird's-eye view feature vector based on the depth estimation information and the environmental feature vector; wherein, the bird's-eye view feature vector represents the feature data based on the bird's-eye view plane after the depth estimation information and the environmental feature vector are fused.

[0072] S1032. Process the bird's-eye view feature vector to obtain the three-dimensional target detection results of the target image.

[0073] For example, the bird's-eye view feature vector can be obtained first, based on the estimated depth information and the converted environmental feature vector, to represent the feature data based on the bird's-eye view plane after the fusion processing of the depth estimation information and the environmental feature vector; then, the bird's-eye view feature vector is processed to obtain the three-dimensional target detection result of the target image.

[0074] In one example, obtaining a bird's-eye view feature vector based on depth estimation information and environmental feature vectors can include:

[0075] S10311. According to the preset weighting coefficients, the depth estimation information is weighted and superimposed onto the environmental feature vector to obtain the fused feature vector; wherein, the fused feature vector represents the feature data obtained after fusing the depth estimation information and the environmental feature vector.

[0076] S10312. Input the fused feature vector into the efficient voxel pool for processing to obtain the bird's-eye view feature vector.

[0077] For example, different weight coefficients can be assigned to features of different parts / locations in the target image captured by each camera. Features with higher weights are considered more important, while features with lower weights are considered less important. The specific values ​​are not limited in this application and can be predetermined based on experimental data or set manually.

[0078] This application's camera-based target detection device can weight and superimpose depth estimation information onto environmental feature vectors according to preset weighting coefficients to obtain a fused feature vector representing the feature data obtained after fusing depth estimation information and environmental feature vectors. This fused feature vector is then input into an efficient voxel pool for processing to obtain a bird's-eye view feature vector. The purpose of the efficient voxel pool is to map the multi-view fused feature vector onto BEV features. It divides the grid space into several uniformly distributed grids and then aggregates the frustum features falling into the same grid to form the corresponding BEV features. In specific processing, a CUDA thread can be allocated to each frustum feature to add the corresponding feature to the corresponding BEV grid, thereby improving computation speed and the overall processing efficiency of the detection algorithm, enabling faster acquisition of target detection results.

[0079] In addition, this application uses different weighting coefficients to quantify the features of different parts / locations in the target image, which can improve the impact of depth estimation information on the target detection results and also improve the robustness of the overall algorithm and the accuracy of the target detection results.

[0080] In one example, processing the bird's-eye view feature vector to obtain the 3D target detection result of the target image can include:

[0081] The bird's-eye view feature vector is input into the target detection head for target detection, and the three-dimensional target detection result of the target image is obtained; where the target detection head is a pre-trained model used to determine the three-dimensional target detection result of the target image.

[0082] For example, after the obtained bird's-eye view feature vector is input into the target detection head, the target detection head will perform target detection based on the bird's-eye view feature vector, thereby obtaining the three-dimensional target detection result of the target image required by this application. The target detection head can be a pre-trained model for determining the three-dimensional target detection result of the target image, and its specific structure is not limited in this application. For example, detection heads such as CenterPoint and SSD3D can be used.

[0083] The camera-based target detection method provided in this application includes: inputting a target image captured by a camera deployed on a vehicle into an image feature extraction network for feature extraction to obtain an image feature vector of the target image; and performing transformation processing on the image feature vector to obtain an environmental feature vector; wherein the image feature vector represents the feature data of the target image in the camera coordinate system, and the environmental feature vector represents the feature data of the target image in the global coordinate system; inputting the image feature vector and the camera's intrinsic parameters into a depth estimation network for depth estimation to obtain depth estimation information of each foreground target in the target image; wherein the depth estimation network is obtained in advance by direct supervision based on sample images labeled with real feature information of foreground targets; and obtaining a three-dimensional target detection result of the target image based on the depth estimation information and the environmental feature vector. The proposed solution, on the one hand, uses only intrinsic camera parameters when estimating the depth information of foreground targets in a target image through a depth estimation network, thereby shielding the influence of extrinsic camera parameters and other factors such as vehicle body posture on depth estimation; on the other hand, the depth estimation network of this application is obtained by direct supervision using sample images labeled with real feature information of foreground targets, which improves the robustness and accuracy of depth estimation, and thus also improves the overall robustness of the algorithm and the accuracy of target detection results.

[0084] Figure 2 This diagram illustrates an architecture example of a camera-based target detection scheme provided in an embodiment of this application. Figure 2As shown, multiple target images (multi-view images) captured by cameras located at different positions on the vehicle are input into the backbone network for feature extraction. The backbone network outputs image features of the target images. Then, the image features are transformed to obtain context features. Simultaneously, the camera intrinsics of each camera and their corresponding image features are input into the depth estimation network for depth estimation. The depth estimation network outputs depth estimation information for the foreground targets. The depth estimation information is weighted and superimposed onto the context features, then processed by an efficient voxel pooling system to obtain more robust bird's-eye view (BEV) features. Finally, a detection head is used to detect the bird's-eye view features to obtain target detection results. As shown in the figure, this application also uses sample images labeled with foreground objects to directly supervise the output of the depth estimation network (bbox2d / class / theta / depth, etc.), thereby improving the accuracy of the depth estimation results.

[0085] in, Figure 2 The two-dimensional detection box indicated by the mark 10 means that there is a foreground target at that location, and it has been filled to distinguish it from other parts. In practical applications, it is sufficient to mark only the two-dimensional detection box in order to determine the category of the foreground target in the detection box. This application does not impose any restrictions.

[0086] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0087] Figure 3 This is a schematic diagram of a camera-based target detection device provided in an embodiment of this application. Figure 3 As shown, the camera-based target detection device 30 provided in this application embodiment includes an image processing unit 301, a depth estimation unit 302, and a target detection unit 303.

[0088] The image processing unit 301 is used to input the target image captured by the camera deployed on the vehicle into the image feature extraction network for feature extraction, obtain the image feature vector of the target image, and perform transformation processing on the image feature vector to obtain the environmental feature vector; wherein, the image feature vector represents the feature data of the target image in the camera coordinate system, and the environmental feature vector represents the feature data of the target image in the global coordinate system.

[0089] The depth estimation unit 302 is used to input the image feature vector and the camera intrinsic parameters of the camera into the depth estimation network to perform depth estimation and obtain the depth estimation information of each foreground target in the target image; wherein, the depth estimation network is obtained in advance by direct supervision based on sample images labeled with real feature information of foreground targets;

[0090] The target detection unit 303 is used to obtain the three-dimensional target detection result of the target image based on the depth estimation information and the environmental feature vector.

[0091] The apparatus provided in this embodiment can be used to execute the methods of the above embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0092] Figure 4 This is a schematic diagram of another camera-based target detection device provided in an embodiment of this application. Figure 4 As shown, the camera-based target detection device 40 provided in this application embodiment includes an image processing unit 401, a depth estimation unit 402, and a target detection unit 403.

[0093] The image processing unit 401 is used to input the target image captured by the camera deployed on the vehicle into the image feature extraction network for feature extraction, obtain the image feature vector of the target image, and perform transformation processing on the image feature vector to obtain the environmental feature vector; wherein, the image feature vector represents the feature data of the target image in the camera coordinate system, and the environmental feature vector represents the feature data of the target image in the global coordinate system.

[0094] The depth estimation unit 402 is used to input the image feature vector and the camera intrinsic parameters of the camera into the depth estimation network to perform depth estimation and obtain the depth estimation information of each foreground target in the target image; wherein, the depth estimation network is obtained in advance by direct supervision based on sample images labeled with real feature information of foreground targets;

[0095] The target detection unit 403 is used to obtain the three-dimensional target detection result of the target image based on the depth estimation information and the environmental feature vector.

[0096] In one example, the target detection unit 403 includes a data processing module 4031 and a detection module 4032.

[0097] The data processing module 4031 is used to obtain the bird's-eye view feature vector based on the depth estimation information and the environmental feature vector; wherein, the bird's-eye view feature vector represents the feature data based on the bird's-eye view plane after the depth estimation information and the environmental feature vector are fused and processed.

[0098] The detection module 4032 is used to process the bird's-eye view feature vector to obtain the three-dimensional target detection results of the target image.

[0099] In one example, the data processing module 4031 includes a weighting module 40311 and a processing module 40312.

[0100] The weighting module 40311 is used to weight and superimpose the depth estimation information onto the environmental feature vector according to the preset weight coefficients to obtain the fused feature vector; wherein, the fused feature vector represents the feature data obtained after fusing the depth estimation information and the environmental feature vector.

[0101] The processing module 40312 is used to input the fused feature vector into the efficient voxel pool for processing to obtain the bird's-eye view feature vector.

[0102] In one example, the detection module 4032 is specifically used to input the bird's-eye view feature vector into the target detection head for target detection, and obtain the three-dimensional target detection result of the target image; wherein, the target detection head is a pre-trained model used to determine the three-dimensional target detection result of the target image.

[0103] In one example, the device 40 also includes a model training unit 400, which includes a sample module 4001 and a training module 4002.

[0104] The sample module 4001 is used to acquire sample images labeled with real feature information of foreground targets; wherein, the real feature information includes the two-dimensional detection box, category, angle and depth of the foreground target, and the size information of the foreground target is labeled on the two-dimensional detection box.

[0105] Training module 4002 is used to directly supervise the depth estimation network using sample images to obtain a trained depth estimation network.

[0106] In one example, the image processing unit 401 includes a vector transformation module 4011.

[0107] The vector transformation module 4011 is used to convert image feature vectors into environment feature vectors according to a preset transformation formula from camera coordinate system to global coordinate system.

[0108] In one example, the 3D target detection result includes one or more of the following: the 3D detection bounding box of the target object, the category of the target object, the position of the target object, the size of the target object, the pose information of the target object, and the number of target objects.

[0109] The apparatus provided in this embodiment can be used to execute the methods of the above embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0110] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. Furthermore, they can be stored as program code in the device's memory, and the data processing modules can be called and executed by a specific processing element. The implementation of other modules is similar. These modules can be fully or partially integrated together, or implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0111] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 50 includes a processor 501 and a memory 502 communicatively connected to the processor.

[0112] The memory 502 stores computer-executable instructions; the processor 501 executes the computer-executable instructions stored in the memory 502 to implement the method as described above.

[0113] In the specific implementation of the aforementioned electronic device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The methods disclosed in the embodiments of this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0114] This application also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the methods described in any of the foregoing embodiments.

[0115] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to computer instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0116] This application also provides a computer program product, including a computer program that, when executed by a processor, is used to implement the method as described in any of the foregoing embodiments.

[0117] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0118] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A camera-based target detection method, characterized in that, The method includes: The target image captured by the camera deployed on the vehicle is input into the image feature extraction network for feature extraction to obtain the image feature vector of the target image. The image feature vector is then transformed to obtain the environmental feature vector. The image feature vector represents the feature data of the target image in the camera coordinate system, and the environmental feature vector represents the feature data of the target image in the global coordinate system. When performing depth estimation, only the camera intrinsic parameters are used as input to shield the influence of camera extrinsic parameters and vehicle pose on the depth estimation. The image feature vector and the camera intrinsic parameters are input into the depth estimation network to perform depth estimation and obtain the depth estimation information of each foreground target in the target image. The depth estimation network is obtained by direct supervision based on sample images labeled with real feature information of the foreground targets. The real feature information includes the two-dimensional detection box, category, angle, and depth of the foreground target, and the size information of the foreground target is labeled on the two-dimensional detection box. Based on the depth estimation information and the environmental feature vector, the three-dimensional target detection result of the target image is obtained.

2. The method according to claim 1, characterized in that, The step of obtaining the 3D target detection result of the target image based on the depth estimation information and the environmental feature vector includes: Based on the depth estimation information and the environmental feature vector, a bird's-eye view feature vector is obtained; wherein, the bird's-eye view feature vector represents the feature data based on the bird's-eye view plane after the depth estimation information and the environmental feature vector are fused. The bird's-eye view feature vector is processed to obtain the three-dimensional target detection result of the target image.

3. The method according to claim 2, characterized in that, The step of obtaining the bird's-eye view feature vector based on the depth estimation information and the environmental feature vector includes: According to preset weighting coefficients, the depth estimation information is weighted and superimposed onto the environmental feature vector to obtain a fused feature vector; wherein, the fused feature vector represents the feature data obtained after fusing the depth estimation information and the environmental feature vector; The fused feature vector is input into an efficient voxel pool for processing to obtain the bird's-eye view feature vector.

4. The method according to claim 2, characterized in that, The process of processing the bird's-eye view feature vector to obtain the 3D target detection result of the target image includes: The bird's-eye view feature vector is input into the target detection head for target detection to obtain the three-dimensional target detection result of the target image; wherein, the target detection head is a pre-trained model used to determine the three-dimensional target detection result of the target image.

5. The method according to claim 1, characterized in that, Before inputting the image feature vector and the camera intrinsic parameters of the camera into a depth estimation network to perform depth estimation and obtain depth estimation information for each foreground object in the target image, the method further includes: Obtain sample images labeled with real feature information of foreground targets; The depth estimation network is directly supervised using the sample images to obtain a trained depth estimation network.

6. The method according to claim 1, characterized in that, The step of transforming the image feature vector to obtain the environmental feature vector includes: The image feature vector is converted into the environment feature vector according to a preset transformation formula from the camera coordinate system to the global coordinate system.

7. The method according to any one of claims 1-6, characterized in that, The three-dimensional target detection results include one or more of the following: the three-dimensional detection bounding box of the target object, the category of the target object, the position of the target object, the size of the target object, the pose information of the target object, and the number of target objects.

8. A camera-based target detection device, characterized in that, The device includes: An image processing unit is used to input the target image captured by a camera deployed on the vehicle into an image feature extraction network for feature extraction, obtain the image feature vector of the target image, and perform transformation processing on the image feature vector to obtain an environmental feature vector; wherein, the image feature vector represents the feature data of the target image in the camera coordinate system, and the environmental feature vector represents the feature data of the target image in the global coordinate system; A depth estimation unit is used to use only camera intrinsic parameters as input when performing depth estimation, thereby shielding the influence of camera extrinsic parameters and vehicle pose on depth estimation. The image feature vector and the camera's intrinsic parameters are input into the depth estimation network to perform depth estimation, obtaining depth estimation information for each foreground target in the target image. The depth estimation network is obtained in advance through direct supervision based on sample images labeled with real feature information of the foreground targets. The real feature information includes the two-dimensional detection box, category, angle, and depth of the foreground target, with the size information of the foreground target labeled on the two-dimensional detection box. The target detection unit is used to obtain the three-dimensional target detection result of the target image based on the depth estimation information and the environmental feature vector.

9. An electronic device, characterized in that, The electronic device includes: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images