Target detection method and device, electronic equipment and readable storage medium

By using preset depth prediction model and depth distribution loss function in 3D object detection technology, the problem of insufficient accuracy of depth estimation is solved, and the accuracy of object detection is improved.

CN119027903APending Publication Date: 2024-11-26INSPUR (BEIJING) ELECTRONICS INFORMATION IND CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310610246.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The accuracy of depth estimation in the existing 3D object detection technology is insufficient, which affects the accuracy of object detection.

Method used

The preset depth prediction model is used to supervise and train based on the foreground depth distribution consistency loss function, the foreground region depth distribution entropy reduction constraint and the confusing region depth distribution entropy increase constraint, to estimate the depth distribution of image features, and to use the depth distribution to draw the image features. Convert to image aerial view features for object detection.

Benefits of technology

The accuracy of depth distribution estimation of image features is improved and more accurate object detection results are achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027903A_ABST
    Figure CN119027903A_ABST
Patent Text Reader

Abstract

The invention discloses a target detection method and device, electronic equipment and a readable storage medium, relates to the technical field of machine learning, and aims to solve the problem of inaccurate target detection result caused by inaccurate image feature depth estimation in the traditional target detection technology, and the method comprises the steps: carrying out the feature extraction of a target image, and obtaining image features; processing the image features by using a preset depth prediction model to obtain depth distribution of the image features; wherein the preset depth prediction model is obtained based on supervision training of a preset depth distribution loss function, and the preset depth distribution loss function comprises one or a combination of more of a foreground depth distribution consistency loss function, a foreground region depth distribution entropy reduction constraint and a confusion region depth distribution entropy increase constraint; converting the image features into image aerial view features by using the depth distribution; and performing target detection on the image aerial view features to obtain a target detection result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a target detection method, device, electronic device and readable storage medium. Background Art

[0002] In recent years, with the continuous advancement of sensor technology and information processing technology, autonomous driving technology has achieved rapid development and has received widespread attention. As an important part of autonomous driving technology, 3D object detection is an important means for autonomous driving systems to perceive and understand the surrounding environment.

[0003] In 3D object detection technology, the view conversion of image features from 2D to 3D (conversion from two-dimensional view to three-dimensional view) is critical. In related technologies, the LSS (Lift, Splat, Shoot) method is generally used to first predict the depth distribution of image features, then use the depth distribution to lift the image features into 3D feature point clouds, and finally convert them into BEV space. Among them, depth estimation is a big challenge. The accuracy of depth estimation directly affects the quality of image features after view conversion, and then affects the accuracy of 3D object detection results. However, the depth estimation accuracy of current technology still has a lot of room for improvement.

[0004] Therefore, how to perform more accurate depth distribution estimation of image features and thus achieve more accurate target detection is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] An object of the present invention is to provide a target detection method, which can perform more accurate depth distribution estimation of image features, thereby achieving more accurate target detection; another object of the present invention is to provide a target detection device, an electronic device and a computer-readable storage medium, all of which have the above-mentioned beneficial effects.

[0006] In a first aspect, the present invention provides a target detection method, comprising:

[0007] Extract features of the target image to obtain image features;

[0008] The image features are processed using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint;

[0009] converting the image features into image bird's-eye view features using the depth distribution;

[0010] Perform target detection on the bird's-eye view features of the image to obtain a target detection result.

[0011] Optionally, the process of constructing the preset depth prediction model based on the foreground depth distribution consistency loss function includes:

[0012] Acquire a target image sample, and determine a bounding box of a target object in the target image sample;

[0013] Taking the inner area of ​​the bounding box as the foreground area;

[0014] Perform random sampling in each of the foreground areas to obtain foreground point pairs;

[0015] Performing depth distribution prediction on the foreground point in each of the foreground point pairs to obtain depth prediction value pairs of the foreground point pairs;

[0016] In combination with the foreground depth distribution consistency loss function, model supervision training is performed using the preset depth distribution dimension and each of the depth prediction value pairs to obtain the preset depth prediction model.

[0017] Optionally, the process of constructing the preset depth prediction model based on the entropy reduction constraint of the foreground area depth distribution includes:

[0018] Acquire a target image sample, and determine a bounding box of a target object in the target image sample;

[0019] Taking the inner area of ​​the bounding box as the foreground area;

[0020] Perform random sampling in each of the foreground areas to obtain foreground points;

[0021] Predicting the depth distribution of each foreground point to obtain a depth prediction value of the foreground point;

[0022] Combined with the entropy reduction constraint of the depth distribution of the foreground area, model supervision training is performed using the preset depth distribution dimension and the depth prediction value of each foreground point to obtain the preset depth prediction model.

[0023] Optionally, the process of constructing the preset depth prediction model based on the entropy increase constraint of the depth distribution of the confusion area includes:

[0024] Acquire a target image sample, and determine a bounding box of a target object in the target image sample;

[0025] Inside the bounding box, a pixel area at a preset number of pixels away from the bounding box is used as a confusion area;

[0026] Perform random sampling in each of the confusion areas to obtain confusion points;

[0027] Performing depth distribution prediction on each of the confusion points to obtain a depth prediction value of the confusion point;

[0028] Combined with the entropy increase constraint of the depth distribution of the confused area, model supervision training is performed using the preset depth distribution dimension and the depth prediction value of each of the confused points to obtain the preset depth prediction model.

[0029] Optionally, the converting the image features into image bird's-eye view features by using the depth distribution includes:

[0030] Expanding the image features into point cloud features;

[0031] Weighting the point cloud features using the depth distribution to obtain weighted point cloud features;

[0032] Converting the weighted point cloud features from a camera coordinate system to a target detection coordinate system;

[0033] The weighted point cloud features in the target detection coordinate system are subjected to voxel pooling processing to obtain the image bird's-eye view features.

[0034] Optionally, performing voxel pooling processing on the weighted point cloud features in the target detection coordinate system to obtain the image bird's-eye view features includes:

[0035] Dividing the coordinate system space of the target detection coordinate system into cubic voxels of a preset size;

[0036] For each of the cubic voxels, performing depth clipping processing on the weighted point cloud features in the cubic voxel according to the depth clipping threshold corresponding to the cubic voxel to obtain clipping features;

[0037] The cropping features corresponding to all the cube voxels are combined into the image bird's-eye view feature.

[0038] Optionally, performing depth clipping processing on the weighted point cloud features in the cubic voxels according to the depth clipping threshold corresponding to the cubic voxels to obtain clipping features includes:

[0039] Determine the depth value of the cube voxel in the coordinate system space;

[0040] Performing depth probability prediction on the depth value to obtain a predicted depth probability;

[0041] When the predicted depth probability does not exceed the depth clipping threshold, all weighted point cloud features in the cube voxel are eliminated;

[0042] When the predicted depth probability exceeds the depth clipping threshold, the weighted point cloud features within the cube voxel are retained as the clipping features.

[0043] Optionally, the process of setting the depth clipping threshold includes:

[0044] Get a set of candidate thresholds;

[0045] For each candidate threshold in the candidate threshold set, calculating the proportion of clipped features and the model evaluation index corresponding to the candidate threshold;

[0046] In the candidate threshold set, the candidate thresholds whose model evaluation index is lower than the preset index threshold are deleted to obtain a retained candidate threshold set;

[0047] In the set of retained candidate thresholds, the candidate threshold corresponding to the largest proportion of clipped features is used as the depth clipping threshold.

[0048] Optionally, the calculation process of the model evaluation index includes:

[0049] For each candidate threshold, obtain the average accuracy value, average translation error, average scale error, average angle error, average velocity error, and average attribute error of the training model corresponding to the candidate threshold;

[0050] The model evaluation index is obtained by calculating the average accuracy value, the average translation error, the average scale error, the average angle error, the average velocity error, and the average attribute error.

[0051] Optionally, extracting features from the target image to obtain image features includes:

[0052] The target image is subjected to feature extraction using a preset image backbone network model to obtain the image features.

[0053] Optionally, performing target detection on the bird's-eye view features of the image to obtain a target detection result includes:

[0054] Use LiDAR to obtain point cloud data;

[0055] Extracting features from the point cloud data to obtain point cloud bird's-eye view features;

[0056] Fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fused features;

[0057] Perform target detection on the fused features to obtain the target detection result.

[0058] Optionally, fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fused features includes:

[0059] The image bird's-eye view features and the point cloud bird's-eye view features are fused using a preset feature fusion network model to obtain the fused features; wherein the preset feature fusion network model is constructed based on a multimodal semantic alignment loss function.

[0060] Optionally, the process of constructing the preset feature fusion network model based on the multimodal semantic alignment loss function includes:

[0061] Get a bird's eye view;

[0062] Acquire an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view;

[0063] Converting the image bird's-eye view feature set into an image bird's-eye view feature semantic coding set;

[0064] Converting the point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic coding set;

[0065] In combination with the multimodal semantic alignment loss function, the image bird's-eye view feature semantic coding set and the point cloud bird's-eye view feature semantic coding set are used to perform model supervision training to obtain the preset feature fusion network model.

[0066] Optionally, the acquiring of an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view includes:

[0067] determining a bounding box about a target object in the bird's-eye view;

[0068] Taking the inner area of ​​the bounding box as the foreground area;

[0069] Perform random sampling in each of the foreground areas to obtain each of the preset sampling foreground positions;

[0070] An image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each of the preset sampling foreground positions are obtained in the bird's-eye view.

[0071] Optionally, converting the image bird's-eye view feature set into an image bird's-eye view feature semantic coding set includes:

[0072] The image bird's-eye view feature set is converted into the image bird's-eye view feature semantic coding set by using a preset image semantic coding model;

[0073] Accordingly, the step of converting the point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic coding set includes:

[0074] The point cloud bird's-eye view feature set is converted into the point cloud bird's-eye view feature semantic coding set by using a preset point cloud semantic coding model.

[0075] Optionally, the extracting features from the point cloud data to obtain point cloud bird's-eye view features includes:

[0076] The point cloud data is subjected to feature extraction using a preset point cloud backbone network model to obtain the point cloud bird's-eye view features.

[0077] Optionally, fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fused features includes:

[0078] Splicing the image bird's-eye view features and the point cloud bird's-eye view features along a feature channel direction to obtain a splicing feature;

[0079] Using a preset feature fusion network model to perform feature fusion processing on the splicing features to obtain initial fusion features;

[0080] Performing dimension reduction processing on the initial fusion feature to obtain the fusion feature.

[0081] Optionally, performing target detection on the fused features to obtain the target detection result includes:

[0082] The preset fusion feature detection model is used to perform target detection on the fusion feature to obtain the target detection result.

[0083] In a second aspect, the present invention further discloses a target detection device, comprising:

[0084] An extraction module is used to extract features from a target image and obtain image features;

[0085] A processing module, used to process the image features using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint;

[0086] A conversion module, configured to convert the image features into image bird's-eye view features using the depth distribution;

[0087] The detection module is used to perform target detection on the bird's-eye view features of the image to obtain target detection results.

[0088] In a third aspect, the present invention further discloses an electronic device, comprising:

[0089] Memory for storing computer programs;

[0090] A processor is used to implement the steps of any one of the target detection methods described above when executing the computer program.

[0091] In a fourth aspect, the present invention further discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the target detection methods described above are implemented.

[0092] A target detection method provided by the present invention includes: extracting features from a target image to obtain image features; processing the image features using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint; converting the image features into image bird's-eye view features using the depth distribution; performing target detection on the image bird's-eye view features to obtain target detection results.

[0093] The application of the technical solution provided by the present invention has the following technical effects:

[0094] After extracting the image features from the target image, firstly use the pre-created depth prediction model to estimate the depth of the image features to obtain the depth distribution of the image features, then perform 2D-3D view conversion on the image features based on the depth distribution to obtain the corresponding image bird's-eye view features, and finally perform target detection on the image bird's-eye view features to achieve target detection for the target image and obtain the target detection result. Wherein, the preset depth prediction model is obtained by supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of the foreground depth distribution consistency loss function, the foreground area depth distribution entropy reduction constraint, and the confusion area depth distribution entropy increase constraint. The foreground depth distribution consistency loss function can be used to constrain different foreground points in the same target foreground area to have similar depth distributions, the foreground area depth distribution entropy reduction constraint can be used to constrain the same target foreground area to have consistent semantic information and not be easily disturbed by the background area, and the confusion area depth distribution entropy increase constraint can be used to guide the depth prediction model to make a more reasonable depth distribution prediction for the target foreground background confusion area. Therefore, based on the preset depth prediction model, a more accurate depth distribution estimation of the image features can be performed, which helps to achieve more accurate target detection.

[0095] The present invention also discloses a target detection device, an electronic device and a computer-readable storage medium, which also have the above-mentioned technical effects, and the present invention will not elaborate on them here. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present invention, the drawings required for describing the prior art and the embodiments of the present invention are briefly introduced below. Of course, the drawings related to the embodiments of the present invention below only describe some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work, and the obtained other drawings also belong to the protection scope of the present invention.

[0097] Figure 1 A schematic diagram of a flow chart of a target detection method provided by the present invention;

[0098] Figure 2 A framework diagram of a target detection system provided by the present invention;

[0099] Figure 3 A schematic diagram of a flow chart of an image feature view conversion method provided by the present invention;

[0100] Figure 4 A schematic diagram of a depth cutting method provided by the present invention;

[0101] Figure 5 A schematic diagram of a flow chart of a multimodal bird's-eye view feature semantic alignment method provided by the present invention;

[0102] Figure 6 A schematic diagram of a process flow of a target detection device provided by the present invention;

[0103] Figure 7 The present invention is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0104] The core of the present invention is to provide a target detection method, which can perform more accurate depth distribution estimation of image features, thereby achieving more accurate target detection; another core of the present invention is to provide a target detection device, electronic device and computer-readable storage medium, all of which have the above-mentioned beneficial effects.

[0105] In order to describe the technical solutions in the embodiments of the present invention more clearly and completely, the technical solutions in the embodiments of the present invention will be introduced below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0106] An embodiment of the present invention provides a target detection method.

[0107] Please refer to Figure 1 , Figure 1 This is a flow chart of a target detection method provided by the present invention. The target detection method may include the following S101 to S104.

[0108] S101: Extract features of the target image to obtain image features.

[0109] This step aims to achieve feature extraction of the target image to obtain its image features. The target image is the image data for target detection, which can be acquired by a camera device or the like. Further, after acquiring the target image, feature extraction can be performed on it to obtain the image features therein, so as to determine whether there is a target object (such as a pedestrian, an animal, an obstacle, etc.) in the target image by performing recognition processing on the image features.

[0110] In a possible implementation, the above-mentioned extracting features from the target image to obtain image features may include: extracting features from the target image using a preset image backbone network model to obtain image features.

[0111] The present embodiment provides a method for extracting features from a target image to obtain image features, which can be implemented based on a pre-created image backbone network model. Specifically, after acquiring the target image, it can be input into a preset image backbone network model for processing, and the output of the model is the image features of the target image. Among them, the preset image backbone network model can be constructed using a traditional convolutional neural network, such as a residual neural network (ResNet, Residual Network), or a Transformer-based network model, such as Swin Transformer.

[0112] S102: Processing the image features using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint;

[0113] This step aims to predict the depth distribution of image features. Specifically, the depth distribution is the probability distribution of a set of discrete depth values ​​defined in the depth dimension, and its prediction process is as follows:

[0114] First, a set of discrete depth values ​​is preset, such as s1 meters to s D Meters, interval s gap meters, and obtain D discrete depth values, where D = (s D -s1) / s gap +1,s D >s1. Then, the depth prediction network is used to take the image features under the camera view as input and output the depth distribution at each feature point. Assume that the size of the image features under the camera view is (H, W, C), where H and W represent the height and width of the image features respectively, and C represents the number of channels of the image features. For each feature point in the H and W dimensions, predict its corresponding depth distribution That is, the depth probability of each feature point at each preset depth value. The size of the depth distribution predicted for the image features under the entire camera view is (H, W, D), where H and W represent the height and width of the image feature respectively, and D represents the number of depth values ​​at each feature point.

[0115] Therefore, in the implementation process, a depth prediction model, that is, the above-mentioned preset depth prediction model, can be created in advance, and then the image features of the target image can be input into the preset depth prediction model for processing, and the output of the model is the depth distribution information of the image features. Among them, the preset depth prediction model can be composed of residual units, and a Softmax activation layer is used to output a multi-dimensional depth distribution prediction. During the construction process, it can be obtained by supervised training based on a preset depth distribution loss function. The preset depth distribution loss function can include a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint.

[0116] Among them, the prior rule for constructing the foreground depth distribution consistency loss function is: different foreground points in the same target foreground area have similar depth distributions. Therefore, the foreground depth distribution consistency loss function can be used to constrain the depth distribution of the two foreground points in each foreground point pair of each foreground area, so that the depth distribution of the two foreground points in each foreground point pair tends to be consistent. The prior rule for constructing the entropy reduction constraint of the foreground area depth distribution is: the same target foreground area has consistent semantic information and is not easily disturbed by the background area. Therefore, the entropy reduction constraint of the foreground area depth distribution can be used to constrain the entropy of the depth distribution at each foreground point in each foreground area, so that the entropy of the depth distribution at each foreground point tends to be smaller. The prior rule for constructing the entropy increase constraint of the confused area depth distribution is: in the confused area of ​​the target foreground and background, the foreground and background semantic information are confused, and the foreground and background generally have different depths, so the depth distribution should tend to be multi-peaked. Therefore, the entropy increase constraint of the confused area depth distribution can be used to constrain the entropy of the depth distribution at the feature point of each foreground background confused area, so that the entropy of the depth distribution at the feature point of each foreground background confused area tends to be larger. Therefore, the depth prediction model constructed based on the above-mentioned depth distribution loss functions can effectively ensure the accuracy of the prediction results and help achieve more accurate target detection.

[0117] S103: Convert the image features into bird's-eye view features using the depth distribution.

[0118] This step aims to achieve the view conversion of image features, which is essentially a 2D-3D view conversion of image features, that is, converting the image features under the two-dimensional view to the three-dimensional view (bird's eye view), thereby converting the image features into image bird's eye view features. Specifically, after obtaining the depth distribution of the image features, the high depth distribution can be used to achieve the above view conversion operation.

[0119] S104: Perform target detection on the bird's-eye view features of the image to obtain a target detection result.

[0120] This step aims to achieve target detection of the image bird's-eye view features and obtain the final target detection result. Specifically, after the image features are converted into image bird's-eye view features, 3D target detection can be performed on them, and the detection result is the target detection result about the target image, which is used to indicate whether a target object exists in the target image.

[0121] It can be seen that the target detection method provided by the embodiment of the present invention, after extracting the image feature from the target image, first uses the pre-created depth prediction model to perform depth estimation on it to obtain the depth distribution of the image feature, then performs 2D-3D view conversion on the image feature based on the depth distribution to obtain the corresponding image bird's-eye view feature, and finally performs target detection on the image bird's-eye view feature to achieve target detection for the target image, and obtains the target detection result. Among them, the preset depth prediction model is obtained by supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of the foreground depth distribution consistency loss function, the foreground area depth distribution entropy reduction constraint, and the confusion area depth distribution entropy increase constraint. The foreground depth distribution consistency loss function can be used to constrain different foreground points in the same target foreground area to have similar depth distributions, the foreground area depth distribution entropy reduction constraint can be used to constrain the same target foreground area to have consistent semantic information and not be easily disturbed by the background area, and the confusion area depth distribution entropy increase constraint can be used to guide the depth prediction model to make a more reasonable depth distribution prediction for the target foreground background confusion area. Therefore, based on the preset depth prediction model, a more accurate depth distribution estimation of the image feature can be performed, which helps to achieve more accurate target detection.

[0122] Based on the above embodiments:

[0123] In one embodiment of the present invention, the process of constructing a preset depth prediction model based on the foreground depth distribution consistency loss function includes:

[0124] Obtain a target image sample and determine a bounding box of a target object in the target image sample;

[0125] The inner area of ​​the bounding box is regarded as the foreground area;

[0126] Perform random sampling in each foreground area to obtain foreground point pairs;

[0127] Predicting the depth distribution of the foreground point in each foreground point pair to obtain a depth prediction value pair of the foreground point pair;

[0128] Combined with the foreground depth distribution consistency loss function, the preset depth distribution dimension and each depth prediction value are used to perform model supervision training to obtain the preset depth prediction model.

[0129] The embodiment of the present invention provides an implementation method for constructing a preset depth prediction model based on a foreground depth distribution consistency loss function. In the implementation process, for a target image sample, first determine the bounding box of the target object, which can be manually calibrated, and use the inner area of ​​the bounding box as the foreground area, and correspondingly, the outer area of ​​the bounding box is the background area; further, for each target image sample, F1 foreground areas can be randomly sampled first, and for each foreground area, 2Q foreground points are randomly sampled to form Q foreground point pairs, D is used to represent the dimension of the depth distribution, and the foreground depth distribution consistency loss function L is depth_consistent is defined as follows:

[0130]

[0131] in, Represents the k-th dimension depth prediction probability value of the depth distribution corresponding to the first foreground point in the j-th foreground point pair of the i-th foreground area.

[0132] Therefore, the construction of a preset depth prediction model can be achieved based on the foreground depth distribution consistency loss function.

[0133] In one embodiment of the present invention, the process of constructing a preset depth prediction model based on the entropy reduction constraint of the depth distribution of the foreground area may include:

[0134] Obtain a target image sample and determine a bounding box of a target object in the target image sample;

[0135] The inner area of ​​the bounding box is regarded as the foreground area;

[0136] Perform random sampling in each foreground area to obtain foreground points;

[0137] Predict the depth distribution of each foreground point and obtain the depth prediction value of the foreground point;

[0138] Combined with the entropy reduction constraint of the depth distribution in the foreground area, the preset depth distribution dimension and the depth prediction value of each foreground point are used to perform model supervision training to obtain the preset depth prediction model.

[0139] The embodiment of the present invention provides an implementation method for constructing a preset depth prediction model based on the entropy reduction constraint of the depth distribution of the foreground area. In the implementation process, for the target image sample, the bounding box of the target object is first determined, which can be manually calibrated, and the inner area of ​​the bounding box is used as the foreground area, and correspondingly, the outer area of ​​the bounding box is the background area; further, for each target image sample, F2 foreground areas can be randomly sampled, and for each foreground area, P foreground points are randomly sampled, then the entropy reduction constraint L of the depth distribution of the foreground area is depth_entropy_decrease is defined as follows:

[0140]

[0141] Among them, d ijk Represents the k-th dimension depth prediction probability value of the depth distribution corresponding to the j-th foreground point of the i-th foreground area.

[0142] Therefore, the construction of a preset depth prediction model can be achieved based on the entropy reduction constraint of the depth distribution of the foreground area.

[0143] In one embodiment of the present invention, the process of constructing a preset depth prediction model based on the entropy increase constraint of the depth distribution of the confusion area may include:

[0144] Obtain a target image sample and determine a bounding box of a target object in the target image sample;

[0145] Inside the bounding box, a pixel area with a preset number of pixels away from the bounding box is used as a confusion area;

[0146] Perform random sampling in each confusion area to obtain confusion points;

[0147] Predict the depth distribution of each confusion point to obtain the depth prediction value of the confusion point;

[0148] Combined with the entropy increase constraint of the depth distribution in the confused area, the preset depth distribution dimension and the depth prediction value of each confused point are used to perform model supervision training to obtain the preset depth prediction model.

[0149] The embodiment of the present invention provides an implementation method for constructing a preset depth prediction model based on the entropy increase constraint of the depth distribution of the confusion area. In the implementation process, for the target image sample, first determine the bounding box of the target object, which can be manually calibrated, and the inner area of ​​the bounding box is used as the foreground area, and correspondingly, the outer area of ​​the bounding box is the background area; then, in the foreground area, the pixel area with a distance of σ (preset number) pixels from the bounding box is defined as the foreground background confusion area; finally, for each target image sample, F3 foreground background confusion areas can be randomly sampled, and for each foreground background confusion area, T foreground points are randomly sampled, then the entropy increase constraint L of the depth distribution of the confusion area depth_entropy_increase is defined as follows:

[0150]

[0151] Among them, d ijk Represents the k-th dimension depth prediction probability value of the depth distribution corresponding to the j-th foreground point in the i-th foreground-background confusion area.

[0152] Therefore, the construction of a preset depth prediction model can be achieved based on the entropy increase constraint of the depth distribution of the confused area.

[0153] In one embodiment of the present invention, the above-mentioned conversion of image features into image bird's-eye view features by using depth distribution may include:

[0154] Expand image features into point cloud features;

[0155] The point cloud features are weighted using the depth distribution to obtain weighted point cloud features;

[0156] Convert the weighted point cloud features from the camera coordinate system to the target detection coordinate system;

[0157] The weighted point cloud features in the target detection coordinate system are subjected to voxel pooling to obtain the image bird's-eye view features.

[0158] The embodiment of the present invention provides a method for converting image features into image bird's-eye view features. First, the image features are expanded into point cloud features. This process can be implemented according to the dimension of the preset depth distribution. By equally dividing the preset depth distribution dimension, a corresponding number of equally spaced discrete depth points are obtained, and then the 2D image features are expanded along the depth direction to obtain 3D point cloud features; further, the point cloud features are weighted using the predicted depth distribution to obtain depth-weighted point cloud features, that is, the above-mentioned weighted point cloud features; finally, since the target image is acquired by a camera device, it is obvious that the weighted point cloud features obtained based on the target image are feature information in the camera coordinate system. Therefore, the weighted point cloud features can be converted from the camera coordinate system to the target detection coordinate system, and then the weighted point cloud features are voxel pooled in the target detection coordinate system to obtain the image bird's-eye view features. Among them, the target detection coordinate system is the coordinate system of the current execution subject of the target detection. For example, to realize target detection in autonomous driving technology, the target detection coordinate system can be a vehicle coordinate system.

[0159] It is understandable that since the 3D image feature point cloud is very dense, the storage and calculation burdens of voxel pooling are very high. Therefore, by performing voxel pooling on the weighted point cloud features, a relatively sparse 3D image feature point cloud can be obtained. That is, the bird's-eye view features of the image obtained by voxel pooling are sparser than the weighted point cloud features, which can effectively reduce the storage and calculation burdens of voxel pooling, thereby helping to improve the efficiency of target detection.

[0160] In one embodiment of the present invention, the voxel pooling process is performed on the weighted point cloud features in the target detection coordinate system to obtain the image bird's-eye view features, which may include:

[0161] Divide the coordinate system space of the target detection coordinate system into cubic voxels of a preset size;

[0162] For each cubic voxel, a depth clipping process is performed on the weighted point cloud features in the cubic voxel according to the depth clipping threshold corresponding to the cubic voxel to obtain a clipping feature;

[0163] The cropped features corresponding to all cube voxels are combined into the image bird's-eye view features.

[0164] The embodiment of the present invention provides a method for performing voxel pooling processing on weighted point cloud features to obtain image bird's-eye view features. Specifically, in the target detection coordinate system, the coordinate system space of the target detection coordinate system can be spatially divided to obtain a plurality of cubic voxels with the same preset size; further, each cubic voxel is preset with its own corresponding depth clipping threshold (after spatial division, the depth of each cubic voxel is different. To ensure the accuracy of depth clipping, different depth clipping thresholds can be set for each cubic voxel). Therefore, for each cubic voxel, all weighted point cloud features therein can be depth clipped according to their corresponding depth clipping threshold to obtain corresponding clipping features. Thus, the clipping features corresponding to all cubic voxels can be combined into the image bird's-eye view features corresponding to the entire weighted point cloud features.

[0165] In one embodiment of the present invention, performing depth clipping processing on weighted point cloud features in cubic voxels according to the depth clipping threshold corresponding to the cubic voxels to obtain clipping features may include:

[0166] Determine the depth value of the cube voxel in the coordinate system space;

[0167] Performing depth probability prediction on the depth value to obtain predicted depth probability;

[0168] When the predicted depth probability does not exceed the depth cut-off threshold, all weighted point cloud features within the cube voxel are removed;

[0169] When the predicted depth probability exceeds the depth clipping threshold, the weighted point cloud features within the cube voxel are retained as clipping features.

[0170] The embodiment of the present invention provides a method for implementing depth clipping of weighted point cloud features. As described above, after space division, the depth of each cubic voxel is different. To ensure the accuracy of depth clipping, different depth clipping thresholds can be set for each cubic voxel. Based on this, the specific value of the depth clipping threshold can be determined by the depth of the corresponding cubic voxel in the coordinate system space. Therefore, for each cubic voxel, the depth value of the cubic voxel in the coordinate system space can be determined first, and each depth value corresponds to a predicted depth probability. Then, when the predicted depth probability corresponding to the current cubic voxel does not exceed the corresponding depth clipping threshold, all weighted point cloud features in the cubic voxel can be removed (that is, all weighted point cloud features in the current cubic voxel are clipped). When the predicted depth probability corresponding to the current cubic voxel exceeds the corresponding depth clipping threshold, the weighted point cloud features in the cubic voxel can be retained (that is, all weighted point cloud features in the current cubic voxel are retained) and used as the clipping feature of the current cubic voxel.

[0171] In one embodiment of the present invention, the process of setting the depth clipping threshold may include:

[0172] Get a set of candidate thresholds;

[0173] For each candidate threshold in the candidate threshold set, calculate the proportion of clipped features and the model evaluation index corresponding to the candidate threshold;

[0174] In the candidate threshold set, the candidate thresholds whose model evaluation index is lower than the preset index threshold are deleted to obtain the retained candidate threshold set;

[0175] In the set of retained candidate thresholds, the candidate threshold corresponding to the largest proportion of clipped features is used as the depth clipping threshold.

[0176] The embodiment of the present invention provides a method for setting a depth clipping threshold. Specifically, a group of candidate thresholds can be preset to generate a candidate threshold set. For the candidate threshold set, all candidate thresholds therein can be traversed to calculate the proportion of clipped features corresponding to each candidate threshold (when the candidate threshold is selected as the depth clipping threshold, the proportion of the weighted point cloud features clipped to all weighted point cloud features) and the model evaluation index (when the candidate threshold is selected as the depth clipping threshold, the evaluation index value of the processing model trained based on the depth clipping threshold); further, in the candidate threshold set, the candidate threshold whose model evaluation index value is lower than the preset index threshold is deleted to obtain a set of retained candidate thresholds, and then in the set of retained candidate thresholds, the candidate threshold corresponding to the maximum proportion of clipped features is used as the depth clipping threshold. In this way, the determination of the depth clipping threshold is achieved.

[0177] It can be understood that the determination of the depth clipping threshold through the above-mentioned screening rules helps to screen and obtain the threshold conditions that are most suitable for the current training model, thereby obtaining a higher-precision training model, thereby achieving more accurate target detection.

[0178] In one embodiment of the present invention, the calculation process of the above model evaluation index may include:

[0179] For each candidate threshold, obtain the average accuracy value, average translation error, average scale error, average angle error, average velocity error, and average attribute error of the training model corresponding to the candidate threshold;

[0180] The model evaluation index is obtained by calculating the average accuracy value, average translation error, average scale error, average angle error, average velocity error, and average attribute error.

[0181] The embodiment of the present invention provides a method for calculating model evaluation indicators. In this embodiment, NDS (NuScenes Detection Score) can be selected as the model evaluation indicator, which is composed of mAP (meanAverage Precision) and a set of five TP metrics (True Positives, a deep learning evaluation indicator): ATE (Average Translation Error), ASE (AverageScale Error), AOE (Average Orientation Error), AVE (Average Velocity Error) and AAE (AverageAttribute Error), and the calculation formula is as follows:

[0182]

[0183]

[0184]

[0185] in, Indicates the target category.

[0186] In one embodiment of the present invention, performing target detection on the bird's-eye view features of the image to obtain the target detection result may include:

[0187] Use LiDAR to obtain point cloud data;

[0188] Extract features from point cloud data to obtain point cloud bird's-eye view features;

[0189] Fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fused features;

[0190] Perform target detection on the fused features to obtain target detection results.

[0191] The embodiment of the present invention provides a target detection method based on feature information to achieve the acquisition of target detection results. In this embodiment, in addition to the image bird's-eye view features obtained based on the camera device, point cloud data can also be obtained based on the laser radar and the point cloud bird's-eye view features can be extracted. Thus, the image bird's-eye view features and the point cloud bird's-eye view features can be fused to obtain the fusion features of the two. Finally, the final target detection result can be obtained by performing target detection on the fusion features. It can be seen that this embodiment realizes multi-modal feature detection, which can further improve the accuracy of the target detection results.

[0192] In one embodiment of the present invention, the above-mentioned fusing of the image bird's-eye view features and the point cloud bird's-eye view features to obtain the fused features may include: using a preset feature fusion network model to fuse the image bird's-eye view features and the point cloud bird's-eye view features to obtain the fused features; wherein the preset feature fusion network model is constructed based on a multimodal semantic alignment loss function.

[0193] The embodiment of the present invention provides a feature fusion method, that is, a feature fusion technology based on a preset feature fusion network model. Specifically, a feature fusion network model can be pre-constructed, and after obtaining the image bird's-eye view features and the point cloud bird's-eye view features respectively, the two can be input into the preset feature fusion network model for processing, and the output of the model is the fusion feature.

[0194] Among them, the preset feature fusion network model is constructed based on the multimodal semantic alignment loss function. It is understandable that the target image and point cloud data belong to two completely different modalities, and there must be differences in the feature distribution between the two. For example, the 3D point cloud features of the two modalities of the same target object do not have similar distributions in the same feature space. This difference needs to be explicitly filled in order to improve the effect of subsequent multimodal feature fusion. Therefore, this embodiment performs model training through a multimodal semantic alignment loss function. Based on the multimodal semantic alignment loss function, the divergence between the two types of features can be effectively constrained, making the divergence between the two types of features tend to be smaller, thereby effectively improving the multimodal feature fusion effect and further improving the accuracy of the target detection results.

[0195] In one embodiment of the present invention, the process of constructing a preset feature fusion network model based on the multimodal semantic alignment loss function may include:

[0196] Get a bird's eye view;

[0197] Obtaining an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view;

[0198] Converting the image bird's-eye view feature set into an image bird's-eye view feature semantic encoding set;

[0199] Converting a point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic encoding set;

[0200] Combined with the multimodal semantic alignment loss function, the image bird's-eye view feature semantic coding set and the point cloud bird's-eye view feature semantic coding set are used for model supervision training to obtain the preset feature fusion network model.

[0201] The embodiment of the present invention provides an implementation method for constructing a preset feature fusion network model based on a multimodal semantic alignment loss function. First, obtain a bird's-eye view of the target image (a 3D view corresponding to a 2D view), and determine each sampling foreground position in the bird's-eye view. The sampling foreground position is pre-set by a technician according to actual conditions, and can be all foreground features in the corresponding foreground area, or can be part of the foreground features selected in the corresponding foreground area; further, obtain an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position, and convert the two into corresponding image bird's-eye view feature semantic coding sets and point cloud bird's-eye view feature semantic coding sets respectively through coding technology, thereby, the image bird's-eye view feature semantic coding set and the point cloud bird's-eye view feature semantic coding set can be used, and combined with the multimodal semantic alignment loss function to construct a multimodal semantic alignment loss function.

[0202] In one embodiment of the present invention, the step of obtaining the image bird's-eye view feature set and the point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view may include:

[0203] Determine a bounding box about a target object in a bird's-eye view;

[0204] The inner area of ​​the bounding box is regarded as the foreground area;

[0205] Perform random sampling in each foreground area to obtain each preset sampling foreground position;

[0206] In the bird's-eye view, an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position are obtained.

[0207] As described above, the preset sampling foreground position can be all foreground features in the corresponding foreground area, or can be some foreground features selected in the corresponding foreground area. In this embodiment, some foreground features in the foreground area are set as the preset sampling foreground position to effectively reduce the amount of data calculation and improve the efficiency of model training. Specifically, the bounding box of the target object in the bird's-eye view can be determined first, and the area in the bounding box can be used as the foreground area. Obviously, the area outside the bounding box is the background area; further, for each foreground area, random sampling can be performed therein to obtain each of the above-mentioned preset sampling foreground positions, and then the image bird's-eye view feature set and the point cloud bird's-eye view feature set are obtained.

[0208] It is understandable that when all foreground features in the corresponding foreground area are set as the preset sampling foreground position, the amount of data calculation is relatively large, but the model training accuracy can be effectively guaranteed. Therefore, the selection of the preset sampling foreground position can be customized by the technician according to the actual training needs, and the present invention does not limit this.

[0209] In one embodiment of the present invention, the above-mentioned converting the image bird's-eye view feature set into the image bird's-eye view feature semantic coding set may include: converting the image bird's-eye view feature set into the image bird's-eye view feature semantic coding set using a preset image semantic coding model;

[0210] Accordingly, the above-mentioned converting the point cloud bird's-eye view feature set into the point cloud bird's-eye view feature semantic coding set may include: converting the point cloud bird's-eye view feature set into the point cloud bird's-eye view feature semantic coding set using a preset point cloud semantic coding model.

[0211] The embodiment of the present invention provides a feature coding technology, that is, a feature coding technology based on a preset semantic coding model. Specifically, an image semantic coding model and a point cloud semantic coding model can be created in advance, and used to perform semantic coding processing on the image bird's-eye view feature set and the point cloud bird's-eye view feature set, respectively, to obtain the image bird's-eye view feature semantic coding set and the point cloud bird's-eye view feature semantic coding set. Among them, both types of semantic coding models can be constructed using a neural network layer (graph residual neural network).

[0212] In one embodiment of the present invention, the above-mentioned feature extraction of point cloud data to obtain point cloud bird's-eye view features may include: using a preset point cloud backbone network model to extract features of the point cloud data to obtain point cloud bird's-eye view features.

[0213] This embodiment provides a method for extracting features from point cloud data to obtain point cloud bird's-eye view features, which can be implemented based on a pre-created point cloud backbone network model. Specifically, after the point cloud data is collected, it can be input into a preset point cloud backbone network model for processing, and the output of the model is the point cloud bird's-eye view features of the point cloud data.

[0214] In one embodiment of the present invention, the above-mentioned fusing of the image bird's-eye view features and the point cloud bird's-eye view features to obtain the fused features may include:

[0215] The image bird's-eye view features and the point cloud bird's-eye view features are spliced ​​along the feature channel direction to obtain splicing features;

[0216] Use the preset feature fusion network model to perform feature fusion processing on the splicing features to obtain the initial fusion features;

[0217] The initial fusion features are subjected to dimensionality reduction processing to obtain fusion features.

[0218] The embodiment of the present invention provides a feature fusion method to achieve the fusion of image bird's-eye view features and point cloud bird's-eye view features. First, for the image bird's-eye view features and point cloud bird's-eye view features, the two can be spliced ​​according to the feature channel direction to obtain the spliced ​​features of the two, and then the spliced ​​features are input into a preset feature fusion network model for feature fusion processing. The output of the preset feature fusion network model is the above-mentioned initial fusion feature, and finally the final fusion feature is obtained through dimensionality reduction processing. Among them, the preset feature fusion network model is a pre-created network model for realizing feature fusion, which is pre-stored in the corresponding storage space and can be directly called when used.

[0219] In one embodiment of the present invention, the above-mentioned performing target detection on the fused features to obtain the target detection result may include: performing target detection on the fused features using a preset fused feature detection model to obtain the target detection result.

[0220] This embodiment provides a method for implementing target detection on fused features to obtain target detection results, which can be implemented based on a pre-created fused feature detection model. Specifically, after obtaining the fused features, they can be input into a preset fused feature detection model for processing, and the output of the model is the final target detection result.

[0221] Based on the above embodiments, an embodiment of the present invention provides another target detection method.

[0222] First, please refer to Figure 2 , Figure 2This is a framework diagram of a target detection system provided by the present invention. The target detection method based on the target detection system takes target detection in driving technology as an example, and its implementation process is as follows:

[0223] 1. Use the image backbone network to extract the image features of the image data; use the point cloud backbone network to extract the point cloud bird's-eye view features of the lidar point cloud data;

[0224] 2. Use the depth prediction model to predict the depth distribution of image features. The depth prediction model is trained using the foreground depth distribution consistency loss function, the foreground area depth distribution entropy reduction constraint, and the confusion area depth distribution entropy increase constraint.

[0225] 3. Based on the depth distribution predicted in step 2, the image features are processed using the LSS technique and the voxel pooling method of depth cutting to obtain the image bird's-eye view features;

[0226] 4. Generate semantic encoding of foreground features of the image bird's-eye view, generate semantic encoding of foreground features of the point cloud bird's-eye view, and use a multimodal semantic alignment loss function to supervise the training of semantic alignment of image bird's-eye view features and point cloud bird's-eye view features;

[0227] 5. Use the feature fusion model to fuse the image bird's-eye view features and the point cloud bird's-eye view features to obtain multimodal fused bird's-eye view features, i.e., fused features;

[0228] 6. Use the 3D object detection task head to perform 3D object detection. You can use the classification loss function and the regression loss function to supervise the training of the 3D object detection model.

[0229] 7. Deploy the trained 3D object detection model into the autonomous driving system.

[0230] For step 2 above, please refer to Figure 3 , Figure 3 This is a flow chart of an image feature view conversion method provided by the present invention, and its implementation process is as follows:

[0231] 2.1. Assume that the 2D image features obtained by the image backbone network are Where H and W represent the length and width of the feature, respectively, C represents the number of channels, and D discrete depth points with equal intervals are predefined, such as 1 meter, 2 meters, ..., D meters. The 2D image features are expanded along the depth direction. The 2D image features can be expanded into 3D point cloud features of size H×W×D×C. Each 3D point contains a C-dimensional image feature vector.

[0232] 2.2. Using 2D image features, predict the depth distribution at each 2D feature point and obtain the depth distribution The depth distribution is used to weight the 3D point cloud features at the corresponding position to obtain the depth-weighted 3D point cloud features.

[0233] 2.3. Use the camera's intrinsic and extrinsic parameters to transform the weighted 3D point cloud features from the camera coordinate system to the vehicle coordinate system.

[0234] 2.4. In the vehicle coordinate system, the 3D space of the coordinate system is divided into a series of cubic voxels of the same size, and all image features in each voxel are pooled to obtain the image bird's-eye view features.

[0235] For step 2.4 above, please refer to Figure 4 , Figure 4 This is a schematic diagram of a depth cutting method provided by the present invention, and its implementation process is as follows:

[0236] Specifically, Figure 4 The depth clipping operation at a single point on the image feature plane is demonstrated. At this point, there are several depth value points with fixed intervals along the depth direction. Each depth value corresponds to a predicted depth probability. A depth clipping threshold is set. For depth value points with a depth probability higher than the depth clipping threshold, the image feature is extended to the spatial point; and for depth value points with a depth probability lower than the depth clipping threshold, the image feature is not extended to the spatial point, that is, the spatial point is empty. After the above-mentioned depth clipping process, a 3D point cloud feature that is sparser than the original 3D point cloud feature can be obtained, thereby reducing the storage and computational burden of voxel pooling. Among them, the depth clipping threshold can be manually pre-set, or obtained through a hyperparameter search method (such as grid search), or learned by the model through autonomous learning. For example, a method for determining a depth clipping threshold is as follows:

[0237] Preset a set of candidate thresholds Traverse all candidate thresholds and calculate each candidate threshold τ i The corresponding proportion of cut features And model evaluation indicators Construct a set of evaluation results for all candidate thresholds and press The rule filters the evaluation results to obtain the filtered evaluation results Δ′, where ε represents the tolerance for the reduction of the model evaluation index. In the embodiment of the present invention, ε=0.01. is the initial model evaluation index value before model training. Further, select the proportion of the cut features in Δ′ The candidate threshold τ corresponding to the largest combination i , as the final depth clipping threshold.

[0238] Through verification, we take the image features of a single camera as an example to estimate the amount of computation and storage reduced by the depth clipping threshold. Assume that the image feature size is H×W×C, where H is the height, W is the width, and C is the number of feature channels; assume that the image feature depth distribution size is H×W×D, where D is the number of depth values, and assume that the bird's-eye view grid size is w×d. Then, the computational cost of generating 3D point cloud features is H×W×D×C, the amount of data to be stored is H×W×D×C, and the computational cost of voxel pooling is On this basis, by setting the depth clipping threshold, the amount of depth retained can be controlled. Taking the retention of 10% high confidence depth value as an example, the amount of calculation to generate 3D point cloud features is H×W×D×C×0.1, the amount of data to be stored is H×W×D×C×0.1, and the amount of calculation for voxel pooling is Therefore, when 10% of high-confidence depth values ​​are retained, after depth cropping, the computational complexity of generating 3D point cloud features is reduced by 90%, the storage complexity is reduced by 90%, and the computational complexity of voxel pooling is reduced by 90%.

[0239] For step 4 above, please refer to Figure 5 , Figure 5 This is a flow chart of a multimodal bird's-eye view feature semantic alignment method provided by the present invention, and its implementation process is as follows:

[0240] 4.1. Define the area within the target bounding box in the bird's-eye view as the foreground area, use the annotation of the target bounding box to obtain the foreground area under the bird's-eye view, and randomly sample Foreground positions, and obtain the corresponding image bird's-eye view feature set at these foreground positions and point cloud bird's eye view feature collection in, Both C b The feature vector of each channel is then processed by the image bird's-eye view feature semantic coding module to obtain the image bird's-eye view feature semantic coding set. Use the point cloud bird's eye view feature semantic encoding module to process the point cloud bird's eye view feature set to obtain the point cloud bird's eye view feature semantic encoding set in, are all semantic encoding vectors with g dimensions.

[0241] 4.2. Using multimodal semantic alignment loss function L alignThe divergence between the semantic encoding of the image bird's-eye view feature at each sampling position and the semantic encoding of the point cloud bird's-eye view at the corresponding position is calculated. The multimodal semantic alignment loss function is used to constrain the divergence between the semantic encoding of the image bird's-eye view feature at each sampling position and the semantic encoding of the point cloud bird's-eye view at the corresponding position, so that the divergence between the semantic encoding of the image bird's-eye view feature at each sampling position and the semantic encoding of the point cloud bird's-eye view at the corresponding position tends to be smaller. Among them, the formula of the multimodal semantic alignment loss function is:

[0242]

[0243] in, represents the dimension of feature semantic encoding, is the j-th dimension encoding value of the semantic encoding of the bird's-eye view feature of the i-th image, The j-th dimension encoding value of the semantic encoding of the i-th point cloud bird's-eye view feature.

[0244] On this basis, when the image bird's-eye view features and the point cloud bird's-eye view features are fused based on the feature fusion model, the image bird's-eye view features and the point cloud bird's-eye view features can be first spliced ​​along the channel direction to obtain the spliced ​​features, and then the neural network layer is used to fuse and reduce the dimension to obtain the final fused features.

[0245] It can be seen that the target detection method provided by the embodiment of the present invention, after extracting the image feature from the target image, first uses the pre-created depth prediction model to perform depth estimation on it to obtain the depth distribution of the image feature, then performs 2D-3D view conversion on the image feature based on the depth distribution to obtain the corresponding image bird's-eye view feature, and finally performs target detection on the image bird's-eye view feature to achieve target detection for the target image, and obtains the target detection result. Among them, the preset depth prediction model is obtained by supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of the foreground depth distribution consistency loss function, the foreground area depth distribution entropy reduction constraint, and the confusion area depth distribution entropy increase constraint. The foreground depth distribution consistency loss function can be used to constrain different foreground points in the same target foreground area to have similar depth distributions, the foreground area depth distribution entropy reduction constraint can be used to constrain the same target foreground area to have consistent semantic information and not be easily disturbed by the background area, and the confusion area depth distribution entropy increase constraint can be used to guide the depth prediction model to make a more reasonable depth distribution prediction for the target foreground background confusion area. Therefore, based on the preset depth prediction model, a more accurate depth distribution estimation of the image feature can be performed, which helps to achieve more accurate target detection.

[0246] An embodiment of the present invention provides a target detection device.

[0247] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of a target detection device provided by the present invention, and the target detection device may include:

[0248] Extraction module 1, used to extract features from the target image to obtain image features;

[0249] Processing module 2, used to process image features using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint;

[0250] A conversion module 3, used for converting the image features into image bird's-eye view features using the depth distribution;

[0251] The detection module 4 is used to perform target detection on the bird's-eye view features of the image to obtain target detection results.

[0252] It can be seen that the target detection device provided by the embodiment of the present invention, after extracting the image feature from the target image, first uses the pre-created depth prediction model to perform depth estimation on the image feature to obtain the depth distribution of the image feature, then performs 2D-3D view conversion on the image feature based on the depth distribution to obtain the corresponding image bird's-eye view feature, and finally performs target detection on the image bird's-eye view feature to achieve target detection for the target image, and obtains the target detection result. Among them, the preset depth prediction model is obtained by supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of the foreground depth distribution consistency loss function, the foreground area depth distribution entropy reduction constraint, and the confusion area depth distribution entropy increase constraint. The foreground depth distribution consistency loss function can be used to constrain different foreground points in the same target foreground area to have similar depth distributions, the foreground area depth distribution entropy reduction constraint can be used to constrain the same target foreground area to have consistent semantic information and not be easily disturbed by the background area, and the confusion area depth distribution entropy increase constraint can be used to guide the depth prediction model to make a more reasonable depth distribution prediction for the target foreground background confusion area. Therefore, based on the preset depth prediction model, a more accurate depth distribution estimation of the image feature can be performed, which helps to achieve more accurate target detection.

[0253] In one embodiment of the present invention, the target detection device may further include:

[0254] The first model building module is used to obtain a target image sample, determine a bounding box about a target object in the target image sample, and use an inner area of ​​the bounding box as a foreground area;

[0255] Random sampling is performed in each foreground area to obtain foreground point pairs; depth distribution prediction is performed on the foreground point in each foreground point pair to obtain depth prediction value pairs of the foreground point pairs; combined with the foreground depth distribution consistency loss function, model supervision training is performed using the preset depth distribution dimension and each depth prediction value pair to obtain a preset depth prediction model.

[0256] In one embodiment of the present invention, the target detection device may further include:

[0257] The second model building module is used to obtain target image samples and determine the bounding box of the target object in the target image samples; use the internal area of ​​the bounding box as the foreground area; perform random sampling in each foreground area to obtain foreground points; perform depth distribution prediction on each foreground point to obtain a depth prediction value of the foreground point; combine the entropy reduction constraint of the depth distribution of the foreground area, use the preset depth distribution dimension and the depth prediction value of each foreground point to perform model supervision training, and obtain a preset depth prediction model.

[0258] In one embodiment of the present invention, the target detection device may further include:

[0259] The third model building module is used to obtain target image samples and determine the bounding box of the target object in the target image samples; inside the bounding box, the pixel area with a preset number of pixels away from the bounding box is used as the confusion area; random sampling is performed in each confusion area to obtain confusion points; depth distribution prediction is performed on each confusion point to obtain a depth prediction value of the confusion point; combined with the entropy increase constraint on the depth distribution of the confusion area, the preset depth distribution dimension and the depth prediction value of each confusion point are used to perform model supervision training to obtain a preset depth prediction model.

[0260] In one embodiment of the present invention, the conversion module 3 may include:

[0261] An expansion unit, used to expand image features into point cloud features;

[0262] A weighting unit, used for weighting point cloud features using depth distribution to obtain weighted point cloud features;

[0263] A conversion unit, used to convert the weighted point cloud features from the camera coordinate system to the target detection coordinate system;

[0264] The processing unit is used to perform voxel pooling processing on the weighted point cloud features in the target detection coordinate system to obtain the image bird's-eye view features.

[0265] In one embodiment of the present invention, the processing unit may include:

[0266] A division subunit, used to divide the coordinate system space of the target detection coordinate system into cubic voxels of a preset size;

[0267] A cutting subunit is used for performing a depth cutting process on the weighted point cloud features in each cubic voxel according to a depth cutting threshold corresponding to the cubic voxel to obtain a cutting feature;

[0268] The combining subunit is used to combine the cropping features corresponding to all cube voxels into an image bird's-eye view feature.

[0269] In one embodiment of the present invention, the above-mentioned cropping subunit can be specifically used to determine the depth value of the cubic voxel in the coordinate system space; perform depth probability prediction on the depth value to obtain a predicted depth probability; when the predicted depth probability does not exceed the depth cropping threshold, all weighted point cloud features in the cubic voxel are eliminated; when the predicted depth probability exceeds the depth cropping threshold, the weighted point cloud features in the cubic voxel are retained as cropping features.

[0270] In one embodiment of the present invention, the above-mentioned cropping sub-unit can also be used to obtain a set of candidate threshold values; for each candidate threshold value in the candidate threshold value set, the cropped feature ratio and the model evaluation index corresponding to the candidate threshold value are calculated; in the candidate threshold value set, the candidate threshold values ​​whose model evaluation index is lower than the preset index threshold value are deleted to obtain a set of retained candidate threshold values; in the retained candidate threshold value set, the candidate threshold value corresponding to the cropped feature ratio with the largest value is used as the depth cropping threshold.

[0271] In one embodiment of the present invention, the above-mentioned cropping subunit can be specifically used to obtain, for each candidate threshold, the average accuracy value, average translation error, average scale error, average angle error, average speed error, and average attribute error of the training model corresponding to the candidate threshold; and obtain the model evaluation index based on the average accuracy value, average translation error, average scale error, average angle error, average speed error, and average attribute error.

[0272] In one embodiment of the present invention, the extraction module 1 can be specifically used to extract features of a target image using a preset image backbone network model to obtain image features.

[0273] In one embodiment of the present invention, the detection module 4 may include:

[0274] An acquisition unit, used for acquiring point cloud data using a laser radar;

[0275] An extraction unit, used for extracting features from point cloud data to obtain features of a bird's-eye view of the point cloud;

[0276] A fusion unit, used for fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fusion features;

[0277] The detection unit is used to perform target detection on the fused features to obtain target detection results.

[0278] In one embodiment of the present invention, the fusion unit may include:

[0279] The fusion subunit is used to fuse the image bird's-eye view features and the point cloud bird's-eye view features using a preset feature fusion network model to obtain fusion features; wherein the preset feature fusion network model is constructed based on a multimodal semantic alignment loss function.

[0280] In one embodiment of the present invention, the above-mentioned fusion unit may further include:

[0281] Construct a subunit for obtaining a bird's-eye view; obtain an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view view; convert the image bird's-eye view feature set into an image bird's-eye view feature semantic coding set; convert the point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic coding set; combine the multimodal semantic alignment loss function, use the image bird's-eye view feature semantic coding set and the point cloud bird's-eye view feature semantic coding set to perform model supervision training, and obtain a preset feature fusion network model.

[0282] In one embodiment of the present invention, the above-mentioned construction subunit can be specifically used to determine the bounding box of the target object in the bird's-eye view; use the internal area of ​​the bounding box as the foreground area; perform random sampling in each foreground area to obtain each preset sampling foreground position; and obtain the image bird's-eye view feature set and the point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view.

[0283] In one embodiment of the present invention, the above-mentioned construction subunit can be specifically used to convert an image bird's-eye view feature set into an image bird's-eye view feature semantic coding set using a preset image semantic coding model; and to convert a point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic coding set using a preset point cloud semantic coding model.

[0284] In one embodiment of the present invention, the extraction unit may be specifically used to perform feature extraction on point cloud data using a preset point cloud backbone network model to obtain point cloud bird's-eye view features.

[0285] In one embodiment of the present invention, the above-mentioned fusion unit can be specifically used to splice the image bird's-eye view features and the point cloud bird's-eye view features along the feature channel direction to obtain the spliced ​​features; use the preset feature fusion network model to perform feature fusion processing on the spliced ​​features to obtain the initial fusion features; perform dimensionality reduction processing on the initial fusion features to obtain the fusion features.

[0286] In one embodiment of the present invention, the above-mentioned detection unit can be specifically used to perform target detection on the fused features using a preset fused feature detection model to obtain a target detection result.

[0287] For an introduction to the device provided in the embodiment of the present invention, please refer to the above method embodiment, and the present invention will not be elaborated here.

[0288] An embodiment of the present invention provides an electronic device.

[0289] Please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided by the present invention, and the electronic device may include:

[0290] Memory for storing computer programs;

[0291] A processor, when used to execute a computer program, can implement the steps of any one of the target detection methods described above.

[0292] like Figure 7 , which is a schematic diagram of the composition structure of an electronic device, the electronic device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through the communication bus 13.

[0293] In the embodiment of the present invention, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices.

[0294] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in the embodiment of the target detection method.

[0295] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present invention, the memory 11 at least stores programs for implementing the following functions:

[0296] Extract features of the target image to obtain image features;

[0297] The image features are processed using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint;

[0298] Using deep distribution to transform image features into bird's-eye view features;

[0299] Perform target detection on the bird's-eye view features of the image to obtain the target detection result.

[0300] In a possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0301] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0302] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0303] Of course, it should be noted that Figure 7 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present invention. In actual applications, the electronic device may include Figure 7 More or fewer components than shown, or combinations of certain components.

[0304] An embodiment of the present invention provides a computer-readable storage medium.

[0305] The computer-readable storage medium provided in the embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above target detection methods can be implemented.

[0306] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0307] For an introduction to the computer-readable storage medium provided in an embodiment of the present invention, please refer to the above method embodiment, and the present invention will not elaborate on it here.

[0308] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0309] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0310] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0311] The technical solution provided by the present invention is described in detail above. Specific examples are used herein to illustrate the principle and implementation of the present invention, and the description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A target detection method, characterized in that: include: Extract features of the target image to obtain image features; The image features are processed using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint; converting the image features into image bird's-eye view features using the depth distribution; Perform target detection on the bird's-eye view features of the image to obtain a target detection result.

2. The target detection method according to claim 1, characterized in that: The process of constructing the preset depth prediction model based on the foreground depth distribution consistency loss function includes: Acquire a target image sample, and determine a bounding box of a target object in the target image sample; Taking the inner area of ​​the bounding box as the foreground area; Perform random sampling in each of the foreground areas to obtain foreground point pairs; Performing depth distribution prediction on the foreground point in each of the foreground point pairs to obtain depth prediction value pairs of the foreground point pairs; In combination with the foreground depth distribution consistency loss function, model supervision training is performed using the preset depth distribution dimension and each of the depth prediction value pairs to obtain the preset depth prediction model.

3. The target detection method according to claim 1, characterized in that: The process of constructing the preset depth prediction model based on the entropy reduction constraint of the depth distribution of the foreground area includes: Acquire a target image sample, and determine a bounding box of a target object in the target image sample; Taking the inner area of ​​the bounding box as the foreground area; Perform random sampling in each of the foreground areas to obtain foreground points; Predicting the depth distribution of each foreground point to obtain a depth prediction value of the foreground point; Combined with the entropy reduction constraint of the depth distribution of the foreground area, model supervision training is performed using the preset depth distribution dimension and the depth prediction value of each foreground point to obtain the preset depth prediction model.

4. The target detection method according to claim 1, characterized in that: The process of constructing the preset depth prediction model based on the entropy increase constraint of the depth distribution of the confusion area includes: Acquire a target image sample, and determine a bounding box of a target object in the target image sample; Inside the bounding box, a pixel area at a preset number of pixels away from the bounding box is used as a confusion area; Perform random sampling in each of the confusion areas to obtain confusion points; Performing depth distribution prediction on each of the confusion points to obtain a depth prediction value of the confusion point; Combined with the entropy increase constraint of the depth distribution of the confused area, model supervision training is performed using the preset depth distribution dimension and the depth prediction value of each of the confused points to obtain the preset depth prediction model.

5. The target detection method according to claim 1, characterized in that: The converting the image features into image bird's-eye view features by using the depth distribution comprises: Expanding the image features into point cloud features; Weighting the point cloud features using the depth distribution to obtain weighted point cloud features; Converting the weighted point cloud features from a camera coordinate system to a target detection coordinate system; Voxel pooling is performed on the weighted point cloud features in the target detection coordinate system to obtain the image bird's-eye view features.

6. The target detection method according to claim 5, characterized in that: The step of performing voxel pooling processing on the weighted point cloud features in the target detection coordinate system to obtain the image bird's-eye view features includes: Dividing the coordinate system space of the target detection coordinate system into cubic voxels of a preset size; For each of the cubic voxels, performing depth clipping processing on the weighted point cloud features in the cubic voxel according to the depth clipping threshold corresponding to the cubic voxel to obtain clipping features; The cropping features corresponding to all the cube voxels are combined into the image bird's-eye view feature.

7. The target detection method according to claim 6, characterized in that: The step of performing a depth clipping process on the weighted point cloud features in the cubic voxels according to the depth clipping threshold corresponding to the cubic voxels to obtain clipping features includes: Determine the depth value of the cube voxel in the coordinate system space; Performing depth probability prediction on the depth value to obtain a predicted depth probability; When the predicted depth probability does not exceed the depth clipping threshold, all weighted point cloud features in the cube voxel are eliminated; When the predicted depth probability exceeds the depth clipping threshold, the weighted point cloud features within the cube voxel are retained as the clipping features.

8. The target detection method according to claim 6, characterized in that: The process of setting the depth clipping threshold includes: Get a set of candidate thresholds; For each candidate threshold in the candidate threshold set, calculating the proportion of clipped features and the model evaluation index corresponding to the candidate threshold; In the candidate threshold set, the candidate thresholds whose model evaluation index is lower than the preset index threshold are deleted to obtain a retained candidate threshold set; In the set of retained candidate thresholds, the candidate threshold corresponding to the largest proportion of clipped features is used as the depth clipping threshold.

9. The target detection method according to claim 8, characterized in that: The calculation process of the model evaluation index includes: For each candidate threshold, obtain the average accuracy value, average translation error, average scale error, average angle error, average velocity error, and average attribute error of the training model corresponding to the candidate threshold; The model evaluation index is obtained by calculating the average accuracy value, the average translation error, the average scale error, the average angle error, the average velocity error, and the average attribute error.

10. The target detection method according to claim 1, characterized in that: The step of extracting features from the target image to obtain image features includes: The target image is subjected to feature extraction using a preset image backbone network model to obtain the image features.

11. The target detection method according to any one of claims 1 to 10, characterized in that: The performing target detection on the bird's-eye view features of the image to obtain a target detection result includes: Use LiDAR to obtain point cloud data; Extracting features from the point cloud data to obtain point cloud bird's-eye view features; Fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fused features; Perform target detection on the fused features to obtain the target detection result.

12. The target detection method according to claim 11, characterized in that: The step of fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fused features includes: The image bird's-eye view features and the point cloud bird's-eye view features are fused using a preset feature fusion network model to obtain the fused features; wherein the preset feature fusion network model is constructed based on a multimodal semantic alignment loss function.

13. The target detection method according to claim 12, characterized in that: The process of constructing the preset feature fusion network model based on the multimodal semantic alignment loss function includes: Get a bird's eye view; Acquire an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view; Converting the image bird's-eye view feature set into an image bird's-eye view feature semantic coding set; Converting the point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic coding set; In combination with the multimodal semantic alignment loss function, the image bird's-eye view feature semantic coding set and the point cloud bird's-eye view feature semantic coding set are used to perform model supervision training to obtain the preset feature fusion network model.

14. The target detection method according to claim 13, characterized in that: The step of obtaining an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view includes: determining a bounding box about a target object in the bird's-eye view; Taking the inner area of ​​the bounding box as the foreground area; Perform random sampling in each of the foreground areas to obtain each of the preset sampling foreground positions; An image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each of the preset sampling foreground positions are obtained in the bird's-eye view.

15. The target detection method according to claim 13, characterized in that: The step of converting the image bird's-eye view feature set into an image bird's-eye view feature semantic coding set comprises: The image bird's-eye view feature set is converted into the image bird's-eye view feature semantic coding set by using a preset image semantic coding model; Accordingly, the step of converting the point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic coding set includes: The point cloud bird's-eye view feature set is converted into the point cloud bird's-eye view feature semantic coding set by using a preset point cloud semantic coding model.

16. The target detection method according to claim 11, characterized in that: The step of extracting features from the point cloud data to obtain point cloud bird's-eye view features includes: The point cloud data is subjected to feature extraction using a preset point cloud backbone network model to obtain the point cloud bird's-eye view features.

17. The target detection method according to claim 11, characterized in that: The step of fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fused features includes: Splicing the image bird's-eye view features and the point cloud bird's-eye view features along a feature channel direction to obtain a splicing feature; Using a preset feature fusion network model to perform feature fusion processing on the splicing features to obtain initial fusion features; Performing dimension reduction processing on the initial fusion feature to obtain the fusion feature.

18. The target detection method according to claim 11, characterized in that: The performing target detection on the fusion feature to obtain the target detection result includes: The preset fusion feature detection model is used to perform target detection on the fusion feature to obtain the target detection result.

19. A target detection device, characterized in that: include: An extraction module is used to extract features from a target image and obtain image features; A processing module, used to process the image features using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint; A conversion module, configured to convert the image features into image bird's-eye view features using the depth distribution; The detection module is used to perform target detection on the bird's-eye view features of the image to obtain target detection results.

20. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the target detection method according to any one of claims 1 to 18 when executing the computer program.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the target detection method according to any one of claims 1 to 18 are implemented.