Target detection method and apparatus, electronic device, and readable storage medium
By using preset depth prediction model and depth distribution loss function in 3D object detection technology, the problem of insufficient accuracy of depth estimation is solved, and more accurate object detection and higher quality image feature conversion are achieved.
Patent Information
- Application Number
- PCT/CN2024/088921
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-26
- Filing Date
- 2024-04-19
- Publication Date
- 2024-12-05
AI Technical Summary
The accuracy of depth estimation in the existing 3D object detection technology affects the conversion quality of image features and the accuracy of object detection results.
The preset depth prediction model is used to supervise training based on the foreground depth distribution consistency loss function, the foreground region depth distribution entropy reduction constraint and the confusing region depth distribution entropy increase constraint, which is used to more accurately estimate the depth distribution of image features and to draw them. Convert to image aerial view features for object detection.
The accuracy of depth distribution estimation and the accuracy of object detection are improved, background interference is reduced, and depth distribution prediction ability for foreground background confusing areas is enhanced.
Smart Images

Figure CN2024088921_05122024_PF_FP_ABST
Abstract
Description
Target detection method, device, electronic device and readable storage medium
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on May 26, 2023, with application number 202310610246.X and titled “A Target Detection Method, Device, Electronic Device and Readable Storage Medium,” the entire contents of which are incorporated by reference into this application. Technical Field
[0003] The present application relates to the field of machine learning technology, and in particular to a target detection method, device, electronic device, and readable storage medium. Background Art
[0004] In recent years, with the continuous advancement of sensor technology and information processing technology, autonomous driving technology has made rapid progress and attracted widespread attention. As an important component of autonomous driving technology, 3D object detection is a key means for autonomous driving systems to perceive and understand their surroundings.
[0005] In 3D object detection technology, the 2D-3D view conversion of image features (conversion from a two-dimensional view to a three-dimensional view) is critical. In related technologies, the LSS (Lift, Splat, Shoot) method is generally used. First, the depth distribution of image features is predicted, and then the image features are lifted into a 3D feature point cloud using the depth distribution, and finally converted into the BEV space. Among them, depth estimation is a big challenge. The accuracy of depth estimation directly affects the quality of image features after view conversion, and thus affects the accuracy of 3D object detection results. However, the depth estimation accuracy of current technologies still has a lot of room for improvement.
[0006] Summary of the Invention
[0007] The purpose of this application is to provide a target detection method, which can perform more accurate depth distribution estimation of image features, thereby achieving more accurate target detection; another purpose of this application is to provide a target detection device, electronic device and computer-readable storage medium, all of which have the above-mentioned beneficial effects.
[0008] In a first aspect, the present application provides a target detection method, comprising:
[0009] Extract features from the target image to obtain image features;
[0010] Processing image features using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint;
[0011] Using depth distribution to convert image features into bird's-eye view features;
[0012] Perform target detection on the bird's-eye view features of the image to obtain the target detection results.
[0013] Optionally, the process of constructing a preset depth prediction model based on the foreground depth distribution consistency loss function includes:
[0014] Obtain a target image sample and determine a bounding box of a target object in the target image sample;
[0015] The inner area of the bounding box is regarded as the foreground area;
[0016] Perform random sampling in each foreground area to obtain foreground point pairs;
[0017] Predicting the depth distribution of the foreground point in each foreground point pair to obtain a depth prediction value pair for the foreground point pair;
[0018] Combined with the foreground depth distribution consistency loss function, the preset depth distribution dimension and each depth prediction value are used to perform model supervision training to obtain the preset depth prediction model.
[0019] Optionally, the process of constructing a preset depth prediction model based on the entropy reduction constraint of the foreground area depth distribution includes:
[0020] Obtain a target image sample and determine a bounding box of a target object in the target image sample;
[0021] The inner area of the bounding box is regarded as the foreground area;
[0022] Perform random sampling in each foreground area to obtain foreground points;
[0023] Predict the depth distribution of each foreground point and obtain the depth prediction value of the foreground point;
[0024] Combined with the entropy reduction constraint of the depth distribution in the foreground area, the preset depth distribution dimension and the depth prediction value of each foreground point are used to perform model supervision training to obtain the preset depth prediction model.
[0025] Optionally, the process of constructing a preset depth prediction model based on the entropy increase constraint of the depth distribution of the confusion area includes:
[0026] Obtain a target image sample and determine a bounding box of a target object in the target image sample;
[0027] Inside the bounding box, the pixel area with a preset number of pixels away from the bounding box is used as the confusion area;
[0028] Perform random sampling in each confusion area to obtain confusion points;
[0029] Perform depth distribution prediction on each confusion point to obtain the depth prediction value of the confusion point;
[0030] Combined with the entropy increase constraint of the depth distribution in the confused area, the preset depth distribution dimension and the depth prediction value of each confused point are used to perform model supervision training to obtain the preset depth prediction model.
[0031] Optionally, the image features are converted into bird's-eye view features using depth distribution, including:
[0032] Expand image features into point cloud features;
[0033] Use depth distribution to weight point cloud features to obtain weighted point cloud features;
[0034] Convert the weighted point cloud features from the camera coordinate system to the target detection coordinate system;
[0035] The weighted point cloud features in the target detection coordinate system are subjected to voxel pooling to obtain the image bird's-eye view features.
[0036] Optionally, voxel pooling is performed on the weighted point cloud features in the target detection coordinate system to obtain image bird's-eye view features, including:
[0037] Divide the coordinate space of the target detection coordinate system into cubic voxels of a preset size;
[0038] For each cubic voxel, depth clipping is performed on the weighted point cloud features within the cubic voxel according to the depth clipping threshold corresponding to the cubic voxel to obtain clipping features;
[0039] The cropped features corresponding to all cube voxels are combined into the image bird's-eye view features.
[0040] Optionally, performing depth clipping processing on the weighted point cloud features within the cubic voxels according to the depth clipping threshold corresponding to the cubic voxels to obtain clipping features includes:
[0041] Determine the depth value of the cube voxel in the coordinate space;
[0042] Perform depth probability prediction on the depth value to obtain predicted depth probability;
[0043] When the predicted depth probability does not exceed the depth clipping threshold, all weighted point cloud features within the cube voxel are removed;
[0044] When the predicted depth probability exceeds the depth clipping threshold, the weighted point cloud features within the cube voxel are retained as clipping features.
[0045] Optionally, the process of setting the depth clipping threshold includes:
[0046] Get a set of candidate thresholds;
[0047] For each candidate threshold in the candidate threshold set, calculate the proportion of clipped features and model evaluation indicators corresponding to the candidate threshold;
[0048] In the candidate threshold set, the candidate thresholds whose model evaluation index is lower than the preset index threshold are deleted to obtain the retained candidate threshold set;
[0049] In the set of retained candidate thresholds, the candidate threshold corresponding to the largest proportion of clipped features is used as the depth clipping threshold.
[0050] Optionally, the calculation process of the model evaluation index includes:
[0051] For each candidate threshold, obtain the average accuracy value, average translation error, average scale error, average angle error, average velocity error, and average attribute error of the training model corresponding to the candidate threshold;
[0052] The model evaluation index is obtained based on the average accuracy value, average translation error, average scale error, average angle error, average velocity error, and average attribute error.
[0053] Optionally, feature extraction is performed on the target image to obtain image features, including:
[0054] The preset image backbone network model is used to extract features of the target image and obtain image features.
[0055] Optionally, target detection is performed on the bird's-eye view features of the image to obtain target detection results, including:
[0056] Use lidar to obtain point cloud data;
[0057] Perform feature extraction on point cloud data to obtain point cloud bird's-eye view features;
[0058] Fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fused features;
[0059] Perform target detection on the fused features to obtain target detection results.
[0060] Optionally, the image bird's-eye view features and the point cloud bird's-eye view features are fused to obtain fused features, including:
[0061] The preset feature fusion network model is used to fuse the image bird's-eye view features and the point cloud bird's-eye view features to obtain fusion features; wherein, the preset feature fusion network model is constructed based on the multimodal semantic alignment loss function.
[0062] Optionally, the process of constructing a preset feature fusion network model based on the multimodal semantic alignment loss function includes:
[0063] Get a bird's-eye view;
[0064] Obtaining an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view;
[0065] Converting the image bird's-eye view feature set into an image bird's-eye view feature semantic encoding set;
[0066] Convert the point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic encoding set;
[0067] Combined with the multimodal semantic alignment loss function, the image bird's-eye view feature semantic coding set and the point cloud bird's-eye view feature semantic coding set are used for model supervision training to obtain the preset feature fusion network model.
[0068] Optionally, obtaining an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view includes:
[0069] Determine a bounding box about the target object in the bird's-eye view;
[0070] The inner area of the bounding box is regarded as the foreground area;
[0071] Perform random sampling in each foreground area to obtain each preset sampling foreground position;
[0072] In the bird's-eye view, an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position are obtained.
[0073] Optionally, converting the image bird's-eye view feature set into an image bird's-eye view feature semantic encoding set includes:
[0074] The image bird's-eye view feature set is converted into an image bird's-eye view feature semantic coding set using a preset image semantic coding model;
[0075] Accordingly, the point cloud bird's-eye view feature set is converted into a point cloud bird's-eye view feature semantic encoding set, including:
[0076] The preset point cloud semantic coding model is used to convert the point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic coding set.
[0077] Optionally, feature extraction is performed on the point cloud data to obtain point cloud bird's-eye view features, including:
[0078] The preset point cloud backbone network model is used to extract features from point cloud data to obtain the point cloud bird's-eye view features.
[0079] Optionally, the image bird's-eye view features and the point cloud bird's-eye view features are fused to obtain fused features, including:
[0080] Splicing the image bird's-eye view features and the point cloud bird's-eye view features along the feature channel direction to obtain splicing features;
[0081] Use the preset feature fusion network model to perform feature fusion processing on the splicing features to obtain the initial fusion features;
[0082] Perform dimensionality reduction on the initial fusion features to obtain fusion features.
[0083] Optionally, target detection is performed on the fused features to obtain target detection results, including:
[0084] The preset fusion feature detection model is used to perform target detection on the fusion features to obtain the target detection results.
[0085] In a second aspect, the present application further discloses a target detection device, comprising:
[0086] Extraction module, used to extract features from the target image and obtain image features;
[0087] A processing module for processing image features using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint;
[0088] A conversion module for converting image features into bird's-eye view features using depth distribution;
[0089] The detection module is used to perform target detection on the bird's-eye view features of the image and obtain target detection results.
[0090] In a third aspect, the present application further discloses an electronic device, comprising:
[0091] Memory for storing computer programs;
[0092] A processor is used to implement the steps of any of the above target detection methods when executing a computer program.
[0093] In a fourth aspect, the present application further discloses a computer non-volatile readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above target detection methods are implemented.
[0094] A target detection method provided by the present application includes: extracting features from a target image to obtain image features; processing the image features using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint; converting image features into image bird's-eye view features using the depth distribution; performing target detection on the image bird's-eye view features to obtain target detection results.
[0095] The application of the technical solution provided by this application has the following technical effects:
[0096] After extracting image features from the target image, the depth is first estimated using a pre-created depth prediction model to obtain the depth distribution of the image features. Then, based on the depth distribution, the image features are converted from 2D to 3D to obtain the corresponding image bird's-eye view features. Finally, target detection is performed on the image bird's-eye view features to achieve target detection of the target image and obtain the target detection result. The preset depth prediction model is obtained by supervised training based on a pre-set depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint. The foreground depth distribution consistency loss function can be used to constrain different foreground points in the same target foreground area to have similar depth distributions. The foreground area depth distribution entropy reduction constraint can be used to constrain the same target foreground area to have consistent semantic information and not be easily disturbed by the background area. The confusion area depth distribution entropy increase constraint can be used to guide the depth prediction model to make a more reasonable depth distribution prediction for the target foreground background confusion area. Therefore, based on the preset depth prediction model, a more accurate depth distribution estimation of the image features can be performed, which helps to achieve more accurate target detection.
[0097] The present application also discloses a target detection device, an electronic device, and a computer-readable storage medium, which also have the above-mentioned technical effects and will not be described in detail in the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0098] In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the following is a brief introduction to the drawings required for describing the prior art and the embodiments of the present application. Of course, the drawings described below in connection with the embodiments of the present application are only part of the embodiments of the present application. For those skilled in the art, other drawings can be obtained based on the provided drawings without inventive effort, and the obtained other drawings also fall within the scope of protection of the present application.
[0099] FIG1 is a flow chart of a target detection method provided in this application;
[0100] FIG2 is a framework diagram of a target detection system provided by the present application;
[0101] FIG3 is a flow chart of an image feature view conversion method provided by the present application;
[0102] FIG4 is a schematic diagram of a depth cutting method provided by the present application;
[0103] FIG5 is a flow chart of a multimodal bird's-eye view feature semantic alignment method provided by the present application;
[0104] FIG6 is a schematic diagram of a flow chart of a target detection device provided by the present application;
[0105] FIG7 is a schematic structural diagram of an electronic device provided in this application. DETAILED DESCRIPTION
[0106] The core of this application is to provide a target detection method, which can perform more accurate depth distribution estimation of image features, thereby achieving more accurate target detection; another core of this application is to provide a target detection device, electronic device and computer-readable storage medium, all of which have the above-mentioned beneficial effects.
[0107] In order to describe the technical solutions in the embodiments of the present application more clearly and completely, the technical solutions in the embodiments of the present application will be introduced below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0108] An embodiment of the present application provides a target detection method.
[0109] Please refer to FIG1 , which is a flow chart of a target detection method provided in the present application. The target detection method may include the following S101 to S104 .
[0110] S101: Extract features from the target image to obtain image features.
[0111] This step aims to extract features from the target image to obtain its image features. The target image is the image data for target detection, which can be acquired through a camera or other device. Furthermore, after acquiring the target image, feature extraction can be performed to obtain the image features. This allows for identification processing of the image features to determine whether a target object (such as a pedestrian, animal, obstacle, etc.) is present in the target image.
[0112] In a possible implementation, the above-mentioned feature extraction of the target image to obtain image features may include: using a preset image backbone network model to extract features of the target image to obtain image features.
[0113] This embodiment provides a method for extracting features from a target image to obtain image features, which can be implemented based on a pre-created image backbone network model. Specifically, after acquiring the target image, it can be input into a preset image backbone network model for processing, and the output of the model is the image features of the target image. Among them, the preset image backbone network model can be constructed using a traditional convolutional neural network, such as a residual neural network (ResNet, Residual Network), or a Transformer-based network model, such as Swin Transformer (a new neural network architecture designed specifically for processing visual tasks).
[0114] S102: Processing the image features using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint;
[0115] This step aims to predict the depth distribution of image features. Specifically, the depth distribution is the probability distribution of a set of discrete depth values defined in the depth dimension. The prediction process is as follows:
[0116] First, a set of discrete depth values is preset, such as s1 meters to s D meters, interval is s gap Meters, D discrete depth values are obtained, where D = (s D -s1) / s gap +1,s D>s1. Then, a depth prediction network is used to take the image features under the camera view as input and output the depth distribution at each feature point. Assume that the size of the image features under the camera view is (H, W, C), where H and W represent the height and width of the image features respectively, and C represents the number of channels of the image features. For each feature point in the H and W dimensions, its corresponding depth distribution is predicted. That is, the depth probability of each feature point at each preset depth value. The size of the depth distribution predicted for the image features under the entire camera view is (H, W, D), where H and W represent the height and width of the image feature respectively, and D represents the number of depth values at each feature point.
[0117] Therefore, during the implementation process, a depth prediction model, that is, the above-mentioned preset depth prediction model, can be created in advance, and then the image features of the target image are input into the preset depth prediction model for processing, and the output of the model is the depth distribution information of the image features. Among them, the preset depth prediction model can be composed of residual units, and a Softmax (normalized exponential function) activation layer is used to output a multi-dimensional depth distribution prediction. During the construction process, it can be obtained by supervised training based on a preset depth distribution loss function. The preset depth distribution loss function can include a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint.
[0118] The prior rule for constructing the foreground depth distribution consistency loss function is that different foreground points in the same target foreground region have similar depth distributions. Therefore, the foreground depth distribution consistency loss function can be used to constrain the depth distribution of the two foreground points in each foreground point pair in each foreground region, so that the depth distributions of the two foreground points in each foreground point pair tend to be consistent. The prior rule for constructing the foreground region depth distribution entropy reduction constraint is that the same target foreground region has consistent semantic information and is not easily disturbed by the background region. Therefore, the foreground region depth distribution entropy reduction constraint can be used to constrain the entropy of the depth distribution at each foreground point in each foreground region, so that the entropy of the depth distribution at each foreground point tends to be smaller. The prior rule for constructing the confusion region depth distribution entropy increase constraint is that in the confusion region of the target foreground and background, the foreground and background semantic information are confused, and the foreground and background generally have different depths, so the depth distribution should tend to be multimodal. Therefore, the confusion region depth distribution entropy increase constraint can be used to constrain the entropy of the depth distribution at each feature point in the foreground-background confusion region, so that the entropy of the depth distribution at each feature point in the foreground-background confusion region tends to be larger. Therefore, the depth prediction model constructed based on the above-mentioned depth distribution loss functions can predict the depth distribution of image features, which can effectively ensure the accuracy of the prediction results and help achieve more accurate target detection.
[0119] S103: Convert the image features into bird's-eye view features using the depth distribution.
[0120] This step aims to achieve view conversion of image features. Essentially, it performs a 2D-to-3D view conversion on image features. That is, it converts image features from a 2D view to a 3D view (a bird's-eye view), thereby converting image features into bird's-eye view features. Specifically, after obtaining the depth distribution of image features, the high depth distribution can be used to achieve the above view conversion operation.
[0121] S104: Perform target detection on the bird's-eye view features of the image to obtain a target detection result.
[0122] This step aims to detect objects using bird's-eye view features and obtain the final object detection results. Specifically, after converting image features into bird's-eye view features, 3D object detection can be performed on them. The detection result is the object detection result for the target image, which is used to indicate whether a target object exists in the target image.
[0123] It can be seen that the target detection method provided in the embodiment of the present application, after extracting image features from the target image, first uses a pre-created depth prediction model to perform depth estimation on it to obtain the depth distribution of the image features, then performs 2D-3D view conversion on the image features based on the depth distribution to obtain the corresponding image bird's-eye view features, and finally performs target detection on the image bird's-eye view features to achieve target detection for the target image and obtain a target detection result. Among them, the preset depth prediction model is obtained by supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint. The foreground depth distribution consistency loss function can be used to constrain different foreground points in the same target foreground area to have similar depth distributions, the foreground area depth distribution entropy reduction constraint can be used to constrain the same target foreground area to have consistent semantic information and not be easily disturbed by the background area, and the confusion area depth distribution entropy increase constraint can be used to guide the depth prediction model to make a more reasonable depth distribution prediction for the target foreground background confusion area. Therefore, based on the preset depth prediction model, a more accurate depth distribution estimation of the image features can be performed, which helps to achieve more accurate target detection.
[0124] Based on the above embodiment:
[0125] In one embodiment of the present application, the process of constructing a preset depth prediction model based on the foreground depth distribution consistency loss function includes:
[0126] Obtain a target image sample and determine a bounding box of a target object in the target image sample;
[0127] The inner area of the bounding box is regarded as the foreground area;
[0128] Perform random sampling in each foreground area to obtain foreground point pairs;
[0129] Predicting the depth distribution of the foreground point in each foreground point pair to obtain a depth prediction value pair for the foreground point pair;
[0130] Combined with the foreground depth distribution consistency loss function, the preset depth distribution dimension and each depth prediction value are used to perform model supervision training to obtain the preset depth prediction model.
[0131] The embodiment of the present application provides an implementation method for constructing a preset depth prediction model based on a foreground depth distribution consistency loss function. In the implementation process, for a target image sample, first determine the bounding box of the target object, which can be manually calibrated, and use the inner area of the bounding box as the foreground area, and correspondingly, the outer area of the bounding box is the background area; further, for each target image sample, F1 foreground areas can be randomly sampled first, and for each foreground area, 2Q foreground points are randomly sampled to form Q foreground point pairs, D is used to represent the dimension of the depth distribution, and the foreground depth distribution consistency loss function L is: depth_consistent is defined as follows:
[0132] in, Represents the k-th dimension depth prediction probability value of the depth distribution corresponding to the first foreground point in the j-th foreground point pair of the i-th foreground area.
[0133] Therefore, the construction of a preset depth prediction model can be achieved based on the foreground depth distribution consistency loss function.
[0134] In one embodiment of the present application, the process of constructing a preset depth prediction model based on the entropy reduction constraint of the foreground area depth distribution may include:
[0135] Obtain a target image sample and determine a bounding box of a target object in the target image sample;
[0136] The inner area of the bounding box is regarded as the foreground area;
[0137] Perform random sampling in each foreground area to obtain foreground points;
[0138] Predict the depth distribution of each foreground point and obtain the depth prediction value of the foreground point;
[0139] Combined with the entropy reduction constraint of the depth distribution in the foreground area, the preset depth distribution dimension and the depth prediction value of each foreground point are used to perform model supervision training to obtain the preset depth prediction model.
[0140] The embodiment of the present application provides an implementation method for constructing a preset depth prediction model based on the entropy reduction constraint of the foreground area depth distribution. In the implementation process, for the target image sample, the bounding box of the target object is first determined, which can be manually calibrated, and the inner area of the bounding box is used as the foreground area, and the corresponding outer area of the bounding box is the background area; further, for each target image sample, F2 foreground areas can be randomly sampled, and for each foreground area, P foreground points are randomly sampled. The definition of the foreground area depth distribution entropy reduction constraint Ldepth_entropy_decrease is as follows:
[0141] Among them, d ijk Represents the k-th dimension depth prediction probability value of the depth distribution corresponding to the j-th foreground point of the i-th foreground area.
[0142] Therefore, the construction of a preset depth prediction model can be achieved based on the entropy reduction constraint of the depth distribution of the foreground area.
[0143] In one embodiment of the present application, the process of constructing a preset depth prediction model based on the entropy increase constraint of the depth distribution of the confusion area may include:
[0144] Obtain a target image sample and determine a bounding box of a target object in the target image sample;
[0145] Inside the bounding box, the pixel area with a preset number of pixels away from the bounding box is used as the confusion area;
[0146] Perform random sampling in each confusion area to obtain confusion points;
[0147] Perform depth distribution prediction on each confusion point to obtain the depth prediction value of the confusion point;
[0148] Combined with the entropy increase constraint of the depth distribution in the confused area, the preset depth distribution dimension and the depth prediction value of each confused point are used to perform model supervision training to obtain the preset depth prediction model.
[0149] The embodiment of the present application provides an implementation method for constructing a preset depth prediction model based on the entropy increase constraint of the depth distribution of the confusion area. In the implementation process, for the target image sample, the bounding box of the target object is first determined, which can be manually calibrated, and the inner area of the bounding box is used as the foreground area, and correspondingly, the outer area of the bounding box is the background area; then, within the foreground area, the pixel area of σ (preset number) pixels away from the bounding box is defined as the foreground background confusion area; finally, for each target image sample, F3 foreground background confusion areas can be randomly sampled first, and for each foreground background confusion area, T foreground points are randomly sampled, and the definition of the entropy increase constraint Ldepth_entropy_increase of the depth distribution of the confusion area is as follows:
[0150] Among them, d ijk Represents the k-th dimension depth prediction probability value of the depth distribution corresponding to the j-th foreground point in the i-th foreground-background confusion area.
[0151] Therefore, the construction of a preset depth prediction model can be achieved based on the entropy increase constraint of the depth distribution of the confused area.
[0152] In one embodiment of the present application, converting image features into bird's-eye view features using depth distribution may include:
[0153] Expand image features into point cloud features;
[0154] Use depth distribution to weight point cloud features to obtain weighted point cloud features;
[0155] Convert the weighted point cloud features from the camera coordinate system to the target detection coordinate system;
[0156] The weighted point cloud features in the target detection coordinate system are subjected to voxel pooling to obtain the image bird's-eye view features.
[0157] The embodiment of the present application provides a method for converting image features into image bird's-eye view features. First, the image features are expanded into point cloud features. This process can be implemented according to a pre-set depth distribution dimension. By equally dividing the preset depth distribution dimension, a corresponding number of equally spaced discrete depth points are obtained. Then, the 2D image features are expanded along the depth direction to obtain 3D point cloud features. Further, the point cloud features are weighted using the predicted depth distribution to obtain depth-weighted point cloud features, i.e., the above-mentioned weighted point cloud features. Finally, since the target image is acquired by a camera device, it is obvious that the weighted point cloud features obtained based on the target image are feature information in the camera coordinate system. Therefore, the weighted point cloud features can be converted from the camera coordinate system to the target detection coordinate system. Then, the weighted point cloud features are voxel pooled in the target detection coordinate system to obtain the image bird's-eye view features. Among them, the target detection coordinate system is the coordinate system of the current execution subject of the target detection. For example, to implement target detection in autonomous driving technology, the target detection coordinate system can be a vehicle coordinate system.
[0158] It is understandable that since the 3D image feature point cloud is very dense, the storage and calculation burdens of voxel pooling are very high. Therefore, by performing voxel pooling on the weighted point cloud features, a relatively sparse 3D image feature point cloud can be obtained. That is, the bird's-eye view features of the image obtained by voxel pooling are sparser than the weighted point cloud features, which can effectively reduce the storage and calculation burdens of voxel pooling, thereby helping to improve the efficiency of target detection.
[0159] In one embodiment of the present application, performing voxel pooling on the weighted point cloud features in the target detection coordinate system to obtain the image bird's-eye view features may include:
[0160] Divide the coordinate space of the target detection coordinate system into cubic voxels of a preset size;
[0161] For each cubic voxel, depth clipping is performed on the weighted point cloud features within the cubic voxel according to the depth clipping threshold corresponding to the cubic voxel to obtain clipping features;
[0162] The cropped features corresponding to all cube voxels are combined into the image bird's-eye view features.
[0163] The embodiment of the present application provides a method for performing voxel pooling processing on weighted point cloud features to obtain image bird's-eye view features. Specifically, in the target detection coordinate system, the coordinate system space of the target detection coordinate system can be spatially divided to obtain a plurality of cubic voxels with the same preset size; further, each cubic voxel is preset with its own corresponding depth clipping threshold (after spatial division, the depth of each cubic voxel is different. In order to ensure the accuracy of depth clipping, different depth clipping thresholds can be set for each cubic voxel). Therefore, for each cubic voxel, all weighted point cloud features therein can be depth clipped according to their corresponding depth clipping threshold to obtain corresponding clipping features. Thus, the clipping features corresponding to all cubic voxels can be combined into the image bird's-eye view features corresponding to the entire weighted point cloud features.
[0164] In one embodiment of the present application, performing depth clipping on the weighted point cloud features within the cubic voxels according to the depth clipping threshold corresponding to the cubic voxels to obtain clipping features may include:
[0165] Determine the depth value of the cube voxel in the coordinate space;
[0166] Perform depth probability prediction on the depth value to obtain predicted depth probability;
[0167] When the predicted depth probability does not exceed the depth clipping threshold, all weighted point cloud features within the cube voxel are removed;
[0168] When the predicted depth probability exceeds the depth clipping threshold, the weighted point cloud features within the cube voxel are retained as clipping features.
[0169] The embodiment of the present application provides a method for implementing depth clipping of weighted point cloud features. As described above, after spatial division, the depth of each cubic voxel is different. To ensure the accuracy of depth clipping, different depth clipping thresholds can be set for each cubic voxel. Based on this, the specific value of the depth clipping threshold can be determined by the depth of the corresponding cubic voxel in the coordinate space. Therefore, for each cubic voxel, the depth value of the cubic voxel in the coordinate space can be determined first, and each depth value corresponds to a predicted depth probability. Then, when the predicted depth probability corresponding to the current cubic voxel does not exceed the depth clipping threshold corresponding to itself, all weighted point cloud features in the cubic voxel can be eliminated (that is, all weighted point cloud features in the current cubic voxel are clipped). When the predicted depth probability corresponding to the current cubic voxel exceeds the depth clipping threshold corresponding to itself, the weighted point cloud features in the cubic voxel can be retained (that is, all weighted point cloud features in the current cubic voxel are retained) and used as the clipping feature of the current cubic voxel.
[0170] In one embodiment of the present application, the process of setting the depth clipping threshold may include:
[0171] Get a set of candidate thresholds;
[0172] For each candidate threshold in the candidate threshold set, calculate the proportion of clipped features and model evaluation indicators corresponding to the candidate threshold;
[0173] In the candidate threshold set, the candidate thresholds whose model evaluation index is lower than the preset index threshold are deleted to obtain the retained candidate threshold set;
[0174] In the set of retained candidate thresholds, the candidate threshold corresponding to the largest proportion of clipped features is used as the depth clipping threshold.
[0175] The embodiment of the present application provides a method for setting a depth clipping threshold. Specifically, a group of candidate thresholds can be preset first to generate a candidate threshold set. For the candidate threshold set, all the candidate thresholds therein can be traversed to calculate the proportion of clipped features corresponding to each candidate threshold (when the candidate threshold is selected as the depth clipping threshold, the proportion of the weighted point cloud features that are clipped to all weighted point cloud features) and the model evaluation index (when the candidate threshold is selected as the depth clipping threshold, the evaluation index value of the processing model obtained by training based on the depth clipping threshold); further, in the candidate threshold set, the candidate threshold whose model evaluation index value is lower than the preset index threshold is deleted to obtain a retained candidate threshold set, and then in the retained candidate threshold set, the candidate threshold corresponding to the maximum proportion of clipped features is used as the depth clipping threshold. In this way, the determination of the depth clipping threshold is achieved.
[0176] It can be understood that the determination of the depth clipping threshold through the above-mentioned screening rules helps to screen and obtain the threshold conditions that are most suitable for the current training model, thereby obtaining a higher-precision training model and achieving more accurate target detection.
[0177] In one embodiment of the present application, the calculation process of the above-mentioned model evaluation index may include:
[0178] For each candidate threshold, obtain the average accuracy value, average translation error, average scale error, average angle error, average velocity error, and average attribute error of the training model corresponding to the candidate threshold;
[0179] The model evaluation index is obtained based on the average accuracy value, average translation error, average scale error, average angle error, average velocity error, and average attribute error.
[0180] The embodiment of the present application provides a method for calculating model evaluation indicators. In this embodiment, NDS (NuScenes Detection Score) can be selected as the model evaluation indicator, which is composed of mAP (mean Average Precision) and a set of five TP metrics (True Positives, a deep learning evaluation indicator): ATE (Average Translation Error), ASE (Average Scale Error), AOE (Average Orientation Error), AVE (Average Velocity Error) and AAE (Average Attribute Error), and the calculation formula is as follows:
[0181] in, Indicates the target category.
[0182] In one embodiment of the present application, performing target detection on the bird's-eye view features of the image to obtain the target detection result may include:
[0183] Use lidar to obtain point cloud data;
[0184] Perform feature extraction on point cloud data to obtain point cloud bird's-eye view features;
[0185] Fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fused features;
[0186] Perform target detection on the fused features to obtain target detection results.
[0187] The embodiment of the present application provides a target detection method based on feature information to achieve the acquisition of target detection results. In this embodiment, in addition to the image bird's-eye view features obtained based on the camera device, point cloud data can also be obtained based on the laser radar and the point cloud bird's-eye view features can be extracted. Thus, the image bird's-eye view features and the point cloud bird's-eye view features can be fused to obtain the fusion features of the two. Finally, the final target detection result can be obtained by performing target detection on the fusion features. It can be seen that this embodiment realizes multimodal feature detection, which can further improve the accuracy of the target detection results.
[0188] In one embodiment of the present application, the above-mentioned fusion of the image bird's-eye view features and the point cloud bird's-eye view features to obtain the fusion features may include: using a preset feature fusion network model to fuse the image bird's-eye view features and the point cloud bird's-eye view features to obtain the fusion features; wherein the preset feature fusion network model is constructed based on a multimodal semantic alignment loss function.
[0189] The present invention provides a feature fusion method, namely, a feature fusion technique based on a pre-set feature fusion network model. Specifically, a feature fusion network model can be pre-built. After obtaining the image bird's-eye view features and the point cloud bird's-eye view features, both can be input into the pre-set feature fusion network model for processing. The output of the model is the fused feature.
[0190] Among them, the preset feature fusion network model is constructed based on the multimodal semantic alignment loss function. It is understandable that the target image and point cloud data belong to two completely different modalities, and there must be differences in the feature distribution between the two. For example, the 3D point cloud features of the two modalities of the same target object do not have similar distributions in the same feature space. This difference needs to be explicitly filled in, so as to improve the effect of subsequent multimodal feature fusion. Therefore, this embodiment uses a multimodal semantic alignment loss function to perform model training. Based on the multimodal semantic alignment loss function, it can effectively constrain the differences between the two types of features, making the differences between the two tend to be smaller, thereby effectively improving the multimodal feature fusion effect and further improving the accuracy of the target detection results.
[0191] In one embodiment of the present application, the process of constructing a preset feature fusion network model based on the multimodal semantic alignment loss function may include:
[0192] Get a bird's-eye view;
[0193] Obtaining an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view;
[0194] Converting the image bird's-eye view feature set into an image bird's-eye view feature semantic encoding set;
[0195] Converting the point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic encoding set;
[0196] Combined with the multimodal semantic alignment loss function, the image bird's-eye view feature semantic coding set and the point cloud bird's-eye view feature semantic coding set are used for model supervision training to obtain the preset feature fusion network model.
[0197] The embodiment of the present application provides an implementation method for constructing a preset feature fusion network model based on a multimodal semantic alignment loss function. First, a bird's-eye view of the target image is obtained (a 3D view corresponding to a 2D view), and each sampling foreground position in the bird's-eye view is determined. The sampling foreground position is pre-set by a technician according to actual conditions. It can be all foreground features in the corresponding foreground area, or it can be part of the foreground features selected in the corresponding foreground area; further, an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position are obtained, and the two are converted into corresponding image bird's-eye view feature semantic coding sets and point cloud bird's-eye view feature semantic coding sets respectively through coding technology. Thus, the image bird's-eye view feature semantic coding set and the point cloud bird's-eye view feature semantic coding set can be used, and combined with the multimodal semantic alignment loss function to construct a multimodal semantic alignment loss function.
[0198] In one embodiment of the present application, the step of obtaining the image bird's-eye view feature set and the point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view may include:
[0199] Determine a bounding box about the target object in the bird's-eye view;
[0200] The inner area of the bounding box is regarded as the foreground area;
[0201] Perform random sampling in each foreground area to obtain each preset sampling foreground position;
[0202] In the bird's-eye view, an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position are obtained.
[0203] As mentioned above, the preset sampling foreground position can be all foreground features in the corresponding foreground area, or it can be some foreground features selected from the corresponding foreground area. In this embodiment, some foreground features in the foreground area are set as the preset sampling foreground position to effectively reduce the amount of data calculation and improve the efficiency of model training. Specifically, the bounding box of the target object in the bird's-eye view can be determined first, and the area within the bounding box can be used as the foreground area. Obviously, the area outside the bounding box is the background area. Furthermore, for each foreground area, random sampling can be performed therein to obtain each of the above-mentioned preset sampling foreground positions, and then the image bird's-eye view feature set and the point cloud bird's-eye view feature set can be obtained.
[0204] It is understood that setting all foreground features in the corresponding foreground area as the preset sampling foreground positions requires relatively large amounts of data computation, but this effectively ensures model training accuracy. Therefore, the selection of the preset sampling foreground positions can be customized by technicians based on actual training needs, and this application does not impose any restrictions on this.
[0205] In one embodiment of the present application, the converting of the image bird's-eye view feature set into the image bird's-eye view feature semantic coding set may include: converting the image bird's-eye view feature set into the image bird's-eye view feature semantic coding set using a preset image semantic coding model;
[0206] Accordingly, the above-mentioned conversion of the point cloud bird's-eye view feature set into the point cloud bird's-eye view feature semantic coding set may include: using a preset point cloud semantic coding model to convert the point cloud bird's-eye view feature set into the point cloud bird's-eye view feature semantic coding set.
[0207] The embodiments of the present application provide a feature coding technology, namely, a feature coding technology based on a preset semantic coding model. Specifically, an image semantic coding model and a point cloud semantic coding model can be pre-created and used to perform semantic coding processing on the image bird's-eye view feature set and the point cloud bird's-eye view feature set, respectively, to obtain the image bird's-eye view feature semantic coding set and the point cloud bird's-eye view feature semantic coding set. Both types of semantic coding models can be constructed using a neural network layer (graph residual neural network).
[0208] In one embodiment of the present application, the above-mentioned feature extraction of point cloud data to obtain point cloud bird's-eye view features may include: using a preset point cloud backbone network model to extract features of the point cloud data to obtain point cloud bird's-eye view features.
[0209] This embodiment provides a method for extracting features from point cloud data to obtain point cloud bird's-eye view features. This method can be implemented based on a pre-created point cloud backbone network model. Specifically, after acquiring point cloud data, it can be input into the pre-created point cloud backbone network model for processing. The output of this model is the point cloud bird's-eye view features of the point cloud data.
[0210] In one embodiment of the present application, the above-mentioned fusion of the image bird's-eye view features and the point cloud bird's-eye view features to obtain the fused features may include:
[0211] Splicing the image bird's-eye view features and the point cloud bird's-eye view features along the feature channel direction to obtain splicing features;
[0212] Use the preset feature fusion network model to perform feature fusion processing on the splicing features to obtain the initial fusion features;
[0213] Perform dimensionality reduction on the initial fusion features to obtain fusion features.
[0214] The embodiment of the present application provides a feature fusion method to achieve the fusion of image bird's-eye view features and point cloud bird's-eye view features. First, for the image bird's-eye view features and point cloud bird's-eye view features, the two can be spliced according to the feature channel direction to obtain the spliced features of the two, and then the spliced features are input into a preset feature fusion network model for feature fusion processing. The output of the preset feature fusion network model is the above-mentioned initial fusion feature. Finally, the final fusion feature is obtained through dimensionality reduction processing. Among them, the preset feature fusion network model is a pre-created network model for realizing feature fusion, which is pre-stored in the corresponding storage space and can be directly called when used.
[0215] In one embodiment of the present application, performing target detection on the fused features to obtain target detection results may include: performing target detection on the fused features using a preset fused feature detection model to obtain target detection results.
[0216] This embodiment provides a method for performing target detection on fused features to obtain target detection results, which can be implemented based on a pre-created fused feature detection model. Specifically, after obtaining the fused features, they can be input into the preset fused feature detection model for processing, and the output of the model is the final target detection result.
[0217] Based on the above embodiments, an embodiment of the present application provides another target detection method.
[0218] First, please refer to FIG2 , which is a framework diagram of a target detection system provided by this application. The target detection method based on this target detection system takes target detection in driving technology as an example, and its implementation process is as follows:
[0219] 1. Use the image backbone network to extract image features of image data; use the point cloud backbone network to extract the point cloud bird's-eye view features of the lidar point cloud data;
[0220] 2. Use a depth prediction model to predict the depth distribution of image features. The depth prediction model is trained using a foreground depth distribution consistency loss function, an entropy reduction constraint on the foreground region depth distribution, and an entropy increase constraint on the confusion region depth distribution.
[0221] 3. Based on the depth distribution predicted in step 2, the image features are processed using LSS technology and depth-cropped voxel pooling method to obtain the image bird's-eye view features;
[0222] 4. Generate semantic encoding of foreground features of the image bird's-eye view and point cloud bird's-eye view, and use a multimodal semantic alignment loss function to supervise the training of semantic alignment of image bird's-eye view features and point cloud bird's-eye view features;
[0223] 5. Use the feature fusion model to fuse the image bird's-eye view features and the point cloud bird's-eye view features to obtain multimodal fused bird's-eye view features, i.e., fused features;
[0224] 6. Use the 3D object detection task head to perform 3D object detection. You can use the classification loss function and the regression loss function to supervise the training of the 3D object detection model.
[0225] 7. Deploy the trained 3D object detection model into the autonomous driving system.
[0226] Regarding step 2 above, please refer to FIG3 , which is a flow chart of an image feature view conversion method provided by this application. The implementation process is as follows:
[0227] 2.1. Assume that the 2D image features obtained by the image backbone network are Where H and W represent the length and width of the feature respectively, C represents the number of channels, and D discrete depth points are predefined at equal intervals, such as 1 meter, 2 meters, ..., D meters. The 2D image features are expanded along the depth direction, and the 2D image features can be expanded into 3D point cloud features of size H×W×D×C. Each 3D point contains a C-dimensional image feature vector.
[0228] 2.2. Using 2D image features, predict the depth distribution at each 2D feature point and obtain the depth distribution The depth distribution is used to weight the 3D point cloud features at the corresponding position to obtain the depth-weighted 3D point cloud features.
[0229] 2.3. Use the camera's intrinsic and extrinsic parameters to transform the weighted 3D point cloud features from the camera coordinate system to the vehicle coordinate system.
[0230] 2.4. In the vehicle coordinate system, the 3D space of the coordinate system is divided into a series of cubic voxels of the same size, and all image features within each voxel are pooled to obtain the image bird's-eye view features.
[0231] Regarding step 2.4 above, please refer to FIG4 , which is a schematic diagram of a depth cropping method provided by this application. The implementation process is as follows:
[0232] Specifically, Figure 4 shows the depth cropping operation at a single point on the image feature plane. At this point, there are several depth value points at fixed intervals along the depth direction. Each depth value corresponds to a predicted depth probability. A depth cropping threshold is set. For depth value points with a depth probability higher than the depth cropping threshold, the image feature is extended to the spatial point; and for depth value points with a depth probability lower than the depth cropping threshold, the image feature is not extended to the spatial point, that is, the spatial point is empty. After the above-mentioned depth cropping process, a 3D point cloud feature that is sparser than the original 3D point cloud feature can be obtained, thereby reducing the storage and computational burden of voxel pooling. Among them, the depth cropping threshold can be manually pre-set, or obtained through a hyperparameter search method (such as grid search), or learned by the model through autonomous learning. For example, a method for determining a depth cropping threshold is as follows:
[0233] Preset a set of candidate thresholds Traverse all candidate thresholds and calculate each candidate threshold τ i The corresponding proportion of cut features and model evaluation metrics Construct a set of evaluation results for all candidate thresholds and press The rule screening evaluation results are obtained to obtain the screened evaluation results Δ′, where ε represents the tolerance for the reduction of the model evaluation index. In the embodiment of the present application, ε=0.01. is the initial model evaluation index value before model training. Further, the proportion of clipped features is selected in Δ′ The candidate threshold τ corresponding to the largest combination i , as the final depth clipping threshold.
[0234] Through verification, we take the image features of a single camera as an example to estimate the computational complexity and storage requirements reduced by the depth cropping threshold. Assuming that the image feature size is H×W×C, where H is the height, W is the width, and C is the number of feature channels; assuming that the image feature depth distribution size is H×W×D, where D is the number of depth values, and assuming that the bird's-eye view grid size is w×d. Then, the computational complexity of generating 3D point cloud features is H×W×D×C, the amount of data to be stored is H×W×D×C, and the computational complexity of voxel pooling is On this basis, by setting the depth clipping threshold, the amount of depth retained can be controlled. Taking the retention of 10% high confidence depth values as an example, the computational complexity of generating 3D point cloud features is H×W×D×C×0.1, the amount of data to be stored is H×W×D×C×0.1, and the computational complexity of voxel pooling is Therefore, when retaining 10% of high-confidence depth values, after depth cropping, the computational cost of generating 3D point cloud features is reduced by 90%, the storage cost is reduced by 90%, and the computational cost of voxel pooling is reduced by 90%.
[0235] Regarding step 4 above, please refer to FIG5 , which is a flowchart of a multimodal bird's-eye view feature semantic alignment method provided by this application. The implementation process is as follows:
[0236] 4.1. Define the area within the target bounding box in the bird's eye view as the foreground area, use the annotation of the target bounding box to obtain the foreground area under the bird's eye view, and randomly sample Foreground positions, and obtain the corresponding image bird's-eye view feature set at these foreground positions and point cloud bird's-eye view feature collection in, All C b The feature vector of each channel is then processed using the image bird's-eye view feature semantic coding module to obtain the image bird's-eye view feature semantic coding set. Use the point cloud bird's eye view feature semantic coding module to process the point cloud bird's eye view feature set to obtain the point cloud bird's eye view feature semantic coding set in, are semantic encoding vectors of g dimensions.
[0237] 4.2. Using multimodal semantic alignment loss function L align The divergence between the semantic encoding of the image bird's-eye view feature at each sampling position and the semantic encoding of the point cloud bird's-eye view at the corresponding position is calculated. The multimodal semantic alignment loss function is used to constrain the divergence between the semantic encoding of the image bird's-eye view feature at each sampling position and the semantic encoding of the point cloud bird's-eye view at the corresponding position, so that the divergence between the semantic encoding of the image bird's-eye view feature at each sampling position and the semantic encoding of the point cloud bird's-eye view at the corresponding position tends to be smaller. The formula of the multimodal semantic alignment loss function is:
[0238] Among them, g represents the dimension of feature semantic encoding, is the j-th dimension encoding value of the semantic encoding of the bird's-eye view feature of the i-th image, The j-th dimension encoding value of the semantic encoding of the i-th point cloud bird's-eye view feature.
[0239] On this basis, when performing feature fusion of image bird's-eye view features and point cloud bird's-eye view features based on the feature fusion model, the image bird's-eye view features and point cloud bird's-eye view features can be first spliced along the channel direction to obtain spliced features, and then the neural network layer is used to fuse and reduce the dimension to obtain the final fusion features.
[0240] It can be seen that the target detection method provided in the embodiment of the present application, after extracting image features from the target image, first uses a pre-created depth prediction model to perform depth estimation on it to obtain the depth distribution of the image features, then performs 2D-3D view conversion on the image features based on the depth distribution to obtain the corresponding image bird's-eye view features, and finally performs target detection on the image bird's-eye view features to achieve target detection for the target image and obtain a target detection result. Among them, the preset depth prediction model is obtained by supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint. The foreground depth distribution consistency loss function can be used to constrain different foreground points in the same target foreground area to have similar depth distributions, the foreground area depth distribution entropy reduction constraint can be used to constrain the same target foreground area to have consistent semantic information and not be easily disturbed by the background area, and the confusion area depth distribution entropy increase constraint can be used to guide the depth prediction model to make a more reasonable depth distribution prediction for the target foreground background confusion area. Therefore, based on the preset depth prediction model, a more accurate depth distribution estimation of the image features can be performed, which helps to achieve more accurate target detection.
[0241] An embodiment of the present application provides a target detection device.
[0242] Please refer to FIG6 , which is a schematic diagram of the structure of a target detection device provided by the present application. The target detection device may include:
[0243] Extraction module 1, used to extract features from the target image and obtain image features;
[0244] Processing module 2, for processing image features using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint;
[0245] A conversion module 3, configured to convert image features into bird's-eye view features using depth distribution;
[0246] The detection module 4 is used to perform target detection on the bird's-eye view features of the image to obtain target detection results.
[0247] It can be seen that the target detection device provided in the embodiment of the present application, after extracting image features from the target image, first uses a pre-created depth prediction model to perform depth estimation on the image features to obtain the depth distribution of the image features, then performs 2D-3D view conversion on the image features based on the depth distribution to obtain the corresponding image bird's-eye view features, and finally performs target detection on the image bird's-eye view features to achieve target detection for the target image and obtain a target detection result. Among them, the preset depth prediction model is obtained by supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint. The foreground depth distribution consistency loss function can be used to constrain different foreground points in the same target foreground area to have similar depth distributions, the foreground area depth distribution entropy reduction constraint can be used to constrain the same target foreground area to have consistent semantic information and not be easily disturbed by the background area, and the confusion area depth distribution entropy increase constraint can be used to guide the depth prediction model to make a more reasonable depth distribution prediction for the target foreground background confusion area. Therefore, based on the preset depth prediction model, a more accurate depth distribution estimation of the image features can be performed, which helps to achieve more accurate target detection.
[0248] In one embodiment of the present application, the target detection device may further include:
[0249] The first model building module is used to obtain a target image sample, determine a bounding box of a target object in the target image sample, and use the inner area of the bounding box as a foreground area;
[0250] Random sampling is performed in each foreground area to obtain foreground point pairs; depth distribution prediction is performed on the foreground point in each foreground point pair to obtain depth prediction value pairs of the foreground point pairs; combined with the foreground depth distribution consistency loss function, the preset depth distribution dimension and each depth prediction value pair are used to perform model supervision training to obtain the preset depth prediction model.
[0251] In one embodiment of the present application, the target detection device may further include:
[0252] The second model construction module is used to obtain target image samples and determine the bounding box of the target object in the target image samples; use the internal area of the bounding box as the foreground area; perform random sampling in each foreground area to obtain foreground points; perform depth distribution prediction on each foreground point to obtain the depth prediction value of the foreground point; combine the entropy reduction constraint of the depth distribution of the foreground area, use the preset depth distribution dimension and the depth prediction value of each foreground point to perform model supervision training, and obtain a preset depth prediction model.
[0253] In one embodiment of the present application, the target detection device may further include:
[0254] The third model construction module is used to obtain target image samples and determine the bounding box of the target object in the target image samples; inside the bounding box, the pixel area with a preset number of pixels away from the bounding box is used as the confusion area; random sampling is performed in each confusion area to obtain confusion points; depth distribution prediction is performed on each confusion point to obtain a depth prediction value of the confusion point; combined with the entropy increase constraint of the depth distribution of the confusion area, the preset depth distribution dimension and the depth prediction value of each confusion point are used to perform model supervision training to obtain a preset depth prediction model.
[0255] In one embodiment of the present application, the conversion module 3 may include:
[0256] Extension unit, used to expand image features into point cloud features;
[0257] A weighting unit, used to weight the point cloud features using the depth distribution to obtain weighted point cloud features;
[0258] A conversion unit, used to convert the weighted point cloud features from the camera coordinate system to the target detection coordinate system;
[0259] The processing unit is used to perform voxel pooling processing on the weighted point cloud features in the target detection coordinate system to obtain the image bird's-eye view features.
[0260] In one embodiment of the present application, the processing unit may include:
[0261] A division subunit, for dividing the coordinate system space of the target detection coordinate system into cubic voxels of a preset size;
[0262] A cropping subunit is configured to perform depth cropping on the weighted point cloud features within each cubic voxel according to a depth cropping threshold corresponding to the cubic voxel to obtain cropping features;
[0263] The combining subunit is used to combine the cropping features corresponding to all cube voxels into the image bird's-eye view features.
[0264] In one embodiment of the present application, the above-mentioned cropping subunit can be specifically used to determine the depth value of the cubic voxel in the coordinate system space; perform depth probability prediction on the depth value to obtain a predicted depth probability; when the predicted depth probability does not exceed the depth cropping threshold, all weighted point cloud features within the cubic voxel are eliminated; when the predicted depth probability exceeds the depth cropping threshold, the weighted point cloud features within the cubic voxel are retained as cropping features.
[0265] In one embodiment of the present application, the above-mentioned cropping sub-unit can also be used to obtain a set of candidate threshold values; for each candidate threshold value in the candidate threshold value set, the cropped feature ratio and model evaluation index corresponding to the candidate threshold value are calculated; in the candidate threshold value set, the candidate threshold values whose model evaluation index is lower than the preset index threshold value are deleted to obtain a set of retained candidate threshold values; in the retained candidate threshold value set, the candidate threshold value corresponding to the cropped feature ratio with the largest value is used as the depth cropping threshold.
[0266] In one embodiment of the present application, the above-mentioned cropping sub-unit can be specifically used to obtain the average accuracy value, average translation error, average scale error, average angle error, average speed error, and average attribute error of the training model corresponding to each candidate threshold value; and obtain the model evaluation index based on the average accuracy value, average translation error, average scale error, average angle error, average speed error, and average attribute error.
[0267] In one embodiment of the present application, the extraction module 1 can be specifically used to extract features of a target image using a preset image backbone network model to obtain image features.
[0268] In one embodiment of the present application, the detection module 4 may include:
[0269] An acquisition unit, used for acquiring point cloud data using a laser radar;
[0270] An extraction unit is used to extract features from point cloud data to obtain features of a bird's-eye view of the point cloud;
[0271] A fusion unit is used to fuse the image bird's-eye view features and the point cloud bird's-eye view features to obtain fusion features;
[0272] The detection unit is used to perform target detection on the fused features to obtain target detection results.
[0273] In one embodiment of the present application, the fusion unit may include:
[0274] The fusion subunit is used to fuse the image bird's-eye view features and the point cloud bird's-eye view features using a preset feature fusion network model to obtain fusion features; wherein the preset feature fusion network model is constructed based on a multimodal semantic alignment loss function.
[0275] In one embodiment of the present application, the fusion unit may further include:
[0276] Construct a subunit for obtaining a bird's-eye view; obtain an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view; convert the image bird's-eye view feature set into an image bird's-eye view feature semantic coding set; convert the point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic coding set; combine the multimodal semantic alignment loss function, use the image bird's-eye view feature semantic coding set and the point cloud bird's-eye view feature semantic coding set to perform model supervision training, and obtain a preset feature fusion network model.
[0277] In one embodiment of the present application, the above-mentioned construction subunit can be specifically used to determine the bounding box of the target object in the bird's-eye view; use the internal area of the bounding box as the foreground area; perform random sampling in each foreground area to obtain each preset sampling foreground position; and obtain the image bird's-eye view feature set and point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view.
[0278] In one embodiment of the present application, the above-mentioned construction subunit can be specifically used to convert the image bird's-eye view feature set into an image bird's-eye view feature semantic coding set using a preset image semantic coding model; and convert the point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic coding set using a preset point cloud semantic coding model.
[0279] In one embodiment of the present application, the extraction unit may be specifically used to perform feature extraction on point cloud data using a preset point cloud backbone network model to obtain point cloud bird's-eye view features.
[0280] In one embodiment of the present application, the above-mentioned fusion unit can be specifically used to splice the image bird's-eye view features and the point cloud bird's-eye view features along the feature channel direction to obtain spliced features; use a preset feature fusion network model to perform feature fusion processing on the spliced features to obtain initial fusion features; perform dimensionality reduction processing on the initial fusion features to obtain fusion features.
[0281] In one embodiment of the present application, the above-mentioned detection unit can be specifically used to use a preset fusion feature detection model to perform target detection on the fusion feature to obtain a target detection result.
[0282] For an introduction to the apparatus provided in the embodiments of this application, please refer to the above method embodiments, which will not be elaborated in this application.
[0283] An embodiment of the present application provides an electronic device.
[0284] Please refer to FIG7 , which is a schematic diagram of the structure of an electronic device provided by the present application. The electronic device may include:
[0285] Memory for storing computer programs;
[0286] The processor is configured to implement the steps of any of the above target detection methods when executing a computer program.
[0287] 7 is a schematic diagram of the structure of an electronic device, which may include a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 communicate with each other via the communication bus 13.
[0288] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic devices.
[0289] The processor 10 may call a program stored in the memory 11 . Specifically, the processor 10 may execute operations in the embodiment of the target detection method.
[0290] The memory 11 is used to store one or more programs. The program may include program code, and the program code includes computer operating instructions. In the embodiment of the present application, the memory 11 stores at least a program for implementing the following functions:
[0291] Extract features from the target image to obtain image features;
[0292] Processing image features using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint;
[0293] Using depth distribution to convert image features into bird's-eye view features;
[0294] Perform target detection on the bird's-eye view features of the image to obtain the target detection results.
[0295] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.
[0296] In addition, the memory 11 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.
[0297] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.
[0298] Of course, it should be noted that the structure shown in FIG7 does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include more or fewer components than those shown in FIG7, or combine certain components.
[0299] An embodiment of the present application provides a computer-readable non-volatile storage medium.
[0300] The computer non-volatile readable storage medium provided in the embodiment of the present application stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned target detection methods can be implemented.
[0301] The computer non-volatile readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0302] For an introduction to the computer non-volatile readable storage medium provided in the embodiments of the present application, please refer to the above method embodiments, and this application will not elaborate on them here.
[0303] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.
[0304] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0305] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0306] The technical solution provided by the present application is described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications may be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.
Claims
1. A target detection method, characterized in that: include: Extract features of the target image to obtain image features; The image features are processed using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint; converting the image features into image bird's-eye view features using the depth distribution; Perform target detection on the bird's-eye view features of the image to obtain a target detection result.
2. The target detection method according to claim 1, characterized in that: The process of constructing the preset depth prediction model based on the foreground depth distribution consistency loss function includes: Acquire a target image sample, and determine a bounding box of a target object in the target image sample; Taking the inner area of the bounding box as the foreground area; Perform random sampling in each of the foreground areas to obtain foreground point pairs; Performing depth distribution prediction on the foreground point in each of the foreground point pairs to obtain depth prediction value pairs of the foreground point pairs; In combination with the foreground depth distribution consistency loss function, model supervision training is performed using the preset depth distribution dimension and each of the depth prediction value pairs to obtain the preset depth prediction model.
3. The target detection method according to claim 1, characterized in that: The process of constructing the preset depth prediction model based on the entropy reduction constraint of the depth distribution of the foreground area includes: Acquire a target image sample, and determine a bounding box of a target object in the target image sample; Taking the inner area of the bounding box as the foreground area; Perform random sampling in each of the foreground areas to obtain foreground points; Predicting the depth distribution of each foreground point to obtain a depth prediction value of the foreground point; Combined with the entropy reduction constraint of the depth distribution of the foreground area, model supervision training is performed using the preset depth distribution dimension and the depth prediction value of each foreground point to obtain the preset depth prediction model.
4. The target detection method according to claim 1, characterized in that: The process of constructing the preset depth prediction model based on the entropy increase constraint of the depth distribution of the confusion area includes: Acquire a target image sample, and determine a bounding box of a target object in the target image sample; Inside the bounding box, a pixel area at a preset number of pixels away from the bounding box is used as a confusion area; Perform random sampling in each of the confusion areas to obtain confusion points; Performing depth distribution prediction on each of the confusion points to obtain a depth prediction value of the confusion point; Combined with the entropy increase constraint of the depth distribution of the confused area, model supervision training is performed using the preset depth distribution dimension and the depth prediction value of each of the confused points to obtain the preset depth prediction model.
5. The target detection method according to claim 1, characterized in that: The converting the image features into image bird's-eye view features by using the depth distribution comprises: Expanding the image features into point cloud features; Weighting the point cloud features using the depth distribution to obtain weighted point cloud features; Converting the weighted point cloud features from a camera coordinate system to a target detection coordinate system; The weighted point cloud features in the target detection coordinate system are subjected to voxel pooling processing to obtain the image bird's-eye view features.
6. The target detection method according to claim 5, characterized in that: The step of performing voxel pooling processing on the weighted point cloud features in the target detection coordinate system to obtain the image bird's-eye view features includes: Dividing the coordinate system space of the target detection coordinate system into cubic voxels of a preset size; For each of the cubic voxels, performing depth clipping processing on the weighted point cloud features in the cubic voxel according to the depth clipping threshold corresponding to the cubic voxel to obtain clipping features; The cropping features corresponding to all the cube voxels are combined into the image bird's-eye view feature.
7. The target detection method according to claim 6, characterized in that: The step of performing a depth clipping process on the weighted point cloud features in the cubic voxels according to the depth clipping threshold corresponding to the cubic voxels to obtain clipping features includes: Determine the depth value of the cube voxel in the coordinate system space; Performing depth probability prediction on the depth value to obtain a predicted depth probability; When the predicted depth probability does not exceed the depth clipping threshold, all weighted point cloud features in the cube voxel are eliminated; When the predicted depth probability exceeds the depth clipping threshold, the weighted point cloud features within the cube voxel are retained as the clipping features.
8. The target detection method according to claim 6, characterized in that: The process of setting the depth clipping threshold includes: Get a set of candidate thresholds; For each candidate threshold in the candidate threshold set, calculating the proportion of clipped features and the model evaluation index corresponding to the candidate threshold; In the candidate threshold set, the candidate thresholds whose model evaluation index is lower than the preset index threshold are deleted to obtain a retained candidate threshold set; In the set of retained candidate thresholds, the candidate threshold corresponding to the largest proportion of clipped features is used as the depth clipping threshold.
9. The target detection method according to claim 8, characterized in that: The calculation process of the model evaluation index includes: For each candidate threshold, obtain the average accuracy value, average translation error, average scale error, average angle error, average velocity error, and average attribute error of the training model corresponding to the candidate threshold; The model evaluation index is obtained by calculating the average accuracy value, the average translation error, the average scale error, the average angle error, the average velocity error, and the average attribute error.
10. The target detection method according to claim 1, characterized in that: The step of extracting features from the target image to obtain image features includes: The target image is subjected to feature extraction using a preset image backbone network model to obtain the image features.
11. The target detection method according to any one of claims 1 to 10, characterized in that: The performing target detection on the bird's-eye view features of the image to obtain a target detection result includes: Use LiDAR to obtain point cloud data; Extracting features from the point cloud data to obtain point cloud bird's-eye view features; Fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fused features; Perform target detection on the fused features to obtain the target detection result.
12. The target detection method according to claim 11, characterized in that: The step of fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fused features includes: The image bird's-eye view features and the point cloud bird's-eye view features are fused using a preset feature fusion network model to obtain the fused features; wherein the preset feature fusion network model is constructed based on a multimodal semantic alignment loss function.
13. The target detection method according to claim 12, characterized in that: The process of constructing the preset feature fusion network model based on the multimodal semantic alignment loss function includes: Get a bird's eye view; Acquire an image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view; Converting the image bird's-eye view feature set into an image bird's-eye view feature semantic coding set; Converting the point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic coding set; In combination with the multimodal semantic alignment loss function, the image bird's-eye view feature semantic coding set and the point cloud bird's-eye view feature semantic coding set are used to perform model supervision training to obtain the preset feature fusion network model.
14. The target detection method according to claim 13, characterized in that: The step of obtaining the image bird's-eye view feature set and the point cloud bird's-eye view feature set corresponding to each preset sampling foreground position in the bird's-eye view includes: determining a bounding box about a target object in the bird's-eye view; Taking the inner area of the bounding box as the foreground area; Perform random sampling in each of the foreground areas to obtain each of the preset sampling foreground positions; An image bird's-eye view feature set and a point cloud bird's-eye view feature set corresponding to each of the preset sampling foreground positions are obtained in the bird's-eye view.
15. The target detection method according to claim 13, characterized in that: The step of converting the image bird's-eye view feature set into an image bird's-eye view feature semantic coding set comprises: The image bird's-eye view feature set is converted into the image bird's-eye view feature semantic coding set by using a preset image semantic coding model; Accordingly, the step of converting the point cloud bird's-eye view feature set into a point cloud bird's-eye view feature semantic coding set includes: The point cloud bird's-eye view feature set is converted into the point cloud bird's-eye view feature semantic coding set by using a preset point cloud semantic coding model.
16. The target detection method according to claim 11, characterized in that: The step of extracting features from the point cloud data to obtain point cloud bird's-eye view features includes: The point cloud data is feature extracted using a preset point cloud backbone network model to obtain the point cloud bird Overview features.
17. The target detection method according to claim 11, characterized in that: The step of fusing the image bird's-eye view features and the point cloud bird's-eye view features to obtain fused features includes: Splicing the image bird's-eye view features and the point cloud bird's-eye view features along a feature channel direction to obtain a splicing feature; Using a preset feature fusion network model to perform feature fusion processing on the splicing features to obtain initial fusion features; Performing dimension reduction processing on the initial fusion feature to obtain the fusion feature.
18. The target detection method according to claim 11, characterized in that: The performing target detection on the fusion feature to obtain the target detection result includes: The preset fusion feature detection model is used to perform target detection on the fusion feature to obtain the target detection result.
19. A target detection device, characterized in that: include: An extraction module is used to extract features from a target image and obtain image features; A processing module, used to process the image features using a preset depth prediction model to obtain a depth distribution of the image features; wherein the preset depth prediction model is obtained through supervised training based on a preset depth distribution loss function, and the preset depth distribution loss function includes a combination of one or more of a foreground depth distribution consistency loss function, a foreground area depth distribution entropy reduction constraint, and a confusion area depth distribution entropy increase constraint; A conversion module, configured to convert the image features into image bird's-eye view features using the depth distribution; The detection module is used to perform target detection on the bird's-eye view features of the image to obtain a target detection result.
20. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the target detection method according to any one of claims 1 to 18 when executing the computer program.
21. A computer non-volatile readable storage medium, characterized in that: The computer non-volatile readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the target detection method according to any one of claims 1 to 18 are implemented.
Citation Information
Patent Citations
Depth shape priori extraction method
CN107945187A
Target detection model training method and device, target detection method and device, equipment and medium
CN113902897A
Perception result acquisition method and device, computer equipment and storage medium
CN115240168A
Target detection method and device and vehicle
CN116012376A
Source-free cross domain detection method with strong data augmentation and self-trained mean teacher modeling
US20230154167A1
Cited By
Heavy-load large holding tool driving wheel multi-mode damage detection method based on machine vision
CN119760646A
Multi-view camera image and laser radar fused road sensing method and system
CN120198881A
Underwater pipeline fault detection method and device
CN120212445A
Method for controlling arc knife of arc knife type tree mover to meet hard objects and arc knife type tree mover
CN121587200A
Semantic map updating method and device, vehicle and storage medium
CN121658487A
