A training method, device and electronic equipment for target detection model
By conducting multi-sensor multi-dataset training on the object detection model, using feature fusion and region of interest loss calculation, the problem of insufficient generalization ability of the object detection model on different datasets is solved, and efficient model training and generalization are achieved.
Patent Information
- Application Number
- CN202411407566.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2044-10-09
AI Technical Summary
The existing object detection model has low generalization ability on different data sets, resulting in degradation of detection accuracy. Multi-sensor multi-data set training requires a large number of labeling resources and human resources.
By acquiring multiple sample data sets, point cloud features and image backbone networks are used to extract image features, and feature fusion is performed, combining the detection results and true values of the region of interest, and calculating model losses and adjusting to achieve multi-sensors and multiple sensors Dataset Training - Single Dataset Test Paradigm.
The generalization ability of the object detection model in different data sets is improved, the training cost is reduced, the application scenario is expanded, and the training efficiency and generalization ability of the model are improved.
Smart Images
Figure CN118918418B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a training method, device and electronic equipment for a target detection model. Background Art
[0002] In the application scenarios of target detection models, it is often necessary to detect multiple types of data collected by multiple devices. For example, in the scenario of autonomous driving, data collected by different sensors will be used, such as using lidar to collect point cloud data and image sensors to collect images. Targets are detected based on multiple types of data collected by each sensor.
[0003] However, in related technologies, most target detection models follow the single dataset training - single dataset testing paradigm design. If a target detection model trained with a single dataset is directly deployed to another type of data for detection, its detection accuracy will be seriously degraded. The paradigm of multi-sensor multi-dataset training - single dataset testing requires targeted training for each type of dataset, which will cause a large amount of labeling resources and human resources to be wasted, making it difficult to truly implement. Therefore, how to improve the generalization ability of target detection models in different datasets has always been one of the problems that need to be solved urgently. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a method, device and electronic device for training a target detection model to improve the generalization ability of the target detection model in different data sets. The specific technical solution is as follows:
[0005] In a first aspect, an embodiment of the present application provides a method for training a target detection model, comprising:
[0006] Acquire a plurality of sample data sets, wherein each of the sample data sets has a respective region of interest and includes respective point cloud data and image data;
[0007] Input each of the sample data sets into the target detection model to be trained, use the three-dimensional backbone network in the target detection model to extract the point cloud features of the point cloud data, and use the image backbone network in the target detection model to extract the image features of the image data;
[0008] For each of the sample data sets, feature fusion is performed on the point cloud features and the image features to obtain the features to be detected of the sample data set;
[0009] Performing target detection on each of the features to be detected to obtain target detection results for each of the sample data sets;
[0010] According to the respective regions of interest of the sample data sets, comparing the target detection results with the true values of the sample data sets, and determining the model loss of the target detection model;
[0011] The object detection model is adjusted based on the model loss.
[0012] In one embodiment of the present application, before inputting each of the sample data sets into the target detection model to be trained, the method further includes:
[0013] Based on the detection range and ground height of each sample data set, point cloud input alignment is performed on each point cloud data to obtain each point cloud data with a unified detection range and ground height;
[0014] De-distortion processing is performed on the image data in each of the sample data sets, and image input alignment is performed on each of the image data to obtain each of the image data with a unified feature space.
[0015] In one embodiment of the present application, for each of the sample data sets, the point cloud features and the image features are fused to obtain the features to be detected of the sample data set, including:
[0016] For each of the sample data sets, estimating the depth information of the image feature, and converting the image feature into a bird's-eye view space based on the depth information to obtain a three-dimensional image feature;
[0017] The point cloud features and the three-dimensional image features are fused using a feature fusion network in the target detection model to obtain the features to be detected.
[0018] In one embodiment of the present application, comparing each of the target detection results with the true value of each of the sample data sets to determine the model loss of the target detection model includes:
[0019] For each of the sample data sets, obtaining a region of interest of the sample data set, and determining a mask of interest of the sample data set according to the region of interest;
[0020] Determine the detection target within the mask of interest in the target detection result, compare the detection target with the true value of the sample data set, and obtain the center point loss and size loss of the target detection model;
[0021] Comparing the classification results of each of the detection targets in the mask of interest with the classification results of each of the detection targets in the true value to obtain the classification loss of the target detection model;
[0022] The model loss is determined based on the center point loss, the size loss, and the classification loss.
[0023] In one embodiment of the present application, the target detection model includes a detection network, the detection network includes multiple levels of detection modules with different classification detection difficulties, and the target detection is performed on each of the features to be detected to obtain the target detection results of each of the sample data sets, including:
[0024] For the detection modules at each level, the detection modules are used to perform target detection on the features to be detected, so as to obtain target detection results corresponding to the detection modules at each level.
[0025] In one embodiment of the present application, the calculating the classification loss of the target detection model according to the mask of interest, the target detection result and the true value of the sample data set includes:
[0026] Obtain a target heat map of the target detection results corresponding to each level of detection module, and a true value heat map corresponding to the true value of the sample data set, wherein the heat points in the target heat map and the true value heat map are detection targets;
[0027] Mapping the mask of interest to the target heat map of the first level, comparing the target heat map with the true value heat map using a classification loss function to obtain a first classification loss, and determining a first heat point mask based on the heat points in the target heat map, wherein the first level represents the level with the lowest classification detection difficulty among the detection modules of each level;
[0028] Mapping the mask of interest and the first heat point mask to the target heat map of the second level, comparing the target heat map with the true value heat map using the classification loss function to obtain a second classification loss, and determining a second heat point mask based on the heat points in the target heat map until the classification losses of the target heat maps of each level are obtained;
[0029] The classification loss of the target detection model is determined based on the classification loss of the target heat map at each level.
[0030] In one embodiment of the present application, the model loss of the target detection model is calculated according to the following formula:
[0031] ;
[0032] ;
[0033] ;
[0034] ;
[0035] Among them, loss represents the model loss, l ctr_mask represents the center point loss, l size_mask represents the size loss, l cls_mask represents the classification loss, mask represents the mask of interest, y t represents the true value, y s represents the target detection result, n represents the detected target in the mask of interest, and i represents the i-th detected target.
[0036] In a second aspect, an embodiment of the present application provides a training device for a target detection model, comprising:
[0037] A data acquisition module, used to acquire a plurality of sample data sets, wherein each of the sample data sets includes respective point cloud data and image data;
[0038] A feature extraction module, used to input each of the sample data sets into the target detection model to be trained, and for each of the sample data sets, to extract point cloud features of the point cloud data using the three-dimensional backbone network in the target detection model, and to extract image features of the image data using the image backbone network in the target detection model;
[0039] A feature fusion module, used for fusing the point cloud features and the image features for each sample data set to obtain the features to be detected of the sample data set;
[0040] A target detection module is used to perform target detection on each of the features to be detected, and obtain target detection results for each of the sample data sets;
[0041] A model loss determination module, used to compare each of the target detection results with the true value of each of the sample data sets to determine the model loss of the target detection model;
[0042] A model adjustment module is used to adjust the target detection model based on the model loss.
[0043] In one embodiment of the present application, it also includes:
[0044] A point cloud data input alignment module, used to perform point cloud input alignment on each of the point cloud data based on the detection range and ground height of each of the sample data sets, to obtain each of the point cloud data with a unified detection range and ground height;
[0045] The image data input alignment module is used to perform dedistortion processing on the image data in each of the sample data sets, and perform image input alignment on each of the image data to obtain each of the image data with a unified feature space.
[0046] In one embodiment of the present application, the feature fusion module includes:
[0047] A feature conversion submodule, for estimating the depth information of the image feature for each of the sample data sets, and converting the image feature into a bird's-eye view space based on the depth information to obtain a three-dimensional image feature;
[0048] The feature fusion submodule is used to fuse the point cloud features and the three-dimensional image features using the feature fusion network in the target detection model to obtain the features to be detected.
[0049] In one embodiment of the present application, the model loss determination module includes:
[0050] An interest mask determination submodule, used for acquiring an interest region of the sample data set for each of the sample data sets, and determining an interest mask of the sample data set according to the interest region;
[0051] A first loss calculation submodule, used to determine the detection target within the mask of interest in the target detection result, compare the detection target with the true value of the sample data set, and obtain the center point loss and size loss of the target detection model;
[0052] A second loss calculation submodule, used to compare the classification results of each detection target in the mask of interest and the classification results of each detection target in the true value to obtain the classification loss of the target detection model;
[0053] A model loss determination submodule is used to determine the model loss based on the center point loss, the size loss and the classification loss.
[0054] In one embodiment of the present application, the target detection model includes a detection network, the detection network includes multiple levels of detection modules with different classification detection difficulties, and the model adjustment module is specifically used to:
[0055] For the detection modules at each level, the detection modules are used to perform target detection on the features to be detected, so as to obtain target detection results corresponding to the detection modules at each level.
[0056] In one embodiment of the present application, the second loss calculation submodule is specifically used to:
[0057] Obtain a target heat map of the target detection results corresponding to each level of detection module, and a true value heat map corresponding to the true value of the sample data set, wherein the heat points in the target heat map and the true value heat map are detection targets;
[0058] Mapping the mask of interest to the target heat map of the first level, comparing the target heat map with the true value heat map using a classification loss function to obtain a first classification loss, and determining a first heat point mask based on the heat points in the target heat map, wherein the first level represents the level with the lowest classification detection difficulty among the detection modules of each level;
[0059] Mapping the mask of interest and the first heat point mask to the target heat map of the second level, comparing the target heat map with the true value heat map using the classification loss function to obtain a second classification loss, and determining a second heat point mask based on the heat points in the target heat map until the classification losses of the target heat maps of each level are obtained;
[0060] The classification loss of the target detection model is determined based on the classification loss of the target heat map at each level.
[0061] In a third aspect, an embodiment of the present application provides an electronic device, including:
[0062] Memory, used to store computer programs;
[0063] The processor is used to implement any of the above-mentioned target detection model training methods when executing the program stored in the memory.
[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned target detection model training methods.
[0065] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described methods for training a target detection model.
[0066] Beneficial effects of the embodiments of the present application:
[0067] The training method of the target detection model provided in the embodiment of the present application obtains multiple sample data sets including point cloud data and image data, inputs each sample data set into the target detection model to be trained, uses the three-dimensional backbone network in the target detection model to extract the point cloud features of the point cloud data, and uses the image backbone network in the target detection model to extract the image features of the image data; for each sample data set, performs feature fusion on the point cloud features and the image features to obtain the features to be detected of the sample data set; performs target detection on each feature to be detected to obtain the target detection result of each sample data set; according to the respective regions of interest of each sample data set, compares each target detection result with the true value of each sample data set, determines the model loss of the target detection model, and only focuses on the target detection results in the regions of interest of each data set, thereby improving the accuracy and efficiency of the model loss calculation; adjusts the target detection model based on the model loss. By performing target detection on the features to be detected obtained by fusing the point cloud features and the image features, it is realized that the target detection model can be trained using multiple types of data at the same time; multiple sample data sets are input into the same target detection model, and target detection and model training are performed on each sample data set respectively, so as to realize the support of multi-sensor, multi-dataset training-single dataset testing paradigm, and improve the training efficiency of the target detection model. The target detection model based on this training can be applied to multiple data sets, which expands the application scenarios of the target detection model and improves the generalization ability of the target detection model in different data sets.
[0068] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.
[0070] Figure 1 A schematic diagram of a flow chart of a first target detection model training method provided in an embodiment of the present application;
[0071] Figure 2 A schematic diagram of a flow chart of a second method for training a target detection model provided in an embodiment of the present application;
[0072] Figure 3 A possible implementation of step S103 provided in an embodiment of the present application;
[0073] Figure 4-1 A possible implementation of step S105 provided in an embodiment of the present application;
[0074] Figure 4-2 An example diagram of a mask of interest of a multi-sample data set provided in an embodiment of the present application;
[0075] Figure 5-1 A possible implementation of step S403 provided in an embodiment of the present application;
[0076] Figure 5-2 An example diagram of a multi-level thermal point mask provided in an embodiment of the present application;
[0077] Figure 6 An example diagram of a method for training a target detection model provided in an embodiment of the present application;
[0078] Figure 7 A schematic diagram of the structure of a training device for a target detection model provided in an embodiment of the present application;
[0079] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0080] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.
[0081] Since the target detection model in the related art has low generalization ability in different data sets, in order to solve this problem, the embodiments of the present application provide a training method, device and electronic device for a target detection model.
[0082] The following is a detailed description of the training method of the target detection model provided in the embodiment of the present application. Figure 1 , Figure 1 A flowchart of a first target detection model training method is provided for an embodiment of the present application, including:
[0083] Step S101, obtaining multiple sample data sets.
[0084] Each of the sample data sets has its own region of interest and includes its own point cloud data and image data.
[0085] The region of interest for each sample data set is different. The region of interest indicates the data range that needs to be paid attention to in the sample data set, which can be set according to user needs. For example, if the user needs data in a certain area, the sensor acquisition range corresponding to the area is used as the region of interest. Specifically, in the training scenario of the target detection model, the region of interest is a known range that can be determined at the same time when the sample data set is obtained; in the application scenario of the trained target detection model, the region of interest can be a known range of the data set to be detected before it is obtained, or a known range used to filter data after it is obtained.
[0086] The sample data set is data collected by multiple sensors on the same scene at the same time, such as multiple types of data collected by various types of sensors such as lidar and image collectors. Specifically, the sample data set includes three-dimensional point cloud data and two-dimensional image data.
[0087] Each sample data set may have no association with each other, and may be data collected in different scenarios, at different collection times, and using different sensors. Each sample data set may include point cloud data collected by its own laser radar, image data collected by a camera, etc.
[0088] The sample data set is data with pre-labeled true values. The true values represent the correct information of the target in the scene corresponding to the sample data set. Specifically, it can include detailed information such as the classification of the target and the number of targets corresponding to each classification. For example, in an autonomous driving scene, the classification of targets can include pedestrians, traffic signs, roadblocks, motor vehicles, non-motor vehicles, etc.
[0089] Step S102, input each sample data set into the target detection model to be trained, use the three-dimensional backbone network in the target detection model to extract the point cloud features of the point cloud data, and use the image backbone network in the target detection model to extract the image features of the image data.
[0090] Multiple sample data sets are input into the same model, i.e., the target detection model to be trained. In one embodiment of the present application, Figure 2 As shown, before each sample data set is input into the target detection model to be trained, the method further includes:
[0091] Step S201, based on the detection range and ground height of each sample data set, performing point cloud input alignment on each point cloud data to obtain each point cloud data having a unified detection range and ground height;
[0092] Step S202, performing dedistortion processing on the image data in each of the sample data sets, and performing image input alignment on each of the image data to obtain each of the image data with a unified feature space.
[0093] Since different data sets may have different detection ranges, sensor installation locations, and sensor parameter settings when collecting data, it is necessary to first perform input alignment on each type of data (point cloud data and image data) in each sample data set. Specifically, this can include alignment processing such as unified detection range, unified ground height, and unified image input.
[0094] Exemplarily, the detection range of each sample data set is obtained, and the union of each detection range is taken as a unified detection range; the coordinate system of the sensor corresponding to each point cloud data is obtained, each coordinate system is translated, and the ground point cloud height of each point cloud data is aligned; each image data is dedistorted. Specifically, any image dedistortion algorithm can be used, such as bilinear interpolation, fixed point iteration and other processing methods. Different means and variances are used to calculate the dedistorted image data, so that they come to the same feature space and unify the image input, thereby avoiding input errors caused by different camera parameter settings and different camera models.
[0095] After input alignment of each sample data set, data enhancement is also performed on each sample data set. Specifically, any data enhancement method can be used, such as rotation and translation of each sample data set, line bundle sampling, etc., and gt-sample (ground truth-sample, target sampling) can also be performed between each sample data set. Specifically, the image target frame is randomly sampled in each image data, and the image corresponding to the image target frame is copied and pasted into each image data to obtain each image data after data enhancement; the point cloud target frame is randomly sampled in each point cloud data, and the point cloud with a preset line bundle value (low line bundle) is randomly sampled for the high line bundle point cloud in the point cloud target frame, and the point cloud is copied and pasted into each point cloud data to obtain each point cloud data after data enhancement. The generalization of the target detection model is improved.
[0096] After input alignment and data enhancement for each sample data set, the three-dimensional backbone network in the target detection model is used to extract the point cloud features of each point cloud data, and the image backbone network in the target detection model is used to extract the image features of each image data.
[0097] In one example, after the 3D backbone network extracts point cloud features, the 2D backbone network further extracts point cloud features from each point cloud data to improve the integrity of the 3D feature extraction.
[0098] Step S103: for each of the sample data sets, feature fusion is performed on the point cloud features and the image features to obtain the features to be detected of the sample data set.
[0099] For each sample data set, the feature fusion network in the target detection model is used to fuse the point cloud features and image features to obtain the features to be detected for each sample data set.
[0100] Step S104, performing target detection on each of the features to be detected to obtain target detection results for each of the sample data sets.
[0101] Step S105 , comparing each of the target detection results with the true value of each of the sample data sets according to the respective regions of interest of each of the sample data sets, and determining the model loss of the target detection model.
[0102] Step S106: adjusting the target detection model based on the model loss.
[0103] When comparing the difference between the target detection result of each sample data set and the true value of the sample data set, only the target detection result in the region of interest of the sample data set is focused on to determine the model loss of the target detection model. The target detection model is adjusted based on the model loss until the target detection model training is completed.
[0104] As can be seen from the above, the training method of the target detection model provided in the embodiment of the present application obtains multiple sample data sets including point cloud data and image data, and inputs each sample data set into the target detection model to be trained, and uses the three-dimensional backbone network in the target detection model to extract the point cloud features of the point cloud data, and uses the image backbone network in the target detection model to extract the image features of the image data; for each sample data set, the point cloud features and the image features are feature fused to obtain the features to be detected of the sample data set; target detection is performed on each feature to be detected to obtain the target detection results of each sample data set; according to the respective regions of interest of each sample data set, each target detection result is compared with the true value of each sample data set to determine the model loss of the target detection model, and only pays attention to the target detection results in the regions of interest of each data set, thereby improving the accuracy and efficiency of the model loss calculation; the target detection model is adjusted based on the model loss. By performing target detection on the features to be detected obtained by fusing point cloud features and image features, it is possible to train the target detection model using multiple types of data at the same time; multiple sample data sets are input into the same target detection model, and target detection and model training are performed for each sample data set respectively, which supports the multi-sensor, multi-dataset training-single-dataset testing paradigm and improves the training efficiency of the target detection model. The target detection model based on this training can be applied to multiple data sets, expanding the application scenarios of the target detection model and improving the generalization ability of the target detection model in different data sets.
[0105] In one embodiment of the present application, Figure 3As shown, the above step S103 performs feature fusion on the point cloud features and the image features for each sample data set to obtain features to be detected, including:
[0106] Step S301 : for each of the sample data sets, estimate the depth information of the image feature, and convert the image feature into a bird's-eye view space based on the depth information to obtain a three-dimensional image feature.
[0107] Step S302: using the feature fusion network in the target detection model to fuse the point cloud features and the three-dimensional image features to obtain the features to be detected.
[0108] According to the depth information of the image features, the two-dimensional image features are converted to the bird's eye view space (BEV, Bird's Eye View), which refers to the decision-friendly space, to obtain the three-dimensional image features, so as to realize the conversion of the image features and the point cloud features into the same three-dimensional coordinate system (BEV coordinate system).
[0109] Exemplarily, a bottom-up approach can be used to convert two-dimensional image features into three-dimensional features in the BEV space. Specifically, the image features can be converted into the bird's-eye view space based on the LSS method (Lift, Splat, Shoot, a bottom-up three-dimensional feature construction method). First, the image features are LIFT-ed, and each pixel in the image is mapped into the three-dimensional space according to the internal and external parameters of the camera, so that the image features are lifted from the two-dimensional features to the three-dimensional feature coordinate system, and the final three-dimensional image features are obtained.
[0110] The feature fusion network in the target detection model is used to fuse the point cloud features and the three-dimensional image features. Specifically, the feature fusion network can adopt any feature fusion method according to actual needs to realize the fusion of point cloud features and three-dimensional image features, such as additive fusion, weighted average fusion, direct splicing fusion, multi-scale feature fusion, etc.
[0111] As can be seen from the above, the training method of the target detection model provided by the embodiment of the present application generates multiple depth values for each pixel in the image data according to the camera parameters, explicitly estimates the depth information of the image data, solves the problem of missing depth information when converting from a two-dimensional image to a three-dimensional space, and realizes the conversion of image features into three-dimensional image features, so that the image features and the point cloud features are in the same feature dimension, and then uses the feature fusion network in the target detection model to fuse the point cloud features and the three-dimensional image features, reasonably combines the point cloud features and the three-dimensional image features, and uses the features of different types of data to perform target detection. Thereby improving the performance and generalization ability of the model.
[0112] In one embodiment of the present application, Figure 4-1 As shown, the above step S105 compares each target detection result with the true value of each sample data set to determine the model loss of the target detection model, including:
[0113] Step S401, for each of the sample data sets, obtaining a region of interest of the sample data set, and determining a mask of interest of the sample data set according to the region of interest;
[0114] Step S402, determining the detection target within the mask of interest in the target detection result, comparing the detection target with the true value of the sample data set, and obtaining the center point loss and size loss of the target detection model;
[0115] Step S403, comparing the classification results of each detection target in the mask of interest and the classification results of each detection target in the true value to obtain the classification loss of the target detection model;
[0116] Step S404: determining the model loss based on the center point loss, the size loss and the classification loss.
[0117] The region of interest of a sample dataset indicates the data range that needs to be paid attention to in the sample dataset. It can be set according to user needs. Different sample datasets can have different regions of interest. The mask of interest of the sample dataset is generated based on the region of interest. When calculating the model loss, only the error of the detection result within the mask of interest is calculated. For example Figure 4-2 As shown, different masks of interest are used for different sample data sets (dataset 1 and data set 2) so that the target detection model can be applied to multiple data sets, thereby improving the generalization ability of the target detection model on multiple data sets.
[0118] Center loss refers to the position error of the center point of the detected target. Specifically, the target detection result includes the target frame of each detected target, including the center point coordinates, width and height of the target frame. The center point coordinates are compared with the center point coordinates of each target in the true value to obtain the error of the center point coordinates of the detected target and the center point loss; the width and height of the target frame of the detected target are compared with the width and height of each target in the true value to obtain the size loss of the detected target.
[0119] Specifically, both the center point loss and the size loss are calculated only for the detected objects within the mask of interest.
[0120] Exemplarily, the parameters of the target frame may include ,in Indicates the coordinates of the center point of the target frame. Represents the width, length, height, and angle of the box.
[0121] Compare the classification results of each detected target in the mask of interest with the classification results in the true value, and calculate the classification loss of the target detection model.
[0122] The model loss is determined based on the center point loss, size loss and classification loss. The center point loss, size loss and classification loss can be directly added together or weighted addition is performed with different weights, so as to obtain the model loss. The specific setting can be based on actual needs.
[0123] As can be seen from the above, the training method of the target detection model provided in the embodiment of the present application obtains the mask of interest based on the range of interest of the data set, and only compares each detection target in the mask of interest with each detection target in the true value to obtain the model loss of the target detection model, thereby realizing the calculation of multiple losses of the target detection model and improving the accuracy of the target detection model. In addition, for different sample data sets, different masks of interest are determined according to their respective ranges of interest, and then the model loss is calculated, which solves the problem of model loss calculation errors that may be caused by inconsistencies in the regions of interest of different data sets, and further improves the generalization ability of the target detection model on multiple data sets.
[0124] In one embodiment of the present application, the target detection model includes a detection network, the detection network includes multiple levels of detection modules with different classification detection difficulties, and the target detection is performed on each of the features to be detected to obtain the target detection results of each sample data set, including:
[0125] For the detection modules at each level, the detection modules are used to perform target detection on the features to be detected, so as to obtain target detection results corresponding to the detection modules at each level.
[0126] When performing target detection on the target to be detected, multi-level classification detection is performed on each detection target. The classification difficulty that can be achieved at each level is different, and the target detection results obtained are also different, thus obtaining multi-level target detection results.
[0127] As can be seen from the above, the training method of the target detection model provided in the embodiment of the present application uses detection modules of multiple levels with different classification detection difficulties to perform multi-level detection on the features to be detected, and obtains multi-level target detection results. The model loss is calculated based on this to adjust the model, thereby further improving the accuracy of the target detection model.
[0128] In one embodiment of the present application, Figure 5-1 As shown, the above step S403 compares the classification results of each detection target in the mask of interest with the classification results of each detection target in the true value to obtain the classification loss of the target detection model, including:
[0129] Step S501, obtaining a target heat map of target detection results corresponding to each level of detection module, and a true value heat map corresponding to the true value of the sample data set.
[0130] Wherein, the thermal points in the target thermal map and the true value thermal map are detection targets;
[0131] Step S502, mapping the mask of interest to the target heat map of the first level, comparing the target heat map with the true value heat map using a classification loss function to obtain a first classification loss, and determining a first heat point mask based on the heat points in the target heat map;
[0132] The first level represents the level with the lowest classification detection difficulty among the detection modules at each level;
[0133] Step S503, mapping the mask of interest and the first thermal point mask to the target thermal map of the second level, using the classification loss function to compare the target thermal map with the true value thermal map to obtain a second classification loss, and determining the second thermal point mask based on the thermal points in the target thermal map, until the classification losses of the target thermal maps of each level are obtained;
[0134] Step S504, determining the classification loss of the target detection model based on the classification loss of the target heat map at each level.
[0135] Thermal spot mask Figure 5-2 As shown in the figure, a multi-level detection head is used to perform target detection, and target heat maps of target detection results of multiple levels are obtained. The heat points (detection targets) that overlap with the target heat map (heatmap1) of the second level and the target heat map (heatmap0) of the first level are covered, and only the classification loss between the heat points in the second level and the true value is calculated, and so on, until the classification loss of the target heat map of the last level is calculated.
[0136] Add the classification losses of each level or set different weights for weighted addition to obtain the classification loss of the target detection model.
[0137] In one embodiment of the present application, the model loss of the target detection model is calculated according to the following formula:
[0138] ;
[0139] ;
[0140] ;
[0141] ;
[0142] Among them, loss represents the model loss, l ctr_mask represents the center point loss, l size_mask represents the size loss, l cls_mask represents the classification loss, mask represents the mask of interest, y t represents the true value, y s represents the target detection result, n represents the detected target in the mask of interest, and i represents the i-th detected target.
[0143] As can be seen from the above, the target detection model training method provided in the embodiment of the present application calculates the classification loss for the target detection results of multiple levels in sequence, thereby improving the accuracy of the model loss calculation; and, the heat point mask of the previous level is mapped to the heat map and the true value heat map of the next level, and then the classification loss calculation of the next level is performed, thereby avoiding repeated calculations for the same detection target, improving the efficiency of the model loss calculation and even the target detection model training, and improving the model's detection capabilities for targets of different difficulty levels.
[0144] In one embodiment of the present application, Figure 6 As shown, an example diagram of a training method for a target detection model is provided. Sample dataset 1 and sample dataset 2 both include point cloud data and image data. The point cloud data of the two datasets are ground aligned and point cloud multi-dataset enhancement processing is performed, and the image data is dedistorted and image multi-dataset enhancement processing is performed. The multi-dataset enhancement processing is achieved by methods such as rotation and translation, line beam sampling, and target sampling between datasets. The point cloud data is sequentially passed through the 3D backbone network and the 2D backbone network to extract features to obtain point cloud features; the image data is passed through the image backbone network to extract features, and the image features are converted to the BEV space and fused with the point cloud features. The fused features are then used for target detection using the multi-dataset detection head.
[0145] See also Figure 7 , the embodiment of the present application also provides a structural schematic diagram of a training device for a target detection model, including:
[0146] The data acquisition module 701 is used to acquire multiple sample data sets, wherein each of the sample data sets includes respective point cloud data and image data;
[0147] A feature extraction module 702 is used to input each of the sample data sets into the target detection model to be trained, extract point cloud features of the point cloud data using the three-dimensional backbone network in the target detection model, and extract image features of the image data using the image backbone network in the target detection model;
[0148] A feature fusion module 703 is used to perform feature fusion on the point cloud feature and the image feature for each sample data set to obtain a feature to be detected of the sample data set;
[0149] The target detection module 704 is used to perform target detection on each of the features to be detected, and obtain target detection results for each of the sample data sets;
[0150] A model loss determination module 705 is used to compare each target detection result with the true value of each sample data set to determine the model loss of the target detection model;
[0151] The model adjustment module 706 is used to adjust the target detection model based on the model loss.
[0152] As can be seen from the above, the training device of the target detection model provided by the embodiment of the present application obtains multiple sample data sets including point cloud data and image data, and inputs each sample data set into the target detection model to be trained, and uses the three-dimensional backbone network in the target detection model to extract the point cloud features of the point cloud data, and uses the image backbone network in the target detection model to extract the image features of the image data; for each sample data set, the point cloud features and the image features are feature fused to obtain the features to be detected of the sample data set; target detection is performed on each feature to be detected to obtain the target detection results of each sample data set; each target detection result is compared with the true value of each sample data set to determine the model loss of the target detection model; and the target detection model is adjusted based on the model loss. By performing target detection on the features to be detected obtained by fusing the point cloud features and the image features, it is possible to train the target detection model using multiple types of data at the same time; multiple sample data sets are input into the same target detection model, and target detection and model training are performed on each sample data set respectively, which supports the multi-sensor, multi-dataset training-single data set testing paradigm and improves the training efficiency of the target detection model. The target detection model based on this training can be applied to multiple data sets, which expands the application scenarios of the target detection model and improves the generalization ability of the target detection model in different data sets.
[0153] In one embodiment of the present application, the device further includes:
[0154] A point cloud data input alignment module, used to perform point cloud input alignment on each of the point cloud data based on the detection range and ground height of each of the sample data sets, to obtain each of the point cloud data with a unified detection range and ground height;
[0155] The image data input alignment module is used to perform dedistortion processing on the image data in each of the sample data sets, and perform image input alignment on each of the image data to obtain each of the image data with a unified feature space.
[0156] In one embodiment of the present application, the feature fusion module 703 includes:
[0157] A feature conversion submodule, for estimating the depth information of the image feature for each of the sample data sets, and converting the image feature into a bird's-eye view space based on the depth information to obtain a three-dimensional image feature;
[0158] The feature fusion submodule is used to fuse the point cloud features and the three-dimensional image features using the feature fusion network in the target detection model to obtain the features to be detected.
[0159] As can be seen from the above, the training device of the target detection model provided by the embodiment of the present application generates multiple depth values for each pixel in the image data according to the camera parameters, explicitly estimates the depth information of the image data, solves the problem of missing depth information when converting from a two-dimensional image to a three-dimensional space, and realizes the conversion of image features into three-dimensional image features, so that the image features and the point cloud features are in the same feature dimension, and then uses the feature fusion network in the target detection model to fuse the point cloud features and the three-dimensional image features, reasonably combines the point cloud features and the three-dimensional image features, and uses the features of different types of data to perform target detection. Thereby improving the performance and generalization ability of the model.
[0160] In one embodiment of the present application, the model loss determination module 705 includes:
[0161] An interest mask determination submodule, used for acquiring an interest region of the sample data set for each of the sample data sets, and determining an interest mask of the sample data set according to the interest region;
[0162] A first loss calculation submodule, used to determine the detection target within the mask of interest in the target detection result, compare the detection target with the true value of the sample data set, and obtain the center point loss and size loss of the target detection model;
[0163] A second loss calculation submodule, used to compare the classification results of each detection target in the mask of interest and the classification results of each detection target in the true value to obtain the classification loss of the target detection model;
[0164] A model loss determination submodule is used to determine the model loss based on the center point loss, the size loss and the classification loss.
[0165] As can be seen from the above, the training device of the target detection model provided by the embodiment of the present application obtains the mask of interest based on the range of interest of the data set, and only compares each detection target in the mask of interest with each detection target in the true value to obtain the model loss of the target detection model, thereby realizing the calculation of multiple losses of the target detection model and improving the accuracy of the target detection model. In addition, for different sample data sets, different masks of interest are determined according to their respective ranges of interest, and then the model loss is calculated, which solves the problem of model loss calculation errors that may be caused by inconsistencies in the regions of interest of different data sets, and further improves the generalization ability of the target detection model on multiple data sets.
[0166] In one embodiment of the present application, the target detection model includes a detection network, the detection network includes multiple levels of detection modules with different classification detection difficulties, and the model adjustment module is specifically used to:
[0167] For the detection modules at each level, the detection modules are used to perform target detection on the features to be detected, so as to obtain target detection results corresponding to the detection modules at each level.
[0168] As can be seen from the above, the training device for the target detection model provided in the embodiment of the present application performs multi-level detection on the features to be detected by using detection modules at multiple levels with different classification detection difficulties, and obtains multi-level target detection results. The model loss is calculated based on this to adjust the model, thereby further improving the accuracy of the target detection model.
[0169] In one embodiment of the present application, the second loss calculation submodule is specifically used to:
[0170] Obtain a target heat map of the target detection results corresponding to each level of detection module, and a true value heat map corresponding to the true value of the sample data set, wherein the heat points in the target heat map and the true value heat map are detection targets;
[0171] Mapping the mask of interest to the target heat map of the first level, comparing the target heat map with the true value heat map using a classification loss function to obtain a first classification loss, and determining a first heat point mask based on the heat points in the target heat map, wherein the first level represents the level with the lowest classification detection difficulty among the detection modules of each level;
[0172] Mapping the mask of interest and the first heat point mask to the target heat map of the second level, comparing the target heat map with the true value heat map using the classification loss function to obtain a second classification loss, and determining a second heat point mask based on the heat points in the target heat map until the classification losses of the target heat maps of each level are obtained;
[0173] The classification loss of the target detection model is determined based on the classification loss of the target heat map at each level.
[0174] As can be seen from the above, the training device of the target detection model provided in the embodiment of the present application calculates the classification loss for the target detection results of multiple levels in sequence, thereby improving the accuracy of the model loss calculation; and, maps the heat point mask of the previous level to the heat map and the true value heat map of the next level, and then performs the classification loss calculation of the next level, thereby avoiding repeated calculations for the same detection target, improving the efficiency of the model loss calculation and even the target detection model training, and improving the model's detection capabilities for targets of different difficulty levels.
[0175] The present application also provides an electronic device, such as Figure 8 As shown, including:
[0176] Memory 801, used for storing computer programs;
[0177] The processor 802 is used to implement any of the above-mentioned target detection model training methods when executing the program stored in the memory 801.
[0178] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 802, the communication interface, and the memory 801 communicate with each other via the communication bus.
[0179] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0180] The communication interface is used for communication between the above electronic device and other devices.
[0181] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0182] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0183] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of the training method of any of the above-mentioned target detection models are implemented.
[0184] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the training method of any target detection model in the above embodiments.
[0185] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.
[0186] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0187] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0188] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. A method for training a target detection model, characterized in that: include: Acquire a plurality of sample data sets, wherein each of the sample data sets has a respective region of interest and includes respective point cloud data and image data; Input each of the sample data sets into the target detection model to be trained, use the three-dimensional backbone network in the target detection model to extract the point cloud features of the point cloud data, and use the image backbone network in the target detection model to extract the image features of the image data; For each of the sample data sets, feature fusion is performed on the point cloud features and the image features to obtain the features to be detected of the sample data set; For each level of detection module, use the detection module to perform target detection on the feature to be detected, and obtain the target detection results corresponding to the detection modules at each level, wherein the target detection model includes a detection network, and the detection network includes multiple levels of detection modules with different classification detection difficulties; For each of the sample data sets, obtaining a region of interest of the sample data set, and determining a mask of interest of the sample data set according to the region of interest; Determine the detection target within the mask of interest in the target detection result, compare the detection target with the true value of the sample data set, and obtain the center point loss and size loss of the target detection model; Obtain a target heat map of the target detection results corresponding to each level of detection module, and a true value heat map corresponding to the true value of the sample data set, wherein the heat points in the target heat map and the true value heat map are detection targets; Mapping the mask of interest to the target heat map of the first level, comparing the target heat map with the true value heat map using a classification loss function to obtain a first classification loss, and determining a first heat point mask based on the heat points in the target heat map, wherein the first level represents the level with the lowest classification detection difficulty among the detection modules of each level; Mapping the mask of interest and the first heat point mask to the target heat map of the second level, comparing the target heat map with the true value heat map using the classification loss function to obtain a second classification loss, and determining a second heat point mask based on the heat points in the target heat map until the classification losses of the target heat maps of each level are obtained; Determining the classification loss of the target detection model based on the classification loss of the target heat map at each level; Determine the model loss based on the center point loss, the size loss and the classification loss; The object detection model is adjusted based on the model loss.
2. The method according to claim 1, characterized in that Before inputting each of the sample data sets into the target detection model to be trained, the method further includes: Based on the detection range and ground height of each sample data set, point cloud input alignment is performed on each point cloud data to obtain each point cloud data with a unified detection range and ground height; De-distortion processing is performed on the image data in each of the sample data sets, and image input alignment is performed on each of the image data to obtain each of the image data with a unified feature space.
3. The method according to claim 1, characterized in that For each of the sample data sets, the point cloud features and the image features are fused to obtain the features to be detected of the sample data set, including: For each of the sample data sets, estimating the depth information of the image feature, and converting the image feature into a bird's-eye view space based on the depth information to obtain a three-dimensional image feature; The point cloud features and the three-dimensional image features are fused using a feature fusion network in the target detection model to obtain the features to be detected.
4. The method according to claim 1, characterized in that: The model loss of the target detection model is calculated according to the following formula: ； ； ； ； Among them, loss represents the model loss, l ctr_mask represents the center point loss, l size_mask represents the size loss, l cls_mask represents the classification loss, mask represents the mask of interest, y t represents the true value, y s represents the target detection result, n represents the detected target in the mask of interest, and i represents the i-th detected target.
5. A training device for a target detection model, characterized in that: include: A data acquisition module, used to acquire a plurality of sample data sets, wherein each of the sample data sets includes respective point cloud data and image data; A feature extraction module, used to input each of the sample data sets into the target detection model to be trained, and for each of the sample data sets, to extract point cloud features of the point cloud data using the three-dimensional backbone network in the target detection model, and to extract image features of the image data using the image backbone network in the target detection model; A feature fusion module, used for fusing the point cloud features and the image features for each sample data set to obtain the features to be detected of the sample data set; A target detection module, for each level of detection module, using the detection module to perform target detection on the to-be-detected feature, and obtaining target detection results corresponding to the detection modules at each level, wherein the target detection model includes a detection network, and the detection network includes detection modules at multiple levels with different classification detection difficulties; Model loss determination module, including: An interest mask determination submodule, used for acquiring an interest region of the sample data set for each of the sample data sets, and determining an interest mask of the sample data set according to the interest region; A first loss calculation submodule, used to determine the detection target within the mask of interest in the target detection result, compare the detection target with the true value of the sample data set, and obtain the center point loss and size loss of the target detection model; A second loss calculation submodule is used to obtain a target heat map of the target detection results corresponding to each level of detection module, and a true value heat map corresponding to the true value of the sample data set, wherein the heat points in the target heat map and the true value heat map are detection targets; Mapping the mask of interest to the target heat map of the first level, comparing the target heat map with the true value heat map using a classification loss function to obtain a first classification loss, and determining a first heat point mask based on the heat points in the target heat map, wherein the first level represents the level with the lowest classification detection difficulty among the detection modules of each level; Mapping the mask of interest and the first heat point mask to the target heat map of the second level, comparing the target heat map with the true value heat map using the classification loss function to obtain a second classification loss, and determining a second heat point mask based on the heat points in the target heat map until the classification losses of the target heat maps of each level are obtained; Determining the classification loss of the target detection model based on the classification loss of the target heat map at each level; A model loss determination submodule, used to determine the model loss based on the center point loss, the size loss and the classification loss; A model adjustment module, used to adjust the target detection model based on the model loss; The device also includes: A point cloud data input alignment module, used to perform point cloud input alignment on each of the point cloud data based on the detection range and ground height of each of the sample data sets, to obtain each of the point cloud data with a unified detection range and ground height; An image data input alignment module, used to perform dedistortion processing on the image data in each of the sample data sets, and perform image input alignment on each of the image data to obtain each of the image data with a unified feature space; The feature fusion module comprises: A feature conversion submodule, for estimating the depth information of the image feature for each of the sample data sets, and converting the image feature into a bird's-eye view space based on the depth information to obtain a three-dimensional image feature; The feature fusion submodule is used to fuse the point cloud features and the three-dimensional image features using the feature fusion network in the target detection model to obtain the features to be detected.
6. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-4 when executing a program stored in a memory.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.