A long-tail object detection method, a computer readable storage medium and a driving device
By combining target detection models trained in different scenarios with feature maps, the problem of limited applicability and low accuracy of long-tail target detection methods is solved, achieving higher detection accuracy and applicability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 安徽蔚来智驾科技有限公司
- Filing Date
- 2023-02-10
- Publication Date
- 2026-05-15
AI Technical Summary
Existing long-tailed object detection methods have limited applicability and low accuracy, and cannot effectively identify rare long-tailed objects.
Two object detection models are used for detection, their feature maps are fused to form a fused feature, and long-tail object detection results are generated through the detection head. The model training set comes from different scenes to enrich scene diversity.
It improves the accuracy and applicability of long-tail target detection, and overcomes the cold start and scene migration problems of single models in long-tail target detection.
Smart Images
Figure CN116071557B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection technology, specifically providing a long-tail target detection method, a computer-readable storage medium, and a driving device. Background Technology
[0002] Real-world scene datasets are fundamental to developing autonomous driving capabilities. Different perception tasks require different data for building vehicle perception capabilities. However, the distribution of data in real-world scenarios exhibits a significant long-tail effect. The long-tail effect refers to a situation where a small subset of categories dominates the training samples, while most categories have only a small number of training samples. For example, in real-world scenarios, common categories such as vehicles, pedestrians, non-motorized vehicles, roads, and buildings appear frequently, while long-tail categories and scenes such as obstacles, animals, and accident scenes appear relatively infrequently. Directly using such imbalanced data to develop perception models can significantly reduce the recognition rate of long-tail objects, potentially leading to safety issues.
[0003] To address the aforementioned issues, some related technologies utilize neural network models to simulate various real-world scenarios and acquire simulation data under different scenarios, then train the perception model based on the simulation data. However, the simulation data obtained through simulation cannot fully reflect the ever-changing real-world scenarios and is not easy to obtain high-value long-tail data. Other related technologies employ adversarial network-based training models to encode multiple training images, obtaining multiple latent variables corresponding to scene information; a candidate image is randomly selected, and based on the latent variables and the candidate image, a second generative adversarial network-based training model generates augmented images of the candidate image under various different scene information; finally, the obtained augmented images are added to the dataset. This method can quickly generate a large amount of combined augmented data using multiple scene template images and multiple candidate images, which can alleviate the problem of low recognition rate of long-tail objects to some extent. However, since the generation range of augmented images is limited by the features of the template and candidate images, it cannot generate unseen scene samples, thus failing to solve the scene transferability problem of long-tail objects. Summary of the Invention
[0004] The present invention aims to solve the above-mentioned technical problems, namely, to solve the problems that existing long-tail target detection methods have limited applicable scenarios and low accuracy.
[0005] In a first aspect, the present invention provides a method for detecting long-tailed targets, which may include:
[0006] The first object detection model and the second object detection model are used to detect the image to be recognized to obtain the first candidate box and the second candidate box, respectively. The first object detection model and the second object detection model are trained using training sets from different scenes.
[0007] Extract the first feature corresponding to the first candidate box from the feature map obtained from the feature extraction layer of the first object detection model, and extract the second feature corresponding to the second candidate box from the feature map obtained from the feature extraction layer of the second object detection model;
[0008] The first feature and the second feature are fused to obtain a fused feature; and,
[0009] The fused features are input into the detection head to obtain the first long-tail target detection result of the image to be identified.
[0010] In some embodiments, extracting the first feature corresponding to the first candidate box from the feature map obtained from the feature extraction layer of the first object detection model includes:
[0011] Extract the first feature corresponding to the first candidate box from the feature map obtained from the backbone network and / or neck network of the first target detection model;
[0012] And / or,
[0013] Extracting the second feature corresponding to the second candidate box from the feature map obtained from the feature extraction layer of the second object detection model includes:
[0014] Extract the second feature corresponding to the second candidate box from the feature map obtained from the backbone network and / or neck network of the second object detection model.
[0015] In some embodiments, extracting the first feature corresponding to the first candidate box from the feature map obtained from the backbone network and / or neck network of the first target detection model includes:
[0016] Extract the initial first feature corresponding to the first candidate box from the feature map obtained from the backbone network and / or neck network of the first target detection model;
[0017] The initial first feature of a relatively small size is upsampled to obtain the first feature of the target size.
[0018] In some embodiments, extracting the second feature corresponding to the second candidate box from the feature map obtained from the backbone network and / or neck network of the second target detection model includes:
[0019] Extract the initial second feature corresponding to the second candidate box from the feature map obtained from the backbone network and / or neck network of the second target detection model;
[0020] The initial second feature of the relatively small size is upsampled to obtain the second feature of the target size.
[0021] In some embodiments, fusing the first feature and the second feature to obtain a fused feature includes:
[0022] The first feature and the second feature are subjected to bilinear pooling to obtain the fused feature.
[0023] And / or,
[0024] After obtaining the fused features and before inputting the fused features into the detection head, the method further includes: performing feature dimensionality reduction and normalization on the fused features.
[0025] In some embodiments, the long-tailed target detection method further includes:
[0026] Based on the first long-tail target detection result and the second long-tail target detection result detected by the second target detection model, the final long-tail target detection result is determined.
[0027] In some embodiments, determining the final long-tail target detection result based on the first long-tail target detection result and the second long-tail target detection result detected by the second target detection model includes:
[0028] When the second long-tail target detection result is one, the second long-tail target detection result and the first long-tail target detection result are compared. If they match, the first long-tail target detection result is determined as the final long-tail target detection result.
[0029] When there are multiple second long-tail target detection results, and these multiple second long-tail target detection results come from multiple second target detection models, a voting mechanism is used to determine the final long-tail target detection result based on the multiple second long-tail target detection results and the first long-tail target detection result.
[0030] In some embodiments, the long-tailed target detection method further includes:
[0031] Obtain the initial sample;
[0032] The initial samples are processed to construct a long-tail target sample set, wherein the long-tail target sample set corresponds to an autonomous driving scenario;
[0033] The first target detection model to be trained is trained based on the long-tail target sample set to obtain the trained first target detection model.
[0034] In some embodiments, processing the initial samples to construct a long-tail target sample set includes:
[0035] Image classification is performed on the image samples in the initial sample to obtain long-tail target samples related to the autonomous driving scenario; and / or
[0036] Image segmentation is performed on the image samples in the initial sample to obtain long-tail target samples related to the autonomous driving scenario; and / or
[0037] Image detection is performed on the image samples in the initial sample to obtain long-tail target samples related to the autonomous driving scenario; and / or
[0038] Image features are extracted from the source image and the image samples in the initial sample using a convolutional neural network, and the feature distance between the source image and the image samples is calculated based on the image features; long-tail target samples related to the autonomous driving scenario are selected based on a comparison between the feature distance and a first distance threshold; and / or
[0039] Text features and image features are extracted from the text samples and image samples in the initial sample, respectively; the extracted text features and target image features are fused to obtain a first multimodal fusion feature; image features are extracted from the source image to obtain source image features; the text features and the source image features are fused to obtain a second multimodal fusion feature; the cosine distance between the second multimodal fusion feature and the first multimodal fusion feature is calculated; long-tail target samples related to the autonomous driving scenario are selected based on the comparison between the cosine distance and a second distance threshold.
[0040] In some embodiments, training the first target detection model to be trained based on the long-tail target sample set to obtain the trained first target detection model includes:
[0041] Initial training phase:
[0042] The long-tailed target sample set is input into the first target detection model to be trained to obtain the detection result of each long-tailed target sample in the long-tailed target sample set, wherein the detection result includes candidate boxes and corresponding categories;
[0043] Iterative training phase:
[0044] When the category is a long-tail target, a pseudo-long-tail target is determined based on the candidate box and the ground truth box corresponding to the category:
[0045] Construct a difficult negative sample set based on the pseudo-long-tail target;
[0046] The first target detection model to be trained is iteratively trained at least using the difficult negative sample set;
[0047] Determine whether the preset conditions are met. If they are met, stop iterative training and obtain the trained first target detection model.
[0048] In some embodiments, determining whether a preset condition is met, and stopping iterative training if the condition is met to obtain a trained first object detection model, includes:
[0049] Determine whether the false recognition rate of the first target detection model to be trained is within a preset threshold range;
[0050] If so, once the preset conditions are met, stop the iterative training.
[0051] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the long-tail target detection method described in any of the preceding claims.
[0052] A third aspect of the present invention provides a driving device including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the long-tail target detection method described in any of the preceding claims.
[0053] By employing the above technical solution, the present invention can extract a first feature corresponding to the first candidate box obtained by detecting the first candidate box in the image to be recognized using the first object detection model, and extract a second feature corresponding to the second candidate box obtained by detecting the second candidate box in the image to be recognized using the second object detection model, and extract a second feature corresponding to the second candidate box from the feature map obtained by the feature extraction layer of the second object detection model; by fusing the first feature and the second feature, a fused feature is obtained; finally, the fused feature is input into the detection head to obtain the first long-tail object detection result of the image to be recognized; wherein, the first object detection model and the second object detection model are trained using training sets from different scenes. This method, by combining object detection models trained using training sets from different scenes, is beneficial to enriching the diversity of scenes and improving the accuracy and applicability of long-tail object detection.
[0054] Solution 1. A long-tail target detection method, comprising:
[0055] The first object detection model and the second object detection model are used to detect the image to be recognized to obtain the first candidate box and the second candidate box, respectively. The first object detection model and the second object detection model are trained using training sets from different scenes.
[0056] Extract the first feature corresponding to the first candidate box from the feature map obtained from the feature extraction layer of the first object detection model, and extract the second feature corresponding to the second candidate box from the feature map obtained from the feature extraction layer of the second object detection model;
[0057] The first feature and the second feature are fused to obtain a fused feature; and,
[0058] The fused features are input into the detection head to obtain the first long-tail target detection result of the image to be identified.
[0059] Solution 2. According to the method described in Solution 1, extracting the first feature corresponding to the first candidate box from the feature map obtained from the feature extraction layer of the first target detection model includes:
[0060] Extract the first feature corresponding to the first candidate box from the feature map obtained from the backbone network and / or neck network of the first target detection model;
[0061] And / or,
[0062] Extracting the second feature corresponding to the second candidate box from the feature map obtained from the feature extraction layer of the second object detection model includes:
[0063] Extract the second feature corresponding to the second candidate box from the feature map obtained from the backbone network and / or neck network of the second object detection model.
[0064] Solution 3. According to the method described in Solution 2, extracting the first feature corresponding to the first candidate box from the feature map obtained from the backbone network and / or neck network of the first target detection model includes:
[0065] Extract the initial first feature corresponding to the first candidate box from the feature map obtained from the backbone network and / or neck network of the first target detection model;
[0066] The initial first feature of a relatively small size is upsampled to obtain the first feature of the target size.
[0067] Solution 4. According to the method described in Solution 3, extracting the second feature corresponding to the second candidate box from the feature map obtained from the backbone network and / or neck network of the second target detection model includes:
[0068] Extract the initial second feature corresponding to the second candidate box from the feature map obtained from the backbone network and / or neck network of the second target detection model;
[0069] The initial second feature of the relatively small size is upsampled to obtain the second feature of the target size.
[0070] Solution 5. According to the method described in Solution 1, the step of fusing the first feature and the second feature to obtain a fused feature includes:
[0071] The first feature and the second feature are subjected to bilinear pooling to obtain the fused feature.
[0072] And / or,
[0073] After obtaining the fused features and before inputting the fused features into the detection head, the method further includes: performing feature dimensionality reduction and normalization on the fused features.
[0074] Option 6. The method according to Option 1, further comprising:
[0075] Based on the first long-tail target detection result and the second long-tail target detection result detected by the second target detection model, the final long-tail target detection result is determined.
[0076] Solution 7. According to the method described in Solution 6, determining the final long-tail target detection result based on the first long-tail target detection result and the second long-tail target detection result detected by the second target detection model includes:
[0077] When the second long-tail target detection result is one, the second long-tail target detection result and the first long-tail target detection result are compared. If they match, the first long-tail target detection result is determined as the final long-tail target detection result.
[0078] When there are multiple second long-tail target detection results, and these multiple second long-tail target detection results come from multiple second target detection models, a voting mechanism is used to determine the final long-tail target detection result based on the multiple second long-tail target detection results and the first long-tail target detection result.
[0079] Option 8. The method according to any one of Options 1 to 7, further comprising:
[0080] Obtain the initial sample;
[0081] The initial samples are processed to construct a long-tail target sample set, wherein the long-tail target sample set corresponds to an autonomous driving scenario;
[0082] The first target detection model to be trained is trained based on the long-tail target sample set to obtain the trained first target detection model.
[0083] Solution 9. According to the method described in Solution 8, the step of processing the initial sample to construct a long-tail target sample set includes:
[0084] Image classification is performed on the image samples in the initial sample to obtain long-tail target samples related to the autonomous driving scenario; and / or
[0085] Image segmentation is performed on the image samples in the initial sample to obtain long-tail target samples related to the autonomous driving scenario; and / or
[0086] Image detection is performed on the image samples in the initial sample to obtain long-tail target samples related to the autonomous driving scenario; and / or
[0087] Image features are extracted from the source image and the image samples in the initial sample using a convolutional neural network, and the feature distance between the source image and the image samples is calculated based on the image features; long-tail target samples related to the autonomous driving scenario are selected based on a comparison between the feature distance and a first distance threshold; and / or
[0088] Text features and image features are extracted from the text samples and image samples in the initial sample, respectively; the extracted text features and target image features are fused to obtain a first multimodal fusion feature; image features are extracted from the source image to obtain source image features; the text features and the source image features are fused to obtain a second multimodal fusion feature; the cosine distance between the second multimodal fusion feature and the first multimodal fusion feature is calculated; long-tail target samples related to the autonomous driving scenario are selected based on the comparison between the cosine distance and a second distance threshold.
[0089] Solution 10. According to the method described in Solution 8, the step of training the first target detection model to be trained based on the long-tail target sample set to obtain the trained first target detection model includes:
[0090] Initial training phase:
[0091] The long-tailed target sample set is input into the first target detection model to be trained to obtain the detection result of each long-tailed target sample in the long-tailed target sample set, wherein the detection result includes candidate boxes and corresponding categories;
[0092] Iterative training phase:
[0093] When the category is a long-tail target, a pseudo long-tail target is determined based on the candidate box and the ground truth box corresponding to the category.
[0094] Construct a difficult negative sample set based on the pseudo-long-tail target;
[0095] The first target detection model to be trained is iteratively trained at least using the difficult negative sample set;
[0096] Determine whether the preset conditions are met. If they are met, stop iterative training and obtain the trained first target detection model.
[0097] Solution 11. According to the method described in Solution 10, the step of determining whether a preset condition is met, and stopping iterative training if the condition is met, to obtain a trained first object detection model, includes:
[0098] Determine whether the false recognition rate of the first target detection model to be trained is within a preset threshold range;
[0099] If so, once the preset conditions are met, stop the iterative training.
[0100] Scheme 12. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the long-tail target detection method as described in any one of Schemes 1 to 11.
[0101] Option 13. A driving device comprising a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the long-tail target detection method as described in any one of Options 1 to 11. Attached Figure Description
[0102] The preferred embodiments of the present invention are described below with reference to the accompanying drawings, in which:
[0103] Figure 1 This is a schematic diagram of a long-tail target detection method provided in an embodiment of the present invention;
[0104] Figure 2 This is a schematic diagram of a long-tailed target detection model architecture provided in an embodiment of the present invention;
[0105] Figure 3 This is a schematic diagram of the fusion process of the first feature and the second feature provided in an embodiment of the present invention;
[0106] Figure 4 This is a schematic diagram of the training method of the first target detection model provided in this embodiment of the invention.
[0107] Figure 5 This is a schematic diagram of the training architecture of the first target detection model provided in an embodiment of the present invention;
[0108] Figure 6This is a schematic diagram of another long-tailed target detection model architecture provided in an embodiment of the present invention;
[0109] Figure 7 This is a schematic diagram of a long-tail target detection method provided in another embodiment of the present invention;
[0110] Figure 8 This is a schematic diagram of the driving equipment structure provided in an embodiment of the present invention. Detailed Implementation
[0111] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.
[0112] See Figure 1 As shown, Figure 1 This is a schematic flowchart of a long-tail target detection method provided in an embodiment of the present invention, which may include:
[0113] Step S11: Use the first object detection model and the second object detection model to detect the image to be recognized to obtain the first candidate box and the second candidate box, respectively. The first object detection model and the second object detection model are trained using training sets from different scenes.
[0114] Step S12: Extract the first feature corresponding to the first candidate box from the feature map obtained from the feature extraction layer of the first object detection model, and extract the second feature corresponding to the second candidate box from the feature map obtained from the feature extraction layer of the second object detection model;
[0115] Step S13: Fuse the first feature and the second feature to obtain the fused feature;
[0116] Step S14: Input the fused features into the detection head to obtain the first long-tail target detection result of the image to be identified.
[0117] The first object detection model is trained on a training set from a first scene, while the second object detection model is trained on a training set from a second scene. The first and second scenes are different. Combining the first and second object detection models trained on training sets from different scenes helps to enrich the diversity of scenes and improve the accuracy and applicability of long-tail object detection. This overcomes the problems of cold start or large scene transitions encountered when performing long-tail object detection based on a single object detection model, due to the low probability of long-tail objects appearing or occurring in real-world scenes.
[0118] In some embodiments, the second scenario can be a non-driving scenario, and the first scenario can be a driving scenario. As an example, the second object detection model can be a model trained on a dataset of tens of millions of images, such as ImageNet or OpenImage; the first object detection model can be Faster-RCNN (Faster Region Convolutional Neural Network) or SSD (Single Shot MultiBox Detector).
[0119] In embodiments of the present invention, it can be based on Figure 2 The long-tailed object detection model shown performs long-tailed object detection and may include:
[0120] The system includes an input module; a first target detection model and a second target detection model set in parallel; a feature fusion module; a detection head; and an output module.
[0121] The following text will be based on Figure 2 The long-tailed target detection model shown describes the long-tailed target detection method provided by this invention.
[0122] In some embodiments, step S11 may specifically involve the input module inputting the image to be recognized into the first target detection model and the second target detection model respectively, and the first target detection model and the second target detection model detecting the image to be recognized respectively to obtain the first candidate box and the second candidate box.
[0123] In some embodiments, the first target detection model may include a backbone network and a detection head, wherein the backbone network is used to extract features from the image to be identified, and the detection head is used to obtain the detection result. In other embodiments, the first target detection model may include a backbone network, a neck network, and a detection head, wherein the neck network may be used to fuse the feature maps output by the backbone network, etc.
[0124] In some embodiments, extracting the first feature corresponding to the first candidate box from the feature map obtained from the feature extraction layer of the first object detection model in step S12 may include: extracting the first feature corresponding to the first candidate box from the feature map obtained from the backbone network and / or neck network of the first object detection model.
[0125] In some embodiments, extracting the first feature corresponding to the first candidate box from the feature map obtained from the backbone network and / or neck network of the first object detection model may include extracting initial first features corresponding to the first candidate box from the feature map obtained from the backbone network and / or neck network of the first object detection model; upsampling the relatively small initial first features to obtain the first feature of the target size. Multiple initial first features can be obtained from multiple feature maps of different sizes based on the first candidate box, and these multiple initial first features constitute a first feature pyramid, such as... Figure 3 As shown, Figure 3 The diagram illustrates the process of fusing the first feature and the second feature according to an embodiment of the present invention; the first feature of the target size is obtained by upsampling the relatively small initial first feature.
[0126] As a specific example, the first feature corresponding to the first candidate box can be extracted from the feature maps obtained from the backbone network and neck network of the first object detection model. The backbone network can be a ResNet network (Residual Network), and the neck network can be a cascaded RPN (Region Proposal Network) and ROI Poling (Region Of Interest Pooling). Specifically, multiple feature maps obtained by feature extraction by ResNet and which are one-half, one-quarter, and one-eighth of the original size of the image to be detected, as well as the feature map output by ROI Poling, can be selected to obtain the initial first feature of multiple sizes. The first feature of the target size is obtained by upsampling the relatively small initial first features, where the target size can be one-half of the original size.
[0127] In some embodiments, extracting the second feature corresponding to the second candidate box from the feature map obtained from the feature extraction layer of the second object detection model in step S12 may include extracting the second feature corresponding to the second candidate box from the feature map obtained from the backbone network and / or neck network of the second object detection model.
[0128] In some embodiments, extracting the second feature corresponding to the second candidate box from the feature map obtained from the backbone network and / or neck network of the second object detection model may include: extracting the initial second feature corresponding to the second candidate box from the feature map obtained from the backbone network and / or neck network of the second object detection model; and upsampling the relatively small initial second feature to obtain the second feature of the target size. Multiple initial second features can be obtained from multiple feature maps of different sizes based on the second candidate box, and these multiple initial second features constitute a second feature pyramid, such as... Figure 3 As shown.
[0129] As a concrete example, a second feature corresponding to the second candidate box can be extracted from the feature map obtained from the neck network of the second object detection model. The neck network can employ a cascaded RPN and ROI Poling. The output of ROI Poling is a fixed-size feature map, which can be directly upsampled to obtain the second feature of the target size.
[0130] like Figure 3 As shown, in some embodiments, step S13 may specifically involve performing bilinear pooling on the first feature and the second feature to obtain the fused feature. In other embodiments, after obtaining the fused feature, feature dimensionality reduction and normalization can be performed on the fused feature to subsequently obtain the first long-tail target detection result of the image to be identified based on the dimensionality-reduced and normalized fused feature. As an example, the dimensionality of the fused feature can be reduced by sum pooling and PCA (Principal Component Analysis).
[0131] In some embodiments, the detection head can employ an existing network; for example, the detection head can be a fully connected layer. Step S14 can specifically involve inputting the fused features into the fully connected layer to obtain the first long-tailed target detection result of the image to be identified. The output module is used to output the first long-tailed target detection result.
[0132] In some embodiments, before using the first object detection model to extract features from the image to be identified, the first object detection model can also be trained, see [link to documentation]. Figure 4 As shown, Figure 4 This diagram illustrates a training method flowchart for a first target detection model provided in an embodiment of the present invention, which may include:
[0133] Step S41: Obtain the initial sample;
[0134] Step S42: Process the initial samples to construct a long-tail target sample set, where the long-tail target sample set corresponds to the autonomous driving scenario;
[0135] Step S43: Train the first target detection model to be trained based on the long-tail target sample set to obtain the trained first target detection model.
[0136] See Figure 5 As shown, Figure 5 A schematic diagram of the training architecture of the first target detection model provided in an embodiment of the present invention is shown, which may include a cascaded initial sample input module, a cold start module, and a self-iterative module. In the following text, based on... Figure 5 The training architecture of the first object detection model shown describes the training method of the first object detection model.
[0137] In some embodiments, step S41 may specifically involve inputting an initial sample through the initial sample input module. In some embodiments, the initial sample may include an image sample; in other embodiments, the initial sample may include both image and text samples.
[0138] In some embodiments, the cold start module may include at least one long-tail data mining model. The cold start module is used to mine long-tail target samples using at least one long-tail data mining model and construct a long-tail target sample set based on the obtained long-tail target samples. As an example, the cold start module may include at least one long-tail data mining model selected from image classification, image segmentation, image detection, unimodal image retrieval, and unimodal image retrieval models.
[0139] When the sample is an image, step S42 can specifically involve inputting the image sample into at least one of an image classification model, an image segmentation model, an image detection model, and a single-modal image retrieval model, processing the image sample to obtain long-tailed target samples, and constructing a long-tailed target sample set based on the long-tailed target samples.
[0140] In some embodiments, an image classification model can be used to classify the image samples in the initial sample to obtain long-tail target samples related to the autonomous driving scenario. The image classification model can be used to classify the full-image features of the autonomous driving scenario, such as weather, location, road, lighting, or image quality.
[0141] In some embodiments, an image segmentation model can be used to segment image samples in the initial sample to obtain long-tail target samples related to the autonomous driving scenario. The image segmentation model can be used to segment non-rigid objects in the long-tail target samples, such as segmenting at least one of lane lines, roads, guardrails, mud puddles, and greenery in the image samples.
[0142] In some embodiments, an image detection model can be used to perform image detection on image samples in the initial sample to obtain long-tail target samples related to the autonomous driving scenario. The image detection model can be used to detect rigid objects such as dynamic obstacles and static obstacles, such as vehicles, pedestrians, pillars, or animals.
[0143] In some embodiments, a single-modal image retrieval model can be used to perform large-scale retrieval of source images in a network image library. Specifically, a CNN (Convolutional Neural Network) can be used to extract image features of the source image and image samples respectively, and the feature distance between the source image and the image samples can be calculated based on the image features. Based on the comparison between the feature distance and a first distance threshold, long-tail target samples related to the autonomous driving scenario can be selected. For example, a threshold filtering method can be used based on the first distance threshold to select images that are similar to the image samples as long-tail target samples.
[0144] When the initial sample contains both image and text samples, step S42 can specifically involve inputting the text and image samples into a multimodal image retrieval model; using the multimodal image retrieval model to search for source images in a network image database to obtain long-tail target samples; and constructing a long-tail target sample set based on the long-tail target samples. Specifically, the multimodal image retrieval model can be used to extract text features from the input text samples using a text encoding module and to extract image features from the input image samples using an image encoding module; fusing the extracted text features and target image features to obtain a first multimodal fusion feature; extracting image features from the source images using the image encoding module to obtain source image features; fusing the text features and source image features to obtain a second multimodal fusion feature; calculating the cosine distance between the second and first multimodal fusion features; and selecting long-tail target samples relevant to the autonomous driving scenario based on a comparison of the cosine distance and a second distance threshold. Specifically, based on the second distance threshold, a threshold filtering method can be used to obtain source images that are similar to the image samples, which are then used as long-tail target samples. This method can be applied when image samples are scarce or the model image retrieval recall rate is low, in order to effectively retrieve long-tail target samples.
[0145] In some preferred embodiments, the cold start module may simultaneously include an image classification model, an image segmentation model, an image detection model, a unimodal image retrieval model, and a multimodal image retrieval model to obtain more long-tail target samples, thereby improving the recognition accuracy of the first target detection model.
[0146] In some embodiments, constructing a long-tailed target sample set based on long-tailed target samples can be done by selecting at least a portion of the obtained long-tailed target samples to construct the long-tailed target sample set as needed.
[0147] In some embodiments, step S43 may specifically include:
[0148] Preliminary training phase: Input the long-tail target sample set into the first target detection model to be trained, and obtain the detection result of each long-tail target sample in the long-tail target sample set; where the detection result includes candidate boxes and corresponding categories;
[0149] Iterative training phase: When the category is a long-tailed target, determine the pseudo long-tailed target based on the candidate box and ground truth box corresponding to the category; construct a difficult negative sample set based on the pseudo long-tailed target; use at least the difficult negative sample set to iteratively train the first target detection model to be trained; determine whether the preset conditions are met, and if they are met, stop iterative training and obtain the trained first target detection model.
[0150] In some embodiments, the long-tail target sample set can be expanded through data augmentation before initial training. Data augmentation can be performed using methods such as flipping, scaling, and cropping. The first target detection model to be trained can be used to detect targets on the long-tail target samples, obtaining candidate boxes and the confidence scores of the candidate boxes belonging to the long-tail target category. When the confidence score corresponding to a candidate box is greater than a preset confidence score, the category corresponding to the candidate box is determined to be a long-tail target.
[0151] In some embodiments, the long-tailed target sample set may include multiple long-tailed target samples. Before inputting the long-tailed target sample set into the first target detection model to be trained, the long-tailed target samples may be labeled, wherein the true category and ground truth box of the long-tailed target samples may be labeled.
[0152] In some embodiments, when the detected category is a long-tailed target, determining the pseudo long-tailed target based on the candidate box and ground truth box corresponding to the category can specifically involve calculating the intersection-union ratio (IU) of the candidate box and ground truth box corresponding to the long-tailed target. When the IU is zero, the long-tailed target is determined as a pseudo long-tailed target, that is, a hard negative sample.
[0153] In some embodiments, constructing a difficult negative sample set based on pseudo-long-tailed targets, and iteratively training the first target detection model to be trained using at least this difficult negative sample set, can specifically involve using data augmentation methods such as Mixup or Cutmix to paste the candidate boxes corresponding to the pseudo-long-tailed targets into the long-tailed target negative samples, resulting in multiple difficult negative samples. A difficult negative sample set can be constructed from these multiple difficult negative samples, and iterative training of the first target detection model to be trained using at least this difficult negative sample set is performed. Here, long-tailed target samples can be divided into long-tailed target positive samples and long-tailed target negative samples, and long-tailed target negative samples can be images that do not contain long-tailed targets. In other embodiments, constructing a difficult negative sample set based on pseudo-long-tailed targets can also involve cropping the corresponding region of the long-tailed target sample from the candidate boxes corresponding to the pseudo-long-tailed targets as a difficult negative sample, and constructing a difficult negative sample set based on the obtained multiple difficult negative samples. In some embodiments, the difficult negative sample set and the long-tailed target sample set can also be used for iterative training of the first target detection model to be trained.
[0154] In some embodiments, determining whether a preset condition is met, and stopping iterative training if the condition is met, to obtain a trained first object detection model may include: determining whether the false recognition rate of the first object detection model to be trained is within a preset threshold range; if yes, determining that the preset condition is met, and stopping iterative training. If no, then continuing to construct a hard negative sample set based on the falsely identified pseudo-long-tail targets using data augmentation methods, and labeling the hard negative samples in the currently constructed hard negative sample set, so as to train the first object detection model to be trained based on the labeled hard negative samples.
[0155] The false recognition rate of the first target detection model undergoing iterative training can be determined by the following steps: the false recognition rate is obtained by the proportion of the number of false long-tailed targets misidentified to the total number of input samples to the first target detection model to be trained.
[0156] When the false recognition rate is low, it can be determined that the first target detection model has been trained. The preset threshold range can be set as needed.
[0157] In this embodiment of the invention, by identifying pseudo-long-tailed targets, constructing a difficult negative sample set using the pseudo-long-tailed targets, and training the first target detection model to be trained based at least on the difficult negative sample set, the ability of the first target detection model to identify pseudo-long-tailed targets can be improved, which is beneficial to improving the accuracy of long-tailed target detection in the future.
[0158] In some embodiments, to further improve the accuracy of long-tail target detection, ensemble learning can be performed by combining the detection results of the second target detection model to obtain the final long-tail target detection result. See details in [link to documentation]. Figure 6 and Figure 7 As shown.
[0159] Figure 6 This is a schematic diagram of another long-tailed target detection model architecture provided in an embodiment of the present invention, which is in Figure 2 Based on the example shown, an ensemble learning module may also be included. This ensemble learning module is used to perform ensemble learning on the second long-tail target detection result obtained from the second target detection model and the first long-tail target detection result obtained based on the fusion features, so as to obtain the final long-tail target detection result.
[0160] Figure 7 This is a schematic flowchart of a long-tail target detection method provided in another embodiment of the present invention, which may include:
[0161] Step S71: Use the first object detection model and the second object detection model to detect the image to be recognized to obtain the first candidate box and the second candidate box respectively;
[0162] Step S72: Extract the first feature corresponding to the first candidate box from the feature map obtained from the feature extraction layer of the first object detection model, and extract the second feature corresponding to the second candidate box from the feature map obtained from the feature extraction layer of the second object detection model;
[0163] Step S73: Fuse the first feature and the second feature to obtain the fused feature;
[0164] Step S74: Input the fused features into the detection head to obtain the first long-tailed target detection result of the image to be identified;
[0165] Step S75: Determine the final long-tail target detection result based on the first long-tail target detection result and the second long-tail target detection result detected by the second target detection model.
[0166] Steps S71-S74 can be executed in the same way as steps S11-S14.
[0167] In some embodiments, the second long-tailed object detection result can be obtained in step S71 by detecting the image to be recognized using a second object detection model. The second object detection result may include a second candidate bounding box.
[0168] When a second target detection model is set, a second long-tail target detection result is obtained accordingly. Step S75 can be specifically as follows: compare the first long-tail target detection result and the second long-tail target detection result; when the comparison is consistent, determine the second long-tail target detection result as the final long-tail target detection result.
[0169] When multiple second target detection models are set, multiple second long-tail target detection results are obtained accordingly. Step S75 can be specifically determined by using a voting mechanism based on multiple second long-tail target detection results and first long-tail target detection results to determine the final long-tail target detection result.
[0170] The second and first long-tail object detection results can both include the corresponding categories of candidate boxes. A voting mechanism is used to count the categories of candidate boxes in multiple second and first long-tail object detection results, and the category with the highest number of votes is taken as the final long-tail object detection result.
[0171] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0172] Another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the long-tail target detection method described in any of the above embodiments. This computer-readable storage medium can be a storage device comprising various electronic devices; optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.
[0173] See Figure 8 As shown, another aspect of the present invention provides a driving device, which may include a memory 81 and a processor 82. The memory 81 stores a computer program, which, when executed by the processor 82, implements the long-tail target detection method described in any of the above embodiments.
[0174] The memory 81 and the processor 82 can be connected via a bus or other means. Figure 8 The example shows a configuration where memory 81 and processor 82 are connected via a bus, and processor 82 is configured with only one instance.
[0175] In other embodiments, the driving device may include multiple memories 81 and multiple processors 82. The program executing the long-tail target detection method of any of the above embodiments can be divided into multiple subroutines, each of which can be loaded and run by a processor to perform different steps of the long-tail target detection method of the above method embodiments. Specifically, each subroutine can be stored in a different memory 81, and each processor 82 can be configured to execute programs in one or more memories 81 to jointly implement the long-tail target detection method of the above method embodiments.
[0176] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.
Claims
1. A method for detecting long-tailed targets, characterized in that, include: The first object detection model and the second object detection model are used to detect objects in the image to be recognized to obtain a first candidate box and a second candidate box, respectively. The first object detection model and the second object detection model are trained using training sets from different scenarios. The first object detection model is trained using a training set from an autonomous driving scenario. Extract the first feature corresponding to the first candidate box from the feature map obtained from the feature extraction layer of the first object detection model, and extract the second feature corresponding to the second candidate box from the feature map obtained from the feature extraction layer of the second object detection model; The first feature and the second feature are fused to obtain the fused feature; as well as, The fused features are input into the detection head to obtain the first long-tail target detection result of the image to be identified.
2. The method according to claim 1, characterized in that, Extracting the first feature corresponding to the first candidate box from the feature map obtained from the feature extraction layer of the first object detection model includes: Extract the first feature corresponding to the first candidate box from the feature map obtained from the backbone network and / or neck network of the first target detection model; And / or, Extracting the second feature corresponding to the second candidate box from the feature map obtained from the feature extraction layer of the second object detection model includes: Extract the second feature corresponding to the second candidate box from the feature map obtained from the backbone network and / or neck network of the second object detection model.
3. The method according to claim 2, characterized in that, Extracting the first feature corresponding to the first candidate box from the feature map obtained from the backbone network and / or neck network of the first target detection model includes: Extract the initial first feature corresponding to the first candidate box from the feature map obtained from the backbone network and / or neck network of the first target detection model; The initial first feature of a relatively small size is upsampled to obtain the first feature of the target size.
4. The method according to claim 3, characterized in that, Extracting the second feature corresponding to the second candidate box from the feature map obtained from the backbone network and / or neck network of the second target detection model includes: Extract the initial second feature corresponding to the second candidate box from the feature map obtained from the backbone network and / or neck network of the second target detection model; The initial second feature of the relatively small size is upsampled to obtain the second feature of the target size.
5. The method according to claim 1, characterized in that, The step of fusing the first feature and the second feature to obtain the fused feature includes: The first feature and the second feature are subjected to bilinear pooling to obtain the fused feature. And / or, After obtaining the fused features and before inputting the fused features into the detection head, the method further includes: performing feature dimensionality reduction and normalization on the fused features.
6. The method according to claim 1, characterized in that, The method further includes: Based on the first long-tail target detection result and the second long-tail target detection result detected by the second target detection model, the final long-tail target detection result is determined.
7. The method according to claim 6, characterized in that, The determination of the final long-tail target detection result based on the first long-tail target detection result and the second long-tail target detection result detected by the second target detection model includes: When the second long-tail target detection result is one, the second long-tail target detection result and the first long-tail target detection result are compared. If they match, the first long-tail target detection result is determined as the final long-tail target detection result. When there are multiple second long-tail target detection results, and these multiple second long-tail target detection results come from multiple second target detection models, a voting mechanism is used to determine the final long-tail target detection result based on the multiple second long-tail target detection results and the first long-tail target detection result.
8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Obtain the initial sample; The initial samples are processed to construct a long-tail target sample set, wherein the long-tail target sample set corresponds to an autonomous driving scenario; The first target detection model to be trained is trained based on the long-tail target sample set to obtain the trained first target detection model.
9. The method according to claim 8, characterized in that, The process of processing the initial samples to construct a long-tail target sample set includes: Image classification is performed on the image samples in the initial sample to obtain long-tail target samples related to the autonomous driving scenario; and / or Image segmentation is performed on the image samples in the initial sample to obtain long-tail target samples related to the autonomous driving scenario; and / or Image detection is performed on the image samples in the initial sample to obtain long-tail target samples related to the autonomous driving scenario; and / or Image features are extracted from the source image and the image samples in the initial sample using a convolutional neural network, and the feature distance between the source image and the image samples is calculated based on the image features; long-tail target samples related to the autonomous driving scenario are selected based on a comparison between the feature distance and a first distance threshold; and / or Text features and image features are extracted from the text samples and image samples in the initial sample, respectively; the extracted text features and target image features are fused to obtain a first multimodal fusion feature; image features are extracted from the source image to obtain source image features; the text features and the source image features are fused to obtain a second multimodal fusion feature; the cosine distance between the second multimodal fusion feature and the first multimodal fusion feature is calculated; long-tail target samples related to the autonomous driving scenario are selected based on the comparison between the cosine distance and a second distance threshold.
10. The method according to claim 8, characterized in that, The step of training the first target detection model to be trained based on the long-tail target sample set to obtain the trained first target detection model includes: Initial training phase: The long-tailed target sample set is input into the first target detection model to be trained to obtain the detection result of each long-tailed target sample in the long-tailed target sample set, wherein the detection result includes candidate boxes and corresponding categories; Iterative training phase: When the category is a long-tail target, a pseudo long-tail target is determined based on the candidate box and the ground truth box corresponding to the category. Construct a difficult negative sample set based on the pseudo-long-tail target; The first target detection model to be trained is iteratively trained at least using the difficult negative sample set; Determine whether the preset conditions are met. If they are met, stop iterative training and obtain the trained first target detection model.
11. The method according to claim 10, characterized in that, The determination of whether a preset condition is met, and if so, stopping iterative training to obtain the trained first object detection model, includes: Determine whether the false recognition rate of the first target detection model to be trained is within a preset threshold range; If so, once the preset conditions are met, stop the iterative training.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the long-tail target detection method according to any one of claims 1 to 11.
13. A driving device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, which, when executed by the processor, implements the long-tail target detection method according to any one of claims 1 to 11.