Small sample target detection algorithm based on high-quality synthetic image data

Abstract

The invention belongs to the technical field of machine learning and target detection, and discloses a small sample target detection algorithm based on high-quality synthetic image data, which is used for improving the performance of a small sample target detection model. According to the method, foreground and background decoupling is carried out on a generated target image by utilizing an image segmentation basic model, and a high-quality foreground is selected through a score threshold screening strategy to be recombined with a proper base class image, so that diversified synthetic training data is obtained. Compared with a traditional data enhancement method or random synthesis method, the method can solve the problem of data insufficiency, and effectively improve the detection capability of the model for new category targets. According to the method, the generalization ability of the model can be improved, the quality of synthetic data can be optimized, various models can be flexibly adapted, and the detection precision under the conditions of small samples and extremely small samples is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning and target detection, and specifically relates to a synthetic image data optimization method for improving the performance of a few-shot object detection (FSOD) model. The method uses an image segmentation basic model to decouple the foreground and background of a generated target image, and selects a high-quality foreground through a score threshold screening strategy to recombinant with a suitable base class image, thereby obtaining diverse synthetic training data. Compared with traditional data enhancement methods or random synthesis methods, this method can solve the problem of insufficient data and effectively improve the model's detection capability for new categories of targets. Background Art

[0002] With the rapid development of deep learning technology, object detection tasks in the field of computer vision have achieved breakthrough achievements, especially in application scenarios such as pedestrian detection, face detection, and vehicle detection. Traditional object detection methods usually rely on a large amount of labeled training data, which is a huge challenge in practical applications, especially in fields such as medical imaging, autonomous driving, and wildlife monitoring where data acquisition is difficult. In order to meet the above challenges, small sample object detection came into being, which aims to correctly identify and locate new categories with a small number of samples.

[0003] The current mainstream small sample target detection methods can be roughly divided into two categories: one is to enhance the model's ability to extract features from a small number of samples through model architecture design, which includes transfer learning and meta-learning methods. The meta-learning method updates the model parameters by simulating task learning and using limited support samples, while the transfer learning method adapts to new category samples by fine-tuning the pre-trained model; the other is to expand the training set through data enhancement technology to increase data diversity. However, these methods still face the problems of poor model robustness and insufficient generalization ability, especially when facing new categories, it is still difficult to make full use of limited data for effective learning.

[0004] Small-sample object detection methods based on data augmentation technology have played a key role in improving model performance. By standardizing the original data or generating synthetic data, data augmentation methods can effectively make up for the problem of insufficient data. However, existing data augmentation operations usually rely on basic image transformations [Yun S, Han D, Oh SJ, et al. Cutmix: Regularization strategy to train strong classifiers with localizable features [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 6023-6032.] or directly add synthetic images to training data. This approach has certain limitations, especially when the synthetic data cannot accurately reflect the characteristics of new categories of targets, it may lead to a decrease in model performance. Therefore, how to select or construct high-quality synthetic data and effectively integrate it into the training process is still the main challenge in current small-sample object detection.

[0005] In recent years, generative models (such as DALL E [Ramesh A, Dhariwal P, Nichol A, et al. Hierarchical text-conditional image generation with clip latents [J]. arXiv preprint arXiv: 2204.06125, 2022, 1 (2): 3.]) have made important breakthroughs in the field of image generation and provided new solutions for small sample object detection. These generative models can generate realistic synthetic images and provide more sophisticated generation methods. However, in existing research on using generated images for data augmentation to enrich training datasets, the quality controllability and background adaptability of generated images are usually ignored. For example, Lin [Lin S, Wang K, Zeng X, et al. Explore the power of synthetic data on few-shot object detection [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 638-647.] et al. used Stable Diffusion [Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 10684-10695.] to generate new category samples, cropped the generated foreground through the saliency detection model and pasted it into the original background image to increase the diversity of the dataset. Although this method is effective in increasing data diversity, the generated synthetic images may have distortion problems, and the mismatch between the foreground and the background may mislead the model training, thereby affecting its performance.

[0006] In view of the above problems, the present invention proposes a high-quality synthetic data enhancement method that combines segmentation for decoupling, screening and reorganization of foreground and background. This method generates a new category of generated images through a generative basic model DALL·E, uses the segmentation basic model SEEM to decouple the foreground and background of the generated image, and then combines the score threshold screening strategy to select high-quality new category samples from the decoupled foreground and recombined with the matching background. Compared with traditional data enhancement methods or random synthesis methods, the synthetic data enhancement method based on segmentation decoupling and screening and recombination can more accurately screen out representative and authentic generated data, and improve the diversity and adaptability of synthetic data through the recombination of foreground and background, thereby significantly enhancing the model's detection ability for new category targets, especially showing superior performance and generalization ability in small sample scenarios. Summary of the invention

[0007] In order to solve the data scarcity problem in small sample size object detection (FSOD), this paper proposes an innovative synthetic data selection and utilization framework to improve the generalization ability and detection accuracy of the model in small sample size learning scenarios. Existing small sample size object detection methods usually rely on standard data augmentation methods, but these methods often suffer from performance degradation due to synthetic data quality issues. This paper introduces decoupling and selection strategies to synthesize high-quality data, effectively improving the performance of small sample size object detection models, especially in the case of very few samples and low data quality.

[0008] In Few-Shot Object Detection (FSOD), the base classes are objects with a large number of samples and known categories in the training set, while the new classes are objects of unknown categories with a small number of samples or even only a small number of samples. For example, the base classes may include common vehicles (such as cars, trucks) or animals (such as dogs, cats), while the new classes may be rare species (such as foxes) or objects in special scenes (such as drones).

[0009] The first innovation of the present invention is to propose a foreground-background decoupling module based on segmentation technology. This module uses a segmentation basic model (such as SEEM [Zou X, Yang J, Zhang H, et al. Segment everything everywhere all at once [J]. Advances in Neural Information Processing Systems, 2024, 36.]) to perform fine segmentation on the synthetic image, and selects the highest confidence mask that obeys the threshold interval range from the obtained masks as the best foreground, and gives priority to synthetic samples with clear edges and complete shapes. This process effectively reduces the interference of irrelevant background and ensures the quality of the synthetic image. The second innovation is to propose a synthesis module, which further improves the quality and diversity of the synthetic image by filtering the synthetic foreground image with a fixed threshold through an image encoder. Finally, the synthesis module combines high-quality foreground images with base class images that meet diversity and consistency requirements to enhance sample diversity and improve the positioning capability of the detection model.

[0010] Technical solution of the present invention

[0011] A small sample target detection algorithm based on high-quality synthetic image data, the steps are as follows:

[0012] Step 1: Synthetic Data Generation and Foreground-Background Decoupling

[0013] 1.1 Synthetic Data Generation

[0014] To generate new object images, we feed a small number of new category names into ChatGPT and have it generate detailed text prompts based on each new class. We then feed these generated prompts into the DALL·E model to leverage its ability to generate images. By leveraging ChatGPT's multi-round conversational capabilities, we can generate and gradually optimize text prompts to make them more suitable for generating salient and clear target images. In this way, we are able to ensure that each generated image contains a clear object, while providing higher feasibility and accuracy for the subsequent foreground-background decoupling process.

[0015] 1.2 Foreground-background decoupling

[0016] After generating the synthetic image, we use the foreground-background decoupling module based on segmentation technology to accurately separate the object and background in the synthetic image. In order to solve the problems of background interference and boundary artifacts in traditional methods, the present invention combines SEEM technology to optimize the decoupling process.

[0017] The steps of the algorithm are as follows:

[0018] 1.2.1. Generate initial mask

[0019] Will generate image I gen Input to the segmentation base model (such as SEEM) to generate multiple foreground candidate masks These masks represent different segmented regions, and each mask has a corresponding confidence score n .

[0020] 1.2.2 Confidence screening

[0021] First, set a confidence threshold T (for example, 0.5 or 0.7). If a mask passes the screening, only the confidence score is retained. n Masks above the threshold T; if there are multiple masks that pass the screening, the mask with the highest confidence is selected, and the mask matrix after the screening is recorded as

[0022] 1.2.3. Extraction of foreground image

[0023] In this technical solution, the foreground image is extracted by applying a mask The mask is a matrix with a value range of 0-1, where the area with a value of 1 represents the corresponding image pixel and the area with a value of 0 represents the background. And generate image I gen Perform dot multiplication to obtain the foreground image I fore This step ensures that the foreground extracted from the generated image has clear object boundaries while minimizing background interference, providing a reliable basis for subsequent synthetic data.

[0024] Step 2: Screening and combining foreground and background

[0025] 2.1. Prospect Screening

[0026] The quality of the generated foreground image is crucial to the effectiveness of the synthetic data. In order to ensure that the selected foreground has high quality and representativeness, the present invention adopts a screening strategy based on feature similarity to screen out high-quality foreground samples by matching the features with the real samples. The specific steps are as follows:

[0027] 2.1.1 Feature Extraction

[0028] Use an image encoder (such as CLIP) to generate the foreground image I fore And the foreground image I of the same target in the real dataset real Perform feature extraction, and the extracted feature vectors are represented as f fore With f real .

[0029] 2.1.2 Similarity Calculation

[0030] Calculate the feature similarity sim(f fore ,f real ), where sim represents the cosine similarity formula.

[0031] 2.1.3 Threshold Screening

[0032] Set the similarity threshold T sim (such as 0.7), and filter out the foreground sample set whose similarity is greater than or equal to the threshold

[0033]

[0034] 2.1.4 Diversity Constraints

[0035] To ensure the diversity of foreground samples and avoid over-concentration of the distribution of screened samples, the following strategies can be adopted:

[0036] (1) Feature coverage strategy: Through cluster analysis or dimensionality reduction visualization, samples with wider coverage in the feature space are preferentially selected to ensure diversity.

[0037] (2) Category balance strategy: Ensure that the number of screened samples in each category is relatively balanced to avoid excessive proportion of samples in a specific category, thereby improving the fairness and representativeness of data distribution.

[0038] 2.2 Background screening

[0039] Select images suitable for background from the basic dataset. The quality of the background image has an important impact on the diversity and authenticity of the synthesized image. Specifically, a suitable background image should obey two properties:

[0040] (1) Diversity: When screening background images, we need to increase the diversity of the backgrounds to ensure that they cover a variety of environment types (such as indoors, city streets, natural landscapes, etc.). The distribution of scenes can be controlled by clustering the background images to achieve diversified selection.

[0041] (2) Consistency: The background image should have a certain semantic or visual consistency with the foreground target to avoid unreasonable combinations that affect the authenticity of the synthesized image. For example, when the foreground target is a vehicle, the background should be selected from scenes such as streets and parking lots, rather than natural scenery or indoor scenes. By calculating the feature similarity between the foreground and candidate background images and selecting a background with a high degree of match with the foreground features, the rationality and credibility of the synthesized image can be further improved.

[0042] 2.3 Combination of foreground and background

[0043] The filtered foreground image I fore Background to meeting diversity and consistency requirements I base Recombination. Generate diverse synthetic images. In order to avoid conflict between foreground and background, a position-aware strategy is used to adjust the position and scale of the foreground. The specific steps are as follows:

[0044] 2.3.1 Synthetic Image Generation and Diversity Enhancement

[0045] Randomly paste the adjusted foreground image onto the background image to generate a composite image I syn :

[0046] I syn =Paste(I fore ,I base ,position)

[0047] Among them, Paste indicates the pasting operation, and position indicates the placement position of the foreground object. In order to further improve the diversity of synthetic data, we randomly adjust the rotation angle, brightness, color and other attributes of the foreground image during the synthesis process. These transformations can increase the diversity of synthetic images and enhance the generalization ability of the model.

[0048] 2.3.2 Quality Inspection

[0049] For the generated synthetic image I syn Perform quality checks and remove samples that do not meet the requirements to ensure the validity of the synthetic data.

[0050] Step 3: Construction of synthetic dataset and model training

[0051] 3.1 Construction of synthetic dataset

[0052] The high-quality synthetic image I generated by the above steps syn Construct a synthetic dataset and fuse it with the real base class dataset: the synthetic data and the real data are mixed in a certain ratio (such as 10:1) to form a diversified training set.

[0053] 3.2 Model Training

[0054] We use the mainstream object detection model Faster R-CNN [Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks [J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39 (6): 1137-1149.] for training, and use end-to-end optimization methods to improve the model's ability to detect new categories of objects. In order to better train the model, we use a joint optimization loss function to optimize the classification loss and bounding box regression loss respectively. Among them, the classification loss is used to measure the model's prediction accuracy for the target category, while the bounding box regression loss is used to measure the model's prediction accuracy for the target position and size. By optimizing these two loss terms simultaneously, the model's detection performance on new categories of objects can be effectively improved.

[0055] Beneficial effects of the present invention: First, the generalization ability of the model can be improved. By introducing decoupling and screening strategies, the present invention can make full use of the generated synthetic data and effectively alleviate the overfitting problem caused by data scarcity. In particular, by performing foreground-background decoupling processing and threshold screening strategies on synthetic images, the generalization ability of the model in a small sample environment is effectively improved. The present invention can maintain a high detection accuracy in a variety of complex scenes, and enhances the robustness of the model in different scenes and when the amount of data is insufficient.

[0056] Secondly, the quality of synthetic data can be optimized. Compared with traditional data enhancement methods, the present invention uses two modules to finely select high-quality foreground and background and then reconstruct them, thereby ensuring that the synthetic images used have higher authenticity and representativeness. This high-quality synthetic data not only effectively improves the diversity of training data, but also helps the model better learn the key features of the target object. Especially in the case of scarce samples, this method can better make up for the problem of insufficient data and improve the overall performance of the model.

[0057] The invention can be flexibly adapted to a variety of models. The framework design of the invention is highly flexible and can be seamlessly integrated into a variety of models. This not only makes it applicable to existing small sample detection methods, but also can be effectively combined with the latest fine-tuning-based model architecture, thereby fully tapping the model's potential and maximizing performance. Thanks to this flexible adaptability, the invention provides a wider range of application scenarios and technical support for small sample target detection.

[0058] Finally, the detection accuracy in small and extremely small sample cases is significantly improved. In the case of extremely small samples, traditional detection methods often suffer from performance degradation due to insufficient data. However, the present invention significantly improves the performance of the model in small sample learning by screening high-quality synthetic data and applying it selectively. Especially in the case of a very small number of training samples, the present invention can effectively alleviate the problems caused by data scarcity, greatly improve detection accuracy, and demonstrate a strong advantage under conditions of limited data. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The figure is an overall flow chart of a small sample target detection algorithm based on high-quality synthetic image data.

[0060] Figure 2 This is an application example diagram of a small sample target detection algorithm based on high-quality synthetic image data. DETAILED DESCRIPTION

[0061] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0062] Step 1: Data generation and preprocessing

[0063] First, a small number of new category names are input into ChatGPT, and it is asked to generate detailed text prompts based on each category. Input category: "Bird" ChatGPT initial output: "A sky blue bird standing on a tree stump with rocks in the background." Adjust prompt: "Please remove the rocks in the background and set the background to a single color or blur it to retain the details of the bird and the stump." Optimized output: "The bird's feathers are bright sky blue. It stands on a wooden stump with its claws tightly grasping the stump. The background is a soft brown, simple and blurred, highlighting the main body of the bird, making the picture look peaceful and natural." Use a generative base model (such as DALL·E) to generate images of the target category. During the generation process, the quality of the output image can be optimized by adjusting model parameters (such as the number of diffusion steps or the generation diversity). Ensure that the generated image has a clear foreground and an easy-to-segment background to meet the needs of subsequent tasks.

[0064] Step 2: Foreground-background decoupling

[0065] Using SEEM, the generated target image is finely segmented to achieve the decoupling of foreground targets and backgrounds. Multiple masks are generated during the segmentation process, each mask represents a possible foreground area, thus providing multiple segmentation options. The masks of all segmentation results are scored for confidence, and the masks with the highest scores are selected based on the confidence level that must be greater than 0.8. Masks with complete boundaries and high confidence levels are given priority to effectively reduce background interference, highlight the key features of the target, and lay the foundation for subsequent processing.

[0066] Step 3: Prospect Screening

[0067] Use CLIP to extract the feature vector of the foreground image, and at the same time extract the feature vector of the corresponding category sample in the real data set. The cosine similarity formula is used to calculate the matching degree between the feature vectors, and the proximity between the generated foreground and the real sample in the feature space is evaluated. Set the similarity threshold ≥ 0.7 to filter out foreground images with high feature similarity with the real sample. The screened foreground samples not only have high authenticity, but also retain good diversity in the feature space, thereby enhancing data quality.

[0068] Step 4: Background Screening

[0069] Perform cluster analysis on background images and divide them into different scene categories: urban, indoor, natural, etc. Select appropriate scene categories and then select diverse images from each scene category to ensure that the background data covers a variety of environment types, thereby improving the diversity and representativeness of the synthetic data.

[0070] Step 5: Recombining foreground and background

[0071] The positions of the basic class objects in the background image are detected, and the scale and placement of the foreground image are adjusted according to the detection results to avoid the intersection-over-union ratio between the foreground and background objects being higher than 0.1. This position-aware strategy can effectively reduce the feature interference between the foreground and background objects. Subsequently, the adjusted foreground image is pasted onto the background image to generate a new synthetic image. During the pasting process, the brightness, color, and rotation angle of the foreground object can be randomly adjusted, thereby further improving the diversity and authenticity of the synthetic data.

[0072] Step 6: Synthetic dataset construction

[0073] The generated high-quality synthetic images are fused with the real basic class datasets, and a new training dataset is constructed by mixing them in a 1:1 ratio. During the dataset fusion process, it is necessary to ensure that the category distribution of synthetic samples and real samples is balanced to avoid imbalance in the proportion of samples of certain categories. The generated synthetic datasets are strictly checked for quality, and images with severe distortion, foreground-background conflicts, or invalid features are eliminated to ensure the overall quality of the training data.

[0074] Step 7: Model Training

[0075] The mainstream target detection framework Faster R-CNN is used for model training. The classification and target localization capabilities of the model are gradually optimized by mixing real samples and synthetic samples. During the training process, a joint loss function is used to combine classification loss and bounding box regression loss to improve the model's detection performance for new categories of targets. At the same time, hyperparameters (such as learning rate, weight decay, etc.) are adjusted to improve the model's adaptability to synthetic data.

[0076] This process completely covers all the key steps from data generation, screening, reorganization to model training, and provides a systematic solution to the problem of small sample target detection. Accurate text prompts are generated through ChatGPT, and high-quality images are generated in combination with generative basic models (such as DALL·E). At the same time, technologies such as SEEM and CLIP are used to perform foreground-background decoupling and feature similarity screening to ensure the authenticity and diversity of the data. In addition, background clustering analysis and location-aware strategies further optimize the diversity and synthesis quality of the data, while the strategy of training with mixed real data and synthetic data effectively improves the model's ability to detect new categories of targets. Overall, the process has a clear structure and strong technical innovation, and proposes effective solutions to the problems of insufficient data and synthetic data quality. However, the implementation process may require high computing resources and algorithm performance, and there is still room for efficiency optimization.

Claims

1. A small sample target detection algorithm based on high-quality synthetic image data, characterized by the following steps: Step 1: Synthetic Data Generation and Foreground-Background Decoupling 1.1 Synthetic Image Generation In small-sample object detection, ChatGPT is used to generate textual hints for new categories, where new categories are target categories with few or even unknown samples. The textual hints are then fed into the DALL·E model to generate synthetic images. 1.2 Foreground-background decoupling Separate the new class of targets in the synthesized image from the original background to effectively reduce background interference and boundary artifacts, and combine the SEEM technology to further optimize the decoupling process; the details are as follows: 1.2.

1. Generate initial mask Image I gen Input to the segmentation base model to generate multiple foreground candidate masks The foreground candidate masks represent different segmentation regions, and each foreground candidate mask has a corresponding confidence score score n ; 1.2.2 Confidence screening First, the confidence threshold T is set; if there is a foreground candidate mask that passes the screening, only the foreground candidate mask with a confidence score higher than the confidence threshold T is retained; if there are multiple foreground candidate masks that pass the screening, the foreground candidate mask with the highest confidence score is selected, and the foreground candidate mask matrix after screening is recorded as 1.2.

3. Extraction of foreground image The foreground image is extracted using a mask matrix This is achieved by: The mask is a matrix with a value range of 0-1, where the area with a value of 1 represents the corresponding image pixel, and the area with a value of 0 is the background; the mask matrix With image I gen Perform dot multiplication to obtain the foreground image I fore ; Step 2: Screening and combining foreground and background 2.

1. Prospect Screening A screening strategy based on feature similarity is used to screen out high-quality prospect samples by matching the features with real samples; the details are as follows: 2.1.1 Feature Extraction Use the image encoder to generate the foreground image I fore And the foreground image I of the same target in the real dataset real Perform feature extraction, and the extracted feature vectors are represented as f fore With f real ; 2.1.2 Similarity Calculation Calculate the feature similarity sim(f fore ,f real ), sim represents cosine similarity; 2.1.3 Threshold Screening Set the similarity threshold T sim , and filter out the foreground image set whose feature similarity is greater than or equal to the similarity threshold 2.1.4 Diversity Constraints To ensure the diversity of foreground images and avoid over-concentration of foreground images after screening, the following strategies are adopted: (1) Feature coverage strategy: Through cluster analysis or dimensionality reduction visualization, foreground images with wider coverage in the feature space are selected to ensure diversity; (2) Category balance strategy: ensure that the number of screening samples in each category is relatively balanced; 2.2 Background screening Select suitable background images from the basic class data set. The background images should obey two properties: (1) Diversity: When screening background images, ensure that a variety of environment types are covered to increase the diversity of the background; the control of scene distribution can be achieved by clustering analysis of background images to achieve diversified selection; (2) Consistency: The background image and the foreground image have a certain semantic or visual consistency. By calculating the feature similarity between the foreground image and the candidate background image, the background image with a higher degree of feature matching with the foreground image is selected. 2.3 Combination of foreground and background The filtered foreground image I fore Background images that meet the requirements of diversity and consistency base Recombination is performed to generate diverse synthetic images; in order to avoid the conflict between the foreground image and the background image, a position-aware strategy is used to adjust the position and scale of the foreground image; the details are as follows: 2.3.1 Synthetic Image Generation and Diversity Enhancement Randomly paste the adjusted foreground image onto the background image to generate a composite image I syn : I syn =Paste(I fore ,I base ,position) in, Paste indicates the pasting operation, and position indicates the placement of the foreground image. The rotation angle, brightness, and color of the foreground image are randomly adjusted during the synthesis process. 2.3.2 Quality Inspection For the generated synthetic image I syn Conduct quality checks, remove those that do not meet the requirements, and ensure the validity of the composite images; Step 3: Construction of synthetic dataset and model training 3.1 Construction of synthetic dataset The synthetic image I generated by the above steps syn Build synthetic datasets and fuse them with real base class datasets: synthetic data and real data are mixed in proportion to form a diversified training set; 3.2 Model Training The target detection model Faster R-CNN is used for training, and the end-to-end optimization method is used to improve the detection ability of the target detection model Faster R-CNN for new categories of targets. In order to better train the target detection model Faster R-CNN, a joint optimization loss function is used to optimize the classification loss and bounding box regression loss respectively. The classification loss is used to measure the prediction accuracy of the target category by the target detection model Faster R-CNN, while the bounding box regression loss is used to measure the prediction accuracy of the target position and size by the target detection model Faster R-CNN.

Images