Small sample target detection method and device based on multi-angle optimization

Abstract

The invention discloses a small sample target detection method and device based on multi-angle optimization. The method comprises the following steps: performing data enhancement on small sample category images by combining a BLIP model, a diffusion model and a CLIP model; constructing a small sample target detection model comprising a feature extraction network, a feature fusion network, a region candidate network, a region pooling network, a feature disturbance network, a classification network and a regression network, wherein the feature disturbance network adds disturbance based on feature categories through a feature disturbance mode based on gradient perception; preliminarily training a classifier based on basic category features and small sample category features, freezing classifier parameters, and integrally training a small sample target detection model; and performing target detection by using the trained small sample target detection model. According to the method, through multi-angle optimization design, the problems of unsatisfactory data enhancement scheme, confusion, forgetting and class imbalance are effectively improved, and the performance of small sample target detection tasks is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a small sample target detection method and device based on multi-angle optimization. Background Art

[0002] Object detection, a fundamental task in computer vision, aims to accurately identify and classify objects in images. Traditional object detection methods typically rely on large amounts of labeled data to train models. However, in practical applications, collecting large amounts of high-quality labeled data faces numerous challenges. The labeling process is labor-intensive and time-consuming, increasing project costs. Furthermore, data acquisition in certain scenarios is difficult, leading to a scarcity of labeled data. These challenges limit the widespread application of traditional object detection methods in a wider range of scenarios.

[0003] To overcome the performance bottleneck of traditional object detection methods when labeled data is scarce, Few-Shot Object Detection (FSOD) was developed. Its goal is to narrow the gap between machine vision models and the human visual system in processing small samples, enabling models to accurately detect new categories of objects even with only a small number of labeled examples. This not only reduces the reliance on large-scale labeled data but also opens up the possibility of applying object detection technology in scenarios where data acquisition is difficult.

[0004] Currently, small-shot object detection methods focus on improving model performance in four key areas: First, image enhancement: In recent years, the diffusion model (DM) has demonstrated excellent performance in generating high-quality data. Leveraging the DM to achieve richer image enhancements and expand the diversity of training data has become a research hotspot in the field of small-shot object detection. Second, improving model forgetfulness: In small-shot object detection, the model needs to leverage the basic category knowledge learned during pre-training to assist in learning the small-shot categories. Therefore, maintaining knowledge coherence during the learning process is crucial. Third, improving model confusion: When learning small-shot categories, the model can easily become confused with the basic category knowledge learned during pre-training, making it unable to accurately distinguish which category the target object belongs to. This confusion can seriously affect the accuracy of detection results and reduce the practicality of the model. Fourth, addressing class imbalance: The pre-training stage has a large number of basic category images, while the number of small-shot category images is sparse. This class imbalance can cause the model to favor the basic category during training and neglect the small-shot category. Therefore, balancing the difference in the number of images between the two categories is a key issue that needs to be addressed in small-shot object detection.

[0005] Currently, common small-sample object detection methods use a two-stage training strategy. Stage 1 involves pre-training on a large number of labeled images from basic categories, allowing the model to learn basic visual features and category knowledge. Stage 2 involves freezing the backbone network weights and then fine-tuning the model using small-sample category images, hoping to improve the model's detection capabilities for small-sample categories. However, existing methods still have some issues that need to be addressed:

[0006] First, data augmentation schemes are suboptimal. Data augmentation schemes based on diffusion models either use overly simple cue word designs, failing to fully exploit the model's strengths, or rely on powerful and expensive large language models. Furthermore, existing methods lack a close connection to the original images in the dataset and instead rely heavily on the diffusion model's inherent capabilities for augmentation, introducing additional semantics beyond the dataset's images, potentially defeating the original purpose of small-sample object detection.

[0007] Second, the confusion and forgetting problems have not been effectively solved: Although there are many solutions, most of them start from adjusting the model or optimizing the loss function. The confusion and forgetting problems are still prominent, which limits the improvement of model performance.

[0008] Third, the method of learning category centers directly through training is not friendly to small sample categories: the current method allows the classifier to learn the category centers of different categories of images by itself through gradient backpropagation, but due to the small number of small sample category images, it is often difficult to learn more accurate category centers, which further aggravates the confusion problem.

[0009] Therefore, it is necessary to comprehensively and deeply optimize and innovate the existing small-sample target detection methods to promote the maturity of small-sample target detection technology and achieve wider application. Summary of the Invention

[0010] In view of the above, the purpose of the present invention is to provide a small sample target detection method and device based on multi-angle optimization. By improving the data enhancement method and designing the feature perturbation method and classifier optimization method, the data quality in small sample target detection is improved, and the confusion, forgetfulness and category imbalance problems are improved, so that the performance of small sample target detection is improved.

[0011] To achieve the above-mentioned purpose, the present invention provides the following technical solutions:

[0012] In a first aspect, an embodiment of the present invention provides a small sample target detection method based on multi-angle optimization, comprising the following steps:

[0013] The BLIP model, diffusion model and CLIP model are combined to perform data enhancement on the small sample category image to obtain the small sample category data enhanced image;

[0014] Construct a small sample target detection model that includes a feature extraction network, a feature fusion network, a region candidate network, a region pooling network, a feature perturbation network, a classification network, and a regression network. The feature extraction network is used to extract preliminary image features, the feature fusion network is used to fuse preliminary features to obtain fused features, the region candidate network is used to extract target region candidate boxes based on fused features, the region pooling network is used to extract target features based on fused features and target region candidate boxes and adjust the target feature scale to a specified size, the feature perturbation network is used to add feature category-based perturbations to target features through a gradient-aware feature perturbation method to obtain the final features, the classification network is used to perform classification tasks based on the final features, and the regression network performs regression calculations based on the target region candidate boxes;

[0015] The classifier in the classification network is initially trained based on the basic category features and small sample category features obtained from the CLIP model. The classifier parameters are then frozen, and the small sample target detection model is trained as a whole based on the small sample category data augmented images and basic category images to obtain the trained small sample target detection model.

[0016] Use the trained small-sample target detection model to perform small-sample target detection.

[0017] Preferably, the combined BLIP model, the diffusion model, and the CLIP model perform data enhancement on the small sample category image to obtain the small sample category data enhanced image, including:

[0018] The BLIP model is used to extract the prompt word T used to describe the content of the basic category image. base , the prompt word T base The category keywords in are replaced with the category names of the selected small sample category images to obtain new prompt words T novel , generate a mask image P about the target position according to the image annotation content of the basic category image mask , according to the image annotation content of the small sample category, a sub-image corresponding to the target position is cropped from the small sample category image and resized to cover the target position in the base category image to obtain a new image P novel ;

[0019] Based on the new image P through the diffusion model novel , mask image P mask and the new prompt word T novel Perform image enhancement to obtain several enhanced small sample images P gen ;

[0020] Through the CLIP model based on the enhanced small sample image P genThe target sub-image extracted according to the image annotation content and the target sub-image extracted according to the image annotation content in the small sample category image are subjected to visual feature extraction and similarity calculation, and the enhanced small sample image P corresponding to the similarity greater than the set threshold is selected. gen The qualified images are added to the dataset to participate in the model training, and the remaining enhanced small sample images that are smaller than the set threshold are eliminated as unqualified images.

[0021] Preferably, the feature perturbation network is used to add a feature category-based perturbation to the target feature to obtain the final feature through a gradient-aware feature perturbation method, including:

[0022] In the feature perturbation network, based on the target feature F obtained by fusion feature and target region candidate box extraction target The category to which it belongs calculates the perturbation size F of each target feature dis , and the target feature F target Add up to get the final feature F final , the formula is as follows:

[0023]

[0024] F final =F dis +F target

[0025] Among them, α represents a hyperparameter, n represents the number of images corresponding to the current category, and k represents a positive integer. Represents the classification loss l of the small-shot target detection model cls For the gradient of each target feature, ||·||2 represents the L2 norm.

[0026] Preferably, the preliminary training of the classifier in the classification network based on the basic category features and small sample category features obtained from the CLIP model includes:

[0027] The category names of the small sample category data augmented images and the category names of the basic category images are input into the text encoder of the CLIP model to repeatedly extract several different category features as category centers. Several different category features are used to perform preliminary training on the classifier in the classification network. The cross entropy loss function is used during the preliminary training to enable the classifier to pre-learn the category centers that distinguish different categories.

[0028] Preferably, the original one fully connected layer of the classifier is optimized into two fully connected layers, the first fully connected layer is used to reduce the dimension of the features output by the text encoder of the CLIP model, and the second fully connected layer is used to perform classification based on the features after dimensionality reduction.

[0029] Preferably, only the second fully connected layer is trained during the preliminary training of the classifier, and only the parameters of the second fully connected layer are frozen during the overall training of the small-sample object detection model.

[0030] In a second aspect, to achieve the above-mentioned object of the invention, an embodiment of the present invention further provides a small sample target detection device based on multi-angle optimization, which is implemented using the above-mentioned small sample target detection method based on multi-angle optimization, and includes: a data enhancement module, a model construction module, a model training module, and a target detection module;

[0031] The data enhancement module is used to perform data enhancement on the small sample category image by combining the BLIP model, the diffusion model and the CLIP model to obtain a small sample category data enhanced image;

[0032] The model construction module is used to construct a small sample target detection model including a feature extraction network, a feature fusion network, a region candidate network, a region pooling network, a feature perturbation network, a classification network, and a regression network, wherein the feature extraction network is used to extract preliminary features of the image, the feature fusion network is used to fuse the preliminary features to obtain fused features, the region candidate network is used to extract target region candidate frames based on the fused features, the region pooling network is used to extract target features based on the fused features and the target region candidate frames and adjust the target feature scale to a specified size, the feature perturbation network is used to add feature category-based perturbations to the target features through a gradient-aware feature perturbation method to obtain final features, the classification network is used to perform classification tasks based on the final features, and the regression network performs regression calculations based on the target region candidate frames;

[0033] The model training module is used to initially train the classifier in the classification network based on the basic category features and small sample category features obtained from the CLIP model; then freeze the classifier parameters, and perform overall training on the small sample target detection model based on the small sample category data augmented image and the basic category image to obtain a trained small sample target detection model;

[0034] The target detection module is used to perform small sample target detection using the trained small sample target detection model.

[0035] On the third aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides an electronic device, including a memory and one or more processors, wherein the memory is used to store computer programs, and the processor is used to implement the above-mentioned small sample target detection method based on multi-angle optimization when executing the computer program.

[0036] Fourthly, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned small sample target detection method based on multi-angle optimization is implemented.

[0037] In a fifth aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a computer product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned small sample target detection method based on multi-angle optimization.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] (1) The present invention performs image data enhancement by combining the BLIP model, the diffusion model, and the CLIP model. It does not require a large language model with huge cost like GPT4o. At the same time, it can also make reasonable use of the original basic category images and small sample category images in the dataset, which is more in line with the small sample requirements of small sample target detection tasks.

[0040] (2) Different from the prior art that directly uses the adjustment of loss function or network to improve the confusion and forgetting of the model, the present invention adopts the feature perturbation method based on gradient perception for optimization. Through feature perturbation, the model has stronger generalization ability, further improves the confusion and forgetting, and also makes the image features of small sample categories closer to the category center, alleviating the problem of excessive category center offset caused by category imbalance and too few small sample images.

[0041] (3) Different from the prior art method of using gradient back propagation to enable the model to learn the category center through training, this method often fails to learn relatively accurate small sample category centers. The present invention cleverly utilizes the sufficient semantics of the CLIP model during the training period, and uses the text encoder of the CLIP model to calculate the category centers of different categories. The category center is used to guide the optimization of the classifier, so that the classifier learns to effectively distinguish different category centers, thereby guiding the features of images of different categories to be closer to the category center, further improving the category imbalance problem, and improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 11 is a flow chart of a small sample target detection method based on multi-angle optimization provided by an embodiment of the present invention;

[0044] Figure 2 1 is a flow chart of data enhancement based on a diffusion model according to an embodiment of the present invention;

[0045] Figure 3 Schematic diagram of the structure of a small sample target detection model provided by an embodiment of the present invention;

[0046] Figure 4 3 is a schematic diagram of the structure of a small sample target detection device based on multi-angle optimization provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0048] The inventive concept of the present invention is: in response to the problems that the data enhancement scheme in existing small-sample target detection is not ideal, the confusion and forgetting problems are not effectively solved, and the category center learning method is not friendly to small-sample categories, the embodiment of the present invention provides a small-sample target detection method and device based on multi-angle optimization, which significantly improves data diversity and quality through data enhancement methods based on BLIP model, diffusion model and CLIP model, while maintaining a low computational cost; by constructing a gradient-aware feature perturbation strategy, the robustness of the model to confusion and forgetting problems is successfully enhanced; by using the category center separation method based on the CLIP model text encoder, the discrimination ability of the classifier in small-sample scenarios is more effectively improved compared with the traditional gradient backpropagation training method. These multi-angle optimization innovations jointly improve the performance of small-sample target detection tasks and provide new technical ideas for the field of small-sample target detection.

[0049] like Figure 1 As shown, the embodiment provides a small sample target detection method based on multi-angle optimization, including the following steps:

[0050] S1, the BLIP model, diffusion model and CLIP model are combined to perform data enhancement on the small sample category image to obtain the small sample category data enhanced image.

[0051] Existing data augmentation methods based on diffusion models have two significant issues. The first is the design of prompt words. Some existing work uses very simple prompt words, such as "A photo of [class name]," where classname refers to the name of the category, such as cat or dog. Others use large language models, such as GPT4o. For the former prompt word approach, the diffusion model often cannot generate the required image, while for the latter, using a large language model incurs significant overhead. This research aims to find a balance between ensuring the quality of the generated image and saving costs. Therefore, in this embodiment, a BLIP model (specifically, the BLIP-2 model or other versions), a diffusion model (specifically, the image inpainting version or other versions of the diffusion model), and a CLIP model are designed to jointly accomplish this task. The total size of these three models is much smaller than that of a large language model, effectively saving computational overhead and costs.

[0052] like Figure 2 As shown in the figure, two images are first selected from the dataset, one of which is a basic category image and the other is a small sample category image. Considering the differences caused by different categories, the categories of the two images selected should be similar. For example, the similarity between cats and dogs is higher than that between cats and cars. After obtaining the two images, the BLIP-2 model is used to extract the prompt word T used to describe the content of the basic category image. base , the prompt word T base The category keywords in are replaced with the category names of the selected small sample category images to obtain new prompt words T novel Afterwards, we use the annotations of the images in the dataset to find the location of the target in the base category image and generate a mask image P based on the annotations. mask Then, the same method is used to find the target position in the small sample category image and crop the sub-image corresponding to the target position and cover it to the target position in the basic category image to obtain a new image P novel Since this is a target detection task, the image coverage cannot be performed arbitrarily using the traditional CutMix method, otherwise it will cause the position of the target detection box to change, resulting in errors or requiring additional labor costs for labeling.

[0053] Next, the new image P novel , mask image P mask and the new prompt word T novelThe image is then inpainted in the diffusion model, and the model is allowed to perform image inpainting tasks in the masked area of the new image according to the prompt word, thereby solving the incoordination problem between the foreground and background caused by image coverage, and enhancing the image with the help of the powerful ability of the diffusion model to obtain more enhanced small sample images P. gen This method not only reasonably utilizes the advantage of rich background of basic category images, but also uses small sample category images to generate enhanced images, without allowing the diffusion model to generate images arbitrarily and deviating from the small sample setting.

[0054] Finally, the generated enhanced small sample image P gen The corresponding target sub-images for both the enhanced image and the small sample category images are extracted based on the data's labeled locations. These sub-images are then fed into the CLIP model's visual component for feature extraction. The cosine similarity between the resulting enhanced image features and the small sample category features is calculated, and a threshold is set. When the similarity score is greater than or equal to the threshold, the image is considered qualified and added to the training dataset. The CLIP model is used for screening here because the diffusion model may fail to repair the small sample content during image restoration due to issues such as image clarity. Therefore, samples with similarity below the threshold need to be removed, otherwise the training accuracy will be affected.

[0055] By combining the BLIP-2 model, the image restoration version of the diffusion model, and the CLIP model, a set of small-sample category images enhanced based on the original dataset can be generated. The generated small-sample category images not only have a certain degree of generalization, but are also generated from the original dataset and do not deviate from the basic settings of the small-sample target detection task. At the same time, the CLIP model is used to perform automated image screening, remove images of insufficient quality, save human resources, and make this method more transferable.

[0056] S2, builds a small sample target detection model that includes a feature extraction network, a feature fusion network, a region candidate network, a region pooling network, a feature perturbation network, a classification network, and a regression network. The feature perturbation network adds feature category-based perturbations to the target features through a gradient-aware feature perturbation method.

[0057] like Figure 3As shown in Figure 1, the small sample target detection model includes a feature extraction network, a feature fusion network, a region candidate network, a region pooling network, a feature perturbation network, a classification network, and a regression network. The feature extraction network (specifically using ResNet-101 or other versions as the backbone network) is used to extract preliminary image features, the feature fusion network (specifically using the FPN network (Feature Pyramid Network)) is used to fuse preliminary features to obtain fused features, the region candidate network (specifically using the RPN network (Region Proposal Network)) is used to extract target region candidate frames based on fused features, and the region pooling network (specifically using the ROI Pooling network (Region of Interest)) is used to extract target region candidate frames based on fused features. Pooling (region of interest pooling) is used to extract target features based on fusion features and target area candidate boxes and adjust the target feature scale to a specified size. The feature perturbation network is used to add feature category-based perturbations to the target features through a gradient-aware feature perturbation method to obtain the final features. The classification and regression network (including classifiers and bounding box regressors) is used to perform target detection based on the final features. Specifically, the classification network performs classification tasks based on the final features, and the regression network performs regression calculations based on the target area candidate boxes.

[0058] Confusion in small sample target detection refers to the fact that small sample category images are easily judged as similar basic categories during reasoning due to insufficient training (for example, cat images are classified as dogs). The reason for the confusion is that the separation of class centers of different categories is not enough; forgetfulness refers to the fact that small sample target detection will forget some of the knowledge that has been learned during training. The reason for the forgetfulness is that the compactness within the class is not enough and the feature distribution around the class center is loose. For the small sample target detection task, due to the lack of small sample category images, the model will have deviations in its judgment of the category center during training. Therefore, in order to guide the model to learn more accurate category centers of small sample categories and to make the model more generalizable, a feature perturbation method based on gradient perception is designed in the embodiment for the target features obtained by fusion features and target region candidate box extraction. After obtaining the target features, the perturbation size F of each target feature is calculated in the feature perturbation network according to the category to which the target feature belongs. dis , and the target feature F target Add up to get the final feature F final , the formula is as follows:

[0059]

[0060] F final =F dis +F target

[0061] Wherein, α represents a hyperparameter, n represents the number of images corresponding to the current category; k represents a positive integer, and in the embodiment, a value of 2 or 4 is used to standardize the range of feature perturbation to prevent it from being too large or too small; Represents the classification loss l of the small-shot target detection model cls For the gradient of each target feature; ||·||2 represents the L2 norm, which is used to normalize the gradient to prevent excessive gradients from having a negative impact on feature offset, resulting in the failure of the model training process to converge.

[0062] Here, since there may be different categories of target objects in an image, and the feature perturbations of different target objects are different, if the feature perturbations are performed directly on the image, the perturbations of different categories will interfere with each other. Therefore, it is necessary to use the region candidate network to give different category region predictions first, and then obtain the sub-image based on the region predictions before feature perturbation can be performed. Then the final feature F final Input into the corresponding classifier and the predicted box obtained by the region candidate network is input into the bounding box regressor for subsequent classification and bounding box regression calculation.

[0063] As can be seen from the above formula, when the total number of images in the category to which the target image belongs is large, the amplitude of the feature perturbation is small, and when the total number of images in the category to which the target image belongs is small, the amplitude of the feature perturbation is large. This is because in categories with abundant images, the images contain sufficient semantics, and even without feature perturbation, good results can be achieved in target detection. Perturbation of these categories is only performed to further enhance robustness and generalization. For small sample categories, although there are many images generated by the diffusion model, due to the limitations of the diffusion model itself, the resulting images still cannot extract the same rich semantics as the basic category images. Therefore, performing a larger perturbation on the images of these categories will greatly promote the learning of small sample categories.

[0064] By perturbing the features in the direction of decreasing the loss gradient, the features can be more concentrated in the class center, while strengthening the intra-class compactness of each category feature and further expanding the separation between classes, thereby improving the confusion and forgetting problems in small sample target detection tasks.

[0065] S3, based on the basic category features and small sample category features obtained from the CLIP model, the classifier in the classification network is first preliminarily trained; then the classifier parameters are frozen, and the small sample target detection model is trained as a whole based on the small sample category data augmented images and basic category images to obtain the trained small sample target detection model.

[0066] In current mainstream work, classifiers for small-sample object detection are trained using gradient backpropagation. This method can achieve relatively good results when sufficient training images are available. However, for small-sample category images, direct training often fails to accurately determine the small-sample category center, as these images do not represent the actual distribution of all images and may deviate from the true category center.

[0067] To address this issue, previous work has proposed using a simplex equiangular tight frame (Simplex ETF) to address this challenge. Specifically, this approach addresses this problem by forcing the separation between category centers of different categories to be the same. This approach still has a headache, because the correlation between different categories is not the same, and it is not reasonable to forcibly require the same separation between category centers of different categories. For example, the similarity between cat category images and dog category images must be greater than the similarity between cat category images and car category images. Therefore, although this method can alleviate the confusion problem of small sample category centers to a certain extent, it also brings new problems.

[0068] Therefore, in order to find a more accurate degree of separation between different category centers, the present invention utilizes the rich semantics of the CLIP model pre-training period to achieve it. During the pre-training period, the CLIP model trains a large number of text pairs with images, so that the CLIP model can help distinguish the degree of separation between different category centers with the help of the rich semantics learned in the past. In an embodiment, the category name of the image with small sample category data enhancement and the category name of the basic category image are first input into the text encoder of the CLIP model, and the CLIP model is allowed to calculate the features of different categories, where the features are actually the centers of the categories, and the text encoder of the CLIP model is repeatedly allowed to generate features 30 times. Subsequently, the classifier is preliminarily trained using these data to allow the classifier to learn to distinguish the category centers of different categories. The classifier adopts traditional cross entropy loss. Subsequently, the weight parameters of the classifier are frozen, and the entire model is trained again. At this time, the classifier has learned how to distinguish the category centers of different categories and will also guide the features of the different category images towards the direction of the category centers.

[0069] In actual operation, since the features extracted from the image are of 1024 dimensions, while the features calculated by the text encoder of the CLIP model are of 512 dimensions, the features of the text encoder cannot be directly used to train the original classifier. Because the classifier is optimized from the original one fully connected layer to two fully connected layers, the first fully connected layer reduces the 1024-dimensional features to 512-dimensional features, and the second fully connected layer classifies the 512-dimensional features. Therefore, the above-mentioned method only needs to train the second fully connected layer. At the same time, after the training is completed, the weights of the second fully connected layer need to be frozen to prevent the learned category center separation knowledge from being forgotten during subsequent training.

[0070] Finally, after preliminary training and overall training, the trained small sample target detection model is obtained.

[0071] S4, uses the trained small sample target detection model to perform small sample target detection.

[0072] After completing the model training process and ensuring that the small-sample target detection model reaches the ideal performance state, the trained small-sample target detection model is officially used to carry out comprehensive and accurate target detection for a given small-sample dataset.

[0073] In summary, the embodiment of the present invention provides a small sample target detection method based on multi-angle optimization. First, it uses advanced data enhancement methods to combine the BLIP-2 model, the image restoration version of the diffusion model, and the CLIP model to perform multiple and diverse data enhancement operations on small sample category images, thereby obtaining more rich and different small sample category images, and effectively adding these carefully enhanced images to the training data set. This measure not only expands the data scale, but also cleverly solves the common category imbalance problem in small sample target detection tasks, providing a more balanced data foundation for model training. Subsequently, this method innovatively designs a feature perturbation method. By performing detailed and targeted perturbation processing of different degrees on images of different categories, the model can obtain more robust feature representations when extracting features. At the same time, it also promotes features to be more closely clustered in the feature space, and the feature differences between images of different categories are significantly amplified, thereby effectively improving the classification ability and discrimination of the model. Finally, through deep optimization and careful training of the classifier, the entire small-sample target detection model has achieved significant improvements in multiple key indicators such as accuracy, recall rate, and generalization ability, providing an efficient and practical solution for the field of small-sample target detection.

[0074] Based on the same inventive concept, Figure 4As shown, an embodiment of the present invention further provides a small sample target detection device 400 based on multi-angle optimization, including: a data enhancement module 410, a model construction module 420, a model training module 430, and a target detection module 440.

[0075] The data enhancement module 410 is used to perform data enhancement on the small sample category image by combining the BLIP model, the diffusion model and the CLIP model to obtain a small sample category data enhanced image.

[0076] The model construction module 420 is used to construct a small sample target detection model including a feature extraction network, a feature fusion network, a region candidate network, a region pooling network, a feature perturbation network, a classification network, and a regression network, wherein the feature extraction network is used to extract preliminary features of the image, the feature fusion network is used to fuse the preliminary features to obtain fused features, the region candidate network is used to extract the target region candidate box based on the fused features, the region pooling network is used to extract the target features based on the fused features and the target region candidate box and adjust the target feature scale to a specified size, the feature perturbation network is used to add feature category-based perturbations to the target features through a gradient-aware feature perturbation method to obtain the final features, the classification network is used to perform classification tasks based on the final features, and the regression network performs regression calculations based on the target region candidate box.

[0077] The model training module 430 is used to perform preliminary training on the classifier in the classification network based on the basic category features and small sample category features obtained from the CLIP model; then freeze the classifier parameters, and perform overall training on the small sample target detection model based on the small sample category data augmented image and the basic category image to obtain the trained small sample target detection model.

[0078] The target detection module 440 is used to perform small sample target detection using the trained small sample target detection model.

[0079] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device, including a memory and one or more processors, the memory is used to store computer programs, and the processor is used to implement the above-mentioned small sample target detection method based on multi-angle optimization when executing the computer program.

[0080] Based on the same inventive concept, an embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a computer, the above-mentioned small sample target detection method based on multi-angle optimization is implemented.

[0081] Based on the same inventive concept, an embodiment of the present invention further provides a computer product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned small sample target detection method based on multi-angle optimization.

[0082] It should be noted that the small sample target detection device based on multi-angle optimization, electronic device, computer-readable storage medium, and computer product provided in the above embodiments all belong to the same inventive concept as the small sample target detection method based on multi-angle optimization. The specific implementation process is detailed in the embodiment of the small sample target detection method based on multi-angle optimization, which will not be repeated here.

[0083] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A small sample target detection method based on multi-angle optimization, characterized in that: The following steps are involved: The BLIP model, diffusion model and CLIP model are combined to perform data enhancement on the small sample category image to obtain the small sample category data enhanced image; Construct a small sample target detection model that includes a feature extraction network, a feature fusion network, a region candidate network, a region pooling network, a feature perturbation network, a classification network, and a regression network. The feature extraction network is used to extract preliminary image features, the feature fusion network is used to fuse preliminary features to obtain fused features, the region candidate network is used to extract target region candidate boxes based on fused features, the region pooling network is used to extract target features based on fused features and target region candidate boxes and adjust the target feature scale to a specified size, the feature perturbation network is used to add feature category-based perturbations to target features through a gradient-aware feature perturbation method to obtain the final features, the classification network is used to perform classification tasks based on the final features, and the regression network performs regression calculations based on the target region candidate boxes; The classifier in the classification network is initially trained based on the basic category features and small sample category features obtained from the CLIP model. The classifier parameters are then frozen, and the small sample target detection model is trained as a whole based on the small sample category data augmented images and basic category images to obtain the trained small sample target detection model. Use the trained small-sample target detection model to perform small-sample target detection.

2. The small sample target detection method based on multi-angle optimization according to claim 1 is characterized in that: The combined BLIP model, diffusion model, and CLIP model perform data enhancement on the small sample category image to obtain a small sample category data enhanced image, including: The BLIP model is used to extract the prompt word T used to describe the content of the basic category image. base , the prompt word T base The category keywords in are replaced with the category names of the selected small sample category images to obtain new prompt words T novel , generate a mask image P about the target position according to the image annotation content of the basic category image mask , according to the image annotation content of the small sample category, a sub-image corresponding to the target position is cropped from the small sample category image and resized to cover the target position in the base category image to obtain a new image P novel ; Based on the new image P through the diffusion model novel , mask image P mask and the new prompt word T novel Perform image enhancement to obtain several enhanced small sample images P gen ; Through the CLIP model based on the enhanced small sample image P gen The target sub-image extracted from the image annotation content and the target sub-image extracted from the image annotation content in the small sample category image are subjected to visual feature extraction and similarity calculation, and the enhanced small sample image P corresponding to the similarity greater than the set threshold is selected. gen The qualified images are added to the dataset to participate in the model training, and the remaining enhanced small sample images that are smaller than the set threshold are eliminated as unqualified images.

3. The small sample target detection method based on multi-angle optimization according to claim 1 is characterized in that: The feature perturbation network is used to add feature category-based perturbations to target features through a gradient-aware feature perturbation method to obtain final features, including: In the feature perturbation network, based on the target feature F obtained by fusion feature and target region candidate box extraction target The category to which it belongs calculates the perturbation size F of each target feature dis , and the target feature F target Add up to get the final feature F final , the formula is as follows: F final =F dis +F target Among them, α represents a hyperparameter, n represents the number of images corresponding to the current category, and k represents a positive integer. Represents the classification loss l of the small-shot target detection model cls For the gradient of each target feature, ||·||2 represents the L2 norm.

4. The small sample target detection method based on multi-angle optimization according to claim 1 is characterized in that: The classifier in the classification network is initially trained based on the basic category features and small sample category features obtained from the CLIP model, including: The category names of the small sample category data augmented images and the category names of the basic category images are input into the text encoder of the CLIP model to repeatedly extract several different category features as category centers. Several different category features are used to perform preliminary training on the classifier in the classification network. The cross entropy loss function is used during the preliminary training to enable the classifier to pre-learn the category centers that distinguish different categories.

5. The small sample target detection method based on multi-angle optimization according to claim 1 or 4, characterized in that: The original fully connected layer of the classifier is optimized into two fully connected layers. The first fully connected layer is used to reduce the dimensionality of the features output by the text encoder of the CLIP model, and the second fully connected layer is used to perform classification based on the features after dimensionality reduction.

6. The small sample target detection method based on multi-angle optimization according to claim 5 is characterized in that: Only the second fully connected layer is trained during the initial training of the classifier, and only the parameters of the second fully connected layer are frozen during the overall training of the few-shot object detection model.

7. A small sample target detection device based on multi-angle optimization, implemented using the small sample target detection method based on multi-angle optimization according to any one of claims 1 to 6, characterized in that: include: Data augmentation module, model building module, model training module, and target detection module; The data enhancement module is used to perform data enhancement on the small sample category image by combining the BLIP model, the diffusion model and the CLIP model to obtain a small sample category data enhanced image; The model construction module is used to construct a small sample target detection model including a feature extraction network, a feature fusion network, a region candidate network, a region pooling network, a feature perturbation network, a classification network, and a regression network, wherein the feature extraction network is used to extract preliminary features of the image, the feature fusion network is used to fuse the preliminary features to obtain fused features, the region candidate network is used to extract target region candidate frames based on the fused features, the region pooling network is used to extract target features based on the fused features and the target region candidate frames and adjust the target feature scale to a specified size, the feature perturbation network is used to add feature category-based perturbations to the target features through a gradient-aware feature perturbation method to obtain final features, the classification network is used to perform classification tasks based on the final features, and the regression network performs regression calculations based on the target region candidate frames; The model training module is used to initially train the classifier in the classification network based on the basic category features and small sample category features obtained from the CLIP model; then freeze the classifier parameters, and perform overall training on the small sample target detection model based on the small sample category data augmented image and the basic category image to obtain a trained small sample target detection model; The target detection module is used to perform small sample target detection using the trained small sample target detection model.

8. An electronic device comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that: The processor is configured to implement the small sample target detection method based on multi-angle optimization according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer, the small sample target detection method based on multi-angle optimization according to any one of claims 1 to 6 is implemented.

10. A computer product comprising a computer program, characterized in that When the computer program is executed by a processor, the small sample target detection method based on multi-angle optimization according to any one of claims 1 to 6 is implemented.

Images