Small sample target detection method based on multilevel similarity transfer and adaptive fusion data enhancement

Through multi-level similarity transfer and adaptive fusion data enhancement, the problem of true and false positives of new samples in small sample object detection is solved, which improves the classification performance and detection accuracy of the model, and enhances the sample in Generalization ability in scarce situations.

CN118918309APending Publication Date: 2024-11-08NANJING UNIV OF POSTS & TELECOMM
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410909853.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In small sample object detection, the true and false positives of new class samples are serious, resulting in a decline in model performance. This is mainly due to the similarity between shallow visual knowledge and deep feature knowledge between different categories, which leads to the model's mistaken classification of new classes into a basic basis. Classes or backgrounds and base classes are divided into new classes.

Method used

Using a small sample object detection method based on multi-level similarity transfer and adaptive fusion data augmentation, we will increase sample diversity through adaptive fusion data augmentation strategy, and establish inter-class connections through the similarity transfer strategy of shallow visual knowledge and deep feature knowledge. Dynamic prototype correction library improves model efficiency and convergence speed.

Benefits of technology

The classification performance and generalization ability of the model are improved, false positive problems are reduced, and the detection accuracy and efficiency of the model in the case of scarcity of samples are enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918309A_ABST
    Figure CN118918309A_ABST
Patent Text Reader

Abstract

The invention provides a novel small sample target detection method based on similarity transfer. According to the method, the correlation between different categories can be effectively utilized. Specifically, firstly, a self-adaptive fusion data enhancement strategy is utilized to enhance the diversity of new-class samples, and the false positive problem is further relieved. Then, similarity transfer of shallow visual knowledge and deep feature knowledge between classes is utilized to relieve the problem of false positive new class samples; experimental results show that the method has good performance in small sample target detection tasks, and has stronger competitiveness compared with the existing method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and machine learning, and specifically relates to a small sample target detection method under the condition of scarce samples. Background Art

[0002] In the past few decades, the rapid development of deep learning has promoted great progress in object detection. However, traditional object detection methods require a large number of manually labeled data samples to achieve the desired results, which usually costs a lot of money and time. In addition, the learning process of traditional object detection methods is very complicated because it not only needs to locate image objects, but also needs to classify image objects. Traditional object detection models perform poorly in the case of few samples, but humans can observe, learn and recognize new objects with scarce samples. Therefore, small-sample object detection has received more and more attention and is widely used in medical image pathology analysis, industrial product quality inspection, crop growth status detection and other fields. Small-sample object detection aims to narrow the gap between humans and models, so that the model can have good generalization ability even when the samples are sparse.

[0003] To solve the above problems, small sample target detection has become a very popular solution. At present, most small sample target detection methods are mainly based on the two-stage fine-tuning strategy. This is because the two-stage fine-tuning strategy can effectively alleviate the problem of sample imbalance during training, greatly improve detection accuracy, and enhance generalization ability.

[0004] However, there is a common and very serious problem in small sample size object detection, namely, the true positives and false positives of new class samples. That is, the model mistakenly classifies the new class as the base class, resulting in a decrease in model performance. The main reason for this problem is the similarity between the shallow visual knowledge and deep feature knowledge that may exist between different classes, which is an unavoidable problem in the model training process. The false positive problem can be divided into the following two categories: (1) mistakenly classifying the new class as the base class, and (2) mistakenly classifying the background and base class as the new class. Summary of the invention

[0005] The present invention proposes a small sample target detection method based on multi-level similarity transfer and adaptive fusion data enhancement. The method comprises: (1) A small sample target detection method based on multi-level similarity transfer and adaptive fusion data enhancement is proposed, which helps to improve the model performance. (2) An adaptive fusion data enhancement strategy for data enhancement is proposed, which increases the diversity of samples and helps the model further learn the relationship between different categories. (3) A similarity transfer strategy of shallow visual knowledge and deep feature knowledge is proposed, which establishes connections between classes and improves the classification performance of the model. (4) In order to improve the efficiency of the model, we propose a dynamic prototype correction library to improve the convergence speed of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 This is the framework diagram of the method proposed in this paper;

[0007] Figure 2 Schematic diagram of adaptive fusion data enhancement;

[0008] Figure 3 It is a structural diagram of the similarity of shallow visual knowledge;

[0009] Figure 4 A structural diagram of the similarity of deep visual knowledge DETAILED DESCRIPTION

[0010] The technical solution of the present invention is described in detail below with reference to the accompanying drawings:

[0011] The purpose of this invention is to provide a small sample target detection based on multi-level similarity transfer and adaptive fusion data enhancement. The method proposed in this paper aims to provide small sample target detection under sample scarcity. Specifically, it includes the following steps:

[0012] S1: The performance of the proposed method is evaluated on two public small sample detection datasets: PASCAL VOC and MS COCO. Faster-RCNN is selected as the backbone network of the small sample model proposed in this paper, the number of training samples in each batch is set to 4, and ResNet-101 is selected as the feature extractor pre-trained on ImageNet. The maximum feature storage sequence of the dynamic prototype correction library is set to 2048, and the positive proportionality factor β is set to 5. The k value in the feature distribution similarity transfer is set to 2, and the sampling set Z is set to 4. c Set to 10. The hyperparameter λ in the loss r and λ f Both are set to 0.1. In addition, after 250 iterations of the fine-tuning phase, the inter-category similarity transfer is started.

[0013] S2: Use the fine-tuning module to infer the base class samples and select the false positive samples whose scores are greater than a given threshold.

[0014] S3: Use the CLIP module to delete base class samples that do not meet the quality requirements, including unlabeled base class samples and base class samples that are easily confused with new class samples.

[0015] S4: Select the randomly generated candidate box (blue box) and the bounding box B of the base class sample o =[b o1 , b o2 , ...] (yellow box) and the bounding box B of the false positive sample F =[b f1, b f2 ,.......](red box)’s minimum IOU is used for data fusion.

[0016] S5: Based on the feature vector obtained by the embedded model F, the similarity between categories is calculated by cosine distance to obtain soft labels.

[0017] S6: The shallow visual knowledge similarity between sample r and all other samples is obtained through soft labels, and then the KL divergence is minimized to optimize the base class proposal area.

[0018] S7: Use the loss function to get P based on the results of S5 and S6 s1 .

[0019] S8: Calculate the mean and variance of the deep feature knowledge of the base class and the new class in the pre-trained feature space, and select the top k base class samples with the highest semantic similarity with each new class according to the Euclidean distance.

[0020] S9: Perform deep feature knowledge similarity transfer by calculating the mean and variance of the deep feature knowledge of the base class and the new class, and perform instance sampling on the transferred results

[0021] S10: Use the loss function to optimize the results of steps S8 and S9 to obtain P s2 ;

[0022] Furthermore, in the steps S2, S3, and S4, the adaptive fusion data enhancement process is as follows:

[0023] M = argmin(IOU(G(CD(I n )),B))

[0024] B represents all bounding boxes, CD represents the cropping and downsampling of new class samples, and G represents the generation of random candidate bounding boxes. In addition, IOU calculates the overlap value between all potential candidate boxes, and M represents the final selected fusion position.

[0025] Furthermore, in step S4, the soft label obtained by calculating the cosine distance can be expressed as:

[0026]

[0027] Among them, C n represents a new set of classes, and β represents a positive proportional factor. c The prototype in the dynamic correction library can be expressed as:

[0028]

[0029] in The feature vector representing the i-th base class.

[0030] Secondly, while paying attention to the similarity of shallow visual knowledge between the base class and the new class, we should also pay attention to the similarity of shallow visual knowledge within the base class. Assume that the distance between samples of different base classes can be expressed by the constant shown below:

[0031]

[0032] Among them, C b Represents a base class set.

[0033] Furthermore, in steps S5, S6, and S7, the shallow visual knowledge similarity between sample r and all other samples is as follows:

[0034]

[0035] Finally, the shallow visual knowledge similarity obtained from the proposed region is used as a soft label to guide and supervise the training process of the proposed small-shot object detection model. The optimization of the base class proposal region is achieved by minimizing the KL divergence, where the divergence is obtained through the soft label and the predicted class probabilities generated by the few-shot object detection model Calculated:

[0036]

[0037] Among them, c bg Represents the background class.

[0038] Furthermore, for the new class c in steps S8 and S9, its deep feature knowledge variance can be approximated as the average variance of the deep feature knowledge of its most similar base class, and the formula is as follows:

[0039]

[0040] Among them, σ iis the variance of the ith base class, and Sc is the set of the first k base classes closest to the new class c. For the mean, we need to consider both the mean of the deep feature knowledge of the new class and the mean of the deep feature knowledge of the base class. Therefore, this paper uses the following formula to calculate the mean:

[0041]

[0042] where μi is the mean of the ith base class, and μc is the mean of the new class c. In this case, the deep feature knowledge distribution of the new class c can be expressed as N(μ' c , σ' c ), and more instances of the new class c are sampled using the following formula:

[0043] Z c ={z|z~N (μ′ c ,σ′ c )}

[0044] The μ' c and σ' c is the mean and variance of the new class c after deep feature knowledge transfer, Z c is the set of sampled instances.

[0045] Finally, the detection model is trained using the cross entropy loss function:

[0046]

[0047] Among them, f θ (z) represents the model inference result.

[0048] Furthermore, in step S10, similarity transfer between different categories is achieved, and the final loss can be expressed by the following formula:

[0049] L=L pre +λ r L sim-rec +λ f L sim-fd

[0050] The λ r and λ f is a hyperparameter that balances different losses.

[0051] Among them, L pre In the pre-training phase, Faster RCNN is selected as the backbone network and a large number of base class samples are used for training. The corresponding training loss function is as follows:

[0052] L pre =L rpn +L cls +L reg

[0053] Among them, L cls represents the cross entropy loss function used in the classification task, L reg Represents the smoothed L1 loss function used for the regression task.

Claims

1. A small sample target detection method based on multi-level similarity transfer and adaptive fusion data enhancement, comprising the following steps: The first step is to use adaptive fusion data enhancement method to alleviate the problem of sample scarcity; In the second step, the connection between classes is established through the similarity transfer strategy of shallow visual knowledge and deep feature knowledge to improve the classification performance of the model; in the third step, the convergence speed of the model is improved through a dynamic prototype correction library.

2. The method as claimed in claim 1 uses the fine-tuning module to infer the base class samples and selects the false positive samples with scores greater than a given threshold, and then uses the CLIP module to delete the base class samples that do not meet the quality requirements, including the base class samples that are not labeled and the base class samples that are easily confused with the new class samples. Then select the randomly generated candidate box (blue box), the bounding box B of the base class sample o =[b o1 , b o2 ,.......](yellow box) and the bounding box B of the false positive sample F =[b f1 , b f2 ,.......](red box) to perform data fusion with the minimum IOU. The process can be expressed as follows M = argmin(IOU(G(CD(I n )),B)) (1) B represents all bounding boxes, CD represents the cropping and downsampling of new class samples, and G represents the generation of random candidate bounding boxes. In addition, IOU calculates the overlap value between all potential candidate boxes, and M represents the final selected fusion position.

3. The method as claimed in claim 1, based on the feature vector obtained by the embedded model F, calculates the similarity between categories through cosine distance to obtain soft labels. Among them, C n represents a new set of classes, and β represents a positive proportional factor. c The prototype in the dynamic correction library can be expressed as: in Characteristic vector representing the i-th base class. Secondly, while paying attention to the similarity of shallow visual knowledge between the base class and the new class, we should also pay attention to the similarity of shallow visual knowledge within the base class. Assume that the distance between samples of different base classes can be expressed by the constant shown below: Among them, C b Represents a base class set.

4. According to the method of claim 1, the shallow visual knowledge similarity between sample r and all other samples is obtained as follows: Finally, the shallow visual knowledge similarity obtained from the proposed region is used as a soft label to guide and supervise the training process of the proposed small-shot object detection model. The optimization of the base class proposal region is achieved by minimizing the KL divergence, where the divergence is obtained through the soft label and the predicted class probabilities generated by the few-shot object detection model Calculated: Among them, c bg Represents the background class.

5. According to the method of claim 1, the variance of the deep feature knowledge can be approximated as the average variance of the deep feature knowledge of its most similar base class, and the formula is as follows: Among them, σ i is the variance of the ith base class, and Sc is the set of the first k base classes closest to the new class c. For the mean, we need to consider both the mean of the deep feature knowledge of the new class and the mean of the deep feature knowledge of the base class. Therefore, this paper uses the following formula to calculate the mean: where μi is the mean of the ith base class, and μc is the mean of the new class c. In this case, the deep feature knowledge distribution of the new class c can be expressed as N(μ' c , σ' c ), and more instances of the new class c are sampled using the following formula: Z c ={z|z~N (μ′ c , σ' c )} (9) The μ' c and σ′ c is the mean and variance of the new class c after deep feature knowledge transfer, Z c is the set of sampled instances. Finally, the detection model is trained using the cross entropy loss function: Among them, f θ (z) represents the model inference result. The final loss can be expressed as follows: L=L pre +λ r L sim-rec +λ f L sim-fd (11) The λ r and λ f is a hyperparameter that balances different losses. Among them, Lpre In the pre-training phase, Faster RCNN is selected as the backbone network and a large number of base class samples are used for training. The corresponding training loss function is as follows: L pre =L rpn +L cls +L reg (12) Among them, L cls represents the cross entropy loss function used in the classification task, L reg Represents the smoothed L1 loss function used for the regression task.

Citation Information

Patent Citations

  • Small sample image classification method and device based on multi-modal symmetric enhancement, and medium

    CN116452895A

  • Flower classification method, system and equipment based on knowledge distillation and medium

    CN117058437A

  • Small sample classification detection method based on hierarchical feature fusion network

    CN118154973A

  • Method for person re-identification based on deep model with multi-loss fusion training strategy

    US20200285896A1