Small sample target detection method oriented to scarce sample scene and based on prototype network feature enhancement

Abstract

According to the small sample target detection method based on prototype network feature enhancement, firstly, a foreground feature aggregation module is adopted, redundant background information is removed in the category prototype construction process, and purer category features are extracted; and a condition information coupling module is utilized, and the category prototype is dynamically adjusted in combination with the characteristics of the query image, so that the adaptability is better. And then, dynamically supplementing the most similar support sample through a support sample expansion module in the training process, improving the expression ability of the category prototype, and enhancing the generalization performance of the model. Finally, an optimization strategy based on meta learning is adopted, and a joint optimization mode of classification loss, regression loss and meta loss is combined, so that the matching precision of the category prototypes is improved. The method has the advantages that the detection precision and generalization ability of small sample target detection are improved under the condition that extra data labeling cost is not increased, and the method is suitable for various application scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and object detection, and particularly relates to a few-shot object detection method based on prototype network feature enhancement for scarce sample scenarios. Background Art

[0002] Few-shot object detection is a technology for achieving object detection with only a small number of labeled samples. The core goal of few-shot object detection technology is to learn the generalization ability of the model so that it can still maintain a high detection accuracy when the number of samples of new category objects is small.

[0003] Currently, the mainstream few-shot object detection methods mainly include two categories: transfer learning and meta-learning. The transfer learning method pre-trains the model on a large-scale dataset and fine-tunes it on small-sample data to improve the detection performance. However, transfer learning is sensitive to hyperparameters and usually requires a large amount of training data to achieve good detection results. In contrast, the meta-learning method simulates few-shot learning tasks to improve the model's fast adaptation ability, can share knowledge between different tasks, and improve the generalization performance. Therefore, meta-learning has become a hot research direction in current few-shot object detection. In 2024, Zichen Wang et al. published the paper "Fine-Grained Prototypes Distillation for Few-Shot Object Detection", in which the FPD model was proposed to enhance the two-branch feature fusion in few-shot object detection based on meta-learning.

[0004] Existing few-shot object detection methods mainly rely on the category features in the support set to construct category prototypes and complete object detection by calculating the similarity between the query image and the category prototypes. However, the traditional category prototype construction method has the following deficiencies: 1) Redundant background information interference: When constructing category prototypes, the existing methods fail to effectively remove background noise, resulting in the extracted prototype features may contain irrelevant information, affecting the accuracy of object detection; 2) Lack of query image adaptation ability: Category prototypes are usually constructed only based on support samples, and the information of the query image is not fully utilized, resulting in the prototypes may not well match the features of the query image; 3) Limited support set samples: In few-shot scenarios, the number of available samples in the support set is limited, restricting the expressive ability of category prototypes and affecting the detection accuracy of the model. Summary of the Invention

[0005] The object of the present invention is to further improve the detection accuracy of small - sample object detection in the scenario of scarce samples, aiming to solve the problems existing in the existing small - sample object detection methods during the construction of class prototypes, such as background interference, insufficient query - image adaptation ability, and insufficient support - set samples. A small - sample object detection method based on prototype - network feature enhancement for scarce - sample scenarios is proposed.

[0006] The technical solution adopted by the present invention includes the following steps:

[0007] A small - sample object detection method based on prototype - network feature enhancement for scarce - sample scenarios includes the following steps:

[0008] Step A: Obtain the training set and test set in the small - sample object detection task, and pre - process the data, including image normalization and data augmentation operations. Then, use the support - sample expansion module to amplify the samples in the support set, select additional samples based on similarity, and improve the stability of the class prototypes.

[0009] Step B: Use the backbone network to extract image features, and process the samples in the support set and query set respectively. Remove the background features through the foreground - feature aggregation module, and extract the class prototypes to improve the discriminative ability of the prototypes.

[0010] Step C: Calculate the class prototypes of the support - set samples, and combine the query - image information through the conditional - information coupling module to optimize the class prototypes to make them more suitable for the current query image. Calculate the similarity between the optimized class prototypes and the query - image features to complete object classification and localization.

[0011] Step D: According to the fused features, calculate the loss function of the model, and then minimize the loss function to update the parameters of the model through the gradient - descent algorithm.

[0012] Step E: Save the optimal model - parameter weights after training for subsequent model testing.

[0013] Step F: Predict the images in the test set. According to the trained model - parameter weights described in Step E, predict the position and class of the target, and then obtain the final prediction result through non - maximum suppression.

[0014] The foreground - feature extraction for each image in the training set and test set samples described in Step B is specifically as follows:

[0015] Step B1: Adopt the support - sample expansion method based on similarity to select the most similar samples in the same training batch for supplementation.

[0016] Step B2: For the given support - sample feature matrix X1 and candidate - sample feature matrix X2, calculate their cosine similarity S, and the formula is: Then, select the K samples with the highest similarity and add them to the support set to form an extended support sample set;

[0017] Step B3. Assume that the feature matrix X obtained by extracting the support samples through the backbone network E , X E ∈R h×w×d , where h and w are the spatial dimensions of the feature map, and d is the number of channels. Define a learnable foreground attention vector q c , q c ∈R n×d′ , and calculate the foreground attention weight A c of class c. The formula is: where softmax is a commonly used mathematical function that maps each element in a vector to a value between 0 and 1, and the sum of all elements is 1. Then, use A c to calculate the fine-grained prototype P c of class c. The formula is: P c = A c ·X E ;

[0018] Step B4. After extracting the class prototypes, they need to be matched with the query image features. Define the feature matrix of the query image as X q , and calculate the affinity matrix A' between it and the class prototype P. The formula is: Then, use A' to adjust the query image. The formula is: X′ q = X q + A′·P.

[0019] Step C optimizes the class prototype by calculating the class prototypes of the support set samples and combining the query image information. Specifically:

[0020] Step C1. Perform key-value pair mapping (K q , V q ) between the optimized query image feature X′ q and the class prototype P to generate an adapted representation Q of the query feature. The formula is: K q = X′ q ·W K + b K ; V q = X′ q ·W V + b V ; Q = P·W Q , where W K , W V , W Q , bK and b V are both learnable parameters, and then calculate the correlation weight matrix A between the query feature and the class prototype S , and the formula is: Then, based on A s reconstruct the query image feature X″ q , and the formula is: X″ q = Conv(A s ·X q ), where Conv is the convolution operation;

[0021] Step C2: To make the class prototype more adaptable, it is necessary to further calculate the class prototype P specific to the query image q . First, calculate the cosine similarity matrix C between the query feature X″ q and the class prototype P. The formula is: C = COS(GAP(X″ q ), P), where COS represents the cosine similarity and GAP represents global average pooling. Then, based on C, calculate the class prototype P specific to the query image q , and the formula is: P q = C⊙X″ q + P, where ⊙ represents element-wise multiplication. Finally, calculate the class prototype P qc , and the formula is: where R(P q,i ) is calculated by a fully connected layer and the Sigmoid function, and is used to measure the importance of each support sample;

[0022] Step C3: Combine the optimized class prototype P qc with the query image feature X″ q . The class score S c is calculated by the formula: The target bounding box is generated by the region proposal network and ROI pooling, and finally the target detection result is obtained.

[0023] Step D calculates the loss function of the model, specifically:

[0024] Step D1: The loss function consists of three parts. The first part is the classification loss L cls , which is used to evaluate the accuracy of target classification and is calculated using cross-entropy loss. The formula is: where y i is the true class label, and p i is the probability of the predicted class;

[0025] Step D2: The second part is the regression loss L reg, which is used to evaluate the prediction accuracy of the target bounding box and is calculated using the L1 loss. The formula is: where y i is the value of the ground truth bounding box, and p i is the value predicted by the model;

[0026] Step D3: The third part is the meta-loss L meta , which is used to optimize the class prototype to make it more matched with the query image features. It is calculated using the cross-entropy-based loss. First, calculate the cross-entropy loss for each query sample q. The formula is where p(y = k|q) is the probability that the query sample q belongs to class k, calculated using the softmax function. Then, calculate the meta-loss L meta , and the formula is: where N is the size of the support set, and the total loss function is a linear weighted combination of these three losses:

[0027] L = L cls + L reg + αL meta

[0028] where α is the loss weight coefficient, set to 0.1.

[0029] The present invention mainly has the following advantages: 1) In order to remove background information and improve the quality of class prototypes, a foreground feature aggregation module is adopted to remove the image background features and construct more accurate class prototypes; 2) In order to make the prototypes adapt to the query images and improve the feature matching degree, a conditional information coupling module is introduced to dynamically adjust the class prototypes in combination with the query image features to make them more conform to the feature distribution of the query samples; 3) In order to enhance the utilization rate of support samples and improve the detection accuracy, a support sample expansion module is designed to dynamically supplement the most similar support samples during the training process to improve the expressive ability of class prototypes; 4) In order to accelerate the convergence speed of the model, a joint optimization strategy of classification loss, regression loss, and meta-loss is used. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is the implementation flowchart of the embodiment of the present invention.

[0031] Figure 2 is the schematic diagram of the complete model of the embodiment of the present invention.

[0032] Figure 3 is the schematic diagram of the structure of the foreground feature aggregation module of the embodiment of the present invention.

[0033] Figure 4 is the schematic diagram of the structure of the conditional information coupling module of the embodiment of the present invention.

[0034] Figure 5This is a comparison chart of the visualization results between the embodiments of the present invention and other methods. Detailed implementation manners

[0035] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0036] As Figure 1 、 Figure 2 shown, a few-shot object detection method based on prototype network feature enhancement for scarce sample scenarios in this embodiment includes the following steps:

[0037] Step A: Obtain the training set and test set in the few-shot object detection task, and preprocess the data, including image normalization and data augmentation operations. Then, use the support sample expansion module to amplify the samples in the support set, and select additional samples based on similarity to improve the stability of the class prototypes.

[0038] As Figure 3 shown, Step B: Use the backbone network to extract image features, and process the samples in the support set and query set respectively. Remove the background features through the foreground feature aggregation module to extract the class prototypes and improve the discriminative ability of the prototypes.

[0039] As Figure 4 shown, Step C: Calculate the class prototypes of the support set samples, and combine the query image information through the conditional information coupling module to optimize the class prototypes to make them more suitable for the current query image. Calculate the similarity between the optimized class prototypes and the query image features to complete object classification and localization.

[0040] Step D: Calculate the loss function of the model according to the fused features, and then minimize the loss function to update the parameters of the model through the gradient descent algorithm.

[0041] Step E: Save the optimal model parameter weights after training for subsequent model testing.

[0042] Step F: Predict the images in the test set, predict the position and class of the object according to the trained model parameter weights in Step E, and then obtain the final prediction result through non-maximum suppression.

[0043] The foreground feature extraction for each image in the training set and test set samples described in Step B is specifically as follows:

[0044] Step B1: Adopt a support sample expansion method based on similarity to select the most similar samples in the same training batch for supplementation.

[0045] Step B2: For the given support sample feature matrix X1 and candidate sample feature matrix X2, calculate their cosine similarity S. The formula is: Then, select the K samples with the highest similarity and add them to the support set to form an extended support sample set. Assume the support sample feature matrix X1 and candidate sample feature matrix X2 are [0.2, 0.5, 0.3, 0.7, 0.1] and [0.3, 0.6, 0.4, 0.8, 0.2] respectively, then the cosine similarity

[0046] Step B3: Assume the feature matrix X obtained by extracting the support samples through the backbone network E , X E ∈R h×w×d , where h and w are the spatial dimensions of the feature map, and d is the number of channels. Define a learnable foreground attention vector q c , q c ∈R n×d′ , calculate the foreground attention weight A c of class c. The formula is: where softmax is a commonly used mathematical function that maps each element in a vector to a value between 0 and 1, and the sum of all elements is 1. Then, use A c to calculate the fine-grained prototype P c of class c. The formula is: P c = A c ·X E . For example, given an image feature matrix X E , X E = [0.1 0.2 0.3...1.6 1.7 1.5], and the foreground attention vector q c is [1.0, 0.5, 0.3], then the foreground attention weight A c is [0.0526 0.0610...0.0644 0.0647], and the fine-grained prototype P c is [0.8648, 0.9648, 1.0648];

[0047] Step B4: After extracting the class prototypes, they need to be matched with the query image features. Define the feature matrix of the query image as X q , calculate the affinity matrix A′ between it and the class prototype P. The formula is: Then use A' to adjust the query image. The formula is: X′ q = X q + A′·P. For example, given a query image feature matrix X qis [0.2 0.3 0.1...1.5 1.6 1.8], the class prototype P is [0.8648, 0.9648, 1.0648], then the value of A is [0.85 0.70...0.88 0.78].

[0048] As described in step C, the class prototype of the support set samples is calculated and combined with the query image information to optimize the class prototype. Specifically:

[0049] Step C1: Map the optimized query image feature X′ q to the class prototype P in key-value pairs (K q , V q ), to generate an adapted representation Q of the query feature. The formula is: K q = X′ q ·W K + b K ; V q = X′ q ·W V + b V ; Q = P·W Q , where W K , W V , W Q , b K , b V are all learnable parameters. For example, given the query image feature matrix X q is [0.2 0.3 0.1...1.5 1.6 1.8], given a 3×3 matrix W K = [0.3 0.4 0.1...0.1 0.2 0.6], given b K = [0.1 0.2 0.3], then K q = [0.45 0.60 0.25...1.30 1.50 1.40], and then calculate the correlation weight matrix A S between the query feature and the class prototype. The formula is: Then, based on A s reconstruct the query image feature X″ q . The formula is: X″ q = Conv(A s ·X q ), where Conv is the convolution operation;

[0050] Step C2: To make the class prototype more adaptable, it is necessary to further calculate the class prototype P q specific to the query image. First, calculate the cosine similarity matrix C between the query feature X″ q and the class prototype P. The formula is: C = COS(GAP(X″q ), P), where COS represents the cosine similarity, GAP represents global average pooling, and then the class prototype P specific to the query image is calculated based on C q , and the formula is: P q = C ⊙ X″ q + P, where ⊙ represents element-wise multiplication, and finally the class prototype P is calculated qc , and the formula is: where R(P q,i ) is calculated by a fully connected layer and the Sigmoid function, and is used to measure the importance of each support sample;

[0051] Step C3: Combine the optimized class prototype P qc with the query image feature X″ q to obtain the class score S c , and the calculation formula is: The target bounding box is generated by the region proposal network and ROI pooling, and finally the target detection result is obtained.

[0052] Step D: Calculate the loss function of the model, specifically:

[0053] Step D1: The loss function consists of three parts. The first part is the classification loss L cls , which is used to evaluate the accuracy of target classification and is calculated using cross-entropy loss. The formula is: where y i is the true class label, and p i is the probability of the predicted class;

[0054] Step D2: The second part is the regression loss L reg , which is used to evaluate the prediction accuracy of the target bounding box and is calculated using L1 loss. The formula is: where y i is the value of the true bounding box, and p i is the value predicted by the model;

[0055] Step D3: The third part is the meta-loss L meta , which is used to optimize the class prototype to make it more matched with the query image feature. It is calculated using the loss based on cross-entropy. First, calculate the cross-entropy loss for each query sample q. The formula is where p(y = k|q) is the probability that the query sample q belongs to class k, which is calculated using the softmax function. Then calculate the meta-loss L meta , and the formula is: where N is the size of the support set, and the total loss function is a linear weighted combination of these three losses:

[0056] L = L cls + Lreg +αL meta

[0057] Among them, α is the loss weight coefficient, which is set to 0.1.

[0058] As Figure 5 shown, the present invention selects some representative images to visually demonstrate the advantages of the present invention. The first column, the second column and the third column in the figure respectively show the actual labels of the images, the recognition results of the FPD model and the recognition results of the present invention. In the images of the first row and the third row, the FPD model cannot achieve efficient detection when the target is small or dark; in the second row, the border of the FPD model is obviously deviated and overlapped. Through analysis, it can be obtained that the reason for the poor detection effect may be that the target in the picture is occluded, which results in the model not fully extracting foreground features, and further leads to the foreground area of the query image not being accurately divided in the feature aggregation stage; in the image of the fourth row, the FPD model obviously has missed detection of the target. The present invention is optimized on the basis of the FPD model. In the images of the first row and the third row, more possible targets can already be detected. In the images of the second row and the fourth row, compared with the FPD model, the present invention has more accurate positioning boxes.

[0059] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. A few-shot object detection method based on prototype network feature enhancement for scarce sample scenarios, characterized in that The method includes the following steps: Step A: Obtain the training set and test set in the small-sample object detection task, and preprocess the data, including image normalization and data augmentation operations. Then, use the support sample expansion module to amplify the samples in the support set, select additional samples based on similarity, and improve the stability of the class prototypes; Step B: Use the backbone network to extract image features, and process the samples in the support set and query set respectively. Remove the background features through the foreground feature aggregation module, and extract the class prototypes to improve the discriminative ability of the prototypes; Step C: Calculate the class prototypes of the support set samples, and optimize the class prototypes by combining the query image information through the conditional information coupling module to make them more suitable for the current query image. Calculate the similarity between the optimized class prototypes and the query image features to complete object classification and localization; Step D: Calculate the loss function of the model based on the fused features, and then minimize the loss function to update the model parameters through the gradient descent algorithm; Step E: Save the optimal model parameter weights after training for subsequent model testing; Step F: Predict the images in the test set. According to the trained model parameter weights described in Step E, predict the position and class of the target, and then obtain the final prediction result through non-maximum suppression.

2. The method according to claim 1, wherein The foreground feature extraction for each image in the training set and test set samples described in Step B is specifically: Step B1: Adopt the support sample expansion method based on similarity to select the most similar samples in the same training batch for supplementation; Step B2: For the given support sample feature matrix X1 and candidate sample feature matrix X2, calculate their cosine similarity S using the formula: Then, select the K samples with the highest similarity and add them to the support set to form an extended support sample set; Step B3. Assume that the feature matrix X is obtained by extracting the support samples through the backbone network E , X E ∈R h×w×d , where h and w are the spatial dimensions of the feature map, and d is the number of channels. Define a learnable foreground attention vector q c , q c ∈R n×d′ . Calculate the foreground attention weight A c for class c, and the formula is: where softmax is a commonly used mathematical function that maps each element in a vector to a value between 0 and 1, and the sum of all elements is 1. Then use A c to calculate the fine-grained prototype P c for class c, and the formula is: P c = A c ·X E ; Step B4: After extracting the category prototype, it is necessary to match it with the query image features. Define the feature matrix of the query image as X q , calculate the affinity matrix A' between it and the category prototype P. The formula is: Then use A' to adjust the query image. The formula is: X' q = X q + A'·P.

3. The method according to claim 1, characterized in that The optimization of the class prototypes by calculating the class prototypes of the support set samples and combining the query image information described in Step C is specifically: Step C1: Map the optimized query image feature X′ q to the class prototype P in key-value pairs (K q , V q ) to generate an adapted representation Q of the query feature. The formula is: K q = X′ q ·W K + b K ; V q = X′ q ·W V + b V ; Q = P·W Q , where W K , W V , W Q , b K , b V are all learnable parameters. Then calculate the correlation weight matrix A S between the query feature and the class prototype. The formula is: Then, based on A s , reconstruct the query image feature X″ q . The formula is: X″ q = Conv(A s ·X q ), where Conv is the convolution operation; Step C2: To make the class prototype more adaptable, it is necessary to further calculate the class prototype P specific to the query image q , first calculate the query feature X″ q and the cosine similarity matrix C between the class prototype P, the formula is: C = COS(GAP(X″ q ), P), where COS represents the cosine similarity, GAP represents global average pooling, and then calculate the class prototype P specific to the query image based on C q , the formula is: P q = C⊙X″ q + P, where ⊙ represents element-wise multiplication, and finally calculate the class prototype P qc , the formula is: where R(P q,i ) is calculated by a fully connected layer and a Sigmoid function, and is used to measure the importance of each support sample; Step C3: Combine the optimized class prototype P qc with the query image feature X″ q to obtain the class score S c The calculation formula is as follows: The target bounding box is generated by the region proposal network and ROI pooling, and finally the target detection result is obtained.

4. The method according to claim 1, wherein The calculation of the loss function of the model described in Step D is specifically: Step D1: The loss function consists of three parts. The first part is the classification loss L cls , which is used to evaluate the accuracy of target classification and is calculated using cross-entropy loss. The formula is: where y i is the true class label and p i is the probability of the predicted class; Step D2: The second part is the regression loss L reg , which is used to evaluate the prediction accuracy of the target bounding box and is calculated using the L1 loss. The formula is: where y i is the value of the true bounding box, and p i is the value predicted by the model; Step D3: The third part is the meta-loss L meta , which is used to optimize the class prototypes to make them more matched with the query image features. The loss calculation is based on cross-entropy. First, calculate the cross-entropy loss for each query sample q. The formula is where p(y = k|q) is the probability that the query sample q belongs to class k, calculated using the softmax function. Then calculate the meta-loss L meta , and the formula is: where N is the support set size, and the total loss function is a linear weighted combination of these three losses: L = L cls + L reg + αL meta where α is the loss weight coefficient, which is set to 0.1.

Images