A small sample target detection method based on a double-branch structure

Abstract

The application discloses a small sample target detection method based on a double-branch structure, which comprises the following steps: step 1), scoring instance samples through a scorer; step 2), extracting features of support images and query images; step 3), performing weighted summation on the support image features to obtain absolute features; step 4), obtaining a weighted feature vector of the support features; step 5), obtaining multi-scale query features after fusing information; step 6), outputting a predicted target encoding vector and regression parameters according to the multi-scale query features after fusing information; step 7), obtaining a class-level support feature vector; step 8), performing comparison and matching on the class-level support feature vector and the predicted target encoding vector, outputting a final target matching result, and calculating a relative loss value according to the target matching result; and step 9), updating memory features of corresponding categories stored in a Memory structure. The application can realize target detection on small sample data.

Description

Technical Field

[0001] This invention belongs to the field of small sample target detection technology, specifically relating to a small sample target detection method based on a dual-branch structure. Background Technology

[0002] Object detection tasks based on deep learning methods typically require large amounts of training data, and the detected categories are limited to those predefined in the training set. However, in many practical applications, obtaining large amounts of labeled data can be costly, especially for rare or special object categories where sufficient data samples are difficult to acquire. Therefore, when faced with only a very small amount of labeled data, traditional object detection methods cannot effectively identify and locate new categories under limited data conditions. Thus, few-sample object detection methods, unlike traditional object detection, emphasize the model's generalization ability under limited data conditions, meaning they can classify and locate new categories even with a small number of samples.

[0003] Few-shot object detection tasks typically divide the training set into two sets: a base set and a novel set. The categories in the base set and the novel set have no overlap, and the union of the categories in the base set and the novel set is the entire set of categories. The base set provides abundant training data, so the comparative ability of the few-shot object detection model is trained on the base set. Each novel category in the novel set provides only a limited number of data samples, so the comparative ability of the few-shot object detection model is fine-tuned and validated on the novel set.

[0004] Currently, in order to adapt to the limited conditions of small samples, the small sample target detection method based on the dual-branch structure designs the model to consist of a query network branch and a support network branch. These two network branches process the two inputs, the query image and the support image, respectively. In the query network branch and the support network branch, image processing and feature extraction are performed on the query image and the support image, respectively. Finally, the relationship between the support features and the query features is compared, so that the target of the category contained in the support image can be classified and located on the query image.

[0005] Recently, few-shot object detection methods based on dual-branch structures often employ improved feature aggregation or relation comparison modules to enhance model performance. However, these methods are often insufficient to accurately classify and locate new categories. Their shortcomings lie in the fact that they only use an averaging strategy for instance samples from multiple shots supporting the network branch, failing to distinguish the differences in the impact of different instance samples within the same category on relation comparison. Furthermore, the feature aggregation module cannot fully integrate feature information, failing to transfer the feature information of instance samples supporting the network branch to the image features of the query network branch with fine granularity. Finally, it cannot fully recognize the varying quality of different instance samples supporting the network branch, causing the impact of instance samples of different quality on relation comparison to tend to be averaged, thus preventing high-quality instance samples from playing a more significant role. Summary of the Invention

[0006] To overcome the shortcomings of the existing technologies, the present invention aims to provide a small-sample target detection method based on a dual-branch structure, which solves the problems of using an averaging strategy for multiple-shot instance samples, poor feature aggregation results, and equal participation of instance samples of different quality. The present invention, based on sample scoring and combined with a memory mechanism, uses weighted, finer-grained features for comparison to achieve target detection on small-sample data.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A small-sample target detection method based on a dual-branch structure includes the following steps;

[0009] Step 1): Score the instance samples using a scorer (CLIP);

[0010] Step 2): Construct an image feature extractor based on a dual-branch structure to extract features from the supporting image and the query image respectively;

[0011] Step 3): Based on the score given to the instance sample by the scorer, the supporting image features are weighted and summed to obtain the absolute features;

[0012] Step 4): The supporting image features, the absolute features, and the memory features of the corresponding categories stored in the Memory structure are weighted and summed to obtain the weighted feature vector of the supporting features;

[0013] Step 5): Construct a feature aggregation module to transfer and fuse the information of the weighted feature vector of the supporting features into the features of the query image to obtain multi-scale query features after fusion.

[0014] Step 6): Based on the multi-scale query features after the fusion of information, output the predicted target encoding vector and regression parameters;

[0015] Step 7): Calculate the similarity score based on the weighted feature vector of the supporting features and the predicted target encoding vector, and then sum the weighted feature vectors of the supporting features again based on the similarity score.

[0016] Obtain class-level support feature vectors;

[0017] Step 8): Compare and match the class-level support feature vector and the predicted target encoding vector, output the final target matching result, and calculate the relative loss value based on the target matching result;

[0018] Step 9): Based on the absolute features and the calculated relative loss value, update the memory features of the corresponding category stored in the Memory structure using the modified exponential moving average method.

[0019] The specific steps of step 1) are as follows:

[0020] 1.1) Extract the instance sample region from the labeled image in the supporting branch as the foreground part, replace the original instance sample region in the image with a fixed value, and then use the entire image as the background part;

[0021] 1.2) Based on the characteristic region F of the foreground portion f The class name text T corresponding to the instance sample region is input into the image encoder E of the scorer (CLIP). img and text encoder E text In the middle, the feature region F of the foreground part is obtained. f The similarity score of the foreground portion is obtained based on CLIP, using the corresponding image encoding vector and the text encoding vector of the category name text T corresponding to the instance sample region:

[0022] score1=cos(E img (F f E text (T))

[0023] Here, cos refers to the cosine similarity function;

[0024] 1.3) Based on the specified background feature region F b The category name text T corresponding to the instance sample region is input into the CLIP image encoder E. img and text encoder E text In the middle, the background feature region F is obtained. b The similarity score of the background portion is obtained based on CLIP, using the corresponding image encoding vector and the text encoding vector corresponding to the category name text T of the instance:

[0025] score2=cos(E img (F b E text (T))

[0026] 1.4) Based on the obtained foreground similarity score (score1) and background similarity score (score2), the difference is calculated to obtain the absolute score of the instance sample. This absolute score is then processed using the sigmoid function to limit it to between 0 and 1.

[0027] score=δ((score1-score2)*15)

[0028] Where δ is the sigmoid function.

[0029] The specific steps of step 2) are as follows:

[0030] 2.1) Employing Darknet's backbone with weight sharing e It acts as a feature extractor in the query network branch and the support network branch to extract features from the query image and the support image;

[0031] 2.2) For the support image, the feature maps of the three support features at different scales extracted by the feature extractor are input into the FPN structure for multi-scale feature fusion;

[0032] In the FPN structure, the feature maps obtained from the bottom-up and top-down approaches are fused to obtain feature maps of support features at three different scales after multi-scale feature fusion. For the three multi-scale feature maps output by FPN, one feature map is selected, and the instance samples on the original image are mapped to the corresponding region of the feature map according to the scale. The corresponding mapped region on the feature map is extracted and RoI Align processing is performed. The extracted corresponding mapped region feature map information is aggregated to a fixed size and then average pooling is performed to obtain the original support feature vector F. i It represents the supporting image features, and the dimension of the supporting feature vector is consistent with the number of channels of the feature map of the mapped region;

[0033] 2.3) For the query image, the feature map of the query features at three different scales is obtained by the feature extractor and used as the query image feature map.

[0034] The specific steps of step 3) are as follows:

[0035] 3.1) The original support feature vector F i Multiply by the corresponding fraction s i And add them together:

[0036]

[0037] 3.2) For the weighted absolute feature F abs Numerical normalization is performed to obtain the absolute features corresponding to the supported categories, ensuring a unified measurement scale when fusing information with other features in subsequent steps.

[0038] F absolute =Norm(F abs )

[0039] The specific steps of step 4) are as follows:

[0040] 4.1) Obtain the memory features F corresponding to the supporting categories from the Memory structure. memory ;

[0041] 4.2) The absolute feature F after numerical normalization absolute And the memory feature F of the corresponding category memory Weighted summation according to a fixed ratio:

[0042] F merge =0.1·F absolute +0.9·F memory

[0043] 4.3) Using the epoch value of this iteration and the maximum number of iterations max_epochs as parameters, calculate the absolute feature F during this forward propagation process. absolute and memory characteristics F memory The merged feature F after weighted merging merge Weight control factor w r :

[0044] w r =trend_func(epoch,max_epochs)

[0045] Here, `trend_func` is a linear decreasing function or a cosine function, with its domain defined between 0 and `max_epochs` and its value range between 0 and 1. A weight control factor `w` is also included. r ;

[0046] 4.4) Based on the weight control factor w r Calculate the merged feature F merge and original supporting features F origin The weighted merging result yields the weighted feature vector of the supporting features;

[0047] F integration =α·w r ·F merge +(1-α·w r )·Forigin

[0048] The specific steps of step 5) are as follows:

[0049] 5.1) First, split the multi-scale feature map of the query image into two parts by channel:

[0050]

[0051] 5.2) Take a portion of the multi-scale feature map Processed via Max-Sigmoid:

[0052]

[0053] Where W∈R C×D It is all the supporting feature vectors F merge The set of all supporting feature vectors F, C is the set of all supporting feature vectors F. merge The number of channels is D, which is the dimension of the support feature vector and the number of channels in the multi-scale feature map. δ is the sigmoid function. The correlation between the support feature vector and the query image feature map is calculated, and the multi-scale feature map of the query image is weighted to aggregate the support feature information into the multi-scale feature map of the query image.

[0054] 5.3) The feature map obtained by Max-Sigmoid processing and another part of the initial feature map To splice:

[0055]

[0056] The specific steps of step 6) are as follows:

[0057] 6.1) Design the target encoding output head module using two 3x3 convolutional layers and one 1x1 convolutional layer to ensure that the number of channels in the output feature map is consistent with the dimension of the supporting feature vector;

[0058] 6.2) A regression head module is designed using two 3x3 convolutional layers and one 1x1 convolutional layer to output four regression parameters, which represent the center coordinates (x, y) and height and width (h, w) of the predicted box, respectively, to achieve the localization of the predicted target.

[0059] 6.3) Based on the multi-scale query feature map obtained after fusion information, the target encoding output head module is used to obtain the predicted target encoding feature map O for each scale. embedding ∈R H×W×C ;

[0060] 6.4) Based on the multi-scale query feature map obtained after fusion information, the regression parameters O for each scale feature map are obtained through the regression head module. reg ∈R H×W×4 .

[0061] The specific steps of step 7) are as follows:

[0062] 7.1) Dynamically weight and sum the supporting image features, absolute features, and memory features to obtain a weighted feature vector for multiple shots of each category. and the predicted target encoded feature map O embedding The target encoding vector at each position in Calculate cosine similarity:

[0063]

[0064] 7.2) Softmax processing is applied to the calculated cosine similarity:

[0065]

[0066] 7.3) Based on the similarity score of each shot after softmax Weighted feature vectors of supporting features The weighted sum is taken again to obtain the target encoding feature map O. embedding The contrastive feature vectors of all supporting classes corresponding to each position in the vector:

[0067]

[0068] Where K refers to the number of shots for each supported category;

[0069] 7.4) Perform numerical normalization to obtain the final class-level support feature vector for each support category:

[0070] F support =Norm(F s )

[0071] The specific steps of step 8) are as follows:

[0072] 8.1) Convert the contrastive feature vectors of each support class c and the predicted target encoding vector Perform a similarity comparison:

[0073]

[0074] 8.2) Among the similarity scores of each supporting category, the supporting category corresponding to the highest similarity score is taken as the predicted category:

[0075] cls = argmax 1≤c≤N s c

[0076] Where N refers to the number of supported categories in the supported network branch;

[0077] 8.3) Calculate the loss value of this iteration based on the target matching results, and then use the loss value of this iteration to update and record the average value of recent loss values ​​using the modified exponential moving average method. The average value of this loss value reflects the generality of the loss value in the current time period.

[0078] 8.4) Divide the average of the loss values ​​by the loss value of the current iteration, and control the numerical scale range by exponentiation to obtain the relative loss value μ, which reflects the quality of the current supporting features.

[0079] The specific steps of step 9) are as follows:

[0080] 9.1) Based on the relative loss value μ and the weight decay factor w of the memory structure d Calculate the update factor γ:

[0081] γ=(1-w d )*μ

[0082] 9.2) Based on the update factor γ and the absolute characteristic F absolute Update the memory feature vectors of the currently supported classes stored in the Memory structure:

[0083]

[0084] The beneficial effects of this invention are:

[0085] This invention designs an image instance sample scorer based on CLIP. It scores different instance samples according to the foreground and background parts of the instance sample region, thereby measuring the absolute quality difference between different instance samples. Furthermore, it performs a weighted summation of the original support features of multiple shots based on the quality score. Compared with the average fusion strategy of multiple shots, the weighted summation based on the quality score can distinguish the difference in the role of different instance samples in relational comparison within the same category, allowing high-quality instance samples to play a more significant role.

[0086] This invention integrates original support features, absolute features obtained by weighted summation, and memory features of the corresponding categories stored in the Memory structure by weighted summation. This greatly enriches the support feature information from multiple perspectives. Furthermore, during training, the weight ratio of the original support features is dynamically increased, ensuring that the influence of the original support features gradually increases during training iterations. This reduces the model's dependence on absolute and memory features, thereby significantly increasing the model's generalization ability.

[0087] This invention uses a feature aggregation module to transfer feature information of instance samples supporting network branches to image features of query network branches in a more granular manner.

[0088] This invention utilizes a memory structure to collect common features that support the class well during model training. This enriches the current supporting features with past memorized experience, assisting the model in better feature comparison. During memory structure updates, both absolute features and relative loss values ​​are combined to continuously collect feature information from high-quality and well-performing instance samples for use in subsequent training processes. This demonstrates the diversity of feature information stored in the memory structure from multiple perspectives. Attached Figure Description

[0089] Figure 1 This is a framework diagram of a small sample target detection method based on a dual-branch structure.

[0090] Figure 2 These are example images of instance samples under different scoring conditions. Detailed Implementation

[0091] The present invention will now be described in further detail with reference to the accompanying drawings.

[0092] like Figure 1 As shown, a small-sample target detection method based on a dual-branch structure specifically includes the following steps:

[0093] Step 1) Construct a scorer for image instance samples based on CLIP. The scorer refers to CLIP. The scoring criterion is to calculate the foreground region score and the background region score based on CLIP respectively, and subtract the background region score from the foreground region score to eliminate the influence of the background on the classification of the instance sample (foreground region) into the corresponding category.

[0094] "Score" refers to the final score. In step 3), the supporting features are weighted according to the score. Therefore, the higher the "score", the greater the contribution of the corresponding instance sample features in the weighted features.

[0095] 1.1) Extract the instance sample region from the labeled image in the supporting branch as the foreground part, replace the original instance sample region in the image with a fixed value, and then use the entire image as the background part;

[0096] 1.2) Based on the foreground feature region F f The corresponding category name text T for the instance is input into the CLIP image encoder E. img and text encoder E text In the middle, the feature region F of the foreground part is obtained. f The corresponding image encoding vector and the text encoding vector corresponding to the category name text T of the instance are used to obtain the similarity score of the foreground part based on CLIP:

[0097] score1=cos(E img (F f E text (T))

[0098] Here, cos refers to the cosine similarity function;

[0099] 1.3) Based on the specified background feature region F b The corresponding category name text T for the instance is input into the CLIP image encoder E. img and text encoder E text In the middle, the background feature region F is obtained. b The corresponding image encoding vector and the text encoding vector corresponding to the category name text T of the instance are used to obtain the similarity score of the background part based on CLIP:

[0100] score2=cos(E img (F b E text (T))

[0101] 1.4) Based on the obtained foreground similarity score (score1) and background similarity score (score2), the difference is calculated to obtain the absolute score of the instance sample. This absolute score is then processed using the sigmoid function to limit it to between 0 and 1.

[0102] score=δ((score1-score2)*15)

[0103] Here, δ is the sigmoid function. Multiplying the absolute score by 15 and then passing it through the sigmoid function can increase the score difference between different instance samples, thereby increasing the discrimination between different instance samples.

[0104] Step 2) Construct an image feature extractor based on a dual-branch structure, where the feature extractor shares weights between the query network branch and the support network branch;

[0105] Extract features from the supporting image and the query image respectively.

[0106] 2.1) In the query network branch and the support network branch, the Darknet backbone with weight sharing is adopted. e As a feature extractor, it extracts features from the query image and supporting images;

[0107] 2.2) For the support image, the feature maps of the three support features at different scales extracted by the feature extractor are input into the FPN structure for multi-scale feature fusion;

[0108] In the FPN structure, the feature maps obtained from the bottom up and the feature maps obtained from the top down are fused to obtain feature maps of support features at three different scales after multi-scale feature fusion. For the three multi-scale feature maps output by FPN, one feature map is selected, and the instance samples on the original image are mapped to the corresponding region of the feature map according to the scale. The corresponding mapped region on the feature map is extracted and RoI Align processing is performed. The extracted corresponding mapped region feature map information is aggregated into a fixed size and then average pooling is performed to obtain the original support feature vector. The dimension of the support feature vector is the same as the number of channels of the mapped region feature map.

[0109] 2.3) For the query image, the feature map of the query features at three different scales is obtained by the feature extractor and used as the query image feature map.

[0110] Step 3) The instance sample is scored according to the scorer, and the supporting image features are weighted and summed based on the score to obtain the absolute features.

[0111] 3.1) Based on the score, the original support feature vectors of K shots for each support category are weighted and summed, that is, the original support feature vector F is weighted and summed. i Multiply by the corresponding fraction s i And add them together:

[0112]

[0113] 3.2) For the weighted absolute feature F abs Numerical normalization is performed to obtain the absolute features corresponding to the supported categories, ensuring a unified measurement scale when fusing information with other features in subsequent steps.

[0114] F absolute =Norm(F abs )

[0115] Step 4) The supporting image features, the absolute features, and the memory features of the corresponding categories stored in the Memory structure are weighted and summed to obtain the weighted feature vector of the supporting features.

[0116] 4.1) Obtain the memory features F corresponding to the supporting categories from the Memory structure. memory ;

[0117] 4.2) The absolute feature F after numerical normalization absolute And the memory feature F of the corresponding category memory Weighted summation according to a fixed ratio:

[0118] F merge =0.1·F absolute +0.9·F memory

[0119] 4.3) Using the epoch value of this iteration and the maximum number of iterations max_epochs as parameters, calculate the absolute feature F during this forward propagation process. absolute and memory characteristics F memory The merged feature F after weighted merging merge Weight control factor w r :

[0120] w r =trend_func(epoch,max_epochs)

[0121] Here, `trend_func` is a linear decreasing function or a cosine function, with its domain defined between 0 and `max_epochs` and its value range between 0 and 1. A weight control factor `w` is also included. r This allows for control over the merging feature F. merge The weight α can be dynamically adjusted to merge the features F. merge and original supporting features F origin The weighting ratios between these features ensure that the influence of the original supporting features gradually increases during training iterations, thereby reducing the model's reliance on absolute features F. absolute and memory characteristics F memory Dependency;

[0122] 4.4) Based on the weight control factor w r Calculate the merged feature F merge and original supporting features F origin The weighted merging result yields the weighted feature vector of the supporting features;

[0123] F integration =α·wr ·F merge +(1-α·w r )·F origin

[0124] Step 5) Construct a feature aggregation module to transfer and fuse the information of the weighted feature vector of the supporting features into the features of the query image to obtain the query features after fusion; and transfer the feature information of the instance samples supporting the network branch into the image features of the query network branch in a more granular way.

[0125] 5.1) First, split the multi-scale feature map of the query image into two parts by channel:

[0126]

[0127] 5.2) Take a portion of the multi-scale feature map Processed via Max-Sigmoid:

[0128]

[0129] Where W∈R C×D It is all the supporting feature vectors F merge The set of all supporting feature vectors F, C is the set of all supporting feature vectors F. merge The number of channels is D, which is the dimension of the support feature vector and the number of channels in the multi-scale feature map. δ is the sigmoid function. The correlation between the support feature vector and the query image feature map is calculated, and the multi-scale feature map of the query image is weighted to aggregate the support feature information into the multi-scale feature map of the query image.

[0130] 5.3) The feature map obtained by Max-Sigmoid processing and another part of the initial feature map To splice:

[0131]

[0132] Step 6): Based on the query features after the fusion information, output the predicted target encoding vector and regression parameters. The regression head module and the target encoding output head module have the same structure, but the number of output channels is different. The target encoding output head module outputs C channels, corresponding to the C dimension in the target encoding feature map. The target encoding vector and support feature vector in the target encoding feature map will be used to calculate similarity and predict the category. The regression head module outputs 4 channels, corresponding to the 4 dimension in the regression parameters, which is used to locate the predicted target.

[0133] 6.1) Design the target encoding output head module using two 3x3 convolutional layers and one 1x1 convolutional layer to ensure that the number of channels in the output feature map is consistent with the dimension of the supporting feature vector;

[0134] 6.2) Design a regression head module using two 3x3 convolutional layers and one 1x1 convolutional layer to output four regression parameters, representing the center coordinates (x, y) and height and width (h, w) of the predicted box, respectively.

[0135] 6.3) Based on the multi-scale query feature map obtained after fusion information, the target encoding output head module is used to obtain the predicted target encoding feature map O for each scale. embedding ∈R H×W×C ;

[0136] 6.4) Based on the multi-scale query feature map obtained after fusion information, the regression parameters O for each scale feature map are obtained through the regression head module. reg ∈R H×W×4 .

[0137] Step 7): Calculate the similarity score based on the weighted feature vector of the supporting features in Step 4) and the predicted target encoding vector, and then sum the weighted feature vector of the supporting features again based on the similarity score to obtain the class-level supporting feature vector.

[0138] 7.1) Dynamically weight and sum the supporting image features, absolute features, and memory features to obtain a weighted feature vector for multiple shots of each category. and the predicted target encoded feature map O embedding The target encoding vector at each position in Calculate cosine similarity:

[0139]

[0140] 7.2) Softmax processing is applied to the calculated cosine similarity:

[0141]

[0142] 7.3) Based on the similarity score of each shot after softmax Weighted feature vectors of supporting features The weighted sum is taken again to obtain the target encoding feature map O. embedding The contrastive feature vectors of all supporting classes corresponding to each position in the vector:

[0143]

[0144] Where K refers to the number of shots for each supported category;

[0145] 7.4) Perform numerical normalization to obtain the final class-level support feature vector for each support category:

[0146] F support =Norm(F s )

[0147] Step 8) Compare and match the class-level support feature vector and the predicted target encoding vector, output the final target matching result, and calculate the relative loss value based on the target matching result.

[0148] 8.1) Convert the contrastive feature vectors of each support class c and the predicted target encoding vector Perform a similarity comparison:

[0149]

[0150] 8.2) Among the similarity scores of each supporting category, the supporting category corresponding to the highest similarity score is taken as the predicted category:

[0151] cls = argmax 1≤c≤N S c

[0152] Where N refers to the number of supported categories in the supported network branch;

[0153] 8.3) Calculate the loss value of this iteration based on the target matching results, and then use the loss value of this iteration to update and record the average value of recent loss values ​​using the modified exponential moving average method. The average value of this loss value reflects the generality of the loss value in the current time period.

[0154] 8.4) Divide the average of the loss values ​​by the loss value of the current iteration, and control the numerical scale range by exponentiation to obtain the relative loss value μ, which reflects the quality of the current supporting features.

[0155] Step 9): Based on the absolute features and the calculated relative loss value, update the memory features of the corresponding category stored in the Memory structure using a modified exponential moving average method.

[0156] In the memory structure update, by combining absolute features and relative loss values, feature information of high-quality and well-performing instance samples is continuously collected for use in the subsequent training process, which reflects the diversity of feature information stored in the memory structure from multiple perspectives.

[0157] 9.1) Based on the relative loss value μ and the weight decay factor w of the memory structure d Calculate the update factor γ:

[0158] γ=(1-w d )*μ

[0159] 9.2) Based on the update factor γ and the absolute characteristic F absolute Update the memory feature vectors of the currently supported classes stored in the Memory structure:

[0160]

[0161] like Figure 2 As shown, the example images of instance samples under different scoring conditions are used. The final quality score of the instance sample is the product of the absolute score and the relative loss value during the training process.

[0162] The first row shows example images of instance samples with low final quality scores. It can be seen that the low final quality scores of the instance samples are due to reasons such as occlusion by other categories of targets, incomplete display of targets, or targets that are too small.

[0163] The second row shows example images of instances with high final quality scores. The targets are clearly displayed, appropriately sized, and unobstructed, consistent with the fact that the final quality scores are high.

[0164] Example

[0165] To more clearly illustrate the small-sample target detection method based on the dual-branch structure, the following detailed description of each step in implementing the method of this invention is provided.

[0166] This invention is based on experiments using the PASCALVOC dataset, which contains 20 categories. To accommodate small sample sizes, three different category splits were implemented, with each split using 15 categories as the base class and the remaining 5 categories as the novel class. Additionally, the VOC07+12 training and validation sets were used for training, while the VOC07 test set was used for evaluation.

[0167] Both the training and test sets will be configured according to the base settings. e Classes and novel classes are divided into bas e The small sample object detection model first uses the base set and the novel set. e The training is performed on the dataset, and finally fine-tuned on the novel dataset.

[0168] In bas eDuring training, given a query image without any labels, one category is selected from the categories included in the query image as the positive sample category, and N-1 categories not included in the query image are selected as the negative sample categories. Then, in the training set images, based on the selected positive sample category, K labeled positive sample images are randomly selected (K can be 1, 3, 5, or 10). Based on the selected N-1 negative sample categories, K labeled negative sample images are randomly selected for each of them. These positive and negative sample images are used as support images for the support network branches. Simultaneously, the single query image is used as the query image for the query network branch. During training, the query image for the query network branch and the support image for the support network branch are input into a feature extractor for feature extraction, resulting in query image features and support image features. Then, in the support network branch, the support instance samples are scored according to the scorer built based on CLIP. Based on this score, the support image features for each category are weighted and summed to obtain absolute features. The support image features, absolute features, and memory features of the corresponding categories stored in the Memory structure are then weighted and summed to obtain a weighted feature vector of the support features. Next, in the feature aggregation module, the information of the weighted feature vector of the support features is transferred and fused into the features of the query image to obtain the fused query features. This fused query features output the predicted target encoding vector and regression parameters according to the target encoding output head module and the regression head module. A similarity score is calculated between the weighted feature vector of the support features and the predicted target encoding vector. The weighted feature vector of the support features is then weighted and summed again based on the similarity score to obtain a class-level support feature vector. Finally, the class-level support feature vector and the predicted target encoding vector are compared and matched to output the final target matching result. A relative loss value is calculated based on the target matching result, and then the memory features of the corresponding categories stored in the Memory structure are updated based on the relative loss value and the absolute features.

[0169] After training on the base set is completed, fine-tuning training is performed on the novel set. For each of the N categories in the support network branch that contain positive and negative sample categories of the novel class, K labeled images are provided for each category. These images do not change during the fine-tuning training process.

Claims

1. A small sample target detection method based on a double-branch structure, characterized in that, Includes the following steps; Step 1): Score the instance samples using a scorer; Step 2): Construct an image feature extractor based on a dual-branch structure to extract features from the supporting image and the query image respectively; Step 3): Based on the scoring results of the instance samples by the scorer, the supporting image features are weighted and summed to obtain the absolute features; Step 4): The supporting image features, the absolute features, and the memory features of the corresponding categories stored in the Memory structure are weighted and summed to obtain the weighted feature vector of the supporting features; Step 5): Construct a feature aggregation module to transfer and fuse the information of the weighted feature vector of the supporting features into the features of the query image to obtain multi-scale query features after fusion. Step 6): Based on the multi-scale query features after the fusion of information, output the predicted target encoding vector and regression parameters; Step 7): Calculate the similarity score based on the weighted feature vector of the supporting features and the predicted target encoding vector, and then sum the weighted feature vector of the supporting features again based on the similarity score to obtain the class-level supporting feature vector; Step 8): Compare and match the class-level support feature vector and the predicted target encoding vector, output the final target matching result, and calculate the relative loss value based on the target matching result; Step 9): Based on the absolute features and the calculated relative loss value, update the memory features of the corresponding category stored in the Memory structure using the modified exponential moving average method.

2. The small sample target detection method based on the double-branch structure according to claim 1, characterized in that, The specific steps of step 1) are as follows: 1.1) Extract the instance sample region from the labeled image in the supporting branch as the foreground part, replace the original instance sample region in the image with a fixed value, and then use the entire image as the background part; 1.2) The feature region of the foreground portion and the category name text corresponding to the instance sample area The image encoders are input to the scorer respectively. and text encoder In the middle, the feature regions of the foreground part are obtained. The corresponding image encoding vector and the category name text corresponding to the instance sample region. The corresponding text encoding vector is used to obtain the similarity score of the foreground portion based on CLIP: in, This refers to the cosine similarity function; 1.3) Select the specified background feature region and the category name text corresponding to the instance sample area The images are input to the CLIP image encoder respectively. and text encoder In the middle, the feature regions of the background part are obtained. The corresponding image encoding vector and the class name text corresponding to the instance. The corresponding text encoding vector is used to obtain the similarity score of the background portion based on CLIP: 1.4) Based on the obtained foreground similarity score Similarity score with the obtained background part The difference operation yields the absolute score of the instance sample, which is then processed using the sigmoid function to limit the absolute score to between 0 and 1. in, It is the sigmoid function.

3. The small-sample target detection method based on a dual-branch structure according to claim 1, characterized in that, The specific steps of step 2) are as follows: 2.1) The Darknet backbone with weight sharing is used as a feature extractor in both the query and support network branches to extract features from the query and support images. 2.2) For the support image, the feature maps of the three support features at different scales extracted by the feature extractor are input into the FPN structure for multi-scale feature fusion; In the FPN structure, the feature maps obtained from the bottom-up and top-down approaches are fused to obtain feature maps of support features at three different scales after multi-scale feature fusion. For the three multi-scale feature maps output by FPN, one feature map is selected, and the instance samples on the original image are scaled and mapped onto the corresponding region of the feature map. The corresponding mapped region on the feature map is extracted and RoI Align processing is performed. The extracted corresponding mapped region feature map information is aggregated into a fixed size and then average pooling is performed to obtain the original support feature vector. It represents the supporting image features, and the dimension of the supporting feature vector is consistent with the number of channels of the feature map of the mapped region; 2.3) For the query image, the feature map of the query features at three different scales is obtained by the feature extractor and used as the query image feature map.

4. The small-sample target detection method based on a dual-branch structure according to claim 3, characterized in that, The specific steps of step 3) are as follows: 3.1) Convert the original support feature vector Multiply by the corresponding fraction And add them together: 3.2) Absolute features after weighting Numerical normalization is performed to obtain the absolute features corresponding to the supported categories, ensuring a unified measurement scale when fusing information with other features in subsequent steps. 。 5. A small-sample target detection method based on a dual-branch structure according to claim 4, characterized in that, The specific steps of step 4) are as follows: 4.1) Obtain the memory features corresponding to the supported categories from the Memory structure. ; 4.2) Absolute characteristics after numerical normalization and the memory characteristics of the corresponding categories Weighted summation according to a fixed ratio: 4.3) Based on this iteration Value and maximum number of iterations The value is a parameter to calculate the absolute characteristics during this forward propagation process. and memory characteristics Merging characteristics after weighted merging Weight control factor : in, It is a linearly decreasing function or a cosine function, with its domain defined as 0 to 1. The value range is between 0 and 1, and a weight control factor is designed. ; 4.4) Based on the weighting control factor Calculate merged features and original supporting features The weighted merging result yields the weighted feature vector of the supporting features; 。 6. The small-sample target detection method based on a dual-branch structure according to claim 5, characterized in that, The specific steps of step 5) are as follows: 5.1) First, split the multi-scale feature map of the query image into two parts by channel: 5.2) Take a portion of the multi-scale feature map Processed via Max-Sigmoid: in, It is all supporting feature vectors The set, It is all supporting feature vectors Quantity, It represents the dimension of the supporting feature vectors, and also the number of channels in the multi-scale feature map. It is the sigmoid function, which calculates the correlation between the support feature vector and the query image feature map, and performs weighted processing on the multi-scale feature map of the query image, thereby aggregating the support feature information into the multi-scale feature map of the query image; 5.3) Feature map obtained by Max-Sigmoid processing and another part of the initial feature map To splice: 。 7. The small-sample target detection method based on a dual-branch structure according to claim 6, characterized in that, The specific steps of step 6) are as follows: 6.1) Design the target encoding output head module using two 3x3 convolutional layers and one 1x1 convolutional layer to ensure that the number of channels in the output feature map is consistent with the dimension of the supporting feature vector; 6.2) A regression head module is designed using two 3x3 convolutional layers and one 1x1 convolutional layer to output four regression parameters, representing the center coordinates of the predicted bounding box. Height and width of the frame This is used to locate the predicted target; 6.3) Based on the multi-scale query feature map obtained after fusion information, the target encoding feature map for each scale is obtained through the target encoding output head module. ; 6.4) Based on the multi-scale query feature map obtained after fusion information, the regression parameters of each scale feature map are obtained through the regression head module. .

8. A small-sample target detection method based on a dual-branch structure according to claim 7, characterized in that, The specific steps of step 7) are as follows: 7.1) Dynamically weight and sum the supporting image features, absolute features, and memory features to obtain a weighted feature vector for multiple shots of each category. and the predicted target encoded feature map The target encoding vector at each position in Calculate cosine similarity: 7.2) Softmax processing is applied to the calculated cosine similarity: 7.3) Based on the similarity score of each shot after softmax Weighted feature vectors of supporting features The target encoding feature map is obtained by weighted summation again. The contrastive feature vectors of all supporting classes corresponding to each position in the vector: in, This refers to the number of shots for each supported category; 7.4) Perform numerical normalization to obtain the final class-level support feature vector for each support category: 。 9. A small-sample target detection method based on a dual-branch structure according to claim 8, characterized in that, The specific steps of step 8) are as follows: 8.1) Each supported category Comparison feature vectors and the predicted target encoding vector Perform a similarity comparison: 8.2) Among the similarity scores of each supporting category, the supporting category corresponding to the highest similarity score is taken as the predicted category: in, This refers to the number of supported categories in the supported network branches; 8.3) Calculate the loss value of this iteration based on the target matching results, and then use the loss value of this iteration to update and record the average value of recent loss values ​​using the modified exponential moving average method. The average value of this loss value reflects the general magnitude of the loss value in the current time period. 8.4) Divide the average of the loss values ​​by the loss value of the current iteration, and control the numerical scale range by exponentiation to obtain the relative loss value. This reflects the quality of the current support features.

10. A small-sample target detection method based on a dual-branch structure according to claim 9, characterized in that, The specific steps of step 9) are as follows: 9.1) Based on relative loss value Weight decay factor of memory structure Calculate the update factor : 9.2) Based on the update factor and absolute characteristics Update the memory feature vectors of the currently supported classes stored in the Memory structure: 。

Images