A target retrieval system and method based on contrastive learning

Abstract

The application discloses a target retrieval system based on contrast learning, comprising first to third FPN networks, an RPN network, a feature matching module, a feature fusion module, a classifier and a regressor; the first FPN network inputs a query block; the second and third FPN networks input target images; the output of the first FPN network and the output of the second FPN network are matched by the feature matching module and then input to the RPN network; the RPN network outputs target image candidate frames; the output of the first FPN network and the output of the RPN network are fused by the feature fusion module and then input to the classifier and the regressor respectively; the third FPN network outputs the features of the true value frame and part of the background features randomly cropped; and the third FPN network is combined with the query block and used for contrast training of the first and second FPN networks; the classifier outputs the category of the target; and the regressor outputs the corrected target candidate frame. The application adopts the foreground and background contrast learning method, and solves the problem of low target detection precision caused by excessive false positives.

Description

Technical Field

[0001] This invention relates to the field of machine learning, and in particular to a target retrieval system and method based on contrastive learning. Background Technology

[0002] Currently, the task of object retrieval is to accurately locate and identify objects of a specific category within a large set of images, given a query block (usually an image or a portion of an image). This involves precisely locating targets of a specified category of interest from a collection containing a large number of in-distribution and out-of-distribution images. In-distribution images typically refer to images that the model has encountered during training and belong to known categories and environments. Out-of-distribution images, on the other hand, include images of new categories or styles that the model has not been trained on, placing higher demands on the model's generalization ability.

[0003] Existing methods typically require large amounts of labeled data to train models to identify specific object categories, which is not only costly but also time-consuming. Employing feature fusion strategies to acquire directly transferable meta-knowledge can effectively generalize this meta-knowledge to new categories, which is crucial for rapidly adapting to new recognition tasks. However, existing methods are primarily tested on pairs of query blocks and target images, failing to consider scenarios where targets are retrieved across a broader image set. This leads to frequent false positives in images that do not contain the target of interest.

[0004] In addition, most existing machine learning models are trained based on the closed-world assumption, which assumes that the test data and training data have the same distribution. This assumption often fails in real-world applications, especially when the model is deployed in open-world scenarios. In open-world environments, the main challenge for models is distribution shift, which can stem from semantic shift (e.g., samples covering multiple unseen categories) or covariate shift (e.g., samples coming from different environments or having different styles). This distribution shift significantly increases the difficulty of recognition, requiring models to have high generalization capabilities.

[0005] In summary, to achieve broader applications, such as accurate target retrieval in open-category and open-domain environments, the model's capabilities must be expanded to handle not only diverse object categories but also different visual styles and environmental conditions. Therefore, for target retrieval tasks, in addition to considering the model's robustness to categories, it is also necessary to focus on the model's robustness to styles to achieve true open-category and open-domain target retrieval. Summary of the Invention

[0006] This invention provides a target retrieval system and method based on contrastive learning to solve the technical problems existing in the prior art.

[0007] The technical solution adopted by this invention to solve the technical problems existing in the prior art is as follows:

[0008] A target retrieval system based on contrastive learning includes a first FPN network, a second FPN network, a third FPN network, an RPN network, a feature matching module, a feature fusion module, a classifier, and a regressor;

[0009] The first FPN network inputs a query block; the second and third FPN networks input target images; the first FPN network outputs query block features; the second FPN network outputs target image features.

[0010] The outputs of the first FPN network and the second FPN network are matched by the feature matching module and then input into the RPN network; the RPN network outputs candidate bounding boxes of the target image;

[0011] The outputs of the first FPN network and the RPN network are fused by the feature fusion module and then input into the classifier and regressor, respectively.

[0012] The third FPN network outputs features of the ground truth boxes and randomly cropped background features; a foreground feature queue is formed by combining the features of the ground truth boxes; a background feature queue is formed by combining the randomly cropped background features; the foreground feature queue is combined with the query block for comparative training of the first FPN network; the background feature queue is combined with the query block for comparative training of the second FPN network.

[0013] The classifier outputs the category of the target; the regressor outputs the corrected target candidate boxes.

[0014] Furthermore, the first and second FPN networks are twin networks, with identical structures.

[0015] Furthermore, it also includes feature enhancers for perturbing the mean and variance of shallow features in the first and second FPN networks, the feature enhancers including a first feature enhancement module and / or a second feature enhancement module;

[0016] The first feature enhancement module is used to perform random perturbation transformation on the extracted feature map. The calculation formula for the random perturbation transformation is as follows:

[0017]

[0018]

[0019]

[0020] Will Simplified to: y = αx + (β - α)μ (x) ;

[0021] The second feature enhancement module is used to perform enhancement transformation on the extracted feature map. The calculation formula for its enhancement transformation is as follows:

[0022]

[0023] In the formula:

[0024] x represents the original sample;

[0025] y is a sample generated by transforming x through random perturbation;

[0026] μ (x) Let x be the mean of x;

[0027] μ (y) Let y be the mean;

[0028] σ (x) Let x be the standard deviation;

[0029] σ (y) Let y be the standard deviation;

[0030] α is σ (x) Random noise drawn from a Gaussian distribution;

[0031] β is μ (x) Random noise drawn from a Gaussian distribution;

[0032] This represents the adjusted mean of x.

[0033] The standard deviation of x after adjustment;

[0034] Sty (x) The sample is generated after x undergoes enhancement transformation.

[0035] This invention also provides a target retrieval method based on contrastive learning, which constructs a target retrieval model; the target retrieval model includes a first FPN network, a second FPN network, a third FPN network, an RPN network, a feature matching module, a feature fusion module, a classifier, and a regressor;

[0036] The first FPN network inputs a query block; the second and third FPN networks input the target image; the first FPN network outputs the query block features; the second FPN network outputs the target image features.

[0037] The outputs of the first FPN network and the second FPN network are matched by the feature matching module and then input into the RPN network; the RPN network outputs candidate bounding boxes of the target image;

[0038] The outputs of the first FPN network and the RPN network are fused together by the feature fusion module and then input into the classifier and regressor, respectively.

[0039] The third FPN network outputs features of the ground truth boxes and randomly cropped background features; the features of the ground truth boxes are combined to form a foreground feature queue; the features of the randomly cropped background features are combined to form a background feature queue; the foreground feature queue is combined with the query block for comparative training of the first FPN network; the background feature queue is combined with the query block for comparative training of the second FPN network.

[0040] The classifier outputs the category of the target; the regressor outputs the corrected target candidate boxes.

[0041] Furthermore, the target retrieval model is trained using the following steps:

[0042] Step 1: Use the first and second FPN networks to extract features from the query block samples and the target image samples, respectively;

[0043] Step 2: Perturb the mean and variance of the shallow features in the first and second FPN networks to achieve feature enhancement;

[0044] Step 3: Input the target image sample into the third FPN network and output ground truth bounding box features and randomly cropped background features; create a foreground feature queue from the ground truth bounding box features; create a background feature queue from the randomly cropped background features.

[0045] Step 4: Object features belonging to the same category as the query block in the foreground feature queue are used as positive foreground features, and object features not belonging to the same category as the query block are used as negative foreground features. The positive foreground features are combined with multiple negative foreground features to perform foreground contrastive learning on the first FPN network. The query block and its enhanced features are used as positive background samples, and all features in the background feature queue are used as negative background samples. The positive background features are combined with multiple negative background features to perform background contrastive learning on the second FPN network.

[0046] Step 5: Use the feature fusion module to aggregate the query patch features and the candidate bounding boxes of the target image, classify them using a classifier, and correct the target candidate bounding boxes using a regressor.

[0047] Step six: Repeat steps one through five until training is complete.

[0048] Furthermore, in step one, the first and second FPN networks form a twin network, sharing the same parameters and architecture; the backbone networks of the twin network both adopt the ResNet50 network; the ResNet50 network adopts skip connections; the input data of the twin network undergoes enhanced preprocessing.

[0049] Furthermore, in step two, the method of perturbing the mean and variance of the shallow features in the first and second FPN networks to achieve feature enhancement includes the following steps:

[0050] Set up a feature enhancer to perturb the mean and variance of shallow features in the first and second FPN networks. The feature enhancer includes a first feature enhancement module and / or a second feature enhancement module.

[0051] The first feature enhancement module is used to perform random perturbation transformation on the extracted feature map. The calculation formula for the random perturbation transformation is as follows:

[0052]

[0053]

[0054]

[0055] Will Simplified to: y = αx + (β - α)μ (x) ;

[0056] The second feature enhancement module is used to perform enhancement transformation on the extracted feature map. The calculation formula for its enhancement transformation is as follows:

[0057]

[0058] In the formula:

[0059] x represents the original sample;

[0060] y is a sample generated by transforming x through random perturbation;

[0061] μ (x) Let x be the mean of x;

[0062] μ (y) Let y be the mean;

[0063] σ (x) Let x be the standard deviation;

[0064] σ (y) Let y be the standard deviation;

[0065] α is σ (x) Random noise drawn from a Gaussian distribution;

[0066] β is μ (x) Random noise drawn from a Gaussian distribution;

[0067] This represents the adjusted mean of x.

[0068] The standard deviation of x after adjustment;

[0069] Sty (x) The sample is generated after x undergoes enhancement transformation.

[0070] Furthermore, step three includes the following methodological steps:

[0071] Step B1: Use the third FPN network to extract features from the target image. The backbone network of the third FPN network is combined with the feature pyramid network to update the momentum parameters. The calculation formula for the updated parameters is as follows:

[0072]

[0073] In the formula:

[0074] Backbone_k (t) This is the output of the backbone network of the third FPN network at the t-th iteration.

[0075] Backbone_k (t-1) This is the output of the backbone network of the third FPN network in the (t-1)th iteration;

[0076] Backbone_q (t) This is the output of the backbone network of the second FPN network at the t-th iteration.

[0077] For Backbone_k (t-1) Parameters;

[0078] For Backbone_k (t) Parameters;

[0079] For Backbone_q (t) Parameters;

[0080] t is the number of iterations;

[0081] m is the momentum update parameter;

[0082] Step B2: Use the ground truth bounding boxes of the basic categories to crop the foreground features from the target image and process them through the projection layer; the processed foreground features are stored and updated in the foreground feature queue; each category has a separate foreground feature queue for storing foreground features; a background feature is randomly cropped from each target image and stored as a background feature queue; when creating the foreground feature queue, queues of the same length are created for different categories to ensure that the number of features for each category is the same.

[0083] Furthermore, in step four, when performing foreground contrast learning on the first FPN network, the following loss function is used to supervise the first FPN network:

[0084]

[0085] In the formula:

[0086] L Foreg Foreground contrast learning loss;

[0087] L Foreg,i The foreground contrast learning loss is used for the i-th sample;

[0088] i represents the sample number for foreground contrast learning;

[0089] I represents the foreground contrast learning sample set;

[0090] P(i) is the set of positive samples associated with i;

[0091] 'a' represents the negative sample number;

[0092] p is the positive sample number;

[0093] Learn hyperparameters for foreground contrast;

[0094] z i Let be the feature vector of the i-th sample;

[0095] z p Let p be the feature vector of the p-th positive sample;

[0096] z a Let be the feature vector of the a-th negative sample;

[0097] A(i) represents the set of foreground contrast learning samples that does not contain i;

[0098] When performing background contrast learning on the second FPN network, the following loss function is used to supervise the second FPN network:

[0099]

[0100] In the formula:

[0101] L Backg Learning loss is contrasted with the background;

[0102] L Backg,k The background contrast loss is used for the k-th sample;

[0103] k is the sample number for background contrast learning;

[0104] U represents the background contrast learning sample set;

[0105] b represents the background sample;

[0106] Learn hyperparameters by contrast with the background;

[0107] z k Let k be the feature vector of the k-th sample;

[0108] z j(k) The augmented feature vector of the k-th sample;

[0109] z b Let b be the feature vector of the b-th background sample;

[0110] B(k) is the set of background contrast learning samples excluding k.

[0111] Furthermore, in step five, the method for aggregating the query patch features and candidate bounding boxes of the target image using the feature fusion module includes the following steps:

[0112] Step C1: Concatenate the feature vectors of the query patch feature and the candidate bounding box of the target image end to end;

[0113] Step C2, Weighted Average: Calculate a weighted average of the features based on their importance;

[0114] Step C3 involves transforming the fused features through one or more additional neural network layers.

[0115] The advantages and positive effects of this invention are:

[0116] 1. This invention proposes a foreground and background contrast learning method, which solves the problems of excessive false positives leading to decreased overall accuracy and weak ability to distinguish between targets. The trained model has stronger generality between visible and invisible categories.

[0117] 2. This invention proposes an open-domain feature enhancement method to address the issue of domain distribution differences between the query block and the target image. Without altering the shape and position of objects within the image, various potential patterns are efficiently synthesized by perturbing the shallow channel statistics of the source domain image's CNN, making it suitable for target retrieval tasks. Attached Figure Description

[0118] Figure 1 This is a schematic diagram of the workflow of a target retrieval system based on contrastive learning according to the present invention.

[0119] Figure 2 This is a schematic diagram illustrating the workflow of a method for determining the mean and variance characteristics of shallow features in random perturbation first and second FPN networks according to the present invention.

[0120] Figure 3 This is a schematic diagram of a foreground queue and a background queue structure according to the present invention.

[0121] Figure 2In Chinese: NP represents the first feature enhancement module or the second feature enhancement module. Detailed Implementation

[0122] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0123] Please see Figures 1 to 3 A target retrieval system based on contrastive learning includes a first FPN network, a second FPN network, a third FPN network, an RPN network, a feature matching module, a feature fusion module, a classifier, and a regressor;

[0124] The first FPN network inputs a query block; the second and third FPN networks input target images; the first FPN network outputs query block features; the second FPN network outputs target image features.

[0125] The outputs of the first FPN network and the second FPN network are matched by the feature matching module and then input into the RPN network; the RPN network outputs candidate bounding boxes of the target image;

[0126] The outputs of the first FPN network and the RPN network are fused by the feature fusion module and then input into the classifier and regressor, respectively.

[0127] The third FPN network outputs features of the ground truth boxes and randomly cropped background features; a foreground feature queue is formed by combining the features of the ground truth boxes; a background feature queue is formed by combining the randomly cropped background features; the foreground feature queue is combined with the query block for comparative training of the first FPN network; the background feature queue is combined with the query block for comparative training of the second FPN network.

[0128] The classifier outputs the category of the target; the regressor outputs the corrected target candidate boxes.

[0129] FPN network, short for Feature Pyramid Network, is a neural network architecture used to handle multi-scale object detection problems, commonly used in object detection and semantic segmentation tasks. The structure of an FPN network consists of two main parts:

[0130] 1. Bottom-up Feature Extraction Subnetwork (Backbone Network): This part utilizes existing convolutional neural networks (such as ResNet or VGG) to extract features from the input image. Each convolutional layer produces a feature map; lower-level feature maps have higher resolution and less semantic information, while higher-level feature maps have lower resolution but contain richer semantic information. In this way, the backbone network can extract multi-scale feature maps, providing a foundation for subsequent feature fusion.

[0131] 2. Top-Down Feature Concatenation Subnetwork: This part concatenates low-level feature maps with high-level feature maps to generate a feature pyramid with high resolution and semantic information. This concatenation method can effectively solve the problem of information loss between different scales. Specifically, the top-down path upsamples high-level feature maps and fuses them with low-level feature maps of the corresponding scale to generate multi-scale feature maps containing more information. This process utilizes the complementary properties of feature maps at different levels, enhancing the model's ability to detect targets at different scales.

[0132] RPN (Region Proposal Network): The RPN is a network that uses a fully convolutional neural network to generate candidate bounding boxes, often used in conjunction with subsequent object detection networks (such as Fast R-CNN, Faster R-CNN, etc.). Its main function is to generate candidate object regions, enabling object detection in subsequent networks. The RPN uses an anchor box mechanism, referencing multiple scales and aspect ratios, to predict object boundaries and scores at each location, and through end-to-end training, generates high-quality region proposals.

[0133] Feature matching module: This module uses the edge features of the image as a template, determines the search space according to preset parameters, and searches for targets similar to the template in the image. The feature matching module can be used for tasks such as localization, counting, and presence / absence determination, ensuring accurate location of target objects in complex backgrounds.

[0134] Feature fusion module: This module is used to merge or weight features from different feature extraction methods or models, thereby improving the model's performance or effectiveness. Feature fusion can integrate information from different sources, enabling the model to perform better when handling diverse tasks.

[0135] Classifier: A classifier is used to classify objects in an image. By analyzing the fused feature map, it outputs the category to which the object belongs. Classifiers typically employ fully connected layers or more complex structures to accurately identify various targets in an image.

[0136] Regressor: The regressor is used to calculate the position of candidate boxes, accurately predicting the specific location of objects in an image. Through the regressor, the model can output more accurate bounding box coordinates, thereby improving the accuracy of object detection.

[0137] Through the organic combination of these modules, the FPN network can efficiently detect and segment targets in images at different scales, and is widely used in various tasks in the field of computer vision.

[0138] To improve the performance of the model in target retrieval tasks, this invention proposes a novel contrastive learning branch. The foreground feature contrastive learning branch aims to improve the network's ability to distinguish foreground target features, thereby reducing class confusion, while the background feature contrastive learning branch aims to enhance the network's ability to discriminate background features, thereby mitigating the problem of false positives.

[0139] Specifically, the features of the ground truth bounding box in the target image and the randomly cropped background features are stored in two separate feature libraries. These feature libraries are implemented using queues, and contrastive learning loss is applied to each library to perform contrastive learning with the features of the query block. The aim is to improve the feature extraction capability of the feature extraction network. The method of this invention is inspired by momentum update and contrastive learning in self-supervised learning. This additional branch enhances the model's ability to distinguish between target image features and background features.

[0140] Preferably, the first and second FPN networks can be twin networks, and their structures can be identical.

[0141] Preferably, it may further include feature enhancers for perturbing the mean and variance of shallow features in the first and second FPN networks. The feature enhancers may include a first feature enhancement module and / or a second feature enhancement module.

[0142] The first feature enhancement module is used to perform random perturbation transformation on the feature maps extracted from the first and second FPN networks. The calculation formula for the random perturbation transformation is as follows:

[0143]

[0144]

[0145]

[0146] Will Simplified to: y = αx + (β - α)μ (x) ;

[0147] The second feature enhancement module is used to enhance the feature maps extracted from the first and second FPN networks. The enhancement transformation calculation formula is as follows:

[0148]

[0149] In the formula:

[0150] x represents the original sample;

[0151] y is a sample generated by transforming x through random perturbation;

[0152] μ (x) Let x be the mean of x;

[0153] μ(y) Let y be the mean;

[0154] σ (x) Let x be the standard deviation;

[0155] σ (y) Let y be the standard deviation;

[0156] α is σ (x) Random noise drawn from a Gaussian distribution;

[0157] β is μ (x) Random noise drawn from a Gaussian distribution;

[0158] This represents the adjusted mean of x.

[0159] The standard deviation of x after adjustment;

[0160] Sty (x) The sample is generated after x undergoes enhancement transformation.

[0161] This invention also provides a target retrieval method based on contrastive learning, which constructs a target retrieval model; the target retrieval model includes a first FPN network, a second FPN network, a third FPN network, an RPN network, a feature matching module, a feature fusion module, a classifier, and a regressor;

[0162] The first FPN network inputs a query block; the second and third FPN networks input the target image; the first FPN network outputs the query block features; the second FPN network outputs the target image features.

[0163] The outputs of the first FPN network and the second FPN network are matched by the feature matching module and then input into the RPN network; the RPN network outputs candidate bounding boxes of the target image;

[0164] The outputs of the first FPN network and the RPN network are fused together by the feature fusion module and then input into the classifier and regressor, respectively.

[0165] The third FPN network outputs features of the ground truth boxes and randomly cropped background features; the features of the ground truth boxes are combined to form a foreground feature queue; the features of the randomly cropped background features are combined to form a background feature queue; the foreground feature queue is combined with the query block for comparative training of the first FPN network; the background feature queue is combined with the query block for comparative training of the second FPN network.

[0166] The classifier outputs the category of the target; the regressor outputs the corrected target candidate boxes.

[0167] Preferably, the target retrieval model can be trained using the following methods and steps:

[0168] Step 1: Use the first and second FPN networks to extract features from the query block samples and the target image samples, respectively;

[0169] Step 2: Perturb the mean and variance of the shallow features in the first and second FPN networks to achieve feature enhancement;

[0170] Step 3: Input the target image sample into the third FPN network and output ground truth bounding box features and randomly cropped background features; create a foreground feature queue from the ground truth bounding box features; create a background feature queue from the randomly cropped background features.

[0171] Step four involves using the features of objects belonging to the same category as the query block in the foreground feature queue as positive foreground features and the features of objects not belonging to the same category as the query block as negative foreground features. The positive foreground features are then combined with multiple negative foreground features to perform foreground contrastive learning on the first FPN network. The query block and its enhanced features are then used as positive background samples, and all features in the background feature queue are used as negative background samples. Finally, the positive background features are combined with multiple negative background features to perform background contrastive learning on the second FPN network. This allows the network to fully utilize the features of different object categories and their backgrounds for contrastive learning, enhancing the network's feature discrimination performance without increasing inference parameters or inference time.

[0172] Step 5: Use the feature fusion module to aggregate the query patch features and the candidate bounding boxes of the target image, classify them using a classifier, and correct the target candidate bounding boxes using a regressor.

[0173] Step six: Repeat steps one through five until training is complete.

[0174] The trained model is used for classification and regression. The input is a query block and a set of target images. The model retrieves targets in the target image set that are of the same category as the query block and frames them to obtain the final result.

[0175] Preferably, in step one, the first and second FPN networks can form a twin network, sharing the same parameters and architecture; the backbone network of the twin network can both adopt a ResNet50 network; wherein the ResNet50 network can adopt a skip connection; the input data of the twin network can undergo enhanced preprocessing.

[0176] Preferably, in step two, the method of perturbing the mean and variance of shallow features in the first and second FPN networks to achieve feature enhancement may include the following steps:

[0177] A feature enhancer can be configured to perturb the mean and variance of shallow features in the first and second FPN networks. The feature enhancer may include a first feature enhancement module and / or a second feature enhancement module.

[0178] The first feature enhancement module can be used to perform random perturbation transformation on the extracted feature map. The calculation formula for the random perturbation transformation is as follows:

[0179]

[0180]

[0181]

[0182] Will Simplified to: y = αx + (β - α)μ (x) ;

[0183] The second feature enhancement module can be used to enhance the extracted feature map. The calculation formula for its enhancement transformation is as follows:

[0184]

[0185] In the formula:

[0186] x represents the original sample;

[0187] y is a sample generated by transforming x through random perturbation;

[0188] μ (x) Let x be the mean of x;

[0189] μ (y) Let y be the mean;

[0190] σ (x) Let x be the standard deviation;

[0191] σ (y) Let y be the standard deviation;

[0192] α is σ (x) Random noise drawn from a Gaussian distribution;

[0193] β is μ (x) Random noise drawn from a Gaussian distribution;

[0194] This represents the adjusted mean of x.

[0195] The standard deviation of x after adjustment;

[0196] Sty (x) The sample is generated after x undergoes enhancement transformation.

[0197] The first feature enhancement module and the second feature enhancement module are applied to the first and second FPN networks and run independently, enabling the first and second FPN networks to generate various enhancement modes during training.

[0198] Preferably, step three may include the following method steps:

[0199] Step B1: Use the third FPN network to extract features from the target image. The backbone network of the third FPN network is combined with the feature pyramid network to update the momentum parameters. The calculation formula for the updated parameters is as follows:

[0200]

[0201] In the formula:

[0202] Backbone_k (t) This is the output of the backbone network of the third FPN network at the t-th iteration.

[0203] Backbone_k (t-1) This is the output of the backbone network of the third FPN network in the (t-1)th iteration;

[0204] Backbone_q (t) This is the output of the backbone network of the second FPN network at the t-th iteration.

[0205] For Backbone_k (t-1) Parameters;

[0206] For Backbone_k (t) Parameters;

[0207] For Backbone_q (t) Parameters;

[0208] t is the number of iterations;

[0209] m is the momentum update parameter;

[0210] Step B2: Use the ground truth bounding boxes of the basic categories to crop the foreground features from the target image and process them through the projection layer; the processed foreground features are stored and updated in the foreground feature queue; each category has a separate foreground feature queue for storing foreground features; a background feature is randomly cropped from each target image and stored as a background feature queue; when creating the foreground feature queue, queues of the same length are created for different categories to ensure that the number of features for each category is the same.

[0211] Considering the potential presence of out-of-domain data in the test data, i.e., open-domain issues, this invention adds image feature perturbators to both branches of the Siamese network during training for data augmentation. These perturbations occur randomly, allowing the Siamese network to undergo various augmentation scenarios during training. For example, the query block may not be augmented, but the target image may be augmented; or vice versa; or both may be augmented; or neither may be augmented. The feature augmenters, which perturb the mean and variance of shallow features in the first and second FPN networks, change the statistical properties of the images—specifically, by randomly altering the mean and variance—to diversify the styles of the augmented images. This reduces the model's sensitivity to domain information, aiming to improve its domain generalization ability. This strategy, through diversified training scenarios, enhances the model's robustness to unknown domain data.

[0212] Preferably, in step four, when performing foreground contrast learning on the first FPN network, the following loss function can be used to supervise the first FPN network:

[0213]

[0214] In the formula:

[0215] L Foreg Foreground contrast learning loss;

[0216] L Foreg,i The foreground contrast learning loss is used for the i-th sample;

[0217] i represents the sample number for foreground contrast learning;

[0218] I represents the foreground contrast learning sample set;

[0219] P(i) is the set of positive samples associated with i;

[0220] 'a' represents the negative sample number;

[0221] p is the positive sample number;

[0222] Learn hyperparameters for foreground contrast;

[0223] z i Let be the feature vector of the i-th sample;

[0224] z p Let p be the feature vector of the p-th positive sample;

[0225] z a Let be the feature vector of the a-th negative sample;

[0226] A(i) represents the set of foreground contrast learning samples that does not contain i;

[0227] When performing background contrast learning on the second FPN network, the following loss function can be used to supervise the second FPN network:

[0228]

[0229] In the formula:

[0230] L Backg Learning loss is contrasted with the background;

[0231] L Backg,k The background contrast loss is used for the k-th sample;

[0232] k is the sample number for background contrast learning;

[0233] U represents the background contrast learning sample set;

[0234] b represents the background sample;

[0235] Learn hyperparameters by contrast with the background;

[0236] z k Let k be the feature vector of the k-th sample;

[0237] z j(k) The augmented feature vector of the k-th sample;

[0238] z b Let b be the feature vector of the b-th background sample;

[0239] B(k) is the set of background contrast learning samples excluding k.

[0240] Preferably, in step five, the method for aggregating the query patch features and candidate bounding boxes of the target image using the feature fusion module includes the following steps:

[0241] Step C1: Concatenate the feature vectors of the query patch feature and the candidate bounding box of the target image end to end;

[0242] Step C2, Weighted Average: Calculate a weighted average of the features based on their importance;

[0243] Step C3 involves transforming the fused features through one or more additional neural network layers.

[0244] The structure, working principle, and workflow of the present invention are further illustrated below with reference to a preferred embodiment:

[0245] A target retrieval method based on contrastive learning is proposed, which constructs a target retrieval model. The target retrieval model includes a first FPN network, a second FPN network, a third FPN network, an RPN network, a feature matching module, a feature fusion module, a classifier, and a regressor.

[0246] The first FPN network inputs a query block; the second and third FPN networks input the target image; the first FPN network outputs the query block features; the second FPN network outputs the target image features.

[0247] The outputs of the first FPN network and the second FPN network are matched by the feature matching module and then input into the RPN network; the RPN network outputs candidate bounding boxes of the target image;

[0248] The outputs of the first FPN network and the RPN network are fused together by the feature fusion module and then input into the classifier and regressor, respectively.

[0249] The third FPN network outputs features of the ground truth boxes and randomly cropped background features; the features of the ground truth boxes are combined to form a foreground feature queue; the features of the randomly cropped background features are combined to form a background feature queue; the foreground feature queue is combined with the query block for comparative training of the first FPN network; the background feature queue is combined with the query block for comparative training of the second FPN network.

[0250] The classifier outputs the category of the target; the regressor outputs the corrected target candidate boxes.

[0251] The first and second FPN networks form a twin network, sharing the same parameters and architecture.

[0252] The object categories are divided into a basic category B and a new category N, where The data domain is divided into the visible domain S and the invisible domain U, where Given any query block, the method of the present invention aims to detect targets in an image library containing a large number of complex, unknown images that do not necessarily match the category of the query block.

[0253] The target retrieval model is trained using the following methods and steps:

[0254] (i) Use twin networks to extract features from query blocks and target images.

[0255] (1) Two ResNet50 models are used as twin networks, sharing the same parameters and architecture. Among them, ResNet50 uses skip connections to solve the training problem of deeper networks. These connections allow information to be passed directly from the previous layer to the next layer, which helps gradients flow more efficiently in the network.

[0256] (2) Input data needs to be properly preprocessed before being input into the network. Preprocessing methods include data augmentation strategies such as rotation and cropping.

[0257] (3) Feature extraction. The first and second FPN networks will independently process the input query block and target image, and output a feature map containing important visual and spatial information.

[0258] (II) Feature Enhancement: Mean and Variance of Shallow Features in Randomly Perturbed Siamese Networks:

[0259] Set up a feature enhancer to perturb the mean and variance of shallow features in the first and second FPN networks. The feature enhancer includes a first feature enhancement module and / or a second feature enhancement module.

[0260] To reduce style or domain differences between query blocks and target images, and between training and testing datasets, and to improve the network's ability to search for objects in images of various styles, this invention develops a first feature enhancement module to reduce such differences.

[0261] The first feature enhancement module is used to perform random perturbation transformation on the extracted feature map. The calculation formula for the random perturbation transformation is as follows:

[0262]

[0263]

[0264]

[0265] Will Simplified to: y = αx + (β - α)μ (x) ;

[0266] Since the statistics of image feature channels (mean and standard deviation) are closely related to image style, changing these channel statistics can be seen as an implicit method of changing the style of the input image. This invention develops a second feature enhancement module to change the style of the input image.

[0267] The second feature enhancement module is used to perform enhancement transformation on the extracted feature map. The calculation formula for its enhancement transformation is as follows:

[0268]

[0269] In the formula:

[0270] x represents the original sample;

[0271] y is a sample generated by transforming x through random perturbation;

[0272] μ (x) Let x be the mean of x;

[0273] μ (y) Let y be the mean;

[0274] σ (x)Let x be the standard deviation;

[0275] σ (y) Let y be the standard deviation;

[0276] α is σ (x) Random noise drawn from a Gaussian distribution;

[0277] β is μ (x) Random noise drawn from a Gaussian distribution;

[0278] This represents the adjusted mean of x.

[0279] The standard deviation of x after adjustment;

[0280] Sty (x) The sample is generated after x undergoes enhancement transformation.

[0281] like Figure 2 As shown, the aforementioned feature enhancer is integrated into a Siamese network. This invention employs a simple strategy, applying the feature enhancer to two branches of the Siamese network and allowing them to operate independently, enabling the network to generate various enhancement patterns during training.

[0282] (III) Momentum update of backbone network parameters of the first, second, and third FPN networks and comparative learning using image features.

[0283] Using a large number of features for contrastive learning is beneficial for enhancing the feature extraction and feature discrimination capabilities of the network. This invention adds a foreground-background contrastive learning branch for momentum-updated features, thereby mitigating foreground confusion and background misjudgment. This branch is divided into two parts: a foreground feature queue and a background feature queue, as follows... Figure 3 As shown in the figure, the upper part of the figure is the foreground feature queue, with each category having a separate queue for storing foreground features; the lower part of the figure is the background feature queue, where the network randomly crops a piece of background features from each target image for storage.

[0284] (1) Foreground comparison learning

[0285] This invention uses a foreground feature contrast learning strategy to solve the problem of foreground target ambiguity. For example... Figure 1 As shown, ResNet50 is used to extract features from the target image, and the momentum update parameters are calculated using a Feature Pyramid Network (FPN) as follows:

[0286]

[0287] In the formula:

[0288] Backbone_k(t) This is the output of the backbone network of the third FPN network at the t-th iteration.

[0289] Backbone_k (t-1) This is the output of the backbone network of the third FPN network in the (t-1)th iteration;

[0290] Backbone_q (t) This is the output of the backbone network of the second FPN network at the t-th iteration.

[0291] For Backbone_k (t-1) Parameters;

[0292] For Backbone_k (t) Parameters;

[0293] For Backbone_q (t) Parameters;

[0294] t is the number of iterations;

[0295] m is the momentum update parameter;

[0296] Foreground features are cropped from the image using ground truth bounding boxes of basic categories and processed through a projection layer. These features are stored and updated in a queue, such as... Figure 3 As shown in the image.

[0297] In this process, features of objects belonging to the same category as the query block are considered positive samples, while features of objects from different categories are considered negative samples. For positive / negative samples, this invention uses the following loss function to supervise the first FPN network:

[0298]

[0299] In the formula:

[0300] L Foreg Foreground contrast learning loss;

[0301] L Foreg,i The foreground contrast learning loss is used for the i-th sample;

[0302] i represents the sample number for foreground contrast learning;

[0303] I represents the foreground contrast learning sample set;

[0304] P(i) is the set of positive samples associated with i;

[0305] 'a' represents the negative sample number;

[0306] p is the positive sample number;

[0307] Learn hyperparameters for foreground contrast;

[0308] z i Let be the feature vector of the i-th sample;

[0309] z p Let p be the feature vector of the p-th positive sample;

[0310] z a Let be the feature vector of the a-th negative sample;

[0311] A(i) represents the set of foreground contrast learning samples that does not contain i;

[0312] Through the aforementioned strategy, extensive contrastive learning among base class features enhances the discriminative ability of the first FPN network, thereby mitigating foreground confusion during inference. It is important to note that in designing the foreground feature queues in this invention, queues of the same length are created for different categories. This ensures that each category has the same number of features, thus promoting a more balanced learning process for the entire network.

[0313] (2) Background Comparison Learning

[0314] The basic idea of ​​this invention is to construct an individual recognition task from massive image backgrounds. By maximizing the separation of foreground target features and background features, the network's ability to distinguish between foreground and background regions is enhanced.

[0315] The background contrast learning head of this invention compares positive sample pairs with a large number of negative samples. The query block and its enhanced features are treated as positive samples, while all features in the background feature queue are treated as negative samples.

[0316] Specifically, this invention randomly crops features from the background region of the target image to fill the background feature queue (avoiding the inability to obtain rich background information due to the cropped area being too small), such as Figure 3 As shown in the diagram. By storing these background features in a queue, the diversity of background samples is enriched, effectively leveraging the advantages of individual discrimination tasks and contrastive learning.

[0317] Finally, similar to the prospectus, this invention uses a background contrast learning loss function to supervise the second FPN network:

[0318]

[0319] In the formula:

[0320] L Backg Learning loss is contrasted with the background;

[0321] LBackg,k The background contrast loss is used for the k-th sample;

[0322] k is the sample number for background contrast learning;

[0323] U represents the background contrast learning sample set;

[0324] b represents the background sample;

[0325] Learn hyperparameters by contrast with the background;

[0326] z k Let k be the feature vector of the k-th sample;

[0327] z j(k) The augmented feature vector of the k-th sample;

[0328] z b Let b be the feature vector of the b-th background sample;

[0329] B(k) is the set of background contrast learning samples excluding k.

[0330] (iv) Feature Fusion

[0331] Feature fusion is the process of combining information from two inputs to generate a comprehensive feature representation for subsequent classification and regression tasks. The methods of feature fusion are as follows:

[0332] (1) Concatenate the two feature vectors end to end.

[0333] (2) Weighted average: The features are weighted according to their importance.

[0334] (3) Transform the fused features through one or more additional neural network layers (such as convolutional layers or fully connected layers) to extract the most critical information for the task.

[0335] (v) Obtain the optimal model and use it for target retrieval.

[0336] Specifically, the network is trained using data from the base category and the visible domain. After training, the network can generalize to directly search for objects in an image library using only a single query block. This library includes images from the base category B, images from the new category N, and images from the visible domain S and the invisible domain U. This significantly increases the difficulty and makes the network more closely resemble real-world scenarios.

[0337] (1) This invention uses a stochastic gradient descent (SGD) optimizer to train for 15 rounds.

[0338] (2) The initial learning rate was set to 0.02 and then reduced by a factor of 10 after the 7th round.

[0339] (3) Repeat the above steps until training is complete. Use the trained model to perform classification and regression, that is, input the query block and the target image set, the model retrieves the target objects of the same category as the query block on the target image set, and outlines them to obtain the final result.

[0340] The neural networks or functional modules mentioned above, such as the first FPN network, the second FPN network, the third FPN network, the RPN network, the feature matching module, the feature fusion module, the classifier, the regressor, the background contrast learning head, and the foreground contrast learning head, can all adopt neural networks or functional modules applicable in the prior art, or adopt neural networks or functional modules in the prior art and construct them using conventional technical means.

[0341] The embodiments described above are only used to illustrate the technical ideas and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The patent scope of the present invention should not be limited by these embodiments. That is, any equivalent changes or modifications made in accordance with the spirit disclosed in the present invention still fall within the patent scope of the present invention.

Claims

1. A target retrieval system based on contrastive learning, characterized in that, The system constructs a target retrieval model; the target retrieval model includes a first FPN network, a second FPN network, a third FPN network, an RPN network, a feature matching module, a feature fusion module, a classifier, and a regressor; First FPN network input query block; The second and third FPN networks input the target image; the first FPN network outputs query block features; the second FPN network outputs target image features. The outputs of the first FPN network and the second FPN network are matched by the feature matching module and then input into the RPN network; the RPN network outputs candidate bounding boxes of the target image; The outputs of the first FPN network and the RPN network are fused by the feature fusion module and then input into the classifier and regressor, respectively. Features of the truth box output by the third FPN network and some background features that are randomly cropped; The foreground feature queue is composed of features from the truth boxes; A background feature queue is composed of randomly cropped background features; The foreground feature queue is combined with the query block for comparative training of the first FPN network; The background feature queue is combined with the query block for comparative training of the second FPN network; The classifier outputs the category of the target; the regressor outputs the corrected target candidate boxes. The target retrieval model is trained using the following methods and steps: Step 1: Use the first and second FPN networks to extract features from the query block samples and the target image samples, respectively; Step 2: Perturb the mean and variance of the shallow features in the first and second FPN networks to achieve feature enhancement; Step 3: Input the target image sample into the third FPN network and output the ground truth box features and randomly cropped background features; Create a foreground feature queue from the truth box features; A background feature queue is created from randomly cropped background features; Step 4: Take the object features in the foreground feature queue that belong to the same category as the query block as the positive foreground sample features, and take the object features that do not belong to the same category as the query block as the negative foreground sample features; combine the positive foreground sample features with multiple negative foreground sample features to perform foreground comparison learning on the first FPN network; The query block and its enhanced features are used as positive background samples, and all features in the background feature queue are used as negative background samples. The background positive sample features are combined with multiple background negative sample features to perform background contrast learning on the second FPN network; Step 5: Use the feature fusion module to aggregate the query patch features and the candidate bounding boxes of the target image, classify them using a classifier, and correct the target candidate bounding boxes using a regressor. Step six: Repeat steps one through five until training is complete; In step four, when performing foreground contrast learning on the first FPN network, the following loss function is used to supervise the first FPN network: ； In the formula: Foreground contrast learning loss; For the first Foreground contrast learning loss for each sample; Number the foreground contrast learning samples; Foreground contrast learning sample set; To and The relevant set of positive samples; Number the negative samples; The positive sample number; Learn hyperparameters for foreground contrast; ； ； ； Indicates that it does not contain A set of foreground contrast learning samples; When performing background contrast learning on the second FPN network, the following loss function is used to supervise the second FPN network: ； In the formula: Learning loss is contrasted with the background; For the first Background contrast loss for each sample; Number the background contrast learning samples; This serves as a background contrast learning sample set. Background sample; Learn hyperparameters by contrast with the background; ； for Augmented ; for ; Not including The background comparison learning sample set.

2. The target retrieval system based on contrastive learning according to claim 1, characterized in that, The first and second FPN networks are twin networks, and they have the same structure.

3. The target retrieval system based on contrastive learning according to claim 2, characterized in that, It also includes feature enhancers for perturbing the mean and variance of shallow features in the first and second FPN networks, the feature enhancers including a first feature enhancement module and / or a second feature enhancement module; The first feature enhancement module is used to perform random perturbation transformation on the extracted feature map. The calculation formula for the random perturbation transformation is as follows: ； ； ； Will Simplified to: ; The second feature enhancement module is used to perform enhancement transformation on the extracted feature map. The calculation formula for its enhancement transformation is as follows: ； In the formula: This is the original sample; for Samples generated after random perturbation transformation; for The mean; for The mean; for Standard deviation; for Standard deviation; for Random noise drawn from a Gaussian distribution; for Random noise drawn from a Gaussian distribution; After adjustment The mean; After adjustment Standard deviation; for The sample generated after enhancement transformation.

4. A target retrieval method based on contrastive learning, characterized in that, Construct a target retrieval model; the target retrieval model includes a first FPN network, a second FPN network, a third FPN network, an RPN network, a feature matching module, a feature fusion module, a classifier, and a regressor; Make the first FPN network input query block; The second and third FPN networks input the target image; the first FPN network outputs query block features; the second FPN network outputs target image features. The outputs of the first FPN network and the second FPN network are matched by the feature matching module and then input into the RPN network; the RPN network outputs candidate bounding boxes of the target image; The outputs of the first FPN network and the RPN network are fused together by the feature fusion module and then input into the classifier and regressor, respectively. This enables the third FPN network to output features of the truth box and randomly cropped background features. The foreground feature queue is composed of features from the truth boxes; A background feature queue is composed of randomly cropped background features; The foreground feature queue is combined with the query block for comparative training of the first FPN network; The background feature queue is combined with the query block for comparative training of the second FPN network; To make the classifier output the category of the target; This causes the regressor to output a revised target candidate box. The target retrieval model is trained using the following methods and steps: Step 1: Use the first and second FPN networks to extract features from the query block samples and the target image samples, respectively; Step 2: Perturb the mean and variance of the shallow features in the first and second FPN networks to achieve feature enhancement; Step 3: Input the target image sample into the third FPN network and output the ground truth box features and randomly cropped background features; Create a foreground feature queue from the truth box features; A background feature queue is created from randomly cropped background features; Step 4: Take the object features in the foreground feature queue that belong to the same category as the query block as the positive foreground sample features, and take the object features that do not belong to the same category as the query block as the negative foreground sample features; combine the positive foreground sample features with multiple negative foreground sample features to perform foreground comparison learning on the first FPN network; The query block and its enhanced features are used as positive background samples, and all features in the background feature queue are used as negative background samples. The background positive sample features are combined with multiple background negative sample features to perform background contrast learning on the second FPN network; Step 5: Use the feature fusion module to aggregate the query patch features and the candidate bounding boxes of the target image, classify them using a classifier, and correct the target candidate bounding boxes using a regressor. Step six: Repeat steps one through five until training is complete; In step four, when performing foreground contrast learning on the first FPN network, the following loss function is used to supervise the first FPN network: ； In the formula: Foreground contrast learning loss; For the first Foreground contrast learning loss for each sample; Number the foreground contrast learning samples; Foreground contrast learning sample set; To and The relevant set of positive samples; Number the negative samples; The positive sample number; Learn hyperparameters for foreground contrast; ； ； ； Indicates that it does not contain A set of foreground contrast learning samples; When performing background contrast learning on the second FPN network, the following loss function is used to supervise the second FPN network: ； In the formula: Learning loss is contrasted with the background; For the first Background contrast loss for each sample; Number the background contrast learning samples; This serves as a background contrast learning sample set. Background sample; Learn hyperparameters by contrast with the background; ； for Augmented ; for ; Not including The background comparison learning sample set.

5. The target retrieval method based on contrastive learning according to claim 4, characterized in that, In step one, the first and second FPN networks form a twin network, sharing the same parameters and architecture; the backbone networks of the twin network both adopt the ResNet50 network; the ResNet50 network uses skip connections; the input data of the twin network undergoes enhanced preprocessing.

6. The target retrieval method based on contrastive learning and feature enhancement according to claim 4, characterized in that, Step two involves perturbing the mean and variance of shallow features in the first and second FPN networks to achieve feature enhancement. This includes the following steps: Set up a feature enhancer to perturb the mean and variance of shallow features in the first and second FPN networks. The feature enhancer includes a first feature enhancement module and / or a second feature enhancement module. The first feature enhancement module is used to perform random perturbation transformation on the extracted feature map. The calculation formula for the random perturbation transformation is as follows: ； ； ； Will Simplified to: ; The second feature enhancement module is used to perform enhancement transformation on the extracted feature map. The calculation formula for its enhancement transformation is as follows: ； In the formula: This is the original sample; for Samples generated after random perturbation transformation; for The mean; for The mean; for Standard deviation; for Standard deviation; for Random noise drawn from a Gaussian distribution; for Random noise drawn from a Gaussian distribution; After adjustment The mean; After adjustment Standard deviation; for The sample generated after enhancement transformation.

7. The target retrieval method based on contrastive learning according to claim 4, characterized in that, Step three includes the following steps: Step B1: Use the third FPN network to extract features from the target image. The backbone network of the third FPN network is combined with the feature pyramid network to update the momentum parameters. The calculation formula for the updated parameters is as follows: ； In the formula: This is the output of the backbone network of the third FPN network at the t-th iteration. The backbone network of the third FPN network Output of the next iteration; This is the output of the backbone network of the second FPN network at the t-th iteration. for Parameters; for Parameters; for Parameters; This represents the number of iterations. Update parameters for momentum; Step B2: Use ground truth bounding boxes of the basic categories to crop foreground features from the target image and process them through a projection layer; The processed foreground features are stored and updated in the foreground feature queue; each category has a separate foreground feature queue for storing foreground features; A background feature is randomly cropped from each target image and stored as a background feature queue; When creating the foreground feature queue, create queues of the same length for different categories to ensure that each category has the same number of features.

8. The target retrieval method based on contrastive learning according to claim 4, characterized in that, Step five, the method for aggregating the query patch features and candidate bounding boxes of the target image using the feature fusion module, includes the following steps: Step C1: Concatenate the feature vectors of the query patch feature and the candidate bounding box of the target image end to end; Step C2, Weighted Average: Calculate a weighted average of the features based on their importance; Step C3 involves transforming the fused features through one or more additional neural network layers.

Images