A general object retrieval method based on multi-scale collaborative learning
By employing a multi-scale collaborative learning approach, the problem of feature representation and encoding for small objects in image retrieval is solved. A multi-scale group collaborative learning retrieval model is constructed, which improves the accuracy and versatility of image retrieval and is applicable to object retrieval tasks in large-scale visual corpora.
Patent Information
- Application Number
- CN202410118368.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-29
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-01-29
AI Technical Summary
Existing image retrieval methods perform poorly in representing and encoding the features of small objects in large-scale visual corpora, and are difficult to effectively generalize to tasks that are not independent and identically distributed, resulting in a decline in retrieval performance.
A multi-scale collaborative learning approach is adopted, which extracts latent objects by segmenting all models and constructs a multi-scale group collaborative learning retrieval model. Image features are extracted and encoded using a backbone feature extraction network, a teacher embedding layer, and a student embedding layer. The model is trained by combining contrastive loss, group collaborative distillation loss, and self-distillation loss to optimize the model and improve retrieval accuracy.
It improves image representation and retrieval accuracy, effectively retrieves similar category objects that have appeared or not appeared in the training set, and enhances retrieval performance on different datasets.
Smart Images

Figure CN117932104B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image retrieval technology, specifically relating to a general object retrieval method based on multi-scale collaborative learning. Background Technology
[0002] Image retrieval has always been a research hotspot in computer vision and retrieval fields. How to locate specific objects in a large-scale visual corpus is crucial for finer-grained image retrieval. Therefore, object retrieval that focuses on finer-grained retrieval has significant research value.
[0003] Object retrieval involves two problems: how to retrieve similar images containing a given object, and how to determine the spatial location of the object. Objects often exist in cluttered backgrounds, occupying a relatively small and unpredictable portion of an image. Moreover, due to variations in scale, viewpoint, color, and partial occlusion, the actual object can deviate significantly from the query object, making image retrieval tasks challenging.
[0004] Existing object retrieval methods are mainly developed for specific object retrieval tasks, including pedestrian re-identification, vehicle re-identification, clothing retrieval, and logo retrieval. In these tasks, the distribution of objects follows the assumption of independent and identically distributed objects, which limits the ability of the algorithm to generalize to tasks that are not independent and identically distributed. For example, an algorithm developed based on a pedestrian dataset may perform well in pedestrian retrieval tasks, but it cannot be effectively generalized to vehicle retrieval tasks.
[0005] Object retrieval tasks typically employ supervised learning. In supervised learning's labeled datasets, most labeled objects are large in size. However, in dynamic and open-world retrieval environments, most unlabeled objects are small in size. Therefore, extracting and representing small objects presents significant challenges. While the Segment Anything Model (SAM) offers a potential solution for capturing the spatial context of objects, and current unsupervised representation learning algorithms have demonstrated good performance in general image representations, using these methods to obtain representations of general and small-scale objects may lead to a degraded retrieval performance.
[0006] In summary, most current image retrieval methods suffer from poor feature representation and encoding of arbitrary objects in images, and poor retrieval performance for categories of objects outside the training data distribution. Summary of the Invention
[0007] The purpose of this invention is to provide a general object retrieval method based on multi-scale collaborative learning that improves image representation, enhances retrieval performance, and increases accuracy.
[0008] The general object retrieval method based on multi-scale collaborative learning provided by this invention includes the following steps:
[0009] S1. Obtain the image dataset and preprocess the obtained image dataset;
[0010] S2. Construct a multi-scale group collaborative learning retrieval model;
[0011] S3. Using the image dataset obtained after preprocessing in step S1, train and optimize the multi-scale group collaborative learning retrieval model constructed in step S2 to obtain the final multi-scale group collaborative learning retrieval model.
[0012] S4. Using the final multi-scale group collaborative learning retrieval model constructed in step S3, perform inference testing on the object image to be retrieved to complete the retrieval process;
[0013] Step S1, which involves acquiring the image dataset and preprocessing it, specifically includes:
[0014] Preprocessing includes potential object extraction and grouping.
[0015] (1-1) Potential object extraction processing;
[0016] A segmentation-everything model is used to extract potential objects from an image dataset. Specifically, this involves: segmenting the potential objects using the segmentation-everything model to obtain bounding boxes, defined as [x1, y1; x2, y2]; where (x1, y1) represents the coordinates of the top-left corner of the object's bounding box; and (x2, y2) represents the coordinates of the bottom-right corner of the object's bounding box. Based on the bounding box coordinates, the object is cropped from the original image to obtain the object image.
[0017] Select training images and construct a training dataset by extracting and processing potential object images;
[0018] Select test images, and construct a library of images of objects to be retrieved by extracting information including potential object images, bounding boxes, and original image names, which will be used during the testing process.
[0019] Select the labeled objects in the test image as the query images during the test process;
[0020] (1-2) Group processing;
[0021] Select the training dataset from step (1-1), and group the images in the training dataset according to the size of the objects.
[0022] The size of an object can be represented by the number of pixels in the image obtained by cropping with a bounding box.
[0023] Based on the principle that the number of objects after grouping satisfies a uniform or normal distribution, the training dataset is grouped according to the size of the objects, resulting in four datasets: [400, 900), [900, 3600), [3600, 10000), and greater than or equal to 10000. Here, [400, 900) represents an image whose length multiplied by its width is between 400 and 900 (inclusive); [900, 3600) represents an image whose length multiplied by its width is between 900 and 3600 (inclusive); [3600, 10000) represents an image whose length multiplied by its width is between 3600 and 10000 (inclusive); and greater than or equal to 10000 represents an image whose length multiplied by its width is greater than 10000.
[0024] Step S2, which involves constructing a multi-scale group collaborative learning retrieval model, specifically includes:
[0025] The model includes a backbone feature extraction network and an embedding layer;
[0026] The backbone feature extraction network is used to perform image feature acquisition processing on the extracted potential objects;
[0027] The embedding layer extracts features from the network output based on the backbone features and obtains the object's embedding encoding according to the object's size.
[0028] The embedding layer includes a high-dimensional teacher embedding layer h. t Multi-scale student embedding layer;
[0029] The high-dimensional teacher embedding layer encodes the extracted image features to obtain the teacher image code.
[0030] A multi-scale student embedding layer is used to process the extracted image features based on the size of the object and obtain the student image code.
[0031] Multi-scale student embedding layers include high-dimensional student embedding layers h s Low-dimensional student embedding layer s ;
[0032] The multi-scale group collaborative learning retrieval model involved in training includes a backbone feature extraction network and a teacher embedding layer h. t High-dimensional student embedding layer h s ;
[0033] The multi-scale group collaborative learning retrieval model used in practical applications includes a backbone feature extraction network and a low-dimensional student embedding layer. s ;
[0034] 1) The backbone feature extraction network is selected from convolutional neural networks, specifically the GoogLeNet network;
[0035] The image is processed by extracting features using the Google network to obtain the corresponding feature vector f;
[0036] 2) The teacher embedding layer is a fully connected layer structure, and the resulting embedding encoding is used to evaluate the semantic similarity of objects;
[0037] The teacher embedding layer consists of two parts: pooling and fully connected layers.
[0038] For the feature vector f obtained by feature extraction, average pooling is used to obtain the average pooling result; max pooling is used to obtain the max pooling result; the average pooling result and the max pooling result are added together to obtain the pooling vector; the pooling vector is mapped to the embedding space through a fully connected layer;
[0039] The above processing procedure can be represented by the following formula:
[0040] f t =(averpool(f)+max pool(f))×W T +b
[0041] Among them, f t This represents the obtained teacher code; `averpool(·)` represents average pooling, which preserves the mean of the matrix; `maxpool(·)` represents max pooling, which preserves the maximum value of the matrix; W T b represents the weight parameters of the fully connected network; b represents the bias parameters of the fully connected network.
[0042] 3) The multi-scale student embedding layer is a fully connected layer structure, including two types: a high-dimensional student embedding layer h. s and low-dimensional student embedding layer l s h s Used to update the teacher model during training; s This refers to the embedding layer used in practical applications; f ls f hs This indicates the student's ID number that was retrieved.
[0043] The student embedding layer has several h with the same structure. s and l s The embedding layers encode different groups of object features respectively;
[0044] The student embedding layer consists of two parts: pooling and fully connected layers.
[0045] For the feature vector f obtained by feature extraction, average pooling is used to obtain the average pooling result; max pooling is used to obtain the max pooling result; the average pooling result and the max pooling result are added together to obtain the pooling vector; the pooling vector is mapped to the embedding space through a fully connected layer;
[0046] The above processing procedure can be represented by the following formula:
[0047]
[0048] Among them, f hs This represents the obtained high-dimensional student network encoding; W1 T b1 represents the weight parameters used to obtain the high-dimensional student code for the fully connected layer network; b2 represents the bias parameters used to obtain the high-dimensional student code for the fully connected layer network.
[0049]
[0050] Among them, f ls This represents the acquired low-dimensional network encoding; b1 represents the weight parameters used to obtain the low-dimensional student code for the fully connected layer network; b2 represents the bias parameters used to obtain the low-dimensional student code for the fully connected layer network.
[0051] Step S3, which involves using the image dataset obtained after preprocessing in step S1 to train and optimize the multi-scale group collaborative learning retrieval model constructed in step S2 to obtain the final multi-scale group collaborative learning retrieval model, specifically includes:
[0052] The multi-scale group collaborative learning retrieval model constructed in step S2 is used to calculate the contrast loss, group collaborative distillation loss, and self-distillation loss based on the teacher image code and student image code obtained through the multi-scale group collaborative learning retrieval model.
[0053] By combining the calculated contrastive loss, group cooperative distillation loss, and self-distillation loss, a multi-scale group cooperative learning retrieval model is trained, and the final multi-scale group cooperative learning retrieval model is constructed.
[0054] (3-1) Model training;
[0055] The multi-scale group collaborative learning retrieval model involved in training includes a backbone feature extraction network and a teacher embedding layer h. t Student Embedded Layer h s ;
[0056] A multi-scale grouped collaborative learning retrieval model is trained using a K-nearest neighbor-based batch acquisition method, specifically including: employing a high-dimensional teacher layer h... t The obtained object embedding code f tThe Euclidean distances between all objects are calculated using the following formula, and a distance matrix is constructed:
[0057]
[0058] Where n represents the dimension of the embedding encoding; f i t1 f represents the value of the embedding encoding of object t1 in the i-th dimension; i t2 This represents the value of the object t2 embedding encoding in the i-th dimension;
[0059] Based on the distance of each object to other objects, sort them in ascending order of distance, and select the K nearest samples to form a K-nearest neighbor matrix; the K-nearest neighbor matrix consists of N rows and K columns; where N represents the number of objects in the training set; the i-th row of the K-nearest neighbor matrix stores the indices of the K neighbors corresponding to the i-th object in the training set; randomly select M samples and their corresponding K neighbors from the N training samples, or randomly select M rows of samples from the K-nearest neighbor matrix to obtain a training batch of M×K training samples.
[0060] (3-2) Loss function calculation;
[0061] Loss functions include contrastive loss, group co-distillation loss, and autodistillation loss:
[0062] The self-distillation loss is calculated using the following formula:
[0063]
[0064] in, This indicates loss due to self-distillation; This indicates the embedding layer of the m-th high-dimensional student. The set of object codes obtained through encoding; This represents the embedding layer of the m-th low-dimensional student. The encoded set of objects; ψ(·) represents the softmax activation function; n represents the number of samples used to calculate the loss; m represents the grouping; Low-dimensional student embedding encoding and The relative distance between them is calculated using the following formula:
[0065]
[0066] Where k represents the pair The sequence number to be traversed from the n codes; Represents the traversal of the computation The k-th sample code encountered during the process; Indicates the participants in the calculation The i-th sample is encoded in the data; Indicates the participants in the calculation The j-th sample is encoded in the data; Representing low-dimensional student embedding encoding and The relative distance between them;
[0067] The contrast loss is calculated using the following formula:
[0068]
[0069] in, Indicates comparative loss; w ij Let represent the pairwise similarity score between the i-th object and the j-th object in the teacher embedding space, calculated as follows:
[0070]
[0071] Among them, f i t Indicates the embedding encoding f t The value at the i-th dimension; Indicates the embedding encoding f t The value in the j-th dimension; δ represents the tolerance interval;
[0072] Let represent the square of the hinge function; where,
[0073] The following formula is used to calculate the group's cooperative distillation loss:
[0074]
[0075]
[0076]
[0077] Among them, L CKD The loss represents the group collaborative distillation loss; Q represents the training sample group set; |Q| represents the number of group sets; a and b represent two different groups; L CKD (G a G b ) represents the cooperative distillation of group a on group b; G a G represents the set of training samples for group a in the training batch; b G represents the set of training samples for group b in the training batch; a∩b This represents the intersection of samples from group a and group b, where the area of the training samples in the latter group includes the area of the training samples in the former group; iThis represents the i-th intersection sample in the intersection of two grouped samples; The distillation loss is calculated using the similarity matrix; l represents the l-th cluster center, which is obtained by clustering the low-dimensional student codes of all training samples during model training; L represents the number of cluster centers. Represents the i-th object o in group a. i The similarity matrix with the l-th cluster center; Represents the i-th object o in group b. i The similarity matrix with the l-th cluster center;
[0078] The final loss function is calculated using the following formula:
[0079]
[0080] The model parameters during backpropagation are updated using the final loss function obtained through calculation.
[0081] (3-3) Model optimization;
[0082] The training dataset is used to train the network. The Adam optimizer is used to backpropagate the loss calculated for each training batch to update the network parameters, including the parameters of the backbone feature extraction network, the multi-scale student embedding layer, and the high-dimensional teacher embedding layer. The latent object image set extracted from the training set images is used to train and optimize the model. Finally, a multi-scale group collaborative learning retrieval model is constructed.
[0083] Step S4 describes using the final multi-scale group collaborative learning retrieval model constructed in step S3 to perform inference testing on the object image to be retrieved, completing the retrieval process. Specifically, this includes:
[0084] The multi-scale group collaborative learning retrieval model that completes the inference test includes the backbone feature extraction network and the teacher embedding layer h in the final multi-scale group collaborative learning retrieval model constructed in step S3. t Low-dimensional student embedding layer s ;
[0085] By using the final multi-scale group collaborative learning retrieval model, feature extraction and embedding encoding are performed on the object images to be retrieved to obtain the retrieval database;
[0086] When an image of the object to be retrieved exists, the backbone feature extraction network and the low-dimensional student embedding layer in the final multi-scale group collaborative learning retrieval model are utilized. sFeature extraction and embedding encoding are performed; similarity is obtained by calculating the Euclidean distance between the encoding of the query image and the encoding in the database to be searched; based on a pre-set similarity threshold or by sorting the objects in descending order of similarity and taking the top K objects, the retrieved objects are obtained; based on the bounding box of the retrieved objects and the original image name information, the retrieved objects are drawn in the original image, thus completing the object retrieval; the image of the object to be searched is the query image;
[0087] The general object retrieval method based on multi-scale collaborative learning provided by this invention uses a segmentation model to extract potential objects included in an image, learns an embedded encoding representation for the extracted objects through group collaborative metric learning, and then designs an objective function composed of contrast loss based on Euclidean distance, group collaborative distillation loss, and self-distillation loss to train a feature extraction and encoding model for general object retrieval. The method of this invention can retrieve both objects of similar categories that have appeared in the training set and objects of similar categories that have not appeared in the training set. Attached Figure Description
[0088] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation
[0089] like Figure 1 The diagram shown illustrates the method flow of this invention: This general object retrieval method based on multi-scale collaborative learning, provided by this invention, includes the following steps:
[0090] S1. Obtain the image dataset and perform preprocessing on the obtained image dataset; specifically including:
[0091] Preprocessing includes potential object extraction and grouping.
[0092] (1-1) Potential object extraction processing;
[0093] A segmentation-everything model is used to extract potential objects from an image dataset. Specifically, this involves: segmenting the potential objects using the segmentation-everything model to obtain bounding boxes, defined as [x1, y1; x2, y2]; where (x1, y1) represents the coordinates of the top-left corner of the object's bounding box; and (x2, y2) represents the coordinates of the bottom-right corner of the object's bounding box. Based on the bounding box coordinates, the object is cropped from the original image to obtain the object image.
[0094] Select training images and construct a training dataset by extracting and processing potential object images;
[0095] Select test images, and construct a library of images of objects to be retrieved by extracting information including potential object images, bounding boxes, and original image names, which will be used during the testing process.
[0096] Select the labeled objects in the test image as the query images during the test process;
[0097] (1-2) Group processing;
[0098] Select the training dataset from step (1-1), and group the images in the training dataset according to the size of the objects.
[0099] The size of an object can be represented by the number of pixels in the image obtained by cropping with a bounding box.
[0100] Based on the principle that the number of objects after grouping satisfies a uniform or normal distribution, the training dataset is grouped according to the size of the objects, resulting in four datasets: [400, 900), [900, 3600), [3600, 10000), and greater than or equal to 10000. Here, [400, 900) represents an image whose length multiplied by its width is between 400 and 900 (inclusive); [900, 3600) represents an image whose length multiplied by its width is between 900 and 3600 (inclusive); [3600, 10000) represents an image whose length multiplied by its width is between 3600 and 10000 (inclusive); and greater than or equal to 10000 represents an image whose length multiplied by its width is greater than 10000.
[0101] S2. Construct a multi-scale group collaborative learning retrieval model; specifically including:
[0102] The model includes a backbone feature extraction network and an embedding layer;
[0103] The backbone feature extraction network is used to perform image feature acquisition processing on the extracted potential objects;
[0104] The embedding layer extracts features from the network output based on the backbone features and obtains the object's embedding encoding according to the object's size.
[0105] The embedding layer includes a high-dimensional teacher embedding layer h. t Multi-scale student embedding layer;
[0106] The high-dimensional teacher embedding layer encodes the extracted image features to obtain the teacher image code.
[0107] A multi-scale student embedding layer is used to process the extracted image features based on the size of the object and obtain the student image code.
[0108] Multi-scale student embedding layers include high-dimensional student embedding layers h s Low-dimensional student embedding layers ;
[0109] The multi-scale group collaborative learning retrieval model involved in training includes a backbone feature extraction network and a teacher embedding layer h. t High-dimensional student embedding layer h s ;
[0110] The multi-scale group collaborative learning retrieval model used in practical applications includes a backbone feature extraction network and a low-dimensional student embedding layer. s ;
[0111] 1) The backbone feature extraction network is selected from convolutional neural networks, specifically the GoogLeNet network;
[0112] The image is processed by extracting features using the Google network to obtain the corresponding feature vector f;
[0113] 2) The teacher embedding layer is a fully connected layer structure, and the resulting embedding encoding is used to evaluate the semantic similarity of objects;
[0114] The teacher embedding layer consists of two parts: pooling and fully connected layers.
[0115] For the feature vector f obtained by feature extraction, average pooling is used to obtain the average pooling result; max pooling is used to obtain the max pooling result; the average pooling result and the max pooling result are added together to obtain the pooling vector; the pooling vector is mapped to the embedding space through a fully connected layer;
[0116] The above processing procedure can be represented by the following formula:
[0117] f t =(averpool(f)+max pool(f))×W T +b
[0118] Among them, f t This represents the obtained teacher code; `averpool(·)` represents average pooling, which preserves the mean of the matrix; `maxpool(·)` represents max pooling, which preserves the maximum value of the matrix; W T b represents the weight parameters of the fully connected network; b represents the bias parameters of the fully connected network.
[0119] 3) The multi-scale student embedding layer is a fully connected layer structure, including two types: a high-dimensional student embedding layer h. s and low-dimensional student embedding layer l s h s Used to update the teacher model during training; s This refers to the embedding layer used in practical applications; f ls fhs This indicates the student's ID number that was retrieved.
[0120] The student embedding layer has several h with the same structure. s and l s The embedding layers encode different groups of object features respectively;
[0121] The student embedding layer consists of two parts: pooling and fully connected layers.
[0122] For the feature vector f obtained by feature extraction, average pooling is used to obtain the average pooling result; max pooling is used to obtain the max pooling result; the average pooling result and the max pooling result are added together to obtain the pooling vector; the pooling vector is mapped to the embedding space through a fully connected layer;
[0123] The above processing procedure can be represented by the following formula:
[0124]
[0125] Among them, f hs This represents the obtained high-dimensional student network encoding; W1 T b1 represents the weight parameters used to obtain the high-dimensional student code for the fully connected layer network; b2 represents the bias parameters used to obtain the high-dimensional student code for the fully connected layer network.
[0126]
[0127] Among them, f ls This represents the acquired low-dimensional network encoding; b1 represents the weight parameters used to obtain the low-dimensional student code for the fully connected layer network; b2 represents the bias parameters used to obtain the low-dimensional student code for the fully connected layer network.
[0128] S3. Using the image dataset obtained after preprocessing in step S1, train and optimize the multi-scale group collaborative learning retrieval model constructed in step S2 to obtain the final multi-scale group collaborative learning retrieval model; specifically including:
[0129] The multi-scale group collaborative learning retrieval model constructed in step S2 is used to calculate the contrast loss, group collaborative distillation loss, and self-distillation loss based on the teacher image code and student image code obtained through the multi-scale group collaborative learning retrieval model.
[0130] By combining the calculated contrastive loss, group cooperative distillation loss, and self-distillation loss, a multi-scale group cooperative learning retrieval model is trained, and the final multi-scale group cooperative learning retrieval model is constructed.
[0131] (3-1) Model training;
[0132] The multi-scale group collaborative learning retrieval model involved in training includes a backbone feature extraction network and a teacher embedding layer h. t Student Embedded Layer h s ;
[0133] A multi-scale grouped collaborative learning retrieval model is trained using a K-nearest neighbor-based batch acquisition method, specifically including: employing a high-dimensional teacher layer h... t The obtained object embedding code f t The Euclidean distances between all objects are calculated using the following formula, and a distance matrix is constructed:
[0134]
[0135] Where n represents the dimension of the embedding encoding; f i t1 f represents the value of the embedding encoding of object t1 in the i-th dimension; i t2 This represents the value of the object t2 embedding encoding in the i-th dimension;
[0136] Based on the distance of each object to other objects, sort them in ascending order of distance, and select the K nearest samples to form a K-nearest neighbor matrix; the K-nearest neighbor matrix consists of N rows and K columns; where N represents the number of objects in the training set; the i-th row of the K-nearest neighbor matrix stores the indices of the K neighbors corresponding to the i-th object in the training set; randomly select M samples and their corresponding K neighbors from the N training samples, or randomly select M rows of samples from the K-nearest neighbor matrix to obtain a training batch of M×K training samples.
[0137] (3-2) Loss function calculation;
[0138] Loss functions include contrastive loss, group co-distillation loss, and autodistillation loss:
[0139] The self-distillation loss is calculated using the following formula:
[0140]
[0141] in, This indicates loss due to self-distillation; This indicates the embedding layer of the m-th high-dimensional student. The set of object codes obtained through encoding; This represents the embedding layer of the m-th low-dimensional student. The encoded set of objects; ψ(·) represents the softmax activation function; n represents the number of samples used to calculate the loss; m represents the grouping; Low-dimensional student embedding encoding and The relative distance between them is calculated using the following formula:
[0142]
[0143] Where k represents the pair The sequence number to be traversed from the n codes; Represents the traversal of the computation The k-th sample code encountered during the process; Indicates the participants in the calculation The i-th sample is encoded in the data; Indicates the participants in the calculation The j-th sample is encoded in the data; Representing low-dimensional student embedding encoding and The relative distance between them;
[0144] The contrast loss is calculated using the following formula:
[0145]
[0146] in, Indicates comparative loss; w ij Let represent the pairwise similarity score between the i-th object and the j-th object in the teacher embedding space, calculated as follows:
[0147]
[0148] Among them, f i t Indicates the embedding encoding f t The value at the i-th dimension; Indicates the embedding encoding f t The value in the j-th dimension; δ represents the tolerance interval;
[0149] Let represent the square of the hinge function; where,
[0150] The following formula is used to calculate the group's cooperative distillation loss:
[0151]
[0152]
[0153]
[0154] Among them, L CKD The loss represents the group collaborative distillation loss; Q represents the training sample group set; |Q| represents the number of group sets; a and b represent two different groups; LCKD (G a G b ) represents the cooperative distillation of group a on group b; G a G represents the set of training samples for group a in the training batch; b G represents the set of training samples for group b in the training batch; a∩b This represents the intersection of samples from group a and group b, where the area of the training samples in the latter group includes the area of the training samples in the former group; i This represents the i-th intersection sample in the intersection of two grouped samples; The distillation loss is calculated using the similarity matrix; l represents the l-th cluster center, which is obtained by clustering the low-dimensional student codes of all training samples during model training; L represents the number of cluster centers. Represents the i-th object o in group a. i The similarity matrix with the l-th cluster center; Represents the i-th object o in group b. i The similarity matrix with the l-th cluster center;
[0155] The final loss function is calculated using the following formula:
[0156]
[0157] The model parameters during backpropagation are updated using the final loss function obtained through calculation.
[0158] (3-3) Model optimization;
[0159] The training dataset is used to train the network. The Adam optimizer is used to backpropagate the loss calculated for each training batch to update the network parameters, including the parameters of the backbone feature extraction network, the multi-scale student embedding layer, and the high-dimensional teacher embedding layer. The latent object image set extracted from the training set images is used to train and optimize the model. Finally, a multi-scale group collaborative learning retrieval model is constructed.
[0160] S4. Using the final multi-scale group collaborative learning retrieval model constructed in step S3, perform inference testing on the object image to be retrieved to complete the retrieval process; specifically including:
[0161] The multi-scale group collaborative learning retrieval model that completes the inference test includes the backbone feature extraction network and the teacher embedding layer h in the final multi-scale group collaborative learning retrieval model constructed in step S3. t Low-dimensional student embedding layer s ;
[0162] By using the final multi-scale group collaborative learning retrieval model, feature extraction and embedding encoding are performed on the object images to be retrieved to obtain the retrieval database;
[0163] When an image of the object to be retrieved exists, the backbone feature extraction network and the low-dimensional student embedding layer in the final multi-scale group collaborative learning retrieval model are utilized. s Feature extraction and embedding encoding are performed; similarity is obtained by calculating the Euclidean distance between the encoding of the query image and the encoding in the database to be searched; based on a pre-set similarity threshold or by sorting the objects in descending order of similarity and taking the top K objects, the retrieved objects are obtained; based on the bounding box of the retrieved objects and the original image name information, the retrieved objects are drawn in the original image, thus completing the object retrieval; the image of the object to be searched is the query image;
[0164] In order to evaluate the effectiveness of the method of the present invention, experimental verification was carried out on the following datasets;
[0165] COCO-VOC dataset: The method of this invention uses a test set consisting of 9904 images from the validation sets of COCO2017 and VOC2007 for testing; none of the 9904 images were used in training;
[0166] BelgaLogo Dataset: The BelgaLogo dataset is a dataset for logo retrieval, containing 55 searchable logos and 10,000 images to be searched;
[0167] Visual Genome dataset: The Visual Genome dataset contains 108,077 images, and the method of this invention uses all of these images for testing;
[0168] LVIS dataset: The LVIS dataset uses the same images as the COCO dataset, but the LVIS dataset has finer-grained annotations and contains more annotated images. The method of this invention uses a validation set of 4726 images annotated by LVIS for testing.
[0169] Mean average precision (mPA) and first recall (recall@1) were used as the evaluation metrics for the experiment. Both mean precision and first recall included both object-level and image-level metrics.
[0170] The formula for calculating average accuracy is as follows:
[0171]
[0172] Where q represents the number of objects queried in the test set; p j This represents the rank of the j-th correctly recalled object in the similarity ranking;
[0173] The formula for calculating the first recall rate is as follows:
[0174]
[0175] Where, r i This represents the score of the i-th retrieved object compared to the query target;
[0176] When the first recall is a successful recall, r i =1; otherwise, r i =0;
[0177] At the object retrieval level, a valid recall is defined as an intersection-union (IU) threshold of 0.3, or when the IU of the recalled objects and the bounding boxes of the labeled objects is not less than 0.3. At the image retrieval level, the IU threshold is 10. -10 If an image contains an object that can be recalled, it is considered a valid image recall.
[0178] Table 1 is a comparative table of the retrieval performance of the method of the present invention and other methods on COCO-VOC;
[0179] Table 1
[0180]
[0181] The model was trained on the COCO-VOC training set. Table 1 shows the test results of the model on the COCO-VOC test set. It can be seen that the method of the present invention is superior to other unsupervised metric learning methods, improving the image-level mAP of GoogLeNet and MiT-B2 backbone networks by 2.65% and 10.03%, respectively.
[0182] Table 2 is a comparative table of the retrieval performance of the method of the present invention and other methods on other datasets;
[0183] Table 2
[0184]
[0185] When using the MiT-B2 backbone network, the object-level mAP score is improved by 6.69% on the Visual Genome dataset and by 0.89% on the LVIS dataset. When using the backbone network initialized with MoCo-V3, the highest mAP score and Recall@1 score are achieved on the BelgaLogos, Visual Genome, and LVIS datasets. However, on the Visual Genome dataset, the performance of the MSGCML method using MoCo-V3 is slightly reduced because the embedding space dimension of the present invention is 512, which is lower than the embedding dimension (1000) of MoCo-V3.
[0186] In summary, the method of the present invention can achieve better retrieval results with the same encoding cost.
[0187] This invention is supported by the Hunan Provincial Natural Science Foundation (Youth Fund) for Hierarchical Open Set Target Extraction and Retrieval for Natural Scene Images (2022JJ40632) and the National Natural Science Foundation of China (Youth Fund) for Research on Key Technologies of Open Set Target Extraction and Retrieval for Open World Scenes (62202499).
Claims
1. A general object retrieval method based on multi-scale collaborative learning, comprising the following steps: S1. Obtain the image dataset and preprocess the obtained image dataset; S2. Construct a multi-scale group collaborative learning retrieval model; S3. Using the image dataset obtained after preprocessing in step S1, train and optimize the multi-scale group collaborative learning retrieval model constructed in step S2 to obtain the final multi-scale group collaborative learning retrieval model; specifically including: The multi-scale group collaborative learning retrieval model constructed in step S2 is used to calculate the contrast loss, group collaborative distillation loss, and self-distillation loss based on the teacher image code and student image code obtained through the multi-scale group collaborative learning retrieval model. By combining the calculated contrastive loss, group cooperative distillation loss, and self-distillation loss, a multi-scale group cooperative learning retrieval model is trained, and the final multi-scale group cooperative learning retrieval model is constructed. Model training specifically includes: The multi-scale group collaborative learning retrieval model involved in training includes a backbone feature extraction network and a teacher embedding layer. Student Embedded Layer ; Adopting based on The nearest neighbor batch acquisition method trains a multi-scale group collaborative learning retrieval model, specifically including: employing a high-dimensional teacher layer... The obtained object embedding code The Euclidean distances between all objects are calculated using the following formula, and a distance matrix is constructed: in, Indicates the dimension of the embedding encoding; Represents objects Embedding encoding in the first Values in a dimension; Represents objects Embedding encoding in the first Values in a dimension; Sort each object according to its distance from other objects in ascending order of distance, and select the top one. Composed of the most recent samples Nearest neighbor matrix; The nearest neighbor matrix includes OK, Column; among which, Indicates the number of objects in the training set; The nearest neighbor matrix Row storage The corresponding objects The nearest neighbor indices in the training set; from Randomly selected from training samples Each sample and its corresponding A neighboring, or randomly selected In the nearest neighbor matrix Row samples, obtained A training batch consisting of training samples; Loss function calculation and model optimization, specifically including: (1) Calculation of the loss function: Loss functions include contrastive loss, group co-distillation loss, and autodistillation loss: The self-distillation loss is calculated using the following formula: in, This indicates loss due to self-distillation; Indicates by the first Group High-Dimensional Student Embedded Layer The set of object codes obtained through encoding; Indicates by the first Group low-dimensional student embedding layer The set of object codes obtained through encoding; This represents the softmax activation function; Indicates the number of samples used to calculate the loss; Indicates grouping; Low-dimensional student embedding encoding and The relative distance between them is calculated using the following formula: in, Indicates to middle The sequence number to be traversed using each encoding; Represents the traversal of the computation The first one traversed during the process Each sample is encoded; Indicates the participants in the calculation The first in Each sample is encoded; Indicates the participants in the calculation The first in Each sample is encoded; Representing low-dimensional student embedding encoding and The relative distance between them; The contrast loss is calculated using the following formula: in, Indicates comparative loss; Indicates the first The object and the first The pairwise similarity score of an object in the teacher embedding space is calculated using the following formula: in, Indicates embedding encoding The Middle Values in a dimension; Indicates embedding encoding The Middle Values in a dimension; Indicates the tolerance interval; Let represent the square of the hinge function; where, ; The following formula is used to calculate the group's cooperative distillation loss: in, Indicates the loss during cooperative distillation; Represents the grouping set of training samples; Indicates the number of grouped sets; , This indicates two different groups; Indicates grouping Grouping Cooperative distillation; Indicates training batch group The training sample set; Indicates training batch group The training sample set; Representation group and group The intersection of the samples, where the area size of the training samples in the latter group includes the area size of the training samples in the former group; Denotes the first digit in the intersection of two grouped samples. One intersection sample; This represents the distillation loss calculated using the similarity matrix; Indicates the first Cluster centers are obtained by clustering low-dimensional student codes of all training samples during model training. Indicates the number of cluster centers; Indicates grouping The Middle an object With the first Similarity matrix of cluster centers; Indicates grouping The Middle an object With the first Similarity matrix of cluster centers; The final loss function is calculated using the following formula: The model parameters during backpropagation are updated using the final loss function obtained through calculation. (2) Model optimization: Using centralized data from the training dataset, the Adam optimizer is used to backpropagate and update the network parameters, including the parameters of the backbone feature extraction network, the multi-scale student embedding layer, and the high-dimensional teacher embedding layer, for each training batch. The model is trained and optimized using the latent object image set extracted from the training set images. Finally, a multi-scale group collaborative learning retrieval model is constructed. S4. Using the final multi-scale group collaborative learning retrieval model constructed in step S3, perform inference tests on the object image to be retrieved to complete the retrieval process.
2. The general object retrieval method based on multi-scale collaborative learning according to claim 1, characterized in that... Step S1, which involves acquiring the image dataset and preprocessing it, specifically includes: Preprocessing includes potential object extraction and grouping. (1-1) Potential object extraction processing; A segmentation-everything model is used to extract latent objects from an image dataset. Specifically, this includes segmenting the latent objects using the segmentation-everything model to obtain bounding boxes, which are defined as follows: ;in, Represents the coordinates of the top-left corner of the object's bounding box; This represents the coordinates of the bottom right corner of the object's bounding box; the object is cropped from the original image based on these bounding box coordinates to obtain the object image. Select training images and construct a training dataset by extracting and processing potential object images; Select test images, and construct a library of images of objects to be retrieved by extracting information including potential object images, bounding boxes, and original image names, which will be used during the testing process. Select the labeled objects in the test image as the query images during the test process; (1-2) Group processing; Select the training dataset from step (1-1), and group the images in the training dataset according to the size of the objects. The size of an object can be represented by the number of pixels in the image obtained by cropping with a bounding box. Based on the principle that the number of objects after grouping satisfies a uniform or normal distribution, the training dataset is grouped according to the size of the objects, resulting in four datasets: [400, 900), [900, 3600), [3600, 10000), and greater than or equal to 10000. Here, [400, 900) represents an image whose length multiplied by its width is between 400 and 900 (inclusive); [900, 3600) represents an image whose length multiplied by its width is between 900 and 3600 (inclusive); [3600, 10000) represents an image whose length multiplied by its width is between 3600 and 10000 (inclusive); and greater than or equal to 10000 represents an image whose length multiplied by its width is greater than 10000.
3. The general object retrieval method based on multi-scale collaborative learning according to claim 2, characterized in that... Step S2, which involves constructing a multi-scale group collaborative learning retrieval model, specifically includes: The model includes a backbone feature extraction network and an embedding layer; The backbone feature extraction network is used to perform image feature acquisition processing on the extracted potential objects; The embedding layer extracts features from the network output based on the backbone features and obtains the object's embedding encoding according to the object's size. The embedding layer includes a high-dimensional teacher embedding layer. Multi-scale student embedding layer; The high-dimensional teacher embedding layer encodes the extracted image features to obtain the teacher image code. A multi-scale student embedding layer is used to process the extracted image features based on the size of the object and obtain the student image code. Multi-scale student embedding layers include high-dimensional student embedding layers. Low-dimensional student embedding layer ; The multi-scale group collaborative learning retrieval model involved in training includes a backbone feature extraction network and a teacher embedding layer. High-dimensional student embedding layer ; The multi-scale group collaborative learning retrieval model used in practical applications includes a backbone feature extraction network and a low-dimensional student embedding layer. .
4. The general object retrieval method based on multi-scale collaborative learning according to claim 3, characterized in that... The backbone feature extraction network specifically includes: The backbone feature extraction network was selected from Google's convolutional neural network. The image is processed using Google's network to extract features and obtain the corresponding feature vector. .
5. A general object retrieval method based on multi-scale collaborative learning according to claim 4, characterized in that... High-dimensional teacher embedding layer; specifically includes: The teacher embedding layer is a fully connected layer structure, and the resulting embedding encoding is used to evaluate the semantic similarity of objects; The teacher embedding layer consists of two parts: pooling and fully connected layers. The feature vector obtained from feature extraction The average pooling result is obtained through average pooling; the max pooling result is obtained through max pooling; the average pooling result and the max pooling result are added together to obtain the pooling vector; the pooling vector is mapped to the embedding space through a fully connected layer. The above processing procedure can be represented by the following formula: in, This indicates the teacher's code that was obtained; This indicates average pooling, which preserves the mean of the matrix. This indicates max pooling, which preserves the maximum value of the matrix. These represent the weight parameters of a fully connected network. This represents the bias parameters of a fully connected network.
6. A general object retrieval method based on multi-scale collaborative learning according to claim 5, characterized in that... Multi-scale student embedding layer; specifically including: The multi-scale student embedding layer is a fully connected layer structure, including two types: high-dimensional student embedding layer. and low-dimensional student embedding layer ; Used to update the teacher model during training; This refers to the embedding layer used in practical applications; , This indicates the student's ID number that was retrieved. The student embedding layer has several identical structures and The embedding layers encode different groups of object features respectively; The student embedding layer consists of two parts: pooling and fully connected layers. The feature vector obtained from feature extraction The average pooling result is obtained through average pooling; the max pooling result is obtained through max pooling; the average pooling result and the max pooling result are added together to obtain the pooling vector; the pooling vector is mapped to the embedding space through a fully connected layer. The above processing procedure can be represented by the following formula: in, This represents the obtained high-dimensional student network encoding; This represents the weight parameters used to obtain the high-dimensional student encoding for the fully connected layer network; This represents the bias parameters of the fully connected layer network used to obtain high-dimensional student codes; in, This represents the acquired low-dimensional network encoding; This represents the weight parameters used to obtain the fully connected layer network from the low-dimensional student encoding; This represents the bias parameters of the fully connected layer network used to obtain low-dimensional student encoding.
7. A general object retrieval method based on multi-scale collaborative learning according to claim 6, characterized in that... Step S4 describes using the final multi-scale group collaborative learning retrieval model constructed in step S3 to perform inference testing on the object image to be retrieved, thus completing the retrieval process; specifically, it includes: The multi-scale group collaborative learning retrieval model that completes the inference test includes the backbone feature extraction network and teacher embedding layer in the final multi-scale group collaborative learning retrieval model constructed in step S3. Low-dimensional student embedding layer ; By using the final multi-scale group collaborative learning retrieval model, feature extraction and embedding encoding are performed on the object images to be retrieved to obtain the retrieval database; When an image of the object to be retrieved exists, the backbone feature extraction network and the low-dimensional student embedding layer in the final multi-scale group collaborative learning retrieval model are utilized. Feature extraction and embedding encoding are performed; similarity is obtained by calculating the Euclidean distance between the encoding of the query image and the encoding in the database to be searched; the similarity is then sorted in descending order based on a pre-set similarity threshold or by similarity magnitude, with the top results being selected. The object is retrieved as a recall object; based on the bounding box of the recall object and the original image name information, the retrieved object is drawn in the original image, thus completing the object retrieval; the image of the object to be retrieved is the query image.
Citation Information
Patent Citations
Neurosynaptic network connectivity and collaborative knowledge exchange with visual neural networking and packetized augmented cognition
US20160119582A1
Semisupervised autoencoder for sentiment analysis
US20180165554A1
