Cultural Relics Image Retrieval System Based on Contrastive Learning

Abstract

The present invention discloses a cultural relic image retrieval system based on contrastive learning. First, a supervised contrastive learning algorithm is used to train the model to obtain a good feature extractor, so that the extracted features can accurately express the semantic information contained in the image. Then, the similarity between the feature representations is calculated for retrieval, and the accuracy of the retrieval results is further improved by average query expansion and database-side feature enhancement. The retrieval system of the present invention trains the network to obtain a feature extractor through a supervised contrastive learning algorithm, which can extract effective and discriminative feature representations from images, and further improves the accuracy of retrieval through average query expansion and database-side data feature enhancement.

Description

Technical Field

[0001] The present invention relates to a feature extraction and contrast matching technology of cultural relic image data, and more specifically, to an image retrieval system for cultural relic data based on a contrast learning algorithm. Background Art

[0002] The audit work in each link of folk cultural relics trading and circulation relies too much on empirical analysis and visual judgment, and there are problems such as complicated processes and low efficiency, which has also given rise to the need for computer automatic retrieval of cultural relics images. Image retrieval aims to establish an index between the query image and the image database, and output images in the database that match or are similar to the query image based on a certain measurement method. Based on the current situation of large image data volume and high retrieval demand, it is urgent to propose high-fidelity digital information collection technology that adapts to the diversity of folk cultural relics and the complexity of scenes, and design key feature information extraction methods and comparison and matching methods for cultural relics data. Summary of the invention

[0003] In order to solve the problems in the prior art, the present invention provides a cultural relic image retrieval system based on contrastive learning, which solves the problems in the prior art such as low retrieval accuracy and efficiency, and high requirements on computing power.

[0004] The technical solution of the present invention is:

[0005] A cultural relic image retrieval system based on contrastive learning comprises a feature extractor and a retrieval module, wherein the feature extractor comprises preprocessing and feature extraction, and the retrieval module comprises sorting, indexing and similarity calculation; an image to be retrieved is input, preprocessed and feature extracted to obtain a corresponding feature vector, and all images in an image database are preprocessed and feature extracted to obtain a corresponding image feature library, and then the retrieval module is used to calculate the similarity between the feature vector of the query image and the feature vector in the image feature library, and the images in the image database are sorted and indexed using the similarity to obtain an image matching the query image as a final retrieval result.

[0006] The feature extractor network is trained using a supervised contrastive learning algorithm. The contrastive learning model uses two completely symmetrical and parameter-sharing branches, each of which includes data enhancement, an encoder network, and a projection network. The encoder network and the projection network form a feature extractor. For any image x, it forms two enhanced views x through two different data enhancement methods. i With x j ; Since the upper and lower branches are completely symmetrical, x in the upper branch i First, it is converted into the corresponding feature representation h through the encoder network. i =f θ (x i); then the nonlinear transformation structure--the projection network maps the feature representation to the final feature representation z i =g θ (h i ); Similarly, the enhanced view of the lower branch undergoes two nonlinear transformations to obtain the final feature representation z j =g θ (f θ (x j )).

[0007] The network training is as follows: randomly sampling N samples to form a Batch, denoted as {x k ,y k} k＝1，2，...，N ,y k is x k Through data enhancement, we can get 2N samples in, and It is a data pair obtained by two random data enhancement methods for the same sample. The label information in the data enhancement process will never change. For supervised contrastive learning, one sample corresponds to multiple positive samples, that is, the samples with the same label information in the batch are used as positive samples, and the samples with different label information are used as negative samples. In this way, the known label information can be effectively used for supervised learning, so that samples of the same category are closer in the representation space, and samples of different categories are far away from each other in the representation space, thereby improving the discriminative ability of feature representation. Therefore, the loss function of supervised contrastive learning is defined as:

[0008]

[0009]

[0010] Among them, 1 i≠j ∈{0, 1} is the indicator function, which takes the value 1 when and only when i≠j, otherwise it takes the value 0; τ>0 is the temperature parameter; z j(i) Indicates z i The positive sample, z i ·z j(i) Represents the inner product operation between vectors; Indicates the sample z in the Batch i The total number of samples with the same label information; the network is trained by optimizing the loss function in formula (4), and the trained encoder network and projection network are used as feature extractors to extract features from the query image and the images in the image database.

[0011] The similarity calculation function between the feature vectors uses the dot product of the feature vector after L2 regularization or the cosine similarity between the feature vectors:

[0012]

[0013] Among them, z i With z i represents a one-dimensional vector, and ||·||2 represents the L2 norm of the vector.

[0014] During the indexing and sorting process, average query expansion and database-side feature enhancement are used to further improve the accuracy of retrieval results.

[0015] The average query expansion is to first sort the images in the database according to the similarity between the feature vector of the original query Q0 and the feature vector in the feature library, return the first m (m < 50) results, and then average the original query Q0 and the m results to form a new query Q avg , and use the new query to generate the final retrieval results;

[0016]

[0017] Among them, z0 is the feature vector of the original query, z i is the feature vector of the i-th result.

[0018] The database-side feature enhancement replaces the original image by combining the image in the database with the image that is close to it, aiming to improve the quality of image representation by using the features of the image neighborhood. First, the similarity of the feature vectors in the image feature library is calculated two by two. For any image, the K image features closest to it are added, or the sum is weighted according to the ranking of the features:

[0019]

[0020] Where r is the rank of the image feature and k is the total number of similar images considered.

[0021] Beneficial effects:

[0022] The cultural relic image retrieval system based on contrastive learning proposed in the present invention trains the network to obtain a feature extractor through a supervised contrastive learning algorithm, which can extract effective and discriminative feature representations from images, and further improve the accuracy of retrieval through average query expansion and data feature enhancement at the data end. The user inputs a cultural relic image as a query, and the retrieval system can accurately retrieve and return results (one or multiple in sorted order) that match the query image in the image database. Both quantitative and qualitative results obtained on the common image dataset cifar10 show the effectiveness of the retrieval system. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1Cultural relic image retrieval system based on contrastive learning;

[0024] Figure 2 Contrasting learning models;

[0025] Figure 3 Quantitative and qualitative results of the system. DETAILED DESCRIPTION

[0026] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] The cultural relic image retrieval system based on contrastive learning of the present invention is as follows Figure 1 As shown in the figure, an image to be retrieved is input, and preprocessed such as scale transformation and random flipping is performed to extract the corresponding query image features. At the same time, all images in the image database are preprocessed and feature extracted to obtain the corresponding image feature library. Then, the similarity between the image features and all features in the image feature library is calculated, and the images in the image database are sorted and indexed using the similarity to obtain images that match the query image (one or more sorted images) as the final retrieval results. The following is a detailed introduction to the training of the feature extractor and the implementation of the retrieval module:

[0028] 1. Feature Extractor

[0029] The feature extractor needs to extract features from the query image and the images in the image database to effectively represent the image information. It is the basis for measuring the similarity between images in the subsequent retrieval module based on the similarity between feature representations, and is also the key to the retrieval accuracy of the entire retrieval system. Here, a supervised contrastive learning algorithm is used to train the feature extractor.

[0030] The core idea of ​​contrastive learning is to narrow the distance between samples and positive samples, while widening the distance between samples and negative samples. In supervised contrastive learning algorithms, the label information in the data set is used as supervision. Each sample corresponds to multiple positive samples and negative samples. Training the feature extractor can generate discriminative feature representations, which is conducive to the realization of image retrieval tasks. Figure 2 As shown, two completely symmetrical and parameter-sharing branches are adopted, each of which includes data enhancement, an encoder network and a projection network, where the encoder network and the projection network constitute a feature extractor.

[0031] For any image x, it forms two enhanced views x through two different data augmentation methods i With x j Since the upper and lower branches are completely symmetrical, taking the upper branch as an example, x iFirst, it is converted into the corresponding feature representation h through the encoder network (generally using ResNet as the model structure) i =f θ (x i ). Then the nonlinear transformation structure - the projection network (composed of two layers of MLP [FC->BN->ReLU->FC]) maps the feature representation to the final feature representation z i =g θ (h i ). Similarly, the enhanced view of the lower branch undergoes two nonlinear transformations to obtain the final feature representation z j =g θ (f θ (x j )). The purpose of contrastive learning is to make the distance between positive samples in the representation space closer and the distance between negative samples farther.

[0032] During network training, N samples are randomly sampled to form a Batch, denoted as {x k ,y k} k＝1，2，...，N ,y k is x k Through data enhancement, we can get 2N samples in, and is a data pair obtained by two random data enhancement methods for the same sample. The label information will not change during the data enhancement process. If the category supervision information is not considered, the data pair are positive samples to each other, and Except in Batch Any other 2N-2 samples are negative samples of each other. This is a self-supervised contrastive learning algorithm, and its loss function is defined as:

[0033]

[0034]

[0035] Among them, 1 i≠k ∈{0, 1} is the indicator function, which takes the value 1 when and only when i≠k, otherwise it takes the value 0; τ>0 is the temperature parameter; z j(i) Indicates z i The positive sample, z i ·z j(i) It represents the inner product operation between vectors. It can be seen that the numerator of the loss function encourages the similarity between samples and positive samples to be as high as possible, that is, the closer the distance in the representation space, the better; the denominator encourages the similarity between samples and negative samples to be as low as possible, that is, the farther the distance in the representation space, the better.

[0036] It can be seen that the loss function of self-supervised contrastive learning treats each sample as a separate category and cannot handle the situation where there are labels in the data set, that is, it is known that multiple samples belong to the same category. For supervised contrastive learning, one sample corresponds to multiple positive samples, that is, samples with the same label information in the batch are used as positive samples, and samples with different label information are used as negative samples. In this way, the known label information can be effectively used for supervised learning, so that samples of the same category are closer in the representation space, and samples of different categories are farther away from each other in the representation space, thereby improving the discriminative ability of feature representation. Therefore, the loss function of supervised contrastive learning is defined as:

[0037]

[0038]

[0039] in, Indicates the sample z in the Batch i The total number of samples with the same label information. The network is trained by optimizing the loss function in equation (4), and the trained encoder network and projection network are used as feature extractors to extract features from the query image and the images in the image database.

[0040] 2. Search module

[0041] The feature extractor is used to extract features from all images in the database to obtain the corresponding image feature library. When searching, a query image is input and features are extracted to obtain the corresponding feature vector. Then, the similarity between the feature vector of the query image and the feature vector in the graphic feature library is calculated through the retrieval module, and indexed and sorted according to the similarity, and the sorted images are output as the final result (the number is set manually).

[0042] The similarity calculation function between feature vectors generally uses the dot product of the feature vector after L2 regularization or the cosine similarity between feature vectors:

[0043]

[0044] In the indexing and sorting process, average query expansion and database-side feature enhancement are used to further improve the accuracy of retrieval results. Average query expansion is to first sort the images in the database according to the similarity between the feature vector of the original query Q0 and the feature vector in the feature library, return the first m (m < 50) results, and then average the original query Q0 and the m results to form a new query Q avg , and use the new query to generate the final retrieval results.

[0045]

[0046] Among them, z0 is the feature vector of the original query, z i is the feature vector of the i-th result. Database-side feature enhancement replaces the original image with a combination of images in the database and images that are close to it, aiming to improve the quality of image representation by using the features of the image neighborhood. First, the similarity of feature vectors in the image feature library is calculated pairwise. For any image, the features of the K images closest to it are added, or the sum is weighted according to the ranking of the features:

[0047]

[0048] Where r is the rank of the image feature and k is the total number of similar images considered.

[0049] In the proposed cultural relic image retrieval system based on contrastive learning, the encoder network in the feature extractor adopts the ResNet50 network architecture, and the similarity between image feature vectors is calculated using cosine similarity. When the retrieval module performs average query expansion and database-side feature enhancement, the top five results in feature similarity are selected for calculation. First, the feature extractor is trained based on the supervised contrastive learning algorithm, and the trained feature extractor is used to extract features from images in the image database to obtain a feature database. When a user queries, a query image is input into the system. The system will preprocess and extract features through the feature extractor to obtain query features, and then measure the similarity between the query features and the features in the feature database, and use feature similarity to achieve sorting and indexing, and finally output images (one or multiple sorted ones) that match the query image to the user. The retrieval system can achieve fast and effective retrieval. Table 1 shows its retrieval accuracy on the common image dataset cifar10. Figure 3 The output is shown as 10 search results.

[0050] Table 1

[0051] Precision@1(%) Precision@10(%) Map@all(%) Cifar10 98.0 100 98.1

[0052] The technical solutions disclosed and proposed by the present invention can be realized by those skilled in the art by referring to the contents of this article and appropriately changing the conditions, routes and other links. Although the methods and preparation techniques of the present invention have been described through preferred embodiments, relevant technicians can obviously modify or re-combine the methods and technical routes described herein without departing from the content, spirit and scope of the present invention to achieve the final preparation technology. It is particularly important to point out that all similar substitutions and modifications are obvious to those skilled in the art, and they are all considered to be included in the spirit, scope and content of the present invention. Matters not covered in the present invention belong to the known technology.

Claims

1. A cultural relic image retrieval system based on contrastive learning, characterized in that: It includes a feature extractor and a retrieval module, wherein the feature extractor includes preprocessing and feature extraction, and the retrieval module includes sorting, indexing, and similarity calculation; an image to be retrieved is input, and preprocessed and feature extracted to obtain a corresponding feature vector, and all images in an image database are preprocessed and feature extracted to obtain a corresponding image feature library, and then the retrieval module is used to calculate the similarity between the feature vector of the query image and the feature vector in the image feature library, and the images in the image database are sorted and indexed using the similarity to obtain an image matching the query image as the final retrieval result; The feature extractor network is trained using a supervised contrastive learning algorithm; The contrastive learning model uses two completely symmetrical and parameter-sharing branches, each of which includes data augmentation, an encoder network, and a projection network, where the encoder network and the projection network form a feature extractor. For any image x, it forms two enhanced views x through two different data augmentation methods i With x j ; Since the upper and lower branches are completely symmetrical, x in the upper branch i First, it is converted into the corresponding feature representation h through the encoder network. i =f θ (x i ); then the nonlinear transformation structure--the projection network maps the feature representation to the final feature representation z i =g θ (h i ); Similarly, the enhanced view of the lower branch undergoes two nonlinear transformations to obtain the final feature representation z j =g θ (f θ (x j )); The network training is as follows: randomly sampling N samples to form a Batch, denoted as {x k ,y k } k＝1，2，...，N ,y k is x k Through data enhancement, we can get 2N samples in, and It is a data pair obtained by two random data enhancement methods for the same sample. The label information will not change during the data enhancement process. For supervised contrastive learning, one sample corresponds to multiple positive samples, that is, the samples with the same label information in the batch are regarded as positive samples, and the samples with different label information are regarded as negative samples. In this way, the known label information can be effectively used for supervised learning, so that samples of the same category are closer in the representation space, and samples of different categories are farther away from each other in the representation space, thereby improving the discriminative ability of feature representation; therefore, the loss function of supervised contrastive learning is defined as: Among them, 1 i≠k ∈{0, 1} is the indicator function, which takes the value 1 when and only when i≠k, otherwise it takes the value 0; τ>0 is the temperature parameter; z j(i) Indicates z i The positive sample, z i ·z j(i) Represents the inner product operation between vectors; Indicates the sample z in the Batch i The total number of samples with the same label information; the network is trained by optimizing the loss function in formula (4), and the trained encoder network and projection network are used as feature extractors to extract features from the query image and the images in the image database.

2. The cultural relic image retrieval system based on contrastive learning according to claim 1, characterized in that: The similarity calculation function between the feature vectors uses the dot product of the feature vector after L2 regularization or the cosine similarity between the feature vectors: Among them, z i With z j represents a one-dimensional vector, and ||·||2 represents the L2 norm of the vector.

3. The cultural relic image retrieval system based on contrastive learning according to claim 1, characterized in that: During the indexing and sorting process, average query expansion and database-side feature enhancement are used to further improve the accuracy of retrieval results.

4. The cultural relic image retrieval system based on contrastive learning according to claim 3 is characterized in that: The average query expansion is to first sort the images in the database according to the similarity between the feature vector of the original query Q0 and the feature vector in the feature library, return the first m (m < 50) results, and then average the original query Q0 and the m results to form a new query Q avg , and use the new query to generate the final retrieval results; Among them, z0 is the feature vector of the original query, z i is the feature vector of the i-th result.

5. The cultural relic image retrieval system based on contrastive learning according to claim 3 is characterized in that: The database-side feature enhancement replaces the original image by combining the image in the database with the image that is close to it, aiming to improve the quality of image representation by using the features of the image neighborhood. First, the similarity of the feature vectors in the image feature library is calculated two by two. For any image, the K image features closest to it are added, or the sum is weighted according to the ranking of the features: Where r is the rank of the image feature and k is the total number of similar images considered.

Images