A Silk Artifact Image Retrieval System with a Divided-and-Integrated Architecture

Abstract

This invention relates to a divide-and-conquer integrated architecture for retrieving silk artifact images. It designs a cyclically optimized topic category partitioning scheme, dividing the silk artifact image dataset into several groups based on pre-retrieval results. For each group, a visual Transformer-based silk artifact image retrieval model is constructed, reducing the retrieval pressure on individual models and achieving divide-and-conquer. The semantics of topic categories are analyzed, and semantically similar topic categories are merged to obtain several superclasses. An image superclass recognition model based on ResNet50 is constructed. The confidence scores output by the recognition model are converted into guidance factors, thereby uniformly adjusting the Hamming distance of the output results of all retrieval models. This provides a basis for the joint ranking of the output results of multiple retrieval models, achieving adjustment and integration. Through task decomposition and collaborative integration of multi-path retrieval results, the retrieval of silk artifact images is realized, which helps alleviate negative factors unfavorable to network training, such as small data volume, multiple categories, uneven distribution, and long-tail distribution.

Description

Technical Field

[0001] This invention relates to an image retrieval technology, and more particularly to a silk artifact image retrieval system with a divide-and-conquer integrated architecture. Background Technology

[0002] Silk artifacts are historical witnesses to the exchange of diverse cultures across the country and an important carrier for telling China's story and disseminating Chinese culture. Searching for digitally imaged silk artifacts can be applied to the excavation, extraction, sorting, summarization, and conclusion of the spatiotemporal patterns of silk artifacts, which is conducive to improving the intelligence level of digital reproduction, protection, storage, display, and dissemination of silk artifacts.

[0003] In recent years, data-driven, end-to-end trained deep neural networks have been widely applied in the field of image retrieval, achieving groundbreaking progress. Existing image retrieval methods mainly focus on feature extraction and feature encoding. Lai et al. [HJLai,Y.Pan,Y.Liu,SCYan,Simultaneous Feature Learning and Hash Codingwith Deep Neural Networks,IEEE Conference on Computer Vision and Pattern Recognition (CVPR),2015,3270-3278] used convolutional neural networks to extract semantic features of images and divided the semantic features into several groups by splitting the encoding units. Then, each group of semantic features was learnably mapped to a one-bit hash code. Finally, a triplet loss function was used to measure the relevance of the hash code, thereby preserving the semantics. Wang Zhiming et al. [Wang Zhiming, Zhang Hang. A fast image retrieval method integrating features of multi-layer convolutional neural networks, Journal of Computer-Aided Design & Graphics, 2019, 31(8), 1410-1416] combined the semantic features of deep feature maps and the edge features of shallow feature maps to jointly map them into hash codes, enriching the information contained in the hash codes. The retrieval performance of these methods depends on the quality and scale of the dataset. However, silk artifacts are mostly unique pieces, with rich and varied themes and patterns, resulting in silk artifact images exhibiting characteristics such as small data volume, many categories, uneven distribution, and long-tail distribution, which are not conducive to the training of deep neural networks. Summary of the Invention

[0004] To address the negative impacts of limited data volume, numerous categories, uneven distribution, and long-tail distribution of silk artifact images on network training, a divide-and-conquer integrated architecture silk artifact image retrieval system is proposed. This system constructs multiple image retrieval models based on the visual Transformer. Each model is responsible for retrieving only a few topic categories, reducing the retrieval burden on individual models and lowering the requirements for dataset quality and scale, thus achieving a divide-and-conquer approach. Simultaneously, topic categories are merged according to semantic similarity to obtain several superclasses. A ResNet50-based image superclass recognition model is constructed. The confidence scores output by the recognition models are converted into guidance factors, thereby uniformly adjusting the Hamming distances of the outputs of all retrieval models. This provides a basis for the joint ranking of the outputs of multiple image retrieval models, achieving adjustment and integration.

[0005] The technical solution of the present invention is: a silk cultural relic image retrieval system with a divide-and-conquer integrated architecture, including several silk cultural relic image retrieval models G(a) based on visual Transformer and a silk image super-class recognition model based on ResNet50;

[0006] Each visual Transformer-based silk artifact image retrieval model G(a) is adapted to retrieve a subset of the main topic categories. The query image X is fed into several visual Transformer-based silk artifact image retrieval models G(a) for retrieval, and each model outputs a set of retrieval result images I. a,b And the Hamming distance corresponding to the search result image is H. a,b ;

[0007] The query image X and the search result image I a,b The images are sequentially fed into a ResNet50-based superclass recognition model for silk images, and the confidence scores P corresponding to the query images are obtained respectively. X The confidence level P of the retrieved image a,b The confidence score represents the probability that the input image belongs to the corresponding superclass. The confidence score output by the ResNet50-based silk image superclass recognition model is converted into a guidance factor. The guidance factor is used to adjust the Hamming distance of the retrieval result images output by all visual Transformer-based silk artifact image retrieval models G(a).

[0008] Preferably, the visual Transformer-based silk artifact image retrieval model G(a) consists of a feature extraction module, a hash encoding module, a similarity calculation module, and a loss function calculation module connected in sequence. Specifically, the feature extraction module extracts global features from the query image; the hash encoding module maps continuous numerical global feature codes to discrete numerical target hash codes, ensuring efficient searching and matching between global features; the similarity calculation module uses cosine distance to calculate the global feature similarity between the query image and the retrieved result image; and the loss function calculation module uses the Softmax function to measure the difference between the global feature codes and the target hash codes, thereby guiding the training and updating of the network.

[0009] Preferably, the feature extraction backbone network module of the ResNet50-based silk image superclass recognition model consists of concatenated convolutional blocks and residual blocks; wherein, the convolutional blocks are used to adjust the size of the feature tensor; the residual blocks are configured with skip connections to shorten the error backpropagation path and suppress the gradient vanishing phenomenon during training; the feature extraction backbone network module is also configured with batch normalization and linear correction, batch normalization alleviates the internal covariance effect of the distribution of network output data as the network parameters change, and linear correction helps to preserve the learning results of the network and prevent overfitting; the ResNet50-based silk image superclass recognition model uses cross-entropy loss to measure the difference between the network output result and the true superclass label, which is used to guide the update of network parameters; the calculation method of cross-entropy loss is shown in formula (5-1):

[0010]

[0011] Where Z is the number of superclasses; N is the total number of samples; y n,z Let z be the true supercategory label z corresponding to the nth image of the silk artifact; P n,z This represents the probability that the superclass recognition model classifies the nth image of a silk artifact as the zth superclass.

[0012] Preferably, the method for calculating the guidance factor is as follows:

[0013]

[0014] P X To query the confidence level of the image; P a,b The confidence score is the value of the retrieved image; the confidence score is a vector of length 6, z = 1, 2, ..., 6, where each dimension represents the probability that the input image belongs to the corresponding superclass; the Hamming distance is adjusted according to the guidance factor to obtain the new Hamming distance: H′ a,b =H a,b ·W a,b .

[0015] A method for establishing a retrieval model in a silk artifact image retrieval system with a divide-and-conquer integrated architecture, specifically including the following steps:

[0016] 1) Topic category classification: Constructing a pre-retrieval model G based on a deep neural network. pre The theme categories in the silk cultural relics image database are sorted according to the number of samples. The theme categories with the most samples are ranked higher. The theme categories are divided into multiple rounds according to the iterative optimization scheme. The top P theme categories are selected, and the selected samples are divided into training set and test set.

[0017] The pre-retrieval model G is trained using a training set of P topic categories. pre Then, G was tested using the test set. pre The retrieval performance of each topic category was statistically analyzed, and the Q topic categories with the best retrieval performance were selected to form a group.

[0018] The remaining PQ subject categories are returned to the silk artifact image database. The above iterative optimization process is repeated until all subject categories in the silk artifact image database have obtained corresponding groupings. The silk artifact image data is divided into 5 groups according to subject categories and named as group a, where a = 1, 2, ..., 5.

[0019] Image retrieval models are constructed by group: For group a, the a-th image retrieval model G(a) based on visual Transformer is constructed, where a = 1, 2, ..., 5.

[0020] The beneficial effects of this invention are as follows: The silk artifact image retrieval system of this invention, based on a divide-and-conquer integrated architecture, proposes a divide-and-conquer integrated architecture. On the one hand, it constructs multiple image retrieval models based on visual Transformer, reducing the retrieval pressure of individual models. On the other hand, it uses a guiding factor to uniformly adjust the Hamming distance of all retrieval result images, achieving collaborative integration of multiple retrieval models. The proposed divide-and-conquer integrated architecture alleviates the negative factors that hinder network training, such as the small amount of silk artifact image data, multiple categories, uneven distribution, and long-tail distribution. It helps to improve the intelligent level of digital reproduction, protection, storage, display, and dissemination of silk artifacts, and provides new research ideas for the field of small-sample image retrieval. Attached Figure Description

[0021] Figure 1 This is the overall design diagram of the silk artifact image retrieval system with the divide-and-conquer integrated architecture of the present invention;

[0022] Figure 2 This diagram illustrates the retrieval process of image X in the visual Transformer-based image retrieval model for silk artifacts according to the present invention.

[0023] Figure 3 This is a diagram of the ResNet50-based image super-class recognition model of this invention;

[0024] Figure 4 The integrated architecture diagram of this invention has been adjusted. Detailed Implementation

[0025] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0026] Design a silk artifact image retrieval system with a divide-and-conquer integrated architecture. Design an iterative optimization scheme for topic category partitioning, dividing the silk artifact image dataset into several groups based on pre-retrieval results. For example... Figure 1 The diagram shows the overall design of the silk image recognition and retrieval scheme. It constructs silk artifact image retrieval models based on the Visual Transformer in groups, reducing the retrieval pressure on individual models and achieving a divide-and-conquer approach. The semantics of topic categories are analyzed, and semantically similar topic categories are merged to obtain several superclasses. An image superclass recognition model based on ResNet50 is then constructed. The confidence scores output by the recognition models are converted into guidance factors, which are then used to uniformly adjust the Hamming distance of the output results of all silk artifact image retrieval models. This provides a basis for the joint ranking of the output results of multiple silk artifact image retrieval models, achieving adjustment and integration. This "divide and conquer, adjust and integrate" architecture, through the decomposition of the retrieval task and the collaborative integration of multiple retrieval results, achieves the retrieval of silk artifact images. It helps alleviate negative factors unfavorable to network training, such as insufficient data volume, numerous categories, uneven distribution, and long-tail distribution. The scheme is further elaborated below.

[0027] I. Divide and conquer

[0028] 1. Topic category classification: Constructing a pre-retrieval model G based on a deep neural network. pre The subject categories in the silk artifact image database are sorted according to the number of samples, with the top-ranked categories having more samples. Multiple rounds of subject category partitioning are performed using an iterative optimization scheme. The top P subject categories are selected, and their samples are divided into training and test sets. A pre-retrieval model G is trained using the training set of the P subject categories. pre Then, G was tested using the test set. preThe retrieval performance was assessed. The retrieval performance of each topic category was statistically analyzed, and the Q topic categories with the best retrieval performance were selected to form a group. The remaining PQ topic categories were then returned to the silk artifact image database. This iterative optimization process was repeated until all topic categories in the silk artifact image database had been grouped accordingly. In this embodiment, the silk artifact image data was divided into 5 groups according to topic categories, and each group was named group a, where a = 1, 2, ..., 5.

[0029] 2. Constructing Image Retrieval Models by Group: For group a, construct the a-th image retrieval model G(a) based on the visual Transformer, where a = 1, 2, ..., 5. For example... Figure 2 As shown, the visual Transformer-based image retrieval model for silk artifacts consists of a feature extraction module, a hash encoding module, a similarity calculation module, and a loss function calculation module connected sequentially. The feature extraction module extracts global features from the query image; the hash encoding module maps continuous numerical global feature codes to discrete numerical target hash codes, ensuring efficient searching and matching between global features; the similarity calculation module uses cosine distance to calculate the global feature similarity between the query image and the retrieved image; and the loss function calculation module uses the Softmax function to measure the difference between the global feature codes and the target hash codes, thereby guiding the network's training and updates. The loss function L... rtv The calculation is shown in formula (6-1):

[0030]

[0031] Where N is the number of samples; v n It is the global feature encoding of continuous numerical values ​​output by the feature extraction module; yn ∈[o1,...,o i ,...,o C ] T =O∈{-1,+1} C × K For discrete target hash encoding, C is the number of categories, and K is the length of the encoding vector. The number of categories C depends on the number of topic categories in the current group. Because... θ ni It is a global feature encoding of continuous numerical values ​​v n Discrete target hash encoding o i The angle between them. Regarding v n Perform L2 normalization to obtain ||v n || = 1; due to o i Given a binary discrete vector, normalizing it yields... The loss function L in formula (6-1) rtv It can be represented as:

[0032]

[0033] in, i = y n Global feature encoding of continuous numerical values ​​v n Discrete target hash encoding The angle between them.

[0034] The silk artifact images in group a are divided into training and testing sets. The a-th visual Transformer-based image retrieval model G(a) is trained using the training set of group a, where a = 1, 2, ..., 5. After training, for any query image X, it is fed into each of the five image retrieval models G(a), resulting in five sets of retrieval result images, denoted as I. a,b , a=1,2,…,5; b=1,2,…,N a Among them, N a Let H represent the number of images retrieved by the a-th image retrieval model. Let H be the Hamming distance between the retrieved images. a,b , a=1,2,…,5; b=1,2,…,N a The Hamming distance represents the similarity between the query image and the retrieved image as measured by the image retrieval model; the smaller the value, the more similar they are. For retrieved images from different groups (i.e., different 'a' values), their Hamming distance H... a,b A direct comparison is not possible. It is necessary to compare the Hamming distances H between different groups. a,b Make appropriate adjustments before sorting.

[0035] The "divide and conquer" approach divides the subject categories in the silk artifact image database into several groups through iterative optimization. For each group, a visual Transformer-based image retrieval model for silk artifacts is constructed, reducing the retrieval burden on individual models and lowering the requirements for dataset quality and size.

[0036] II. Adjustment and Integration

[0037] The semantics of topic categories are analyzed, and semantically similar topic categories are merged to obtain several superclasses. An image superclass recognition model based on ResNet50 is constructed. The confidence scores output by the recognition model are converted into guidance factors, which are then used to uniformly adjust the Hamming distances of the outputs of all retrieval models, providing a basis for the joint ranking of the outputs of multiple image retrieval models.

[0038] 1. Semantic Analysis of Topic Categories: Based on the semantic similarity of topic categories, the topic categories in the silk artifact image database are merged into several supercategories. In this example, six supercategories are obtained: flowers, natural scenery, geometry, animals, plants, and others, denoted as z = 1, 2, ..., 6.

[0039] 2. Construct an image super-class recognition model based on RestNet50. For example... Figure 3 As shown, ResNet50 mainly consists of concatenated convolutional blocks and residual blocks. The convolutional blocks are used to adjust the size of the feature tensor; the residual blocks are configured with shortcut connections, which shorten the error backpropagation path and effectively suppress the vanishing gradient phenomenon during training. In addition, ResNet50 is also configured with batch normalization and linear correction. The former alleviates the internal covariance effect of the network output data distribution changing with network parameters, ensuring the stability of network training. The latter helps preserve the network's learning results and prevent overfitting. Finally, cross-entropy loss measures the difference between the network output result and the true superclass label, and is used to guide the update of network parameters. The calculation method of cross-entropy loss is shown in Equation (5-1):

[0040]

[0041] Where Z is the number of superclasses; N is the total number of samples; y n,z Let z be the true supercategory label z corresponding to the nth image of the silk artifact; P n,z This represents the probability that the superclass recognition model classifies the nth image of a silk artifact as the zth superclass.

[0042] 1. Calculate the guiding factor and adjust the Hamming distance: For a query image X, the corresponding search result image is I. a,b Hamming distance H a,b Indicate X and I a,b The similarity is given by a = 1, 2, ..., 5; b = 1, 2, ..., N. a .like Figure 4 As shown, the query image X and the search result image I are... a,b The images are sequentially fed into a ResNet50-based image superclass recognition model to obtain the confidence scores P for each query image. X The confidence level P of the retrieved image a,b The confidence score is a vector of length 6, where each dimension represents the probability that the input image belongs to the corresponding superclass. The guidance factor is calculated using the following formula:

[0043] Where a = 1, 2, ..., 5; b = 1, 2, ..., Na; z = 1, 2, ..., 6; the Hamming distance is adjusted according to the guiding factor to obtain the new Hamming distance: H′ a,b =H a,b ·W a,b The pseudocode for adjusting the Hamming distance is as follows:

[0044] Input: X, I a,b H a,b Where a = 1, 2, ..., 5; b = 1, 2, ..., N a ;

[0045] Output: Adjusted Hamming distance H′ a,b

[0046] 1: Input the query image X into the superclass recognition model to obtain the confidence score P. X ;

[0047] 2: Image I of the search results a,b The data is input sequentially into the superclass recognition model to obtain the confidence score P. a,b Where a = 1, 2, ..., 5; b = 1, 2, ..., N a ;

[0048] 3: Calculate the guiding factor:

[0049] 4: Adjust the Hamming distance H′ using the guiding factor a,b =H a,b ·W a,b

[0050] 5: Return H′ a,b .

[0051] Since all samples in the silk artifact image database were used to train the super-class recognition model, the guidance factor can be used to compensate for the independence between multiple Transformer-based image retrieval models. The adjusted Hamming distance can be jointly ranked, providing a guarantee for integrating the retrieved images.

[0052] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A divide-and-conquer integrated architecture for retrieving images of silk artifacts, characterized in that, Includes several image retrieval models for silk artifacts based on the visual Transformer. G ( a ) and a ResNet50-based superclass recognition model for silk images; Each visual Transformer-based image retrieval model for silk artifacts G ( a These are all retrieval models adapted to retrieve specific topic categories within the overall topic category; query images X Several image retrieval models for silk artifacts based on visual Transformer were respectively input. G ( a The search is performed, and each visual Transformer-based silk artifact image retrieval model outputs a set of retrieval result images. And the Hamming distance corresponding to the search result image is ; Query Image X and search result images The images are sequentially fed into a ResNet50-based superclass recognition model for silk images to obtain the confidence scores corresponding to the query images. and the confidence level of the search result images The confidence score represents the probability that the input image belongs to the corresponding superclass. The confidence score output by the ResNet50-based silk image superclass recognition model is converted into a guidance factor, which is then used to adjust all visual Transformer-based silk artifact image retrieval models. G ( a Output the Hamming distance of the search result image.

2. The silk artifact image retrieval system with a divide-and-conquer integrated architecture according to claim 1, characterized in that, The image retrieval model for silk artifacts based on visual Transformer G ( a The system consists of a feature extraction module, a hash encoding module, a similarity calculation module, and a loss function calculation module connected in sequence. The feature extraction module extracts global features from the query image. The hash encoding module maps the global feature encoding of continuous values ​​to the target hash encoding of discrete values, ensuring efficient search and matching between global features. The similarity calculation module uses cosine distance to calculate the global feature similarity between the query image and the retrieved image. The loss function calculation module uses the Softmax function to measure the difference between the global feature encoding and the target hash encoding, thereby guiding the training and updating of the network.

3. The silk artifact image retrieval system with a divide-and-conquer integrated architecture according to claim 1, characterized in that, The feature extraction backbone network module of the ResNet50-based silk image superclass recognition model consists of concatenated convolutional blocks and residual blocks. The convolutional blocks are used to adjust the size of the feature tensor. Skip-layer connections are configured in the residual blocks to shorten the backpropagation path of errors and suppress the vanishing gradient phenomenon during training. The feature extraction backbone network module is also configured with batch normalization and linear correction. Batch normalization alleviates the internal covariance effect of the distribution of network output data as network parameters change, while linear correction helps preserve the network's learning results and prevents overfitting. The ResNet50-based silk image superclass recognition model uses cross-entropy loss to measure the difference between the network output result and the true superclass label, guiding the update of network parameters. The calculation method of cross-entropy loss is shown in formula (5-1). (5-1) in, Z Number of superclasses; N The total number of samples; For the first n The actual supercategory label corresponding to the image of Zhang Silk artifact z ; This indicates that the super-class recognition model will... n The image of the silk artifact was identified as the first... z The probability of each superclass.

4. The silk artifact image retrieval system with a divide-and-conquer integrated architecture according to claim 3, characterized in that, The method for calculating the guidance factor is as follows: ， To query the confidence level of the image; The confidence score is the score of the retrieved image. The confidence score is a vector of length 6, z=1,2,…,6, where each dimension represents the probability that the input image belongs to the corresponding superclass. The Hamming distance is adjusted based on the guiding factor to obtain the new Hamming distance: .

5. The silk artifact image retrieval system with a divide-and-conquer integrated architecture according to any one of claims 1 to 4, characterized in that, The establishment of the retrieval model specifically includes the following steps: 1) Topic category classification: Constructing a pre-retrieval model G based on a deep neural network. pre The theme categories in the silk cultural relics image database are sorted according to the number of samples. The theme categories with the most samples are ranked higher. The theme categories are divided into multiple rounds according to the iterative optimization scheme. The top P theme categories are selected, and the selected samples are divided into training set and test set. The pre-retrieval model G is trained using a training set of P topic categories. pre Then, G was tested using the test set. pre The retrieval performance of each topic category was statistically analyzed, and the Q topic categories with the best retrieval performance were selected to form a group. The remaining PQ subject categories are returned to the silk artifact image database, and the above iterative optimization process is repeated until all subject categories in the silk artifact image database have obtained corresponding groupings. The image data of silk artifacts were divided into 5 groups according to thematic categories, and each group was named accordingly. a ,in a =1,2,…,5; 2) Constructing image retrieval models by group: For groups a , construct the first a A Visual Transformer-Based Image Retrieval Model for Silk Artifacts G ( a ),in a =1,2,…,5.

Images