A priori enhanced dual semantic-based appearance patent image retrieval method and system

Abstract

This invention discloses a patent image retrieval method based on prior enhanced dual semantics, comprising: inputting image-text pairs from a multimodal image-text dataset into an image encoder and a text encoder to obtain image features and text features; inputting the image features and text features into a Transformer to obtain text-image related features; mapping the text-image related features through a fully connected layer and inputting them into a text decoder, training it using a loss function to obtain an image caption generation network, the output of which is image captions; using the image captions to generate binary hash codes and adding them to a hash code database; and having the user input a query image or text into the image caption generation network, which returns images with high similarity. This invention utilizes a large model for prior visual and textual dual-layer semantic guidance, combined with an image caption generation model to generate captions, avoiding a large amount of manual annotation and enriching the user's retrieval needs.

Description

Technical Field

[0001] This invention relates to the field of image retrieval, and specifically to a method and system for retrieving appearance patent images based on prior enhanced dual semantics. Background Technology

[0002] In recent years, with the popularity of various chat software, heterogeneous data such as images and videos have experienced exponential growth. Given this wealth of image information, helping users retrieve relevant images has become a major challenge. Therefore, establishing a fast and accurate image retrieval system has become an important research direction. After decades of development, image retrieval technology has been widely applied to various aspects of social life, including digital libraries, medical imaging, satellite remote sensing imagery, and e-commerce.

[0003] The development of image retrieval can be traced back to the 1970s. Researchers at that time used text or keywords to provide simple descriptions of images, matching keywords with images—what we now call text-based image retrieval. In the early days of multimedia development, this method was simple, convenient, and easy to understand. However, with the rapid development of the internet and the ever-expanding and enriching databases of text, images, audio, and video, this method has gradually revealed its drawbacks. On the one hand, the traditional method of image annotation is manual annotation; however, for massive image sets, this method is impractical, time-consuming, and requires too much work. On the other hand, human annotation alone cannot fully express the rich visual information of an image. Moreover, due to differences in perception, different people have different semantic understandings of the same image, making annotation highly subjective.

[0004] In the 1990s, to overcome the aforementioned problems, researchers proposed content-based image retrieval (CBIR). This retrieval method involves multiple fields, such as artificial intelligence, pattern recognition, computer vision, and Bayesian and decision tree methods. This approach uses low-level image features, such as color, texture, or shape, to represent images. It automatically extracts image features through computers and uses similarity metrics to find the images the user needs, avoiding manual intervention and significantly reducing the workload of retrieval. Even today, content-based image retrieval (CBIR) remains a hot research topic in the field of image retrieval.

[0005] The primary challenge in CBIR is the extraction of image features, which connect pixel-level cues in an image to semantics from a human perspective. Among numerous methods, deep neural networks have proven capable of learning powerful features; however, directly using high-dimensional features extracted from pre-trained networks (such as ResNet) for computation incurs high computational costs and is impractical in real-world applications.

[0006] Recently, hashing methods have attracted widespread attention because they can significantly speed up the retrieval process when dealing with large-scale datasets. Typically, these methods aim to learn hash functions to project high-dimensional image features into a low-dimensional space, thereby generating a binary code for each data sample. The advantage of the generated low-dimensional binary code is that it can achieve efficient image retrieval through Hamming distance calculation, and it uses less storage space to bring faster retrieval speed.

[0007] Inspired by this, this paper proposes a visual patent image retrieval method based on dual semantic prior enhancement. Pre-trained image and text models are used for prior semantic guidance. The input image and text are each encoded by a pre-trained encoder, then converged into a stacked multi-layer Transformer to obtain a joint visual-text representation. Finally, after dimensional alignment by a fully connected layer, the representation is input into a pre-trained text decoder to obtain the final text. The generated text is hashed and then stored in a hash code database, enabling users to directly retrieve images by inputting text.

[0008] One current technology is a text-based image retrieval method proposed by Li W et al. in "Text-based image retrieval using progressive multi-instance learning". This method uses text or keywords to annotate images, allowing users to find relevant images by entering text. The disadvantages of this method are: with the rapid development of the Internet and the huge databases, the text annotation method has gradually revealed its drawbacks. On the one hand, the traditional method of image annotation is manual annotation, which requires too much work. On the other hand, different people have different semantic understandings of the same image due to differences in perception, which makes the annotation highly subjective. Therefore, this method involves a huge amount of work and has a large bias.

[0009] The second existing technology is a content-based image retrieval method proposed by Gu Wenjiao in "Research on Image Retrieval Technology Based on Content Semantics". This method uses neural networks to analyze images, automatically extracts the visual features of the images, uses the visual features as keywords for indexing, and then compares the distance between the index features and the query conditions, and finally feeds back the query results to the user. The disadvantages of this method are: existing DNN models have difficulty capturing the underlying data structure. In addition, the similarity (visual or semantic) between pairs of images measured by existing deep hashing methods is fuzzy or even uncertain, so the accuracy of existing methods is not high. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of existing methods and propose a method and system for appearance patent image retrieval based on prior-enhanced dual semantics. The main problems addressed by this invention are: 1) how to address the difficulty of applying manual annotation methods to existing text-based image retrieval methods when dealing with large amounts of data; and 2) how to solve the problem that the DNN models used in existing content-based image retrieval methods struggle to capture low-level data, and that the similarity between paired images measured by existing deep hashing methods is highly ambiguous and uncertain.

[0011] To address the aforementioned problems, this invention proposes a method for appearance patent image retrieval based on prior-enhanced dual semantics, the method comprising:

[0012] The image-text pairs in the image-text multimodal dataset are input into the pre-trained image encoder and text encoder respectively to obtain the encoded image features and text features;

[0013] The image features and text features are input into a stacked 6-layer Transformer to obtain text-image related features;

[0014] The text-image related features are mapped through a fully connected layer and then input into a pre-trained text decoder. The network is trained using a loss function to obtain a trained image captioning generation network, whose output is an image caption.

[0015] Using the image captions, a binary hash code is generated and added to the hash code database;

[0016] Using the trained image captioning generation network and the hash code database, users can input query images or text into the image captioning generation network, which will return images with high similarity.

[0017] Preferably, the input of the image features and text features into a stacked 6-layer Transformer to obtain text-image related features is as follows:

[0018] The 6-layer Transformer involves 3 parameters. In the first layer, image features are used as Q, and text features are used as K and V, resulting in the output of the first layer as the image-text joint features. In the remaining 2 to 6 layers, the image-text joint features are used as Q, K, and V as inputs respectively. Structurally, the 6-layer Transformer involves a multi-head self-attention layer, a residual and normalization layer.

[0019] The multi-head self-attention layer employs a multi-head attention mechanism, which decomposes the input sequence into eight subsequences, performs attention calculations on each subsequence separately, and then merges them. The Q, K, and V values ​​of the multi-head self-attention layer are consistent, and the overall attention calculation formula is as follows:

[0020]

[0021] Where K T This represents the transpose of K; Softmax is a common activation function that converts each element of the input vector into a probability value between 0 and 1, while ensuring that the sum of all probabilities is 1; to prevent QK... T The inner product is too large and needs to be divided by a scaling factor constant d. k ;

[0022] The residual and normalization layer consists of a residual layer and a normalization layer. The feedforward layer consists of two fully connected layers with an activation layer in between, using ReLU as the activation function. Treating the residual and normalization layer and the previous layer as a whole, two different scenarios can occur: the layer before the residual and normalization layer is either the multi-head attention layer or the feedforward layer. Let the input vector of the entire process be X, and the entire calculation process is as follows:

[0023] OutPut1=LayerNorm(X+Attention(Q, K, V))

[0024] OutPut2=LayerNorm(X+FeedForward(X))

[0025] FeedForward(X)=Relu(X·W1+b1)W2+b2

[0026] Relu(X) = max(0, X)

[0027] Where OutPut1 and OutPut2 correspond to two different cases where the layer before the residual and normalization layers is either the multi-head attention layer or the feedforward layer; LayerNorm is a normalization technique in deep learning; ReLU is an activation function in deep learning; W1, b1, W2, and b2 represent learnable parameters, W1 and W2 represent the weight matrices of the fully connected layer, and b1 and b2 represent the bias vectors of the fully connected layer.

[0028] Preferably, the text-image related features are mapped through a fully connected layer, input into a pre-trained text decoder, and trained using a loss function to obtain a trained image caption generation network, the output of which is an image caption. Specifically:

[0029] The loss function in this training step is as follows:

[0030]

[0031] The image captions in the dataset are represented as follows: This indicates that it consists of n words, and the generated image caption is represented as follows: It indicates that it consists of m words, and Sim(w1, w2) represents the cosine similarity between the word vectors of w1 and w2. The word vectors are obtained by inputting words into the text encoder.

[0032] Preferably, the step of generating a binary hash code using the image caption and adding it to the hash code database specifically involves:

[0033] The hash code is generated as follows:

[0034] H = hash(R)

[0035] in, R represents the subtitle features of the final layer, and the hash codes are added to the hash code database for subsequent queries.

[0036] Preferably, the step of utilizing the trained image captioning generation network and the hash code database allows users to input query images or text into the image captioning generation network, which then returns images with high similarity. Specifically:

[0037] When a user inputs a query image, the query image will generate descriptive text through the image captioning generation network, then generate a corresponding hash code, and input it into the hash code database for Hamming distance measurement. The image with a distance less than the threshold is returned as the final query result.

[0038] In the Hamming distance measurement process, the threshold k is set to 0.1*L, where L represents the length of the hash code. When the Hamming distance is less than the threshold k, the image is considered to have a high similarity to the original image, and the image is added to the returned result.

[0039] After the user inputs text, the step of inputting the text into the image captioning generation network will be omitted. Instead, the text will be directly input into the hash code database for Hamming distance measurement, and finally, a set of images that match the text description will be output.

[0040] Accordingly, the present invention also provides a design patent image retrieval system based on prior-enhanced dual semantics, comprising:

[0041] The data encoding unit inputs the image-text pairs from the image-text multimodal dataset into the pre-trained image encoder and text encoder, respectively, to obtain the encoded image features and text features;

[0042] The feature calculation unit inputs the image features and text features into a stacked 6-layer Transformer to obtain text-image related features;

[0043] The network training unit maps the text-image related features through a fully connected layer, inputs them into a pre-trained text decoder, and trains it using a loss function to obtain a trained image caption generation network, whose output is image captions.

[0044] The hash code generation unit uses the image captions to generate binary hash codes and adds them to the hash code database;

[0045] The result query unit utilizes the trained image captioning generation network and the hash code database. Users input query images or text into the image captioning generation network, which returns images with high similarity.

[0046] Implementing this invention has the following beneficial effects:

[0047] This invention proposes a novel hashing method for image retrieval, which combines the advantages of current mainstream methods. It utilizes a large model for prior visual and textual semantic guidance, and combines an image captioning generation model to generate captions. This avoids a large amount of manual annotation process, while supporting both text and image retrieval, thus enriching users' retrieval needs. Attached Figure Description

[0048] Figure 1 This is a flowchart of an appearance patent image retrieval method based on prior enhanced dual semantics according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the Transformer network structure according to an embodiment of the present invention;

[0050] Figure 3 This is a flowchart of the image caption generation network training process according to an embodiment of the present invention;

[0051] Figure 4 This is a structural diagram of an appearance patent image retrieval system based on prior enhanced dual semantics, according to an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Figure 1 This is a flowchart of an appearance patent image retrieval method based on prior enhanced dual semantics according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes:

[0054] S1. Input the image-text pairs in the LAION-400M multimodal dataset into the pre-trained image encoder and text encoder respectively to obtain the encoded image features and text features. The image encoder uses ViT and the text encoder uses BERT.

[0055] S2, Input the image features and text features into a stacked 6-layer Transformer to obtain text-image related features;

[0056] S3, the text-image related features are mapped through a fully connected layer and then input into a pre-trained text decoder OPT. The loss function is used for training to obtain a trained image caption generation network, the output of which is an image caption.

[0057] S4. Using the image captions, generate a binary hash code and add it to the hash code database;

[0058] S5, using the trained image captioning generation network and the hash code database, the user inputs a query image or text into the image captioning generation network, which returns images with high similarity.

[0059] Step S2 is as follows:

[0060] S2-1: Input the image features and text features obtained in step S1 into a stacked 6-layer Transformer to obtain text-image related features. The network structure diagram of the Transformer is shown below. Figure 2 As shown, the Transformer requires three input parameters Q, K, and V. In the first layer, image features are used as Q, and text features are used as K and V, resulting in the output of the first layer being the image-text joint features. In the remaining layers 2 to 6, the image-text joint features are used as Q, K, and V respectively as input. Structurally, the 6-layer Transformer involves a multi-head self-attention layer, a residual and normalization layer.

[0061] S2-2: Multi-head attention is a method for processing sequence data. It decomposes the input sequence into multiple subsequences (usually 8), performs attention calculations on each subsequence separately, and then merges them. Since the attention calculations performed on each subsequence are the same, they will be explained uniformly here. Multi-head attention has three input parameters: Q, K, and V. Figure 2The corresponding multi-head attention layer will have 3 arrows pointing inwards. The Q, K, and V values ​​of the multi-head self-attention layer are consistent. The formula for calculating the entire attention is as follows:

[0062]

[0063] Among them, K T This represents the transpose of K; Softmax is a common activation function that converts each element of the input vector into a probability value between 0 and 1, while ensuring that the sum of all probabilities is 1; to prevent QK... T The inner product is too large and needs to be divided by a scaling factor constant d. k Its value is 256;

[0064] S2-3: The residual and normalization layer consists of a residual layer and a normalization layer. The feedforward layer consists of two fully connected layers with an activation layer in between, using ReLU as the activation function. Treating the residual and normalization layer and the previous layer as a whole, two different scenarios can occur: the layer before the residual and normalization layer is either the multi-head attention layer or the feedforward layer. Let the input vector of the entire process be X. The entire calculation process is as follows:

[0065] OutPut1=LayerNorm(X+Attention(Q, K, V))

[0066] OutPut2=LayerNorm(X+FeedForward(X))

[0067] FeedForward(X)=Relu(X·W1+b1)W2+b2

[0068] Relu(X) = max(0, X)

[0069] Where OutPut1 and OutPut2 correspond to two different cases where the layer before the residual and normalization layers is either the multi-head attention layer or the feedforward layer; LayerNorm is a normalization technique in deep learning; ReLU is an activation function in deep learning; W1, b1, W2, and b2 represent learnable parameters, W1 and W2 represent the weight matrices of the fully connected layer, and b1 and b2 represent the bias vectors of the fully connected layer.

[0070] Step S3 is as follows:

[0071] S3-1: The text-image related features obtained in step S2 are mapped through a fully connected layer and then input into the pre-trained text decoder OPT. The network is then trained using a loss function to obtain the trained image caption generation network, whose output is the image caption. The loss function during training is as follows:

[0072]

[0073] The image captions in the dataset are represented as follows: This indicates that it consists of n words, and the generated image caption is represented as follows: This indicates that it consists of m words, and Sim(w1, w2) represents the cosine similarity between the word vectors of w1 and w2. The word vectors are obtained by inputting words into the text encoder BERT. The training process of the image caption generation network from S1 to S3 is as follows: Figure 3 As shown.

[0074] Step S4 is as follows:

[0075] S4-1: Generate binary hash code. Assuming the subtitle feature of the final layer in step S3 is R, use the hash() function in Python to generate the hash code:

[0076] H = hash(R)

[0077] in, This represents a matrix composed of hash codes. The generated hash codes are added to a hash code database for subsequent queries.

[0078] Step S5 is as follows:

[0079] S5-1: The input categories are divided into two cases: image and text.

[0080] S5-2: If the user inputs an image, the image will first be processed by an image captioning network to generate descriptive text, and then the corresponding hash code will be generated using the hash function in step S4. The obtained hash code will be input into the hash code database for Hamming distance measurement, and images with a distance less than the threshold will be returned as the final query result.

[0081] S5-3: Hamming distance measurement. Hamming distance is used in data transmission error control coding. It's a concept representing the number of different characters at corresponding positions in two binary codes of the same length. The Hamming distance is calculated by performing an XOR operation on the two codes and counting the number of 1s in the result. A threshold k is set to 0.1*L, where L is the length of the hash code. When the Hamming distance is less than k, the image is considered to have a high similarity to the original image and is added to the returned result.

[0082] S5-4: If the user inputs text, the input to the image captioning network will be skipped, and a set of images matching the text description will be returned.

[0083] Accordingly, the present invention also provides a design patent image retrieval system based on prior-enhanced dual semantics, such as... Figure 4 As shown, it includes:

[0084] Data encoding unit 1 inputs the image-text pairs in the image-text multimodal dataset into the pre-trained image encoder and text encoder respectively to obtain the encoded image features and text features;

[0085] Feature calculation unit 2 inputs the image features and text features into a stacked 6-layer Transformer to obtain text-image related features;

[0086] Network training unit 3 maps the text-image related features through a fully connected layer, inputs them into a pre-trained text decoder, and trains it using a loss function to obtain a trained image caption generation network, the output of which is an image caption.

[0087] Hash code generation unit 4 uses the image captions to generate binary hash codes and adds them to the hash code database;

[0088] The result query unit 5 utilizes the trained image caption generation network and the hash code database. Users can input query images or text into the image caption generation network, which then returns images with high similarity.

[0089] Therefore, this invention proposes a novel hashing method for image retrieval, which combines the advantages of current mainstream methods. It utilizes a large model for prior visual and textual semantic guidance, and combines an image captioning generation model to generate captions. This avoids a large amount of manual annotation process, while supporting both text and image retrieval, thus enriching users' retrieval needs.

[0090] The foregoing has provided a detailed description of the appearance patent image retrieval method and system based on prior enhanced dual semantics provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for retrieving appearance patent images based on prior-enhanced dual semantics, characterized in that, The method includes: The image-text pairs in the image-text multimodal dataset are input into the pre-trained image encoder and text encoder respectively to obtain the encoded image features and text features; The image features and text features are input into a stacked 6-layer Transformer to obtain text-image related features; The text-image related features are mapped through a fully connected layer, then input into a pre-trained text decoder, and trained using a loss function to obtain a trained image caption generation network, whose output is image captions. Using the image captions, a binary hash code is generated and added to the hash code database; Using the trained image captioning generation network and the hash code database, users can input query images or text into the image captioning generation network, which will return images with high similarity. Specifically, the input of the image features and text features into a stacked 6-layer Transformer to obtain text-image related features is as follows: The 6-layer Transformer involves 3 parameters. In the first layer, image features are used as Q, and text features are used as K and V, resulting in the output of the first layer as the image-text joint features. In the remaining 2 to 6 layers, the image-text joint features are used as Q, K, and V as inputs respectively. Structurally, the 6-layer Transformer involves a multi-head self-attention layer, a residual and normalization layer. The multi-head self-attention layer employs a multi-head attention mechanism, which decomposes the input sequence into eight subsequences, performs attention calculations on each subsequence separately, and then merges them. The Q, K, and V values ​​of the multi-head self-attention layer are consistent, and the overall attention calculation formula is as follows: where K T represents the transpose of K; Softmax is a common activation function that converts the value of each element of an input vector into a probability value between 0 and 1, while ensuring that the sum of all probability values is 1; to prevent the inner product of QK T from being too large, a scaling factor constant d k is needed to be divided; Specifically, the text-image related features are mapped through a fully connected layer, input into a pre-trained text decoder, and trained using a loss function to obtain a trained image caption generation network, the output of which is the image caption. The loss function during training is as follows: The image captions in the dataset are represented as follows: This indicates that it consists of n words, and the generated image caption is represented as follows: It indicates that it consists of m words, and Sim(w1,w2) represents the cosine similarity between the word vectors of w1 and w2. The word vectors are obtained by inputting words into the text encoder.

2. The appearance patent image retrieval method based on prior enhanced dual semantics as described in claim 1, characterized in that, The process of generating a binary hash code using the image captions and adding it to the hash code database is as follows: The hash code is generated as follows: H = hash(R) Wherein, H represents a matrix composed of hash codes, R represents the subtitle features of the final layer of the text decoder, and the hash codes are added to the hash code database for subsequent queries.

3. The appearance patent image retrieval method based on prior enhanced dual semantics as described in claim 1, characterized in that, The trained image captioning generation network and the hash code database are used to allow users to input query images or text into the image captioning generation network, which then returns images with high similarity. Specifically: When a user inputs a query image, the query image will generate descriptive text through the image captioning generation network, then generate a corresponding hash code, and input it into the hash code database for Hamming distance measurement. The image with a distance less than the threshold is returned as the final query result. In the Hamming distance measurement process, the threshold k is set to 0.1*L, where L represents the length of the hash code. When the Hamming distance is less than the threshold k, the image is considered to have a high similarity to the original image, and the image is added to the returned result. After the user inputs text, the step of inputting the text into the image captioning generation network will be omitted. Instead, the text will be directly input into the hash code database for Hamming distance measurement, and finally, a set of images that match the text description will be output.

4. A design patent image retrieval system based on prior-enhanced dual semantics, characterized in that, The system includes: The data encoding unit inputs the image-text pairs from the image-text multimodal dataset into the pre-trained image encoder and text encoder, respectively, to obtain the encoded image features and text features; The feature calculation unit inputs the image features and text features into a stacked 6-layer Transformer to obtain text-image related features; The network training unit maps the text-image related features through a fully connected layer, inputs them into a pre-trained text decoder, and trains it using a loss function to obtain a trained image caption generation network, whose output is image captions. The hash code generation unit uses the image captions to generate binary hash codes and adds them to the hash code database; The result query unit utilizes the trained image caption generation network and the hash code database. Users input query images or text into the image caption generation network, which returns images with high similarity. Specifically, the feature calculation unit is as follows: The 6-layer Transformer involves 3 parameters. In the first layer, image features are used as Q, and text features are used as K and V, resulting in the output of the first layer as the image-text joint features. In the remaining 2 to 6 layers, the image-text joint features are used as Q, K, and V as inputs respectively. Structurally, the 6-layer Transformer involves a multi-head self-attention layer, a residual and normalization layer. The multi-head self-attention layer employs a multi-head attention mechanism, which decomposes the input sequence into eight subsequences, performs attention calculations on each subsequence separately, and then merges them. The Q, K, and V values ​​of the multi-head self-attention layer are consistent, and the overall attention calculation formula is as follows: Where K T This represents the transpose of K; Softmax is a common activation function that converts each element of the input vector into a probability value between 0 and 1, while ensuring that the sum of all probabilities is 1; to prevent QK... T The inner product is too large and needs to be divided by a scaling factor constant d. k ; Specifically, the network training unit is: The loss function during training is as follows: The image captions in the dataset are represented as follows: This indicates that it consists of n words, and the generated image caption is represented as follows: It indicates that it consists of m words, and Sim(w1,w2) represents the cosine similarity between the word vectors of w1 and w2. The word vectors are obtained by inputting words into the text encoder.

5. The appearance patent image retrieval system based on prior-enhanced dual semantics as described in claim 4, characterized in that, The process of generating a binary hash code using the image captions and adding it to the hash code database is as follows: The hash code is generated as follows: H = hash(R) Wherein, H represents a matrix composed of hash codes, R represents the subtitle features of the final layer of the text decoder, and the hash codes are added to the hash code database for subsequent queries.

6. The appearance patent image retrieval system based on prior-enhanced dual semantics as described in claim 4, characterized in that, The trained image captioning generation network and the hash code database are used to allow users to input query images or text into the image captioning generation network, which then returns images with high similarity. Specifically: When a user inputs a query image, the query image will generate descriptive text through the image captioning generation network, then generate a corresponding hash code, and input it into the hash code database for Hamming distance measurement. The image with a distance less than the threshold is returned as the final query result. In the Hamming distance measurement process, the threshold k is set to 0.1*L, where L represents the length of the hash code. When the Hamming distance is less than the threshold k, the image is considered to have a high similarity to the original image, and the image is added to the returned result. After the user inputs text, the step of inputting the text into the image captioning generation network will be omitted. Instead, the text will be directly input into the hash code database for Hamming distance measurement, and finally, a set of images that match the text description will be output.

Images