A text generation and iterative matching-based image-text retrieval method

Abstract

The application discloses a kind of text generation and iterative matching-based image-text retrieval method, including extracting initial text feature vector and initial image feature vector in turn using BI-GRU network model and CNN network model.The text generation and iterative matching-based image-text retrieval method of this based on by training BI-GRU network model and CNN network model respectively, using trained BI-GRU network model and CNN network model respectively extracts text feature and image feature, then respectively with the image feature and text feature in database Similarity is calculated, the first several image features and text features with maximum similarity in database are regarded as the result output of retrieval, realize image-text retrieval;Text generation module enhances the overall semantic relevance of image and text through image-text information interaction, effectively solve the problem that image and text similarity cannot be directly measured in cross-modal image-text retrieval, improve image-text retrieval precision.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of cross-modal retrieval, and particularly relates to a picture-text retrieval method based on text generation and iterative matching. BACKGROUND

[0002] With the rapid development of mobile networks and self-media platforms, a large amount of image and text data is constantly emerging, which makes the demand for cross-modal image-text retrieval increasingly large. Cross-modal image-text retrieval is a task of retrieving relevant samples in a retrieval library of another modality according to one modality information, which has a wide range of applications in the academic and industrial fields. However, due to the different feature representation methods of images and texts, cross-modal picture-text retrieval faces the problem of modality heterogeneity, and it is difficult to measure the similarity of images and texts.

[0003] With the rapid development of mobile networks and self-media platforms, a large amount of image and text data is constantly emerging, which makes the demand for cross-modal image-text retrieval increasingly large. Cross-modal image-text retrieval is a task of retrieving relevant samples in a retrieval library of another modality according to one modality information, which has a wide range of applications in the academic and industrial fields. However, due to the different feature representation methods of images and texts, cross-modal picture-text retrieval faces the problem of modality heterogeneity, and it is difficult to measure the similarity of images and texts. SUMMARY

[0004] The application aims to solve the problems in the background art, and provides a picture-text retrieval method based on text generation and iterative matching.

[0005] To achieve the above-mentioned purpose, the technical solution adopted by the application is as follows:

[0006] The picture-text retrieval method based on text generation and iterative matching provided by the application comprises the following steps:

[0007] S1, initial text feature vectors and initial image feature vectors are extracted in sequence by using a BI-GRU network model and a CNN network model.

[0008] S2, the BI-GRU network model and the CNN network model are trained as follows:

[0009] S2.1, based on the initial text feature vectors, image feature iterative fusion operations are performed on each image feature in the initial image feature vectors to obtain fused image feature vectors, and K times of iterative fusion are performed to obtain image feature vectors after K times of iterative fusion.

[0010] S2.2, based on the initial image feature vector, each item of text feature in the initial text feature vector is subjected to text feature iterative fusion operation, a fused text feature vector is obtained, and K times of iterative fusion is performed to obtain a text feature vector after K times of iterative fusion.

[0011] S2.3, the image feature vector and the text feature vector after K times of iterative fusion are used to design a triple loss function to optimize the iterative fusion operation process.

[0012] S2.4, the image feature vector after K times of iterative fusion is input into a text generation module including an encoder and a decoder to generate a text feature sequence, and a loss function is designed to optimize the text generation module.

[0013] S3, the image or text to be searched is input into the trained CNN network model or BI-GRU network model, and the image feature or text feature is output correspondingly.

[0014] S4, the output image feature or text feature is subjected to cosine similarity calculation with the database text feature or image feature, and the top several text features or image features with the largest similarity in the database are output as the search result.

[0015] Preferably, the initial text feature vector and the initial image feature vector are extracted in sequence by using the BI-GRU network model and the CNN network model, which comprises:

[0016] Let the initial image feature vector be V0={v 0i |i=1,2,…,m,v 0i ∈R d} and the initial text feature vector be T0={t 0j |j=1,2,…,n,t 0j ∈R d}, wherein R d represents a d-dimensional vector, v 0i represents the i-th image feature in the initial image feature vector, m represents the number of image features, t 0j represents the j-th text feature in the initial text feature vector, and n represents the number of text features.

[0017] Preferably, based on the initial text feature vector, each item of image feature in the initial image feature vector is subjected to image feature iterative fusion operation, a fused image feature vector is obtained, and K times of iterative fusion is performed to obtain an image feature vector after K times of iterative fusion, which comprises:

[0018] The image feature vector subjected to the k-th time of iterative fusion is represented as V k ={v kii = 1, 2, …, m, v ki ∈ R d}, where k represents the number of iterations, and k = 0, 1, 2, …, K, v ki represents the i-th image feature in the image feature vector that is fused in the k-th iteration;

[0019] S2.1.1, for the k-th iteration fusion, respectively calculate the similarity s ki of each image feature v 0j with each text feature t kij in the initial text feature vector T0 using the cosine function, and the calculation formula is as follows:

[0020]

[0021] where T represents transposition, and sim represents the cosine function;

[0022] S2.1.2, then similarity normalization is performed on the similarity s kij , and the calculation formula is as follows:

[0023]

[0024] where represents the similarity s kij after similarity normalization, and relu represents the activation function;

[0025] S2.1.3, for each image feature v ki , an adaptive method is adopted to respectively calculate the relationship weight a kij , i.e. the attention value, of each image feature v ki with each text feature t 0j in the initial text feature vector T0 by using the softmax function, and the calculation formula is as follows:

[0026]

[0027] where represents a temperature coefficient for adjusting the attention distribution, and a kij represents the relationship weight of the i-th image feature v k in the image feature vector V ki in the k-th iteration fusion with the j-th text feature t 0j in the initial text feature vector T0;

[0028] S2.1.4, through the cross-modal attention mechanism, each image feature respectively interacts with each text feature (deleted), and a image feature v ki is calculated in a weighted summation manner, which is in the context relationship of the initial text feature vector T0 and the calculation formula is as follows:

[0029]

[0030] wherein, denotes the i-th image feature v ki in the k-th iteration fusion, and the context relationship image feature of the initial text feature vector T0;

[0031] the image feature vector V k in the k-th iteration fusion, and the context relationship image feature vector of the initial text feature vector T0

[0032] S2.1.5, using a gating mechanism to perform feature distillation on each image feature v ki and each context relationship image feature to obtain the distilled image feature v ki *, and the formula is as follows:

[0033]

[0034] wherein, d(·) denotes a gating mechanism;

[0035] Then, the image feature vector V k and the context relationship image feature vector are aggregated to obtain the distilled image feature vector V k+1 ={v ki *|i=1,2,…,m,v ki *∈R d}, and it is the image feature vector after the k-th iteration fusion, and serves as the input of the next iteration fusion of image features;

[0036] The process of obtaining the context relationship image feature vector and the process of image feature distillation are iterated K times in turn to obtain the image feature vector after K times of iteration fusion.

[0037] Preferably, based on the initial image feature vector, the text feature iteration fusion operation is performed on each text feature in the initial text feature vector to obtain the fused text feature vector, and K times of iteration fusion is performed to obtain the text feature vector after K times of iteration fusion, including:

[0038] In the k-th iteration fusion, the text feature vector is represented as T k ={t kj |j=1,2,…,n,t kj ∈R d}, t kjThis represents the j-th text feature in the text feature vector during the k-th iteration of fusion;

[0039] S2.2.1 For the k-th iteration of fusion, the cosine function is used to calculate the text feature t for each iteration. kj With each image feature v in the initial image feature vector V0 0i similarity s kij ', and the calculation formula is as follows:

[0040]

[0041] S2.2.2, then regarding the similarity s kij Similarity normalization is performed, and the calculation formula is as follows:

[0042]

[0043] in, Representing similarity s kij After similarity normalization;

[0044] S2.2.3, For each text feature t kj An adaptive method is used to calculate each text feature t using the softmax function. kj With each image feature v in the initial image feature vector V0 0i Relationship weight α kij ', which is the attention value, and the calculation formula is as follows:

[0045]

[0046] Where, α kij ' represents the text feature vector T fused in the k-th iteration. k The j-th text feature t kj Compared with the i-th image feature v in the initial image feature vector V0 0i Relationship weights;

[0047] S2.2.4. Using a cross-modal attention mechanism, each image feature interacts with each text feature individually, and a text feature t is summarized by weighted summation. kj Contextual relationships in the initial image feature vector V0 The calculation formula is as follows:

[0048]

[0049] in, t represents the i-th text feature in the k-th iteration of fusion. kj The contextual relationship between the text features and the initial image feature vector V0;

[0050] the text feature vector T k contextual relationship text feature vector

[0051] S2.2.5, using a gating mechanism to fuse each text feature t kj respectively with each contextual relationship text feature feature distillation to obtain distilled text feature t kj * , and the formula is as follows:

[0052]

[0053] Then aggregate the text feature vector T k and the contextual relationship image feature vector to obtain the distilled text feature vector T k+1 ={t kj * |i = 1, 2, …, n, t kj * ∈ R d}, and is the text feature vector after the kth iteration fusion, and is used as the input of the next text feature iteration fusion;

[0054] The process of obtaining the contextual relationship text feature vector and the process of text feature distillation are iterated K times in turn, to obtain the text feature vector after K times of iteration fusion.

[0055] Preferably, the image feature vector and the text feature vector after K times of iteration fusion are used to design a triplet loss function to optimize the iteration fusion operation process, including:

[0056] For the kth iteration fusion, the similarity between the image feature v ki in the image feature vector after iteration fusion and the contextual relationship image feature is calculated, and the similarity between the text feature t kj in the text feature vector after iteration fusion and the contextual relationship text feature is calculated, and then the sum is obtained to obtain the overall image-text similarity F k (V, T) of the kth iteration fusion, and the formula is as follows:

[0057]

[0058] Wherein, V represents the overall image, and T represents the overall text.

[0059] After K iterations, all the overall image-text similarity is summed up to obtain the total similarity F(V, T) between the image and the text, and the formula is as follows:

[0060]

[0061] The triple loss L of the small area strong negative sample is obtained triplet :

[0062]

[0063] Where, [x] + ≡max(x, 0) ensures that the value is not negative, and α represents the edge factor, represents the negative sample of the overall image, represents the negative sample of the overall text.

[0064] Preferably, the image feature vector fused after K iterations is input into a text generation module including an encoder and a decoder to generate a text feature sequence, including:

[0065] The first layer of the encoder is a Linear layer, and the second layer of the encoder is a first GRU layer;

[0066] The first layer of the decoder is an embedding layer, the second layer of the decoder is a Dropout layer, and the third layer of the decoder is a second GRU layer;

[0067] First, the image feature fused after K iterations is input into the Linear layer of the encoder and output as a first image feature after a Dropout operation;

[0068] Then, the first image feature is input into the first GRU layer, which uses an image feature conversion function to convert each feature in the input first image feature into a fixed-dimensional vector , and all features in the first image feature are converted to obtain a sequence feature space , where A represents the total number of steps.

[0069] Each vector is input into the decoder and sequentially processed by the embedding layer, the Dropout layer and the second GRU layer, and in the second GRU layer, the hidden state vector h a containing the previous information is generated using the text feature x a-1 corresponding to the current time step and the hidden state h a generated by the previous hidden unit, and each time step corresponds to a hidden unit, and is represented by the function as follows:

[0070] h a =fθ (x a ,h a-1 )

[0071] Calculate the generation probability of the correct text feature at each generation time step 'a':

[0072] Calculate the current hidden state h using vector dot product. a The similarity with each text feature embedding vector in the vocabulary is used, and the similarity is used as the generation probability p(w|h) for each text feature category. a The calculation formula is as follows:

[0073]

[0074] Where D represents a vocabulary containing N text features, W w Let w represent the learned embedding vector for each text feature w in vocabulary D;

[0075] Treating the N text features in the vocabulary as N categories, at the a-th time step, based on the sample true text feature distribution vector L composed of the N text features encoded in one-hot encoding... a =[l0,l1,…,l N-1 The probability distribution vector of generated text features at the current time step a is obtained by formula (1) as P. a =[p0,p1,…,p N-1 ], where p0, p1, ..., p N-1 The maximum value p in max-a That is, the correct text features s corresponding to each generation time step a. a The generation probability;

[0076] Complete the computation over A time steps, given the hidden state sequence {h1, h2, ..., h...} for each time step. A} and Equation (1) yield the probability distribution vector sequence {P1,P2,…,P} of the generated text features at each time step. A Then, take the maximum value p at each time step a. max-a The corresponding correct text features s a Ultimately, the corresponding generated text sequence S = {s1, ..., s} was generated. A}

[0077] Preferably, the loss function optimization text generation module is designed, including:

[0078] Minimize the cross-entropy between the probability distribution vector of the generated text features and the distribution vector of the real text features at each time step, and then obtain the loss value L for the entire sentence by summing the values ​​at each time step. align :

[0079]

[0080] Where E represents the optimization parameter, L a (i) represents the distribution vector of the i-th real text feature, P a (i) represents the probability distribution vector of the i-th generated text feature.

[0081] Preferably, the overall loss L during the training process of the BI-GRU network model and the CNN network model is... total The design is as follows:

[0082] Hyperparameters are used to aggregate the triplet loss and the text generation module loss, and the specific formula is as follows:

[0083] L total =L triplet +μ*L align .

[0084] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0085] This image-text retrieval method based on text generation and iterative matching trains a BI-GRU network model and a CNN network model respectively. It then uses the trained BI-GRU network model and CNN network model to extract text features and image features respectively. The similarity between these features and the image features and text features in the database is calculated. The top few image features and text features with the highest similarity in the database are used as the retrieval results to achieve image-text retrieval.

[0086] During the training of the BI-GRU and CNN network models, image feature vectors and text feature vectors are first obtained through iterative fusion. Then, the image features after iteration are input into the text generation module to be transformed into a text sequence, thereby optimizing the common semantic space. The text generation module uses the idea of ​​feature transformation to map image features to sentence features. Through the interaction of image and text information, the overall semantic relevance of images and text is enhanced, effectively solving the problem of difficulty in directly measuring the similarity between images and text in cross-modal image and text retrieval, and improving the accuracy of image and text retrieval. Attached Figure Description

[0087] Figure 1 This is a schematic diagram of the image and text retrieval method framework based on text generation and iterative matching of the present invention;

[0088] Figure 2 This is a schematic diagram of the structure of the text generation model of the present invention;

[0089] Figure 3 This is a flowchart illustrating the image and text retrieval method based on text generation and iterative matching according to the present invention. DETAILED DESCRIPTION

[0090] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0091] It should be noted that when a component is referred to as being "connected" with another component, it can be directly connected with the other component or there can be a component in the middle. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application.

[0092] In one embodiment, as shown in FIG. 1, a text generation and iterative matching based image-text retrieval method includes: Figures 1-3

[0093] S1, sequentially extracting an initial text feature vector and an initial image feature vector by using a BI-GRU network model and a CNN network model, specifically including:

[0094] Let the initial image feature vector be V0={v 0i |i=1,2,…,m,v 0i ∈R d} and the initial text feature vector be T0={t 0j |j=1,2,…,n,t 0j ∈R d}, where R d represents a d-dimensional vector, v 0i represents the i-th image feature in the initial image feature vector, m represents the number of image features, t 0j represents the j-th text feature in the initial text feature vector, and n represents the number of text features.

[0095] where the value of k in the initial image feature vector V0 and the initial text feature vector T0 is 0, and when k is 0, the result after this iteration fusion is the image feature vector V1 and the text feature vector T2.

[0096] S2, respectively training the BI-GRU network model and the CNN network model as follows:

[0097] ​S2.1 Based on the initial text feature vector, perform image feature iterative fusion operation on each image feature in the initial image feature vector to obtain the fused image feature vector, and perform K iterations of fusion to obtain the image feature vector after K iterations of fusion.

[0098] S2.1 specifically includes: First, the image feature vector for the k-th iteration of fusion is represented as V. k ={v ki |i=1,2,…,m,v ki ∈R d}, where k represents the iteration number, and k = 0, 1, 2, ..., K, v ki This represents the i-th image feature in the image feature vector during the k-th iteration of fusion.

[0099] S2.1.1 For the k-th iteration of fusion, the cosine function is used to calculate the feature v of each image. ki Each text feature t in the initial text feature vector T0 0j similarity s kij The calculation formula is as follows:

[0100]

[0101] Where T represents transpose and sim represents the cosine function.

[0102] S2.1.2, then regarding the similarity s kij Similarity normalization is performed, and the calculation formula is as follows:

[0103]

[0104] in, Representing similarity s kij After similarity normalization, ReLU represents the activation function;

[0105] S2.1.3, For each image feature v ki An adaptive method is used to calculate each image feature v using the softmax function. ki Each text feature t in the initial text feature vector T0 0j Relationship weight α kij (i.e., attention value), and the calculation formula is as follows:

[0106]

[0107] in, α represents the temperature coefficient used to adjust the distribution of attention. kij V represents the image feature vector in the k-th iteration of fusion. kthe i-th image feature v ki the j-th text feature t 0j in the initial text feature vector T0, and the relationship weight between the i-th image feature v

[0108] S2.1.4, calculate a context relationship image feature vector V ki in the initial text feature vector T0 in a weighted summation manner (i.e. by cross-modal attention mechanism, each image feature interacts with each text feature in the initial text feature vector respectively), and the calculation formula is as follows:

[0109]

[0110] wherein, the i-th image feature v ki in the k-th iteration fusion and the context relationship image feature of the initial text feature vector T0;

[0111] the image feature vector V k in the k-th iteration fusion and the context relationship image feature vector of the initial text feature vector T0

[0112] S2.1.5, use a gating mechanism to perform feature distillation between each image feature v ki and each context relationship image feature to obtain the distilled image feature v ki * , and the formula is as follows:

[0113]

[0114] wherein, d(·) represents a gating mechanism.

[0115] Then, the image feature vector V k and the context relationship image feature vector V are aggregated to obtain the distilled image feature vector V k+1 ={v ki * |i=1,2,…,m,v ki * ∈R d}, and it is the image feature vector after the k-th iteration fusion, and is used as the input of the next image feature iteration fusion.

[0116] The process of obtaining the context relationship image feature vector and the process of image feature distillation are iterated K times in turn, to obtain the image feature vector after K times of iteration fusion.

[0117] S2.2, based on the initial image feature vector, performing a text feature iterative fusion operation on each text feature in the initial text feature vector, obtaining a fused text feature vector, and performing K times of iterative fusion to obtain a text feature vector after K times of iterative fusion.

[0118] S2.2 specifically includes: first, the text feature vector after the kth iterative fusion is represented as T k kj |j=1,2,…,n,t kj ∈R d} and t kj represents the jth text feature in the text feature vector after the kth iterative fusion.

[0119] S2.2.1, for the kth iterative fusion, using the cosine function to calculate the similarity s kij ’ of each text feature t kj and each image feature v 0i in the initial image feature vector V0, and the calculation formula is as follows:

[0120]

[0121] S2.2.2, then similarity normalization is performed on the similarity s kij ’, and the calculation formula is as follows:

[0122]

[0123] wherein, represents the similarity s kij ’ after similarity normalization.

[0124] S2.2.3, for each text feature t kj , an adaptive method is adopted to calculate the relationship weight a kj ’ (i.e. attention value) of each text feature t 0i and each image feature v kij in the initial image feature vector V0 through the softmax function, and the calculation formula is as follows:

[0125]

[0126] wherein, a kij ’ represents the relationship weight of the jth text feature t k in the text feature vector T kj after the kth iterative fusion and the ith image feature v 0i in the initial image feature vector V0.

[0127] ​S2.2.4, summarize a text feature t in a weighted summation manner kj Contextual relationship with the initial image feature vector V0 (that is, by cross-modal attention mechanism, each text feature interacts with each text feature in the initial image feature vector), and the calculation formula is as follows:

[0128]

[0129] Wherein, represents the i-th text feature t in the k-th iteration fusion kj Contextual relationship with the initial image feature vector V0

[0130] The text feature vector T of the k-th iteration fusion k Contextual relationship with the initial image feature vector V0

[0131] S2.2.5, use the gating mechanism to perform feature distillation between each text feature t kj And each contextual relationship text feature To get the distilled text feature t kj * , and the formula is as follows:

[0132]

[0133] Then aggregate the text feature vector T k And the contextual relationship image feature vector To get the distilled text feature vector T k+1 ={t kj * |i=1,2,…,n,t kj * ∈R d}, and is the text feature vector after the k-th iteration fusion, and is used as the input of the next text feature iteration fusion.

[0134] The process of obtaining the contextual relationship text feature vector and the process of text feature distillation are iterated K times in turn, to obtain the text feature vector after K times of iteration fusion.

[0135] S2.3, use the image feature vector and the text feature vector after K times of iteration fusion respectively to design a triple loss function to optimize the iteration fusion operation process.

[0136] S2.3 specifically includes: for the k-th iteration fusion, calculate the image feature v ki In the image feature vector after iteration fusion and the contextual relationship image feature the similarity of the text features of the Kth iteration fusion text feature t kj contextual relationship text features the similarity of the text features of the Kth iteration fusion text feature t k (V, T), and the formula is as follows:

[0137]

[0138] wherein V represents the overall image, and T represents the overall text.

[0139] After K iterations, the overall image-text similarity F(V, T) between the image and the text is obtained by summing all the overall image-text similarities, and the formula is as follows:

[0140]

[0141] to obtain the triplet loss L of the small region strong negative sample triplet :

[0142]

[0143] wherein [x] + ≡max(x, 0) ensures that the value is not negative, and a represents an edge factor, represents the negative sample of the overall image, represents the negative sample of the overall text.

[0144] S2.4, input the image feature vector after K iterations of fusion into a text generation module including an encoder and a decoder, generate a text feature sequence, and design a loss function to optimize the text generation module.

[0145] S2.4 specifically includes that the first layer of the encoder is a Linear layer, and the second layer of the encoder is a first GRU (Gated Recurrent Unit) layer.

[0146] The first layer of the decoder is an embedding layer, the second layer of the decoder is a Dropout layer, and the third layer of the decoder is a second GRU (Gated Recurrent Unit) layer.

[0147] First, the image feature after K iterations of fusion is input into the Linear layer of the encoder and output as the first image feature after the Dropout operation (to prevent overfitting).

[0148] Then, the first image feature is input into the first GRU layer, and the first GRU layer uses the image feature conversion function to convert each feature in the input first image feature into a fixed-dimensional vector Furthermore, all features in the first image feature space are transformed to obtain the sequence feature space. Where A represents the total number of steps.

[0149] Each vector The input is fed into the decoder and passes through the embedding layer (e.g., ...). Figure 2 As shown, each vector is calculated in the embedding layer. The attention value is then compared with the attention values ​​of each vector. The embedded vectors are concatenated and fed into a Dropout layer, a Dropout layer (to prevent overfitting), and a second GRU layer (which includes several hidden units). The second GRU layer then uses the text feature x corresponding to the current time step. a The hidden state h generated by the previous hidden unit a-1 Generate a hidden state vector h that contains the information from the preceding context. a Each time step corresponds to a hidden unit, and is represented by the following function:

[0150] h a =f θ (x a ,h a-1 )

[0151] Calculate the generation probability of the correct text feature at each generation time step a.

[0152] Calculate the current hidden state h using vector dot product. a The similarity with each text feature embedding vector in the vocabulary is used, and the similarity is used as the generation probability p(w|h) for each text feature category. a The calculation formula is as follows:

[0153]

[0154] Where D represents a vocabulary containing N text features, W w Let w represent the learned embedding vector for each text feature w in vocabulary D.

[0155] Treating the N text features in the vocabulary as N categories, at the a-th time step, based on the sample true text feature distribution vector L composed of the N text features encoded in one-hot encoding... a =[l0,l1,…,l N-1 The probability distribution vector of generated text features at the current time step a is obtained by formula (1) as P. a =[p0,p1,…,p N-1 ], where p0, p1, ..., p N-1 The maximum value p in max-aThe probability of generating the corresponding correct text feature s a at each generation time step a.

[0156] After the calculation of A time steps, the probability distribution vector sequence {P1, P2, …, P A A} of the generated text feature at each time step is obtained by the hidden state sequence {h1, h2, …, h A A} and formula (1), and then the maximum value p max-a A corresponding to the correct text feature s a at each time step a is taken, and finally the corresponding generated text sequence S = {s1, …, s A A} is generated (the text sequence is used for comparison with the label text to improve the learning ability of each model).

[0157] The loss function is designed to optimize the text generation module, which includes:

[0158] The cross-entropy of the probability distribution vector of the generated text feature at each time step and the real text feature distribution vector is minimized, and then the loss value L align of the entire sentence is obtained by accumulating each time step.

[0159]

[0160] Where E represents the optimization parameter, L a (i) represents the i-th real text feature distribution vector, and P a (i) represents the i-th probability distribution vector of the generated text feature.

[0161] The overall loss L total of the BI-GRU network model and the CNN network model training process is designed as follows:

[0162] The hyperparameter is used to aggregate the triplet loss and the loss of the text generation module, and the specific formula is as follows:

[0163] L total = L triplet + μ * L align .

[0164] S3, input the image or text to be searched into the trained CNN network model or BI-GRU network model, and output the image feature or text feature.

[0165] Specifically, the trained CNN network model outputs the image feature, and the trained BI-GRU network model outputs the text feature.

[0166] S4, cosine similarity calculation is performed between the output image features or text features and the database text features or image features, and the top several text features or image features with the highest similarity in the database are output as the retrieval results.

[0167] Specifically, the number of the top several with the highest similarity is not limited, such as the top 10. Cosine similarity calculation is performed between the output image features and the database text features, and the top 10 text features with the highest similarity in the database are output as the retrieval results; cosine similarity calculation is performed between the output text features and the database image features, and the top 10 image features with the highest similarity in the database are output as the retrieval results.

[0168] In one embodiment, Figure 1 The framework corresponding to the training model is called the IMTG framework. In the training process of each model using the IMTG framework, two benchmark data sets are used, one of which is Flickr30K: composed of 31,000 pictures and 158,915 English texts. Each image is annotated with 5 English texts. In this paper, 29000 images are used for training, 1000 images are used for verification, and 1000 images are used for testing; the other data set is MS COCO: containing about 123,287 images, each image has at least 5 English texts. In this paper, 122,287 images are used to train all models, 1000 images are used for verification, and another 1000 images are used for testing.

[0169] The evaluation criteria Recall@k (the proportion of samples in which the correct answer appears in the top k returned results to the total test samples, usually measures Recall@1, Recall@5, Recall@10, mainly measures whether the correct answer appears) R@1, R@5 and R@10 evaluation indexes are used to measure the performance of text retrieval and image retrieval, and R@sum (the sum of all evaluation metrics) is counted. On Flickr30K data set and MS COCO data set, respectively, compared with existing models (double path convolution network DCP, semantic ranking method SCO, cross attention network SCAN, visual semantic network VSRN, iterative matching based on cycle attention IMRAM and multi semantic visual embedding network PVSE), and the experimental results of each comparison are shown in Table 1 (compared with existing models on Flickr30K data set) and Table 2 (compared with existing models on MS COCO data set):

[0170] Table 1

[0171]

[0172] Table 2

[0173]

[0174]

[0175] From Table 1 and Table 2, it can be seen that the IMTG used in the present method has the best performance in text retrieval and image retrieval compared with the existing models.

[0176] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, but it should be considered that any combination of the technical features is within the scope of the present disclosure, as long as the combination does not result in contradictions.

[0177] The above-described embodiments are merely representative of the present disclosure and should not be considered as limiting the scope of the patent. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present disclosure, and these modifications and improvements are within the scope of the present disclosure. Therefore, the scope of the patent of the present disclosure should be determined by the appended claims.

Claims

1. A method for generating and iteratively matching text-based image search, characterized in that: The text-based generation and iterative matching-based image-text retrieval method comprises the following steps: S1, sequentially extracting initial text feature vectors and initial image feature vectors by using a BI-GRU network model and a CNN network model, comprising: Let the initial image feature vector be , and the initial text feature vector be , where denotes a vector of dimension , denotes the th image feature in the initial image feature vector, denotes the number of image features, denotes the th text feature in the initial text feature vector, denotes the number of text features. S2, training the BI-GRU network model and the CNN network model as follows: S2.1, based on the initial text feature vector, for each image feature in the initial image feature vector, image feature iterative fusion operation is performed to obtain a fused image feature vector, and S2.2, based on the fused image feature vector, the initial text feature vector is updated to obtain a second iteration fused text feature vector. S2.3, based on the second iteration fused text feature vector, the initial image feature vector is updated to obtain a second iteration fused image feature vector. S2.2, based on the initial image feature vector, for each text feature in the initial text feature vector, perform text feature iterative fusion operation, obtain the fused text feature vector, and perform second iterative fusion to obtain the text feature vector after the second iterative fusion. second iterative fusion to obtain the text feature vector after the second iterative fusion. S2.3, using the respective image feature vectors and text feature vectors obtained in S2.1 and S2.2 designing a triplet loss function using the image feature vectors and text feature vectors obtained after the secondary iteration fusion, and optimizing the iterative fusion operation process; S2.4, inputting the image feature vector fused through the sub-iteration to a text generation module including an encoder and a decoder, generating a text feature sequence, and designing a loss function to optimize the text generation module; S2.4, inputting the image feature vector fused through the sub-iteration to a text generation module including an encoder and a decoder, generating a text feature sequence, and designing a loss function to optimize the text generation module; S3, inputting the image or text to be retrieved into the trained CNN network model or BI-GRU network model to output the image feature or text feature; S4, performing cosine similarity calculation on the output image feature or text feature and the database text feature or image feature, and outputting the top several text features or image features with the largest similarity in the database as the retrieval result.

2. The text generation and iterative matching based graphic-text retrieval method according to claim 1, wherein: The image feature iteration fusion operation is performed on each image feature in the initial image feature vector based on the initial text feature vector, a fused image feature vector is obtained, and second iteration fusion, to obtain an image feature vector after the second iteration fusion. The image feature iteration fusion operation is performed on each image feature in the initial image feature vector based on the initial text feature vector, a fused image feature vector is obtained, and The image feature vector after performing the mth iteration fusion is represented as wherein m represents the iteration number, and n represents the number of image features in the image feature vector after performing the mth iteration fusion. , The image feature vector after performing the mth iteration fusion is represented as wherein m represents the iteration number, and n represents the number of image features in the image feature vector after performing the mth iteration fusion. S2.1.1, for the first iteration fusion, the cosine function is used to calculate the similarity of each image feature and each text feature in the initial text feature vector , and the calculation formula is as follows: ​ ； wherein denotes the transpose, denotes the cosine function; S2.1.2, then similarity is normalized by similarity, and the calculation formula is as follows: ； wherein, representing similarity after similarity normalization, represents an activation function; S2.1.3、for each image feature , the adaptive method is adopted, and the relationship weight, i.e. the attention value, of each text feature in the initial text feature vector is calculated by a softmax function respectively, and the calculation formula is as follows: , the adaptive method is adopted, and the relationship weight, i.e. the attention value, of each text feature in the initial text feature vector is calculated by a softmax function respectively, and the calculation formula is as follows:​​​ ； wherein, denotes a temperature coefficient for adjusting the attention distribution, denotes the image feature vector in the th iteration fusion, th image feature, the relationship weight of the th text feature in the initial text feature vector, th text feature, the relationship weight of the S2.1.4, each image feature is interacted with each text feature through a cross-modal attention mechanism, and an image feature is calculated in a weighted summation manner In the initial text feature vector Contextual relationship in , and the calculation formula is as follows: ； wherein, represents the th iteration fusion th image feature and the initial text feature vector contextual relationship image feature; then the first image feature vector of the second iteration contextual relationship image feature vector contextual relationship image feature vector ; S2.1.5, utilize a gating mechanism to combine each image feature with each contextual relationship image feature perform feature distillation to obtain distilled image features , and the formula is as follows: ； wherein represents a gating mechanism; Then the image feature vectors are aggregated and the context-related image feature vectors , to obtain distilled image feature vectors , and the image feature vectors after the th iteration of fusion, and as input for the next iteration of image feature fusion. The process of obtaining the contextual image feature vector and the process of image feature distillation are iterated in sequence times, and an image feature vector after fusion of the previous times of iteration is obtained times of iteration is obtained.

3. The text generation and iterative matching based graphic-text retrieval method according to claim 2, wherein: The method comprises the following steps: performing text feature iterative fusion operation on each text feature in the initial text feature vector based on the initial image feature vector to obtain a fused text feature vector, and performing second iterative fusion to obtain a text feature vector after the second iterative fusion, comprising: second iterative fusion to obtain a text feature vector after the second iterative fusion, comprising: The text feature vector after the first iteration of fusion is represented as , The text feature vector after the first iteration of fusion is represented as , The text feature vector after the first iteration of fusion is represented as , The text feature vector after the first iteration of fusion is represented as , The text feature vector after the first iteration of fusion is represented as , S2.2.1, for the first iteration fusion, the cosine function is used to calculate the similarity of each text feature and each image feature in the initial image feature vector , and the calculation formula is as follows: ​​ ； S2.2.2, then regarding similarity Similarity normalization is performed, and the calculation formula is as follows: ； wherein representing similarity after similarity normalization; S2.2.3, For each text feature An adaptive method is used to calculate each text feature separately using the softmax function. With the initial image feature vector Each image feature Relationship weight This refers to the attention value, and the calculation formula is as follows: ； wherein, represents the th iteration of fusing the text feature vector th text feature th image feature with the initial image feature vector th image feature th image feature weight of the relationship S2.2.4, each image feature interacts with each text feature through the cross-modal attention mechanism, and a text feature is summarized in a weighted sum manner In the initial image feature vector Contextual relationship in , and the calculation formula is as follows: ； in, Indicates the first In the next iteration of fusion Text features With the initial image feature vector Contextual text features; then the first text feature vector of the second iteration contextual relationship text feature vector contextual relationship text feature vector ; S2.2.5, utilize a gating mechanism to obtain each text feature respectively with each context relationship text feature perform feature distillation to obtain distilled text features , and the formula is as follows: ； Then the text feature vector is aggregated and the context relationship image feature vector , to obtain the distilled text feature vector , and the text feature vector after the first iteration fusion, and as the input of the next text feature iteration fusion; The process of obtaining the context-related text feature vector and the process of text feature distillation are iterated in sequence After the first iteration, the text feature vector after fusion is obtained After the second iteration, the text feature vector after fusion is obtained 4. The text generation and iterative matching based graphic-text retrieval method of claim 3, wherein: The utilization is respectively through The image feature vector and the text feature vector after the secondary iteration fusion are used to design a triple loss function to optimize the iteration fusion operation process, including: For the Each iteration of fusion calculates the image features in the image feature vector after iterative fusion. Image features related to context The similarity, and the text features in the text feature vector after iterative fusion. Text features related to context The similarity is then summed to obtain the first... The overall image-text similarity of the next iteration fusion And the formula is as follows: ； wherein, represents an overall image, represents an overall text; go through After each iteration, the overall image-text similarity is summed to obtain the total similarity between the image and the text. And the formula is as follows: ； triplet loss for small region strong negative samples : ； wherein, the guaranteed value is not negative, denotes an edge factor, denotes a negative sample of the overall image, denotes a negative sample of the overall text.

5. The text generation and iterative matching based graphic-text retrieval method of claim 4, wherein: The method comprises the following steps: The image feature vector after the secondary iteration fusion is input into a text generation module comprising an encoder and a decoder to generate a text feature sequence, comprising: The first layer of the encoder is a Linear layer, and the second layer of the encoder is a first GRU layer; The first layer of the decoder is an embedding layer, the second layer of the decoder is a Dropout layer, and the third layer of the decoder is a second GRU layer; First, the image features after the second iteration fusion are input into the Linear layer of the encoder and output the first image features after the Dropout operation. First, the image features after the second iteration fusion are input into the Linear layer of the encoder and output the first image features after the Dropout operation. Then the first image features are input into a first GRU layer, which converts the image features by using an image feature conversion function Each feature in the input first image features is converted into a fixed-dimension vector , , and all features in the first image features are converted to obtain a sequence feature space , wherein represents the total number of steps Each vector The input is fed into the decoder and processed sequentially through the embedding layer, the Dropout layer, and the second GRU layer. The second GRU layer uses the text features corresponding to the current time step. The hidden state generated by the previous hidden unit Generate a hidden state vector containing the information above. Each time step corresponds to a hidden unit, and is represented by the following function: ； Compute each generated time step The generation probability of the corresponding correct text feature: The current hidden state is calculated by vector dot product The similarity with each text feature embedding vector in the vocabulary is calculated, and the similarity is taken as the generation probability of each text feature category , and the calculation formula is as follows: （1）； wherein, represents a vocabulary comprising text features, represents a vocabulary learning embedding vectors for each text feature in the vocabulary. The vocabulary list Each text feature is regarded as The category, in the At each time step, according to Each text feature constitutes a sample true text feature distribution vector encoded in one-hot encoding. The current time step can be obtained using formula (1). The probability distribution vector of the generated text features is ,in The maximum value in That is, for each generation time step The corresponding correct text features The generation probability; complete the calculation of each time step, by each time step hidden state sequence and formula (1), get the probability distribution vector sequence of each time step generated text feature then take the maximum value on each time step corresponding to the correct text feature , finally generated the corresponding generated text sequence .

6. The text generation and iterative matching based graphic-text retrieval method of claim 5, wherein: The design loss function optimizes the text generation module, comprising: Minimize the cross-entropy of the probability distribution vector of the generated text feature and the real text feature distribution vector at each time step, and then get the loss value of the whole sentence by accumulating each time step : ； wherein, denotes an optimization parameter, denotes the th real text feature distribution vector, denotes the th generated text feature probability distribution vector.

7. The text generation and iterative matching based graphic-text retrieval method of claim 6, wherein: The overall loss of the BI-GRU network model and the CNN network model training process The design is as follows: Using hyperparameters to aggregate the triple loss with the loss of the text generation module, and the specific formula is as follows: 。

Images