Cross-modal retrieval method and device based on transformer

Abstract

The application discloses a kind of based on the cross-modal retrieval method and equipment of Transformer, belong to the cross field of image and text technology, including steps: preparing data set;Data pre-processing;Using pre-training network feature extraction;Using image encoder and text encoder, secondary coding extraction is more rich information to feature;According to the similarity of extracted feature;Network training;Predict similarity score;Based on similarity score ranking.The application effectively overcomes the problem that information richness in the prior art is not captured, slow, noisy and the like, greatly improves the performance of the network, improves the training efficiency of the network, can be used for image-text matching, image-text retrieval and the like, and can effectively improve the image-text cross-modal matching and retrieval accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the cross field of image and text technology, and more particularly to a cross-modal retrieval method and device based on Transformer. BACKGROUND

[0002] The general process of cross-modal image-text retrieval can be roughly divided into three stages of training, indexing and querying:

[0003] (1) Training stage (extracting features): learn embedding network using picture and text data X and Y to obtain network parameters θ and ω.

[0004] (2) Indexing stage (database data): database data X ′ and Y ′ get embedding features through embedding network.

[0005] (3) Query stage (query data): calculate the similarity between query samples and index data, and the result is the ranking of the most similar sample.

[0006] Cross-modal image-text retrieval is very challenging, which needs to narrow the distance between image and text feature spaces and reduce the heterogeneity of different modalities to improve retrieval accuracy. The present application aims to provide a new solution to this problem. SUMMARY

[0007] The purpose of the present application is to overcome the shortcomings of the prior art and provide a cross-modal retrieval method and device based on Transformer, which has achieved advanced retrieval performance.

[0008] The purpose of the present application is achieved by the following scheme:

[0009] A cross-modal retrieval method based on Transformer, comprising the following steps:

[0010] Step 1: use Transformer as the backbone network to encode the extracted image and text features, model the fine-grained relationship, and obtain features with region-language clues; the image part uses the encoder part of pure Transformer, the text part uses the pruned Transformer encoder with added prior information, the encoding cross is used to increase the connectivity of the words before and after, accelerate the network to capture and learn the relationship between words, and the Transformer structure is optimized to remove a layer of residual, forward propagation layer and normalization layer;

[0011] Step 2: model the similarity measurement function to calculate the similarity of pictures and texts;

[0012] Step three, training the network based on the Transformer in step one based on the similarity calculated in step two;

[0013] Step four, loading the weight trained in step three into the network to obtain the similarity probability score of each image-text pair, and predicting the matching probability based on the similarity probability score;

[0014] Step five, according to the similarity score obtained in step four, sorting the search objects to obtain the target search object.

[0015] Further, in step two, the modeling similarity measure function calculates the similarity of the picture and the text, including the sub-steps: using the dot product similarity to measure the similarity of the fine-grained features of the image and the text obtained in step one, generating a region-word similarity matrix, then finding the maximum value of the region similarity corresponding to each word as the similarity of the word and the image, and then summing up the similarity of all words and images to obtain the similarity of the picture and the text.

[0016] Further, in step three, the network based on the Transformer in step one is trained based on the similarity calculated in step two, including the sub-steps: the model uses a triple loss based on difficult samples, and based on the similarity score obtained in step two, the similarity scores of a matching pair and a difficult non-matching pair are obtained by mining, and then the network is optimized by the triple loss based on difficult samples.

[0017] Further, before step one, it also includes the steps:

[0018] S1, preparing a data set;

[0019] S2, data preprocessing;

[0020] S3, image feature extraction;

[0021] S4, text feature extraction.

[0022] Further, in step S1, the data set is prepared by using Faster R-CNN as a visual feature extractor to extract fine-grained region features, and 2048-dimensional region features and region position coordinate information are obtained by the extraction of Faster R-CNN.

[0023] Further, in step S2, the data preprocessing includes the sub-steps: for each image, calculate the mean and standard deviation of each channel, then subtract the mean from each data in each channel, and divide by the variance to obtain the normalized result.

[0024] Further, in step S3, the image feature extraction specifically extracts 2048-dimensional region-level image features and position coordinate information of the region using a Faster R-CNN target extraction network; and fuses the image features with the position coordinates and area size information to obtain fused image features.

[0025] Further, in step S4, the text feature extraction includes the following sub-steps: word features required by the model are extracted by a pre-trained "bert-base-uncased" model, and a sentence is encoded into tokens, which are input into BERT, and the output is the word feature representation.

[0026] Further, in step S1, the data set includes a Flickr30K data set.

[0027] A Transformer-based cross-modal retrieval device, the computer device comprising a processor and a memory, the memory storing a computer program, when the computer program is loaded by the processor and executes the method of any one of the above.

[0028] The beneficial effects of the present application include:

[0029] (1) The present application adopts a prior information acceleration strategy and a network pruning strategy, adds prior position information in text encoding, makes the correlation between front and rear words greater, saves a residual structure to make the model more lightweight, and converges faster.

[0030] (2) The present application adopts a network pruning strategy, subtracts redundant Transformer encoding layers, uses fewer Transformer encoding layers, captures the relationship between fine-grained features, accelerates the convergence rate and retrieval rate, and improves the network performance.

[0031] (3) The network structure adopted by the present application achieves advanced performance and speed.

[0032] (4) The present application effectively overcomes the problems of not capturing information richness, slow speed, and noise in the prior art, greatly improves the performance of the network, improves the training efficiency of the network, can be used for image-text matching, image-text retrieval, etc., and effectively improves the image-text cross-modal matching and retrieval accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0034] Figure 1 Flowchart of the method of an embodiment of the present application;

[0035] Figure 2 Flickr30K training set data used in the simulation experiment of the present application;

[0036] Figure 3 First schematic diagram of the image encoding network structure of the present application;

[0037] Figure 4 Second schematic diagram of the image encoding network structure of the present application;

[0038] Figure 5 Schematic diagram of the text encoding network of the present application;

[0039] Figure 6 Recall rate comparison of the training process of an embodiment of the present application; wherein subgraph (a) is Text-to-Image Retrieval, and subgraph (b) is Image-to-Text Retrieval;

[0040] Figure 7 Recall rate comparison of the training process of an embodiment of the present application; wherein subgraph (c) is Text-to-Image Retrieval, and subgraph (d) is Image-to-Text Retrieval. DETAILED DESCRIPTION

[0041] All features disclosed in the specification of all embodiments, or all steps of all methods or processes disclosed in the specification, can be combined and / or extended, replaced, or substituted, in any manner, unless other features and / or steps are mutually exclusive.

[0042] In view of the technical problem that the retrieval accuracy of the cross-modal retrieval method in the background art needs to be improved, the inventors of the present application have made creative thinking and proposed a cross-modal retrieval scheme based on Transformer, and have verified the technical effect, and have achieved advanced performance on the Flickr30K data set.

[0043] In the process of forming the present application, the inventors noticed that the Transformer structure is a model that fully utilizes the self-attention mechanism. Compared with the recurrent neural networks (RNN), LSTM, and GRU commonly used in the field of NLP, the Transformer can not only effectively prevent the gradient disappearance problem existing in RNN, but also can solve the long-distance dependence problem by using the parallel input mechanism, and can speed up the calculation efficiency. Nowadays, many models with excellent performance in the field of NLP, such as BERT, are constructed based on the Transformer model, and the Transformer structure can give each word in the sentence a context meaning. The Transformer not only performs well in the field of NLP, but also as a visual feature extractor, it has shown excellent performance and is widely used in the field of CV.

[0044] The paper "Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders." (ACM Trans. Multim. Comput. Commun. Appl. 17(4): 128:1-128:23, 2021) published by NICOLA MESSINA et al. proposes a Transformer-based fine-grained image-text retrieval model TERAN (Transformer Encoder Reasoning and Alignment Network). This method narrows the regional-level features of the image and the word-level features of the text to realize image-text alignment at the fine-grained level, and uses the state-of-the-art Transformer encoder as the secondary encoder of the image and the text to model the fine-grained segment relationship, which can obtain rich visual-linguistic clues. However, the inventors further found the following technical problems: this method uses too many stacked layers of Transformer, has a large number of parameters, and the model is large, which not only leads to slow convergence speed, but also easily introduces noise to affect the retrieval effect.

[0045] In order to solve the technical problems discovered above, in continuous exploration, the present application proposes a lightweight cross-modal retrieval technology scheme based on Transformer, and advanced retrieval performance is achieved. It mainly involves image and text cross-modal matching and retrieval, and can be used for image-text matching and image-text retrieval, etc., and can effectively improve the image-text cross-modal matching and retrieval accuracy. Specifically, in the specific embodiment, the prior information acceleration strategy and network pruning strategy are used on the basis of the TERAN model, the prior position information is added in the text encoding, the correlation between the front and rear words is larger, the network is pruned and optimized, the network is more lightweight, the convergence is faster, the interference of noise is effectively relieved, and the network performance is improved. In a further embodiment, the steps include:

[0046] (1) Prepare the data set: use Faster R-CNN as a visual feature extractor to extract region features at a fine-grained level, and obtain 2048-dimensional region features and region position coordinate information through the extraction of Faster R-CNN.

[0047] (2) Data preprocessing: in order to prevent gradient disappearance, gradient explosion and overfitting phenomenon in the training process, data preprocessing is performed: for each image, the mean and standard deviation of each channel are calculated, and then each data in each channel is subtracted from the mean and divided by the variance to obtain the normalized result.

[0048] (3) Image feature extraction: use Faster R-CNN: target extraction network to extract 2048-dimensional region-level image features and region position coordinate information; and fuse the image features with the position coordinates and area size information to obtain fused image features;

[0049] (4) Text feature extraction: the word features required by the model are extracted through the pre-trained "bert-base-uncased" model. We encode the sentence into tokens (tokens), input it into BERT, and the output is the word feature representation.

[0050] (5) Image and text fine-grained encoding: use Transformer as the main network to re-encode the extracted image and text features, model the fine-grained relationship, and obtain features with rich region-language clues. The image part uses the encoder part of pure Transformer, the text part uses the pruned Transformer encoder with added prior information, the correlation between the front and rear position words is increased through encoding cross, and the network learning of the relationship between words is accelerated. And optimize the Transformer structure, remove one layer of residual, forward propagation layer and normalization layer.

[0051] (6) Modeling similarity measure function: The model uses dot product similarity to measure the similarity of fine-grained segment features. And through the way of first max-pooling and then summing, the image-sentence similarity is obtained. The fine-grained features of the image and text obtained in step (5) are used to measure their similarity using dot product similarity to generate a region-word similarity matrix. Then find the maximum value of the region similarity corresponding to each word as the similarity of the word and the image, and then sum the similarity of all words and images to obtain the similarity of the picture and the text.

[0052] (7) Training network: The model uses a triplet loss based on difficult samples. Based on the similarity scores obtained in step (6), the similarity scores of a matching pair and a difficult mismatching pair are obtained by mining, and then the network is optimized by a triplet loss based on difficult samples. Triplet loss can help the neural network pay more attention to detailed information and distinguish inputs with small differences, which can help the image-text retrieval task achieve better results. However, the data distribution selected by the triplet loss may not be uniform, which can cause instability in the training process. The triplet loss based on difficult samples only selects the best mismatching pair, which can effectively alleviate this problem.

[0053] (8) Predict the matching probability: Load the weights obtained in step (7) into the network to obtain the similarity probability score of each image-text pair.

[0054] (9) Similarity-based ranking score: Sort the retrieval objects according to the similarity score.

[0055] (10) Evaluate network performance: Calculate R@1, R@5, R@10 on the Flickr30K dataset, and evaluate the network performance on this dataset through these three indicators.

[0056] The Transformer-based cross-modal image-text retrieval method is an important basic research in the field of artificial intelligence, which plays an important role in image-text retrieval, image-text matching and other applications. TERAN is one of the current mainstream algorithms for cross-modal image-text retrieval based on deep learning, which uses the most advanced Transformer encoder as the secondary encoder of image and text, models the fine-grained segment relationship, and can obtain rich visual-linguistic clues. However, this method uses too many stacked layers of Transformer, which has large parameters and a large model, resulting in slow convergence speed and easy introduction of noise affecting the retrieval effect. The present application carries out research and innovation in view of the above research status, and proposes a lightweight Transformer-based cross-modal image-text retrieval method, which has improved performance and speed, see Figure 3 、 Figure 4 、 Figure 5In other specific embodiments of the present application, the following steps are included:

[0057] (1) Prepare the data set: the present application uses the Flickr30K data set, which is composed of 31783 pictures collected from the Flickr website. Each picture is equipped with 5 sentences of human annotation. Among them, 1000 images are verified, 1000 are used for testing, and the rest are used for training.

[0058] (2) Data preprocessing: in order to prevent gradient disappearance, gradient explosion and overfitting phenomenon in the training process, the data is preprocessed: for each image, the mean and standard deviation of each channel are calculated, and then each data in each channel is subtracted from the mean and divided by the variance to obtain the normalized result.

[0059] (3) Image feature extraction: using Faster R-CNN: target extraction network to extract 2048-dimensional region-level image features and region position coordinate information; and the image features are fused with the position coordinates and area size information to obtain the fused image features;

[0060] (4) Text feature extraction: using the 6-layer hidden layer parameters of the fine-tuned "bert-base-uncased" BERT model to obtain 768-dimensional word-level features of the text;

[0061] (5) Image encoding network structure: reference Figure 4 The 2048-dimensional features of n regional blocks of an image and the regional auxiliary information are spliced and then restored to 2048-dimensional through a fully connected layer. Then pass through two Transformer encoding layers, and use a fully connected layer in the middle of the Transformer encoding layer to adjust the dimension, the purpose is to unify the dimension of image and text to 1024-dimensional, and output the image fine-grained features containing rich information;

[0062] (6) Text encoding network structure: reference Figure 5 The 768-dimensional features of m words of a text are input into two lightweight Transformer encoding layers, and a fully connected layer is used in the middle of the Transformer encoding layer to adjust the dimension, the purpose is to unify the dimension of image and text to 1024-dimensional, and output the text fine-grained features containing rich information. New position information is added to the lightweight Transformer encoder to help the model converge faster. The new position information can make the connection between the front and rear words closer. The specific operation is to exchange a part of the continuous codes of the front and rear positions. And in order to reduce network parameters and speed up convergence, compared with the existing Transformer encoder, one layer of residual is removed, and the forward propagation and normalization are removed.

[0063] (7) Modeling similarity measure function: use dot product similarity to measure the similarity of the fine-grained features of the image and text obtained in step (5) and step (6), generate region-word similarity matrix. Then find the maximum value of the region similarity corresponding to each word as the similarity of the word and the image, and then sum up the similarity of all words and images to obtain the similarity of the picture and the text.

[0064] (8) Training network: first we mine the most difficult negative samples according to the similarity matrix of the image and the text obtained in step (7), and then calculate the similarity loss through the triple loss based on the most difficult negative samples.

[0065] (9) Predict matching probability: load the weights obtained in step (8) into the network to obtain the similarity probability score of each input image-text pair. This branch reflects the similarity of the input image-text pair, and the greater the score, the more similar the input image-text pair.

[0066] (10) Evaluate network performance: sort the results of the Flickr30K dataset test set based on the similarity score, and evaluate the performance of the network through R@1, R@5, R@10.

[0067] In further embodiments of the application, the specific operation of the image and text feature extraction step is as follows:

[0068] The features of a single picture obtained using Faster R-CNN can be represented as:

[0069] I={r cls ,r1,...,r n}

[0070] Where I represents the picture, r represents the region, and n represents the nth region.

[0071] The coordinate information can be represented as

[0072] B={x min ,y min ,x max ,y max}

[0073] Where B represents auxiliary information, x min , y min , x max , y max represent the left lower horizontal coordinate, the left lower vertical coordinate, the right upper horizontal coordinate, and the right upper vertical coordinate, respectively.

[0074] After concatenating the region position and shape features with the visual features, the fusion process can be represented as:

[0075] Area=(ymax - y min )*(x max - x min )

[0076] r1 = cat(r1, B, Area)

[0077] R1 = W H * r1 + b H

[0078] where Area represents the area of the region, cat() represents concatenation, W H , b H represent the weights and biases of the fully connected layer respectively, and R1 is the newly obtained region feature of the aggregated visual and position information.

[0079] The word features required by the model are extracted by a pre-trained "bert-base-uncased" model. We encode the sentence into tokens (tokens) and input it into BERT, and the output is the word feature representation. Only the parameters of the 6-layer hidden layer are used, which is fine-tuned through experiments.

[0080] The sentence feature can be represented as:

[0081] T = {w cls , w1,..., w m , w esk , w sep}

[0082] where T represents a sentence, w represents a word, and m represents a total of m words.

[0083] In a further embodiment of the present application, the image and text encoding part of the backbone network is as follows:

[0084] The backbone network is stacked by a Transformer, through a self-attention mechanism, each output segment can pay attention to all input segments, and the distance between each segment is only one.

[0085] Reference Figure 4 , image Tansformer encoder:

[0086] After normalizing the output after the multi-head attention module, the output

[0087] O j = LayerNorm(I j + (MultiHead(I j ))

[0088] where O j = [o1,..., on ]∈R n*d .

[0089] Then, the position feed-forward network and normalization are applied, and the output of each region image is

[0090] z k = LayerNorm(o k + FFN(o k )), k = 1,..., n

[0091] Referring to Figure 5 , the lightweight Transformer encoder of the text:

[0092] The multi-head attention part and the forward propagation and normalization part are similar to the image Transformer encoder, and the new position information is additionally added to enable the connection between the front and rear words to be more close:

[0093]

[0094] And in order to reduce the network parameters and accelerate the convergence, the original Transformer encoder is removed by one layer of residual, forward propagation and normalization.

[0095] In a further embodiment of the present application, the specific way of similarity measurement is as follows:

[0096] For the i-th sentence and the j-th picture, the dot product similarity is used to measure their similarity, and the region-word similarity matrix is generated as:

[0097]

[0098] Wherein,

[0099] The maximum pooling is used first, and then the sum is obtained, to obtain the similarity of the picture and the sentence, and the specific operation is:

[0100] For the i-th sentence and the j-th picture, the similarity is calculated as:

[0101]

[0102] Sim j-i It is found that the maximum value of each word corresponding to the region similarity is the similarity of the word and the picture, and then the similarity of all words and the picture is summed to obtain the similarity of the picture and the text.

[0103] In a further embodiment of the present application, the loss function is as follows:

[0104] The model employs a triplet loss based on difficult samples. Triplet loss helps neural networks focus more on detailed information and distinguish inputs with small differences, thus improving performance in image and text retrieval tasks. However, the data distribution selected for triplet loss may not be uniform, potentially causing instability during training. The triplet loss based on difficult samples, which selects only the best-matching unmatched pairs, effectively mitigates this problem.

[0105] The triplet loss based on hard samples can be expressed as:

[0106]

[0107] j h =argmax j≠i S(j,I)

[0108]

[0109] Where B is the batch size, in image and text retrieval... and It is the similarity score of the most difficult negative sample (i.e., the best-matching least-matching pair), Sim 1ii Sim 1jj is the similarity of positive samples (paired data), and m is the margin. For all i ≠ j, this loss forces the true image and text pairs to have similarity. 1ii The similarity.

[0110] Higher than negative sampling Sim 1ij or Sim 1ji The similarity is at least m.

[0111] The experiment used the Adam optimizer with a learning rate of 0.00001, and employed linear warm-up optimization of the learning rate.

[0112] Simulation experiments regarding the technical solution of this invention:

[0113] Simulation experimental conditions: The hardware platform for the simulation experiment of this invention is: the model is implemented using an NVIDIA GeForce GTX 2080Ti GPU under the PyTorch framework; the simulation software platform is: PyCharm software (Python 3.7).

[0114] Simulation Experiment Content and Result Analysis: The simulation experiment of this invention is specifically divided into three simulation experiments. This invention uses the publicly available Flickr30K dataset. This dataset consists of 31,783 images collected from the Flickr website. Each image is accompanied by 5 sentences of human annotation. Of these, 1,000 images are used for validation, 1,000 for testing, and the rest are used for training.

[0115] Simulation Experiment 1: Performance comparison of the present application and prior art on the Flickr30K benchmark dataset. The present application and the existing mainstream technology are compared, the same training set samples are used for training, and the same test set samples are used for evaluation of various methods. The experimental results of other models are taken from the corresponding papers or other people's experimental results. The evaluation results are shown in Table 1, and the data in the table is the recall value of each method on the test set. Among them, R@1 represents the proportion of paired sample queries ranked in the first 1 in 1000 retrieval tests, and R@5 and R@10 are the same. The larger R@K, the higher the network accuracy.

[0116] Table 1 Comparison of indicators of various models on the Flickr30k dataset

[0117]

[0118]

[0119] Among them, Stu represents the results of the present application, and the bold indicates the best result. As can be seen from Table 1, the results of the model of the present application are better than those of other algorithm models. Although the R@1 of Image-to-Text Retrieval is not the best, the difference is very small. Table 1 shows the advanced performance achieved by the present application.

[0120] Simulation Experiment 2: In order to verify the effectiveness of the prior information acceleration strategy, the present application does an ablation experiment. Stu(6-2) uses the same network structure as the TERAN model and uses the prior information acceleration strategy to optimize the text encoder. The experimental results are as follows:

[0121] Table 2 Influence of modifying the text Transformer on the indicators

[0122]

[0123] From Table 2 and Figure 6 It can be seen that (wherein subgraph (a) is Text-to-Image Retrieval, and subgraph (b) is Image-to-Text Retrieval). This strategy has little effect on the model indicators while accelerating the convergence speed of the model. The reason may be that the strong association information of the words before and after added by the present application can help the network to train text features with rich information in the early stage, so the convergence speed is fast.

[0124] Simulation experiment 3: In practical application, the accuracy and speed of the model are particularly important, and it is not advisable to only pursue one side effect to be optimal and the other side effect to be poor. In order to make the model perform relatively best in recall rate, convergence speed and calculation rate, the network pruning strategy is adopted, the TERAN image encoder structure is pruned and optimized, and the ablation experiment is carried out, and the experimental results are as follows:

[0125] Table 3 Model size comparison

[0126]

[0127] From table 3 and Figure 7 It can be seen from the experimental results (wherein, subgraph (c) is Text-to-Image Retrieval, and subgraph (d) is Image-to-Text Retrieval.), the TERAN model is nearly twice as large as the Stu model, and under the condition of using 2080Ti and maximum utilization of video memory, the TERAN model needs to spend 27 hours to achieve optimal effect after training for 30 rounds, while for the Stu model, under the same conditions, training only takes 7 hours. The results show that after pruning and optimizing the image encoder part of the TERAN model, the Stu greatly reduces the parameter quantity and calculation quantity, so that the model convergence speed and training speed are accelerated.

[0128] Table 4 TERAN and Stu result comparison

[0129]

[0130] As can be seen from table 4, the performance of Stu is better than that of TERAN. It shows that Stu not only lightens the model, but also avoids the problem that the introduction of too much irrelevant information feature noise by complex network structure leads to the loss of original feature distribution characteristics of data. Stu not only converges fast, but also has better model effect.

[0131] In summary, the cross-modal retrieval scheme based on Transformer proposed in the application achieves advanced speed and performance in the cross-modal retrieval problem of natural optical images and English texts.

[0132] It should be noted that the following embodiments can be combined and / or extended, replaced by logic in the protection scope defined in the claims of the application, for example, disclosed technical principles, disclosed technical features or implicitly disclosed technical features.

[0133] Embodiment 1

[0134] A cross-modal retrieval method based on Transformer, comprising the following steps:

[0135] Step one, using Transformer as the backbone network, the extracted image and text features are encoded and processed to model the fine-grained relationship, and the features with region-language clues are obtained; the image part uses the encoder part of the pure Transformer, the text part uses the pruned Transformer encoder with added prior information, through encoding cross, increasing the connection between the front and rear position words, accelerating the network to capture and learn the relationship between words, and optimizing the Transformer structure, removing a layer of residual, forward propagation layer and normalization layer;

[0136] Step two, modeling the similarity measure function to calculate the similarity of the picture and the text;

[0137] Step three, training the network based on Transformer in step one based on the similarity calculated in step two;

[0138] Step four, loading the weights trained in step three into the network to obtain the similarity probability score of each image-text pair, and predicting the matching probability based on the similarity probability score;

[0139] Step five, according to the similarity score obtained in step four, sorting the retrieval objects to obtain the target retrieval object.

[0140] Embodiment 2

[0141] Based on embodiment 1, in step two, the modeling similarity measure function calculates the similarity of the picture and the text, including the sub-steps: using the dot product similarity of the fine-grained features of the image and text obtained in step one to measure their similarity, generating a region-word similarity matrix, then finding the maximum value of the region similarity corresponding to each word as the similarity of the word and the image, and then summing up the similarity of all words and images to obtain the similarity of the picture and the text.

[0142] Embodiment 3

[0143] Based on embodiment 1, in step three, the network based on Transformer in step one is trained based on the similarity calculated in step two, including the sub-steps: the model uses a triplet loss based on difficult samples, and based on the similarity score obtained in step two, the similarity scores of a matching pair and a difficult mismatching pair are obtained by mining, and then the network is optimized by the triplet loss based on difficult samples.

[0144] Embodiment 4

[0145] Based on any of embodiments 1-3, before step one, it further includes the step:

[0146] S1, preparing a data set;

[0147] S2, data preprocessing;

[0148] S3, image feature extraction;

[0149] S4, text feature extraction.

[0150] Embodiment 5

[0151] On the basis of embodiment 4, in step S1, the preparing a data set specifically uses Faster R-CNN as a visual feature extractor to extract region features at a fine-grained level, and 2048-dimensional region features and region position coordinate information are obtained through the extraction of Faster R-CNN.

[0152] Embodiment 6

[0153] On the basis of embodiment 4, in step S2, the data preprocessing includes a sub-step of: for each image, calculating the mean and standard deviation of each channel, and then subtracting the mean from each data in each channel and dividing by the variance to obtain the normalized result.

[0154] Embodiment 7

[0155] On the basis of embodiment 4, in step S3, the image feature extraction specifically uses a Faster R-CNN target extraction network to extract 2048-dimensional region-level image features and region position coordinate information; and fuses the image features with the position coordinates, area size information to obtain fused image features.

[0156] Embodiment 8

[0157] On the basis of embodiment 4, in step S4, the text feature extraction includes a sub-step of: word features required by the model are extracted through a pre-trained “bert-base-uncased” model, and a sentence is encoded into tokens, which are input into BERT, and the output is the word feature representation.

[0158] Embodiment 9

[0159] On the basis of embodiment 4, in step S1, the data set includes a Flickr30K data set.

[0160] Embodiment 10

[0161] A cross-modal retrieval device based on a Transformer, the computer device comprising a processor and a memory, the memory storing a computer program, when the computer program is loaded by the processor and executes the method as described in any one of embodiments 1-9.

[0162] The units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. The units described can also be located in a single processor. In some cases, the names of the units do not limit the units themselves.

[0163] According to an aspect of the embodiments of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method provided in the various optional implementation manners described above.

[0164] As another aspect, the embodiments of the present application also provide a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the method described in the above embodiments.

[0165] The parts of the present application not involved are the same as or can be implemented by the prior art.

[0166] The technical solution described above is only one embodiment of the present application. Based on the application method and principle disclosed in the present application, those skilled in the art can easily make various types of improvements or modifications, and the method described in the above embodiment is not limited to the method described in the above embodiment. Therefore, the above description is only preferred, and is not limited in meaning.

[0167] In addition to the above examples, those skilled in the art can obtain other embodiments by being inspired by the above disclosure or by making changes using knowledge or technology in related fields. The features of each embodiment can be interchanged or replaced. The changes and variations made by those skilled in the art do not deviate from the spirit and scope of the present application, and should be within the protection scope of the claims of the present application.

Claims

1. A cross-modal retrieval method based on Transformer, characterized in that, Includes the following steps: Step 1: Use Transformer as the backbone network to encode the extracted image and text features, and model the fine-grained relationship between the encoded image and text features to obtain features with image region-text language cues. The image part uses a pure Transformer encoder, and the text part uses a pruned Transformer encoder with added prior information. By encoding crossover, the connection between words in the preceding and following positions is increased, which accelerates the network's learning of the relationship between words. The Transformer structure is also optimized by removing one residual layer, a forward propagation layer, and a normalization layer. Step two: Model a similarity metric function to calculate the similarity between the image and the text; Step 3: Train the network built in Step 1 based on the similarity calculated in Step 2; Step 4: Load the weights trained in Step 3 into the network to obtain the similarity probability score for each image-text pair, and predict the matching probability based on the similarity probability score. Step 5: Based on the similarity scores obtained in Step 4, sort the search objects to obtain the target search objects.

2. The cross-modal retrieval method based on Transformer according to claim 1, characterized in that, In step two, the modeling similarity metric function calculates the similarity between the image and the text, including the following sub-steps: using dot product similarity to measure the similarity between the fine-grained features of the image and the text obtained in step one, generating a region-word similarity matrix; then finding the maximum value of the region similarity corresponding to each word as the similarity between that word and the image; and finally summing the similarities between all words and the image to obtain the similarity between the image and the text.

3. The cross-modal retrieval method based on Transformer according to claim 1, characterized in that, In step three, the training of the network built in step one based on the Transformer based on the similarity calculated in step two includes the following sub-steps: the model adopts the triple loss based on hard samples, and based on the similarity score obtained in step two, the similarity score of a matching pair and a hard mismatch pair is obtained by mining, and then the network is optimized by the triple loss based on hard samples.

4. The cross-modal retrieval method based on Transformer according to any one of claims 1 to 3, characterized in that, Before step one, the following steps are also included: S1, Prepare the dataset; S2, Data Preprocessing; S3, Image Feature Extraction; S4, Text Feature Extraction.

5. The cross-modal retrieval method based on Transformer according to claim 4, characterized in that, In step S1, the prepared dataset specifically uses Faster R-CNN as a visual feature extractor to extract fine-grained region features. Through the extraction by Faster R-CNN, 2048-dimensional region features and region location coordinate information are obtained.

6. The cross-modal retrieval method based on Transformer according to claim 4, characterized in that, In step S2, the data preprocessing includes the following sub-steps: for each image, calculate the mean and standard deviation of each channel, then subtract the mean from each data point in each channel, and then divide by the variance to obtain the normalized result.

7. The cross-modal retrieval method based on Transformer according to claim 4, characterized in that, In step S3, the image feature extraction specifically uses the Faster R-CNN object extraction network to extract 2048-dimensional region-level image features and the location coordinate information of the region; and the image features are fused with the location coordinates and area size information to obtain fused image features.

8. The cross-modal retrieval method based on Transformer according to claim 4, characterized in that, In step S4, the text feature extraction includes the following sub-steps: the word features required by the model are extracted through a pre-trained "bert-base-uncased" model, and the sentences are encoded into tokens, which are then input into BERT. The output is the word feature representation.

9. The cross-modal retrieval method based on Transformer according to claim 4, characterized in that, In step S1, the dataset includes the Flickr30K dataset.

10. A cross-modal retrieval device based on Transformer, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when loaded by the processor, executes the method as described in any one of claims 1 to 9.

Images