News scene multi-level image-text retrieval method based on mask guidance information fusion

Abstract

The invention relates to the technical field of multi-modal information retrieval, and particularly discloses a news scene multi-level image-text retrieval method based on mask guidance information fusion. According to the method, a mask-based cross attention module is introduced, and information of two modalities of an image and a text is guided to be better fused by optimizing mask language model loss; by selecting the text and image features with the highest correlation degree, performing finer-grained and efficient alignment between the two modals; according to the method, data characteristics of news data, namely weak alignment relationships between pictures in news articles and all text segments, are fully considered, and parameters are updated by fitting real distribution of data and model output distribution with reference to the idea of comparative learning, so that hierarchical association relationships between news images and texts are better learned.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal information retrieval, and in particular to a news scene multi-level image and text retrieval method based on mask-guided information fusion. Background Art

[0002] With the continuous development and popularization of the Internet and mobile devices, the Internet has become one of the main ways for the public to obtain news. These news are presented in a variety of modal forms, including text, images, videos, etc., which complement each other and serve the same theme. Multimodal data can convey news events to readers more intuitively, but it also poses new challenges to today's retrieval technology, such as cross-modal search operations such as searching for images with text and searching for text with images.

[0003] Thanks to the rapid development of the Transformer model and graphics card computing power, the field of multimodal image and text retrieval has made great progress in recent years. However, most existing methods do not consider the hierarchical relationship between images and text in news articles. An article contains multiple images and text paragraphs. If only the correspondence between images and image titles is considered, the weak alignment relationship between the images and other text segments will be ignored. If all correspondences between images and text segments are considered, too much irrelevant noise will be introduced. In addition, most existing methods directly align the global features extracted from data of two different modalities, ignoring the distribution differences of image and text features in different modalities and the need for more fine-grained alignment.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The purpose of the present invention is to overcome the problems existing in the above-mentioned prior art and to provide a news scene multi-level image and text retrieval method based on mask-guided information fusion.

[0006] In order to achieve the above object, the present invention provides a news scene multi-level image and text retrieval method based on mask-guided information fusion, comprising the following steps:

[0007] S1. Obtain the data set required for model training and process it into a specific format according to the distance between the image, title and text;

[0008] S2. Obtain a batch of training data, perform mask processing on the text, input the image and the masked text into the image encoder and the text encoder respectively, and obtain the feature representation of the corresponding image and text respectively;

[0009] S3, input the image and text features into the cross-modal fusion layer, obtain the text feature representation after fusion of visual information, and calculate the mask loss;

[0010] S4, calculating the similarity between the image and the text through fine-grained alignment operations between image features and text features;

[0011] S5. Calculate the matching relationship between the image and text output by the model within the current batch of data.

[0012] The KL divergence between the true distribution of the data is used as the similarity distribution loss;

[0013] S6. Based on the total loss, back-propagate and update the model parameters until the model converges to the best result.

[0014] Preferably, the data set is a data set that has undergone a special processing operation, wherein the special processing operation specifically

[0015] To: mark the distance between the image and the text segment according to their relative position in the news article, as the image and

[0016] Measuring weakly consistent semantic matching between texts.

[0017] Preferably, the image encoder is a pre-trained ViT model. Given an input image I∈R H×W×C , and get the corresponding image feature representation The text encoder is a pre-trained BERT model, with input

[0018] Corresponding text sequence, get text feature representation Among them, n1 and n2 represent the number of encoded image features and the length of the text sequence respectively.

[0019] Preferably, the cross-modal fusion layer performs cross-modal fusion on image and text features.

[0020] The specific steps of state fusion are: seek cross attention between image and text features, and predict the masked

[0021] The masked words are manipulated to guide the fusion of information between the two different modalities of image and text, and the image information is obtained.

[0022] Fusion text features of information The specific operation is expressed as:

[0023]

[0024] Among them, CA(·) is the cross attention process:

[0025]

[0026] LN(·) represents the layer normalization operation, is the encoded text feature, and The encoded image

[0027] Image features, d is the dimension of the encoded text features.

[0028] Preferably, the structure of the cross-modal fusion layer includes: a multi-head cross attention layer for image and

[0029] The fusion of the two modal information of the text; and 4 repeated transformer blocks (transformer block), with

[0030] To fully extract the global feature information.

[0031] Preferably, the specific steps of calculating the mask loss are: for each masked position Map it to the dimensions of a predetermined vocabulary using a multi-layer perceptron After the softmax operation, the probability distribution of the original word in the vocabulary is used to calculate the cross entropy with the true representation of the masked word in the vocabulary:

[0032] in, is the set of all masked words, For the entire vocabulary, represents the size of the vocabulary, m i is the feature representation of the masked word predicted after encoding and modality fusion, y i is its true embedding e in the vocabulary.

[0033] Preferably, the specific steps of calculating the similarity between the image and the text are: for each text feature, the similarity between it and each image feature is measured by the inner product between vectors; the similarity between the text feature and all image features is calculated, and the maximum value is selected as the most relevant description of the text feature to the picture, that is, the similarity between the text feature and the entire image; the above steps are repeated for all text features, and finally the average value is taken as the similarity between the image and the text.

[0034] Preferably, the specific steps of the similarity distribution loss are: in a small batch of data, each image and all texts in the current batch are combined into a set of image-text representation pairs; the similarities between all image-text pairs are used as the data distribution of the model prediction output, and the true matching probability between the image-text pairs can be measured by the distance between the image and the text. The farther the distance, the lower the matching probability is set, which is used as the true distribution between the data; finally, the KL divergence between the two distributions is calculated as the similarity distribution loss.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention introduces a mask-based cross-attention module, which guides the better fusion of information of the two modalities of image and text by optimizing the mask language model loss; by selecting the most relevant text and image features, a more fine-grained and efficient alignment is performed between the two modalities; the data characteristics of news data, namely the weak alignment relationship between pictures and all text segments in news articles, are fully considered, and then the idea of ​​contrastive learning is borrowed to update parameters by fitting the real distribution of data and the output distribution of the model, so as to better learn the hierarchical association relationship between news pictures and texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of a method for training a picture-text retrieval model in a news scenario provided by an example of the present invention;

[0038] Figure 2 A schematic diagram of preprocessing training data provided by an example of the present invention;

[0039] Figure 3 A schematic diagram of the structure of the image-text retrieval model provided for the example of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Embodiment 1

[0042] The embodiment of the present invention provides a news scene multi-level image and text retrieval method based on mask-guided information fusion, such as Figure 1 As shown, the specific steps are as follows:

[0043] Step S1, obtaining the data set required for model training, and processing it into a specific format according to the distance between the image, title and text;

[0044] In the embodiment of the present invention, in order to fully utilize the many-to-many matching relationship between images and texts in news articles and the weak alignment relationship between images and full text semantics, additional processing is performed on the data.

[0045] Step S2, obtaining a batch of training data, performing mask processing on the text, inputting the image and the masked text into the image encoder and the text encoder respectively, and obtaining feature representations of the corresponding image and text respectively;

[0046] In the embodiment of the present invention, the training data is obtained after processing in step S1, and the masked words in the text sequence need to be recorded. The image encoder used is the pre-trained ViT model. Given an input image I∈R H×W×C , and get the corresponding image feature representation The text encoder is a pre-trained BERT model. The corresponding text sequence is input to obtain the text feature representation. Among them, n1 and n2 represent the number of encoded image features and the length of the text sequence respectively.

[0047] Step S3: input the image and text features into the cross-modal fusion layer to obtain the text feature representation after fusion of visual information, and calculate the mask loss;

[0048] In the embodiment of the present invention, cross attention is sought between image and text features, and the fusion of information between two different modalities of image and text is guided by predicting the words masked by the mask operation, so as to obtain a fused text feature containing image information.

[0049] Step S4, calculating the similarity between the image and the text through a fine-grained alignment operation between the image features and the text features;

[0050] In the embodiment of the present invention, by focusing on the image features that are most relevant to each text feature, the image and text are aligned efficiently and fine-grained.

[0051] Step S5: Calculate the KL divergence between the matching relationship between the image and text output by the model and the real distribution of the data in the current batch of data as the similarity distribution loss;

[0052] In the embodiment of the present invention, the matching degree distribution between the image and the text is calculated by the image-text similarity obtained in step 4, and the real distribution of the data is calculated by the distance between the image and the text.

[0053] Step S6: According to the total loss, back-propagate and update the model parameters until the model converges to the best result.

[0054] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with specific implementation steps below.

[0055] In step S1, special processing operations are performed on the data set such as Figure 2As shown in the figure, taking a news article on a web page as an example, in the Cartesian product of a set of pictures and a set of text paragraphs (including picture titles) in an article, each sample pair includes a picture and a text paragraph, and their relative positions in the original article are used as the distance measure between them. Specifically, for any picture, the picture title is the most accurate description of the picture, so it is closest to the picture and is recorded as d1. The other text paragraphs are recorded as d2, d3, etc. as the distance from the picture gets farther and farther. In particular, for the centered picture, such as Figure 2 As shown, its distance to the previous text is also relatively close, so its distance to the previous text is symmetrical with the distance to the following text. In this embodiment, d1, d2... are {1, 2, 3, ...}. Figure 2 Taking the centered picture as an example, there are four lines of processed data, which are: 1. The file directory where the picture is located: "…… / Figure 2 .jpg"; 2. Text content: "News article paragraph 1..."; 3. The unique number of each image in the dataset: "1"; 4. The distance between the image and the text: "2".

[0056] Most existing image-text retrieval methods directly align images and image captions as positive samples, but the content of the caption may be missing, and the image may express additional context-based information. Therefore, consider aligning the image with all text paragraphs in the article, but this will introduce new problems. Images often only reflect key targets or events. Directly using the original text for training will bring a lot of redundancy and introduce too much irrelevant noise. Therefore, the relative position of the preprocessed image and text in step S1 is used as a correlation measure between the image and text during the alignment below.

[0057] In step S3, the text feature representation after fusion of visual information is obtained according to the image and text features. The specific operation is: As a query Image features As a query Sum After input into the cross-modal fusion layer, the final text feature representation is output.

[0058] The structure of the cross-modal fusion layer is as follows: Figure 3 The left half shows a multi-head cross attention layer and a 4-layer transformer block. After the cross attention layer, self-attention layer and feedforward neural network layer, the masked text features that fully integrate the visual information are obtained. The specific operation is expressed as:

[0059]

[0060] Among them, CA(·) is the cross attention process:

[0061]

[0062] LN(·) represents the layer normalization operation, is the encoded text feature, and is the encoded image feature, and d is the dimension of the encoded text feature.

[0063] Then, the MLM (masked language model loss) is calculated. Specifically, for each masked position Map it to the dimensions of a predetermined vocabulary using a multi-layer perceptron After the softmax operation, the probability distribution of the original token in the vocabulary is used to calculate the cross entropy with the real representation of the masked token in the vocabulary:

[0064]

[0065] in, is the set of all masked words, For the entire vocabulary, represents the size of the vocabulary, m i is the feature representation of the masked word predicted after encoding and modality fusion, y i is its true embedding in the vocabulary.

[0066] In this embodiment, the function of step S3 is as follows: the image and text features come from different modalities, and their features are distributed in different high-dimensional spaces. In order to make these two parts of features fully interact, cross-attention is sought, so that the model can establish connections between different modalities, thereby better understanding and integrating multimodal information. At the same time, studies have shown that in language models, MLM optimizes two goals: (1) the alignment of the embedded representation of masked words in the text with the meaning of the text context; (2) the consistency of the feature representation of the masked words with the embedding space where the vocabulary is located. Therefore, by using the text features after the masking operation as Image features as and Cross attention is sought and the MLM loss is finally optimized. On the one hand, the mask is used as an anchor point so that the model can learn contextual information related to the image and text at the same time. On the other hand, the fusion of two different modal information of image and text is realized.

[0067] In step 4, the similarity between the image and the text is calculated as follows: Figure 3As shown in the upper part, for any one of the finally obtained masked text features fused with visual information where \(0\leq k < n_2\), its similarity with the entire image is The similarity between the picture and the text is where \(i\) and \(j\) respectively represent the indices of the picture and the text sequence in a batch of data

[0068] In this embodiment, the function of step S4 is mainly to calculate the fine-grained similarity between the image and the text. The information contained in the picture and the corresponding text description is often not equivalent. It is difficult to summarize all the image information with simple text, and the information represented by the image may be redundant compared to the text description. Therefore, when calculating the similarity between the image and the text, the following approach is adopted: for each text feature, the similarity between it and each image feature is measured by the inner product between vectors, and the maximum value selected from them can be regarded as the correspondence of the text feature in a certain part of the picture, as the similarity between the text feature and the entire image. Repeat the above steps for all text features, and finally take the average value as the similarity \(s\) between the image and the text i,j .

[0069] In step S5, the process of calculating the similarity distribution loss is as Figure 3 shown in the lower right part. In a small batch of data, for each image representation \(f\) i v , and all text features in the current batch can be combined into a set of picture-text representation pairs where \(i\) and \(j\) respectively represent the index positions of the image and the text in the current batch, and \(x\) i,j is the similarity probability between the \(i\)-th picture and the \(j\)-th text, and \(N\) is the size of the entire batch. Specifically, the calculation steps of \(x\) i,j are as follows: during the data preprocessing process, the distance between each picture in the news article and other text segments can be obtained; the result obtained by passing the distance through a scaling function is used as the similarity probability \(x\) between them i,j . In the best example of the present invention, the adopted scaling function is \(f(d)=e\) -d+1 . After the above process, the true distribution of the data in the current batch can be obtained At the same time, the output distribution of the data after passing through the model can be obtained where \(s\) i,j is the similarity between the picture and the text obtained in step 4, and \(\tau\) is the temperature parameter for adjusting the probability distribution. Finally, by calculating the KL divergence between the model output distribution and the data true distribution, it is used as the loss for fitting the data distribution:

[0070]

[0071] Similarly, the distribution loss of text and all images in a batch can be obtained symmetrically The final total similarity distribution loss

[0072] In this embodiment, step S5 mainly performs contrastive learning between image and text features in the same batch. Contrastive Learning is a self-supervised learning method that aims to extract useful feature representations by learning the similarities and differences between data samples. The core idea of ​​contrastive learning is to learn a good feature representation space by maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs. In this example, the input image and the corresponding text description are semantically similar positive sample pairs, and the image and other texts in the same batch are negative sample pairs. The loss function is optimized to draw the positive sample pairs closer while distinguishing the negative sample pairs. In particular, due to the special nature of the news articles mentioned above, the images may appear multiple times in the same batch and correspond to different text descriptions. If they are directly regarded as the same positive sample pairs without distinction, the weakly correlated data will interfere with the optimization of the model. Therefore, the present invention proposes an operation for data preprocessing, saving the distance between the image and the text as a measure of the weak correlation between them, and on this basis, the distance between the data distribution is used as a loss function to train the model.

[0073] In step 6, the final total loss is:

[0074]

[0075] A more detailed description of the training example is provided below.

[0076] The data used for model training in the present invention example comes from the open source GoodNews dataset, which includes 466,000 pictures with article titles, picture descriptions, and text paragraphs. In the present invention example, 42,400 pictures are selected for training, 18,000 pictures are used for verification, and 23,000 pictures are used for testing. R@K and rsum, which are commonly used in retrieval tasks, are used as evaluation indicators.

[0077] In step S1, data is sampled from the processed data set and input into the model after processing. Specifically, the RGB (red, green and blue three-channel) color natural images are uniformly adjusted to a resolution of 224×224, and data enhancement processing is performed on them, including random cropping and scaling, random color jittering, random horizontal flipping and normalization, so that the model can learn better feature representation. The maximum length of the input text sequence is limited to 77 words, and the excess part is directly discarded, and the text is segmented using BPE encoding and a vocabulary of size 49408. At the same time, the following masking operation is performed on each text sequence: given a text sequence T, the words in it are randomly masked with a probability of 15%. The specific masking operation is as follows: 80% probability of replacing the original word with the [MASK] tag, 10% probability of replacing it with a random word, and 10% probability of keeping it unchanged.

[0078] In the example of the present invention, the image encoder selects the pre-trained ViT-B / 16, where B represents the model size of Base, and 16 represents the patch size of the input image of 16×16. The model has 12 layers, a width (hidden size) of 768, and 12 attention heads. The text encoder part selects the pre-trained bert-base-uncased, which also has 12 layers, a hidden layer dimension of 768, and 12 attention heads. The cross-modal fusion layer contains a cross attention layer and a 4-layer Transformerblock, with a hidden layer dimension of 768 and 12 attention heads.

[0079] In this example, the model uses the Adam optimizer and the learning rate is initialized to 1×10 -5 , using cosine learning rate decay, the temperature parameter τ is set to 0.02, and a total of 30 rounds of training are performed. This series of hyperparameters can achieve good performance on the above dataset.

[0080] Embodiment 2

[0081] An embodiment of the present invention provides a picture and text retrieval system in a news scenario, including an image encoder, a text encoder and a cross-modal fusion layer obtained after training as above.

[0082] Related explanation:

[0083] Cross-Entropy: In information theory, the cross-entropy of two probability distributions p and q based on the same event measure refers to the average number of bits required to uniquely identify an event in a set of events when encoding based on an unnatural (relative to the true distribution p) probability distribution q. Given two probability distributions p and q, the cross-entropy of p relative to q is defined as: H(p,q) = E p [-log q].

[0084] KL (Kullback-Leibler) divergence: KL divergence is a measure of the asymmetry of the difference between two probability distributions P and Q. KL divergence is used to measure the average number of extra bits required to encode samples from a distribution based on Q to those from a distribution based on P. Typically, P represents the true distribution of the data, and Q represents the theoretical distribution of the data, an estimated model distribution, or an approximate distribution of P.

[0085] Softmax function: In mathematics, especially probability theory and related fields, the softmax function, or normalized exponential function, is a generalization of the logic function. It can compress a K-dimensional vector z containing any real number into another K-dimensional real vector σ(z), so that each element ranges between (0, 1) and the sum of all elements is 1. This function is often used to represent the distribution of probability in multi-classification problems.

[0086] Multilayer Perceptron (MLP): It is a feed-forward artificial neural network that maps a set of input vectors to a set of output vectors.

[0087] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A news scene multi-level image and text retrieval method based on mask-guided information fusion, characterized in that: The following steps are involved: S1. Obtain the data set required for model training and process it into a specific format according to the distance between the image, title and text; S2. Obtain a batch of training data, perform mask processing on the text, input the image and the masked text into the image encoder and the text encoder respectively, and obtain the feature representation of the corresponding image and text respectively; S3, input the image and text features into the cross-modal fusion layer, obtain the text feature representation after fusion of visual information, and calculate the mask loss; S4, calculating the similarity between the image and the text through fine-grained alignment operations between image features and text features; S5. Calculate the KL divergence between the matching relationship between the image and text output by the model and the real distribution of the data in the current batch of data as the similarity distribution loss. S6. Based on the total loss, back-propagate and update the model parameters until the model converges to the best result.

2. The news scene multi-level image and text retrieval method based on mask-guided information fusion according to claim 1 is characterized in that: The data set is a data set that has undergone special processing operations, and the special processing operations are specifically: marking the distance between the image and the text segment according to their relative positions in the news article as a measure of weak consistency semantic matching between the image and the text.

3. The news scene multi-level image and text retrieval method based on mask-guided information fusion according to claim 1 is characterized in that: The image encoder is a pre-trained ViT model. Given an input image I∈R H×W×C , and get the corresponding image feature representation The text encoder is a pre-trained BERT model, which inputs the corresponding text sequence to obtain the text feature representation Among them, n1 and n2 represent the number of encoded image features and the length of the text sequence respectively.

4. The news scene multi-level image and text retrieval method based on mask-guided information fusion according to claim 3 is characterized in that: The cross-modal fusion layer performs cross-modal fusion on image and text features. The specific steps of the cross-modal fusion are: seeking cross attention between image and text features, and guiding the fusion of information between two different modalities of image and text by predicting words masked by masking operations, and obtaining fused text features containing image information The specific operation is expressed as: Among them, CA(·) is the cross attention process: LN(·) represents the layer normalization operation, is the encoded text feature, and is the encoded image feature, and d is the dimension of the encoded text feature.

5. The news scene multi-level image and text retrieval method based on mask-guided information fusion according to claim 4 is characterized in that: The structure of the cross-modal fusion layer includes: a multi-head cross-attention layer for fusing the two modal information of image and text; and 4 repeated transformer block encoders for fully extracting global feature information.

6. The news scene multi-level image and text retrieval method based on mask-guided information fusion according to claim 5 is characterized in that: The specific steps of calculating the mask loss are: for each masked position Map it to the dimensions of a predetermined vocabulary using a multi-layer perceptron After the softmax operation, the probability distribution of the original word in the vocabulary is used to calculate the cross entropy with the true representation of the masked word in the vocabulary: in, is the set of all masked words, For the entire vocabulary, represents the size of the vocabulary, m i is the feature representation of the masked word predicted after encoding and modality fusion, y i is its true embedding in the vocabulary.

7. The news scene multi-level image and text retrieval method based on mask-guided information fusion according to claim 6 is characterized in that: The specific steps of calculating the similarity between the image and the text are as follows: for each text feature, the similarity between it and each image feature is measured by the inner product between vectors; the similarity between the text feature and all image features is calculated, and the maximum value is selected as the most relevant description of the text feature to the picture, that is, the similarity between the text feature and the entire image; the above steps are repeated for all text features, and finally the average value is taken as the similarity between the image and the text.

8. The news scene multi-level image and text retrieval method based on mask-guided information fusion according to claim 7 is characterized in that: The specific steps of the similarity distribution loss are as follows: in a small batch of data, each image and all texts in the current batch are combined into a set of image-text pairs; the similarities between all image-text pairs are used as the data distribution output by the model prediction, and the true matching probability between the image-text pairs can be measured by the distance between the image and the text. The farther the distance, the lower the matching probability is set, which is used as the true distribution between the data; finally, the KL divergence between the two distributions is calculated as the similarity distribution loss.

Images