A multimodal summarization method for images and texts based on two-level gated coding

Abstract

The present invention discloses a multimodal summary method of text and image based on secondary gated coding. In the multimodal coding stage, a multimodal hidden layer vector is obtained by a dual encoder, and a grid is divided into a single-mode text summary to provide grid-level visual features. The hidden layer information of the text is obtained through a recurrent neural network as the primary gated information; the secondary hidden layer vector is obtained by a gated recurrent neural network, and a more diversified summary is generated by a more fine-grained fusion and processing of text and image information; in the part where data is processed between sequences, the positive and negative context vectors of the text and image are obtained by a self-attention mechanism, and a filter is used to further perform noise reduction processing on the text and image from both positive and negative directions. The present invention can increase the number of text and image fusion channels and the compensation effect, and use the positive and negative attention mechanisms to eliminate the parts of the text and image that are not related to the summary, so as to generate a multimodal summary with a more fitting text and image.

Description

Technical Field

[0001] The invention relates to a multimodal summary method for images and texts, and in particular to a multimodal summary method for images and texts with two-level gated coding. Background Art

[0002] With the rapid growth of multimedia data on the Internet, the huge amount of information is constantly impacting our daily lives. In order to reduce information redundancy to improve readability, and thus solve the problem of how to extract key information from an article, text summarization technology has received widespread attention. Existing text summarization mainly encodes the entire article through a recurrent neural network, then extracts the word attention in the article in combination with a self-attention mechanism, and finally decodes the article summary through a decoder, which is called an encoder-decoder model. However, the encoder-decoder model has certain problems. When used, it usually generates unnatural summaries consisting of repeated phrases, which is particularly obvious for long text summary generation.

[0003] The output of current multimodal summarization technology is usually expressed in text form. Through a large number of experiments, it is found that multimodal output, that is, the output contains not only text summaries but also summaries of other modalities, can significantly improve users' satisfaction with the informativeness of summaries. However, in existing work, the correlation between summary text and summary images is not reflected, that is, there is no real alignment between image summaries and text summaries, which cannot alleviate the semantic gap between text and images, and it is impossible to select the most suitable image as information compensation through the text summarization process. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a multimodal summarization method for images and texts with a two-level gated coding and reverse attention mechanism that can increase the number of image-text fusion channels and the compensation effect.

[0005] Technical solution: The multimodal summarization method of the present invention comprises the following steps:

[0006] S10, obtaining a data set, ensuring that each piece of data includes a text and several pictures;

[0007] S20, using convolutional neural network to extract image feature vectors v1 to v of all j images respectively j , the long short-term memory recurrent neural network encoder is used to process the text to obtain the hidden layer h of the first-level text encoding i ; At the same time, the grid-level divided pictures are passed through the hidden layer vector h layer by layer i , with each image grid-level feature v1 to v j Combine and take the largest image grid;

[0008] S30 uses a gated recurrent neural network to model each word of the source text with the entire summary text and image feature vector to obtain a secondary gated hidden layer vector

[0009] S40, using a gated recurrent neural network to decode the text, using a positive attention mechanism to obtain a text positive attention vector and a picture positive attention vector, and using a negative attention mechanism to obtain a text negative attention vector and a picture negative attention vector;

[0010] S50, processing the positive weights through a positive correlation filter to discard irrelevant information and enhance relevant information, and processing the irrelevant weights through an irrelevant filter to enhance irrelevant information and discard relevant information;

[0011] S60, adding the forward text context denoising vector to the forward image context denoising vector to obtain a multimodal context forward fusion vector Add the reverse text context vector to the reverse image context vector to get the multimodal context reverse fusion vector

[0012] S70, forward fusion vector of multimodal context Multimodal context reverse fusion vector Combined with the visual features of the image, the positively correlated pointer P is obtained through the positive attention mechanism gen , the irrelevant pointer P is obtained through the reverse attention mechanism regen To assist in generating key words.

[0013] Furthermore, in step S20, the grid-level divided picture is layer by layer through the hidden layer vector h i , with each image grid-level feature v1 to v j Combined with each other, the largest image grid is taken, and its expression is:

[0014]

[0015] in, is the gate feature, σ is the sigmoid activation function, v i Indicates the feature vector of the i-th picture among all j pictures, and the feature vector of this picture satisfies all pictures. Maximum, W r with U r They are all weight matrices, and their specific values ​​are obtained through neural network training.

[0016] Further, it is characterized in that in the step S30, the hidden vector generated by the secondary gating is obtained The expression is as follows:

[0017]

[0018] in, As the hidden layer of the previous step of the gated recurrent neural network; GRU s represents the gated recurrent neural network kernel; “⊙” represents the Hadamard product; z t As the memory gate of the gated neural network, the expression is as follows:

[0019]

[0020] W1 and W2 are weight matrices obtained by neural network training; σ represents the sigmoid activation function, and tanh represents another activation function.

[0021] Further, in step S40, when decoding the text, a positive and negative self-attention mechanism is used to decode the text through the secondary gate hidden layer vector and the previous decoder hidden layer vector s t-1 Add together to get the hidden layer vector s of the current step t , and then get: text forward context vector Text reverse context vector and the image forward context vector Image reverse context vector

[0022] Further, in step S50, the text forward context vector After processing, multiply it by the text filter n represents text, and the positive text context denoising vector is obtained is the image forward context vector After processing, multiply the image filter m is represented by img; the reverse weight step is processed by an uncorrelated filter to keep consistent with the forward step, and the forward image context denoising vector is obtained.

[0023] Compared with the prior art, the present invention has the following significant effects:

[0024] 1. The present invention uses a two-level encoder to extract more unimodal information and introduces a new filter. The positive and negative attention mechanisms are used on the encoder to guide the multimodal pointer generation network to generate more focused words.

[0025] 2. The present invention adopts self-evaluation strategy gradient reinforcement learning to allow the neural network to automatically select the most appropriate picture as the output picture summary, which can increase the number of picture and text fusion channels and compensation effects. At the same time, it uses positive and negative attention mechanisms to eliminate parts of the picture and text that are irrelevant to the summary, so as to generate a multimodal summary with a more appropriate picture and text. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the present invention;

[0027] Figure 2 Schematic diagram of the first channel fusion and information compensation of text and pictures through the first-level encoder;

[0028] Figure 3 A schematic diagram of performing the second channel fusion and information compensation of text and images through the second-level encoder;

[0029] Figure 4 Schematic diagram of the process of obtaining context vectors of text and images through the forward and reverse self-attention mechanism;

[0030] Figure 5 Schematic diagram of the process of obtaining a fused context vector through a new type of filter and context vector weight;

[0031] Figure 6 A schematic diagram of the final text summary generated by the multimodal pointer generation network using positive and negative self-attention vectors; DETAILED DESCRIPTION

[0032] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0033] The present invention proposes a new image selection method based on reinforcement learning, which can select the most suitable image for the current step when the decoder generates each step of text, drive the change of image selection probability, and finally select the most suitable image summary.

[0034] The present invention adopts a two-level gated text-image channel compensation method, a reverse self-attention mechanism, and a self-evaluation-based strategy gradient reinforcement learning to eliminate the influence of irrelevant words and enhance the contribution of relevant words to the summary. Through multiple text-image fusion and information compensation, a text summary that is more in line with the picture can be obtained. Figure 1 As shown, the present invention comprises the following steps:

[0035] Step S10, collect the dataset, and use the manually written key points provided by the Daily Mail as the reference text summary. From the Daily Mail, randomly select articles within a week, and find that 2917 out of 2930 articles contain pictures, and each article contains about five to ten pictures with captions.

[0036] Step S20, using a long short-term memory neural network (LSTM), the text is encoded for the first time to obtain a hidden layer vector h iAt the same time, it is believed that text can be related to visual information at the level of image parts and image overall suggestions. The image grid-level features make up for the gap between text and image piece by piece. Therefore, the image is divided into grid levels, and then layer by layer through the hidden layer vector h i Respectively with each image grid level features v1 to v j Combine and take the largest image grid (i.e., gated feature), the specific formula is:

[0037]

[0038] in, is the gate feature, σ is the sigmoid activation function; v i Represents the feature vector of the i-th picture among all j pictures (the grid-level features of each picture are v1 to v j , v i is the eigenvector of the only picture that satisfies formula (1), and the eigenvector of this picture satisfies all pictures that use Maximum, all of the following v i The feature vector of a picture selected by this formula; W r with U r They are all weight matrices, and their specific values ​​are obtained by neural network training. Different dimensions of the hidden layer represent different semantics and correspond to different image parts. The maximum operation can capture the most relevant image grid area of ​​each text hidden layer.

[0039] Step S30, using a GRU gated recurrent neural network, each word of the source text is modeled with the entire summary text and image feature vector, and the gated features of each time step in the input sequence are calculated based on the multimodal information itself, the saliency of the image, and the redundancy of the text. Get the hidden vector generated by the secondary gating The specific formula is:

[0040]

[0041] in, As the hidden layer of the previous step of the gated recurrent neural network; GRU s represents the gated recurrent neural network kernel; “⊙” represents the Hadamard product; z t As the memory gate of the gated neural network, the expression is as follows:

[0042]

[0043] W1 and W2 are weight matrices obtained by neural network training; σ represents the sigmoid activation function, and tanh is also an activation function.

[0044] Step S40, decoder stage, uses LSTM to decode the text, and the hidden vector at time t The hidden vector generated by the encoder secondary gating By performing "Soft max (normalized exponential function)" processing and following the processing method of the self-attention mechanism, we can obtain the contextual positive attention vectors of the image and text respectively. At the same time, by using the self-attention mechanism in reverse, while paying attention to the correlation between text and image, we use the "1-sigmoid" algorithm to characterize the irrelevance of multimodal information, and we can obtain: Represent the text positive attention context vector and text anti-attention context vector respectively, Image positive attention context vector, image negative attention context vector.

[0045] Step S50, the present invention introduces a filtering mechanism, which refers to recording the past attention weights in the decoder and preventing the decoder from focusing on the same part of the input text in future decoding. At the same time, the present invention takes into account the relationship between the input information in the encoder and the words already generated by the decoder, so a main encoder and a sub-encoder are designed in the encoding stage. The sub-encoder outputs based on the input and the generation of the previous main encoder, so that the output includes more meaningful information and can give better output. A new type of text image denoising filter is used, and the filter formula is:

[0046]

[0047] Wherein, s0 is the initial state of the decoder, which is directly initialized to 0 in the present invention; s t-1 is the decoder hidden layer vector of the previous step, v i As the feature vector of the selected image; They represent weight matrices respectively and their values ​​are obtained by neural network training.

[0048] Through the positive correlation filter (n represents img / text) Process the forward multimodal fusion weights to discard irrelevant information and enhance relevant information, through irrelevant filters (m represents reimg / retext) to process the reverse multimodal fusion weights to enhance irrelevant information and discard relevant information. By denoising the channel information from these two directions, the forward text context denoising vector can be obtained. Forward Image Context Denoising Vector Inverse Text Context Denoising Vector Reverse Image Context Denoising Vector

[0049] Step S60, by using the context fusion vector obtained in step S50, the forward text context denoising vector is added to the forward image context denoising vector, and the reverse text context vector is added to the reverse image context vector, to obtain a multimodal context forward fusion vector Backward fusion vector with multimodal context

[0050] Step S70, using the forward fusion feature vector of text and image And reverse fusion feature vector Combined with the visual features of the image, the positive attention mechanism is used to obtain the positively correlated pointer P gen , use the reverse attention mechanism to get the irrelevant pointer P regen To assist in generating key words, a complete vocabulary P is introduced. vocab (w) is used to encourage the network to generate words that are not in the text. The formula for the pointer generation network is as follows:

[0051]

[0052]

[0053]

[0054] Among them, b′ and b, V′ and V are weight matrices, and their specific values ​​are obtained by neural network training; According to the self-attention mechanism formula (9) (10), s t is the hidden layer vector of the current decoder decoding the sentence, x t Represents the current word encoding vector, P regen From formula (7) becomes get.

[0055] like Figure 2 As shown, img1, img2, img i Representing pictures 1, 2, ...j respectively, the first fusion and complementary method of text data and picture data is as follows: i The encoding of each word vector in the text, v1 to v t The image feature vectors at the maximum gate are obtained by formula 1; the word vector encoding is passed through the LSTM recurrent neural network to obtain the first-level hidden layer vector h t , and then multiply it with the most relevant grid-level image feature vector to get the gate of the first-level gate The detailed formula is shown in formula (1).

[0056] From the perspective of time series, image information can guide the selection and compensation of text information at each step. For example, at a certain time point t, the LSTM long short-term memory recurrent neural network is used to extract the hidden layer of this time step, and the grid-level image feature information is used as the gate to perform the first compensation on the text hidden layer.

[0057] like Figure 3 As shown, the schematic diagram of the two-level gated multimodal data fusion and channel information compensation method is as follows: Input h′ t First-level gating Multiply by the first hidden layer vector h t The obtained r represents the image grid-level features. Since the present invention only uses the image grid-level features, Equal to sGate in the figure i , the formula is:

[0058]

[0059] Input M img The sum of the image information is added, and then the hidden layer vector of the secondary gate is obtained through GRU See formula (2) for details. The text feature vector is further processed using a GRU gated recurrent neural network, and the entire image information is used to participate in the selection of the recurrent neural network gate. The image saliency information and each word in the source text and the entire text are modeled as importance weights z t , obtained from formula (3), is used to represent the gating coefficient to determine how much information from the previous step needs to be inherited by the current time step. t How much attention should be given.

[0060] like Figure 4 As shown, the positive and negative attention mechanism is used to pay attention to the correlation between text and image while depicting the irrelevance of multimodality. The schematic diagram is as follows: In the decoding stage, the current encoder secondary gated hidden layer vector and the encoder hidden layer vector at the previous time step To calculate the self-attention mechanism, the formula is:

[0061]

[0062] in, W a 、V a , b a is the weight matrix and tanh is the activation function.

[0063] For image features, use the image features and the hidden state s of the decoder at the previous time step t-1After Soft max activation, the visual attention distribution α is obtained, thereby obtaining the positive text context vector of the text modality At the same time, the reverse text context vector of the text modality can be obtained through the sigmoid function The formula is:

[0064]

[0065]

[0066] Formula (11) calculated are the forward and reverse context vectors, and n can represent text, retext, img, and reimg respectively. Formulas (9), (10), and (11) are the specific algorithms of the self-attention mechanism.

[0067] like Figure 5 As shown, the process of using filters to update the image-text vector fusion weights is as follows:

[0068] A new text image filter I is used. The filter gain and dynamic state update are used to iterate the filter at each time step. The positive weight is processed by the positive correlation filter to discard irrelevant information and enhance relevant information. The reverse weight is processed by the irrelevant filter to enhance irrelevant information and discard relevant information. The channel information is denoised from these two directions. The initial weight is obtained by the decoder hidden layer vector and the context vector. Finally, the positive and negative multimodal context vectors are obtained by the weighted sum of the text and the image. The specific implementation formula is:

[0069]

[0070]

[0071] where s t is the current decoder hidden state, is the context vector, s0 is initialized to 0, s t-1 is the hidden state of the decoder in the previous step. v i is the image feature vector, and the rest are weight matrices. n can represent text, retext, img, and reimg respectively.

[0072] Among them, the forward and reverse context denoising vectors n can represent text, retext, img, and reimg respectively. Add them together to get the fused context attention vector The formulas are shown in 14 and 15 below.

[0073]

[0074]

[0075] like Figure 6 As shown, the schematic diagram of the multimodal pointer generation network with positive and negative attention mechanism:

[0076] Image information is added to the traditional unimodal pointer generation network, and the attention distribution obtained by the forward attention mechanism is used to construct a multimodal pointer to point to key words. The attention distribution obtained by the reverse attention pointer is used to construct a multimodal pointer to point to irrelevant words, and a complete vocabulary containing all words is constructed at the same time. The purpose of the pointer and the vocabulary is to encourage the network to generate relevant words at the decoder stage, while generating words from a fixed vocabulary to solve the OOV (Out-Of-Vocabulary) problem. The irrelevant pointer further offsets the impact of irrelevant words on the article summary.

[0077] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A multimodal summarization method for images and texts based on two-level gated coding, characterized in that: The following steps are involved: S10, obtaining a data set, ensuring that each piece of data includes a text and several pictures; S20, using convolutional neural network to extract image feature vectors v1 to v of all j images respectively j , the long short-term memory recurrent neural network encoder is used to process the text to obtain the hidden layer h of the first-level text encoding i ; At the same time, the grid-level divided pictures are passed through the hidden layer vector h layer by layer i , with each image grid-level features v1 to v j Combine and take the largest image grid; S30 uses a gated recurrent neural network to model each word of the source text with the entire summary text and image feature vector to obtain a secondary gated hidden layer vector S40, using a gated recurrent neural network to decode the text, using a positive attention mechanism to obtain a text positive attention vector and a picture positive attention vector, and using a negative attention mechanism to obtain a text negative attention vector and a picture negative attention vector; S50, processing the positive weights through a positive correlation filter to discard irrelevant information and enhance relevant information, and processing the irrelevant weights through an irrelevant filter to enhance irrelevant information and discard relevant information; S60, adding the forward text context denoising vector to the forward image context denoising vector to obtain a multimodal context forward fusion vector Add the reverse text context vector to the reverse image context vector to get the multimodal context reverse fusion vector S70, forward fusion vector of multimodal context Multimodal context reverse fusion vector Combined with the visual features of the image, the positively correlated pointer P is obtained through the positive attention mechanism gen , the irrelevant pointer P is obtained through the reverse attention mechanism regen To assist in generating key words.

2. The method for multimodal summarization of images and texts based on secondary gated coding according to claim 1, characterized in that: In step S20, the grid-level divided image is layer by layer through the hidden layer vector h i , with each image grid-level features v1 to v j Combined with each other, the largest image grid is taken, and its expression is: in, is the gate feature, σ is the sigmoid activation function; v i It represents the feature vector of the i-th picture among all j pictures, and the feature vector of this picture satisfies all the pictures. Maximum; W r with U r They are all weight matrices, and their specific values ​​are obtained by neural network training.

3. The method for multimodal summarization of images and texts based on secondary gated coding according to claim 1, characterized in that: In step S30, the hidden vector generated by the secondary gating is obtained. The expression is as follows: in, As the hidden layer of the previous step of the gated recurrent neural network; GRU s represents the gated recurrent neural network kernel; "⊙" represents the Hadamard product; z t As the memory gate of the gated neural network, the expression is as follows: W1 and W2 are weight matrices obtained by neural network training; σ represents the sigmoid activation function, and tanh represents another activation function.

4. The method for multimodal summarization of images and texts based on secondary gated coding according to claim 1, characterized in that: In step S40, when decoding the text, a positive and negative self-attention mechanism is used to decode the text through the secondary gate hidden layer vector and the previous decoder hidden layer vector s t-1 Add together to get the hidden layer vector s of the current step t , and then get: text forward context vector Text reverse context vector and the image forward context vector Image reverse context vector 5. The method for multimodal summarization of images and texts based on secondary gated coding according to claim 1, characterized in that: In step S50, the text forward context vector After processing, multiply it by the text filter n represents text, and the positive text context denoising vector is obtained is the image forward context vector After processing, multiply the image filter m is represented by img; the reverse weight step is processed by an uncorrelated filter to keep consistent with the forward step, and the forward image context denoising vector is obtained.

Images