Multi-modal automatic abstracting method based on deep learning

Abstract

The invention relates to a multi-modal automatic abstracting method based on deep learning, and belongs to the technical field of computer natural language processing. The method comprises the following steps: 1, extracting text features by utilizing BERT, and extracting image features by utilizing GoogleNet; 2, the extracted image-text features are connected in series to form a series vector; 3, fusing the text features and the image features; 4, inputting the fused text features into a decoder of the text abstract, calculating by adopting the cross attention and self-attention with masks of the decoder to obtain autoregression results of word vectors of the text abstract, and selecting the autoregression result with the maximum probability in an autoregression sequence by utilizing beam search to output the text abstract; 5, inputting the fused image features into a Softmax layer, and performing image abstract output on a sample image with the maximum probability in the samples; compared with the prior art, the method has the advantages that the text abstract is generated and the picture most related to the abstract is selected by adopting deep learning, so that the text abstract and the related picture can be accurately generated, and the practicability is very high.

Description

Technical Field

[0001] The present invention relates to a multimodal automatic summarization method based on deep learning, and belongs to the technical field of computer natural language processing. Background Art

[0002] Multimodal Summarization aims to condense articles containing multiple modalities (such as text and images) into short summaries, thereby helping users quickly grasp the key information of the article. However, most current news articles contain both text and images. As an important component of an article, the modal information of images plays a very important role in the task of generating accurate summaries. Therefore, based on the ordinary text summarization task, the multimodal summarization task was derived. In the field of plain text summarization, many studies have been devoted to improving indicators such as ROUGE that measure vocabulary overlap. Many multimodal summarization models also focus on improving the overlap with the source text, treating visual content as auxiliary data that is processed independently. However, visual content can convey additional information beyond the text content, which may result in the summary being unable to effectively cover all the information of the text and visual content in the source article, thereby reducing the coverage of multimodal semantics.

[0003] Therefore, how to generate concise mixed text and image summaries from multimodal input information has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present invention is to solve the technical problem of generating a concise mixed text and image summary from multimodal input information. A multimodal automatic summarization method based on deep learning is proposed.

[0005] The workflow of the present invention is as follows: first, the features of the text part of the article are extracted using the Chinese pre-trained BERT to generate a feature vector, and the features of the image part of the article are extracted using the pre-trained GoogleNet to generate a feature vector. Then, the extracted text feature vector and the image feature vector are connected in series and input into the cross-modal feature fusion module, and the module performs multi-head self-attention calculation to fully fuse and interact the features within and between the two modalities. Then, they are respectively input into the feedforward neural network for constructing image features and for constructing text features to obtain the fused image features and text features. Finally, a text summary and the most relevant picture are generated by a decoder and Softmax, respectively.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A multimodal automatic summarization method based on deep learning of the present invention comprises the following steps:

[0008] Step 1: Use the BERT Chinese pre-trained language model to extract text features, and use the GoogleNet visual pre-trained model to extract image features;

[0009] Step 2: Use a linear layer to uniformly map the extracted text features and image features into a vector of uniform dimension, and concatenate the image and text features in the manner shown in formula (1) to form a concatenated vector;

[0010]

[0011] in, is the text node embedding matrix, is the image node embedding matrix and D is the hidden layer dimension.

[0012] Step 3: Fusion of text features and image features;

[0013] Step 3.1: Input the concatenated vector into the cross-modal feature fusion encoder of Transformer, and obtain the feature fusion vector through the self-attention calculation of the multi-head self-attention layer as shown in equations (2) to (4);

[0014] Q=XW Q ,K=XW K ,V=XW V (2)

[0015] in, is the linear mapping matrix used to generate the attention mechanism operation, and Z is the output obtained after the self-attention calculation. Press N S and N i The fused text feature vector and the fused image feature vector are obtained by splitting.

[0016]

[0017] Among them, i,j∈[1,N s +N i ], d is the dimension of the vector.

[0018] Z=X+SVW O (4)

[0019] Among them, W O is the weight matrix.

[0020] Step 3.2: Perform vector splitting on the text features and image features described in step 1 by length to obtain text feature vectors and image feature vectors respectively.

[0021] Step 3.3: Normalize the text feature vector and the image feature vector respectively and input them into the feedforward neural network, as shown in equations (5) and (6), and obtain the fused text feature and fused image feature by iterating step 3;

[0022]

[0023] Among them, FFN t and FFN i They represent the feedforward neural network for constructing image features and the feedforward neural network for constructing text features respectively.

[0024] Step 4: Input the fused text features obtained in step 3 into the decoder of the text summary, and use the decoder to calculate the autoregressive result of the text summary word vector with masked cross attention and self-attention as shown in equations (7) to (8). Use beam search to select the autoregressive result with the highest probability in the autoregressive sequence to output the text summary;

[0025]

[0026] in, It is the linear mapping matrix used to generate the attention mechanism operation, which is obtained from the matrix required for attention calculation.

[0027]

[0028] Among them, i,j∈[1,N s ], d is the dimension of the vector;

[0029] Z c =Z s +SVW o (9)

[0030] in, is the linear mapping matrix used to generate the attention mechanism operation, Z c This is the output obtained by cross-attention calculation. Then the output is normalized through residual connection and then input into the feedforward neural network to obtain the output Z of the decoder. d .

[0031] Step 5: Input the fused image features obtained in step 3 into the Softmax layer, and output the image summary of the sample image with the highest probability among the samples;

[0032] Beneficial effects:

[0033] Compared with the prior art, the method of the present invention adopts a deep learning algorithm to generate text summaries and select the most relevant pictures for the summaries. It can accurately generate text summaries and related pictures and has strong practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of the present invention;

[0035] Figure 2 This is a structural diagram of the cross-modal feature fusion module of the present invention;

[0036] Figure 3 It is a structural diagram of a decoder of the present invention;

[0037] Figure 4 This is a visualization effect diagram of the present invention. DETAILED DESCRIPTION

[0038] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.

[0039] Example

[0040] like Figure 1 As shown, the multimodal automatic summarization method based on deep learning of the present invention is specifically implemented in the following steps:

[0041] First, the text and image of the news article are collected. In this embodiment, the input text is the word vector of all the words in the news article, and the input image is an RGB three-channel color image with a resolution of 1000×1000 pixels.

[0042] Step 1: Use the BERT Chinese pre-trained language model to extract text features, and use the GoogleNet visual pre-trained model to extract image features;

[0043] In this embodiment, for the text X input to the model t ={x1,x2,…,x N}, and input it into the Chinese pre-trained language model BERT. First, it will pass through the embedding layer to obtain the embedding vector of the text, and then pass through the forward propagation of the model to obtain the hidden state of the final layer output, which is the text feature vector extracted by the model. Input the input image into GoogleNet to obtain the encoded image feature vector

[0044] Step 2: Use a linear layer to uniformly map the extracted text features and image features into a vector of uniform dimension, and concatenate the image and text features in the manner shown in formula (1) to form a concatenated vector;

[0045]

[0046] in, is the text node embedding matrix, is the image node embedding matrix and D is the hidden layer dimension.

[0047] In the embodiment, the text feature vector extracted in step 1 is and the image feature vector Perform concatenation to obtain the concatenated vector X.

[0048] Step 3: Fusion of text features and image features;

[0049] Step 3.1: Input the concatenated vector into the cross-modal feature fusion encoder of Transformer, and obtain the feature fusion vector through the self-attention calculation of the multi-head self-attention layer as shown in equations (2) to (4);

[0050] Q=XW Q ,K=XW K ,V=XW V (2)

[0051] in, is the linear mapping matrix used to generate the attention mechanism operation, and Z is the output obtained after the self-attention calculation. Press N S and N i The fused text feature vector and the fused image feature vector are obtained by splitting.

[0052]

[0053] Among them, i,j∈[1,N s +N i ], d is the dimension of the vector.

[0054] Z=X+SVW O (4)

[0055] Among them, W O is the weight matrix.

[0056] Step 3.2: Perform vector splitting on the text features and image features described in step 1 by length to obtain text feature vectors and image feature vectors respectively.

[0057] Step 3.3: Normalize the text feature vector and the image feature vector respectively and input them into the feedforward neural network, as shown in equations (5) and (6), and obtain the fused text feature and fused image feature by iterating step 3;

[0058]

[0059] Among them, FFNt and FFN i They represent the feedforward neural network for constructing image features and the feedforward neural network for constructing text features respectively.

[0060] In the embodiment, Figure 2 As shown, the concatenated vector X is input into the feature fusion module. This process first needs to calculate the Q, K, and V matrices required for self-attention calculation. Then the attention coefficient S is calculated based on these three matrices. i,j Finally, the final result is calculated according to the attention coefficient, and the fused text features are obtained after the result is split. And the image features after fusion Input them into the feedforward neural network to calculate and get the final output fused text features And the fused image features

[0061] Step 4: Input the fused text features obtained in step 3 into the decoder of the text summary, and use the decoder to calculate the autoregressive result of the text summary word vector with masked cross attention and self-attention as shown in equations (7) to (8). Use beam search to select the autoregressive result with the highest probability in the autoregressive sequence to output the text summary;

[0062]

[0063] in, It is the linear mapping matrix used to generate the attention mechanism operation, which is obtained from the matrix required for attention calculation.

[0064]

[0065] Among them, i,j∈[1,N s ], d is the dimension of the vector;

[0066] Z c =Z s +SVW O (9)

[0067] in, is the linear mapping matrix used to generate the attention mechanism operation, Z c This is the output obtained by cross-attention calculation. Then the output is normalized through residual connection and then input into the feedforward neural network to obtain the output Z of the decoder. d .

[0068] In the embodiment, Figure 3 As shown, the fused text features are input into the decoder. This process will go through self-attention calculation and cross-attention calculation again to get the output Zc Then Z c Input into the feedforward neural network to get the final output Z of the decoder d Then Z d The probability distribution of word vectors is calculated through the Softmax layer, and then the beam search mechanism is used to generate the final text summary.

[0069] Step 5: Input the fused image features obtained in step 3 into the Softmax layer, and output the image summary of the sample image with the highest probability among the samples;

[0070] In the embodiment, the fused image features obtained in step 3 are The image is input into the Softmax layer to obtain the probability distribution of each image in the batch, and the image with the highest probability is selected as the most relevant image for the summary predicted by the model.

[0071] In order to further illustrate the advantages of the present invention, visual experimental results are used for illustration.

[0072] like Figure 4 As shown in the figure, it includes multiple pictures and text content. The original text content is the content input into the model, the summary label is the golden summary of the original text content, and the summary generated by the model is the result generated by the model. It can be seen that the summary generated by the model is very consistent with the golden summary and the original content. The image enclosed by the red dotted box in the figure is the most relevant image predicted by the model. It can be seen that the image predicted by the model has similar semantics to the text summary content generated by the model.

[0073] The results show that the method proposed in this invention can generate accurate text summaries and image summaries for multimodal input content.

Claims

1. A multimodal automatic summarization method based on deep learning, characterized by: The following steps are included: Step 1: Use the BERT Chinese pre-trained language model to extract text features, and use the GoogleNet visual pre-trained model to extract image features; Step 2: Use a linear layer to uniformly map the extracted text features and image features into a vector of uniform dimension, and concatenate the image and text features in the manner shown in formula (1) to form a concatenated vector; in, is the text node embedding matrix, is the image node embedding matrix, D is the hidden layer dimension; Step 3: Fusion of text features and image features; Step 4: Input the fused text features obtained in step 3 into the decoder of the text summary, and use the decoder to calculate the autoregressive result of the text summary word vector with masked cross attention and self-attention as shown in equations (7) to (8). Use beam search to select the autoregressive result with the highest probability in the autoregressive sequence to output the text summary; in, It is the linear mapping matrix used to generate the attention mechanism operation, which is obtained from the matrix required for attention calculation after operation; Among them, i,j∈[1,N s ], d is the dimension of the vector; WITH c =Z d +SVW O (9) in, is the linear mapping matrix used to generate the attention mechanism operation, Z c This is the output obtained by cross-attention calculation; the output is then normalized through the residual connection and then input into the feedforward neural network to obtain the output Z of the decoder. d ; Step 5: Input the fused image features obtained in step 3 into the Softmax layer, and output the image summary of the sample image with the highest probability in the sample.

2. The multimodal automatic summarization method based on deep learning as claimed in claim 1, characterized in that: Step 3 is implemented as follows: Step 3.1: Input the concatenated vector into the cross-modal feature fusion encoder of Transformer, and obtain the feature fusion vector through the self-attention calculation of the multi-head self-attention layer as shown in equations (2) to (4); Q=XW Q ,K=XW K ,V=XW V (2) in, is the linear mapping matrix used to generate the attention mechanism operation, and Z is the output obtained by the self-attention calculation; Press N S and N i Split and obtain the fused text feature vector and the fused image feature vector; Among them, i,j∈[1,N s +N i ], d is the dimension of the vector; Z=X+SVW O (4) Among them, W O is the weight matrix; Step 3.2: Perform vector splitting on the text features and image features described in step 1 by length, and obtain text feature vectors and image feature vectors respectively; Step 3.3: Normalize the text feature vector and the image feature vector respectively and input them into the feedforward neural network, as shown in equations (5) and (6), and obtain the fused text feature and fused image feature by iterating step 3; Among them, FFN t and FFN i They represent the feedforward neural network for constructing image features and the feedforward neural network for constructing text features respectively.

Images