Abstract generation method for multi-modal data based on deep learning

Abstract

A multi-modal data-oriented abstract generation method based on deep learning comprises the following steps: acquiring news sample data and preprocessing the news sample data to obtain news pictures and news text descriptions; extracting picture features from the news pictures through the trained convolutional neural network based on the residual mechanism; extracting text features from the news text description through the trained BERT-base model; sending the picture features to a trained visual attention layer based on a Transfomer module to obtain a picture context vector; sending the text features to a trained text attention layer based on bidirectional LSTM to obtain a text context vector; inputting the picture context vector and the text context vector into a multi-modal attention layer to realize fusion of text features and picture features; and S5, inputting the fused features into a multi-mode decoder to generate an abstract. The method has high universality, the efficiency and accuracy of information processing can be improved, and a new solution is provided for processing big data.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and natural language processing, and in particular to a summary generation method for multimodal data based on deep learning. Background Art

[0002] With the rapid development of the Internet and the advent of the big data era, the data on the Internet has exploded. These data include not only text, but also pictures, videos, audio and other forms, forming multimodal data. Compared with single-modal data, multimodal data can express information more comprehensively and intuitively, so it has been widely used in many fields.

[0003] In the field of news, multimodal data such as social media and online news are widely used in news reporting and dissemination. Usually, news editors need to describe and report news events by observing a large number of news pictures, videos and supporting texts. These contents will form news articles to help the public understand the progress of events. The description of multimodal data in news articles usually needs to follow news writing rules, and news editors need to have rich news knowledge and keen event insight to accurately grasp the key information of news events and complete news descriptions. Therefore, this work is very boring and tedious. Especially when there are many news events, the increase in editorial workload will lead to fatigue of news editors, or the lack of experience of news editors will lead to errors in news descriptions. At the same time, when users are faced with a large amount of multimodal information, they also need to spend a lot of time to extract key information from it. Multimodal summarization methods can play an important role in this field. By automatically extracting key information from news pictures, videos and supporting texts and generating descriptive texts, the workload of news editors can be greatly reduced, and the accuracy and efficiency of news reports can be improved. At the same time, users can also quickly obtain key information with the help of multimodal summarization technology.

[0004] At present, the common methods in the multimodal field are mainly divided into different stages. The early fusion method fuses the features of different modalities at the input layer, and then processes them through a unified model. The advantage of this method is that it is simple and easy to implement, but the disadvantage is that it may lead to information loss because the features of different modalities are not fully analyzed and processed before fusion. Late fusion first extracts and processes the features of data of different modalities, and then fuses these results at the output layer to generate a summary. This method fully considers the independence of different modalities, but the disadvantage is that it may make the model too complicated, requiring a lot of computing resources and time, and processing multimodal data to generate summaries has low training efficiency, unbalanced modal contributions, and inadequate modal fusion. Summary of the invention

[0005] The present invention aims to provide a summary generation method for multimodal data based on deep learning, so as to solve the problems of low training efficiency, unbalanced modal contribution and inadequate modal fusion in processing multimodal data to generate summaries.

[0006] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0007] A method for generating a summary for multimodal data based on deep learning comprises the following steps:

[0008] Step S1: Obtain news sample data and perform preprocessing to obtain news pictures and news text descriptions, and scale the news pictures to a size of 224x224 pixels;

[0009] Step S2: extract image features from news images through the trained convolutional neural network based on residual mechanism; extract text features from news text descriptions through the trained BERT-base model;

[0010] Step S3: Send the image features to the trained visual attention layer based on the Transformer module to obtain the image context vector; send the text features to the trained text attention layer based on the bidirectional LSTM to obtain the text context vector;

[0011] Step S4: Input the image context vector and the text context vector into the multimodal attention layer to achieve the fusion of text features and image features;

[0012] Step S5: Input the fused features into the multimodal decoder to generate a summary.

[0013] Furthermore, the convolutional neural network based on the residual mechanism in step S2 has a total of 4 residual blocks, each block contains 2 convolutional layers, plus the input layer, the first convolutional layer, the average pooling layer and the last fully connected layer, a total of 12 weight layers; the convolutional layer and the residual block are used to extract the feature information of the image, the pooling layer is used to reduce the size of the image features and improve the calculation efficiency, and the fully connected layer outputs the image feature vector.

[0014] Furthermore, the Transformer encoder of the visual attention layer is composed of n identical layers stacked together, each layer has two sub-layers; the first sub-layer is multi-head self-attention; the second sub-layer is a position-based bit-by-bit feedforward network; the Transformer decoder is also composed of n identical layers stacked together. The first multi-head attention layer of the decoder is a masked multi-head self-attention, and the second multi-head attention of the decoder is the encoder-decoder attention layer.

[0015] Further, the encoder layer formula is as follows:

[0016] e mid =LayerNorm(e input +MultiHeadAttention(e input ))

[0017] e output =LayerNorm(e mid +FFN(e mid ))

[0018] Among them, e input represents the input of the encoder layer, e output represents the output of the encoder layer, e mid Represents the output of the first sublayer of the encoder, MultiHeadAttention(·) is a multi-head attention mechanism, FFN is a bit-by-bit feedforward neural network, and LayerNorm(·) represents the layer normalization performed by the model in this field;

[0019] Input to the encoder layer Where h is the number of attention heads, N is the length of the input sequence, and d is model is the embedding dimension size; the query vector sequence Q, key vector sequence K and value vector sequence V are all set to e input .

[0020] MultiHeadAttention(e input ) is represented as MultiHead(Q, K, V)

[0021] MultiHead(Q,K,V)=Contact(head1,...,head i ,..head h )W O

[0022] head i =Attention(QW i Q , KW i K , VW i V )

[0023] MultiHead(Q, K, V) represents the attention matrix of all head information, Contact represents concatenation; head i represents the i-th attention matrix, are all weight matrices learned during training; d q represents the query vector dimension, d k represents the dimension of the key vector, d vIndicates the dimension of the value vector;

[0024] The self-attention formula is as follows:

[0025]

[0026] Softmax represents the normalized exponential function; T represents transpose;

[0027] The sequence obtained after processing by the multi-head attention model is added to the original sequence and then layer normalized. The resulting sequence is input into the bit-by-bit feedforward neural network and then layer normalized again. This process is repeated n times to obtain the output of the encoder.

[0028] Furthermore, the sequence in the decoder first inputs the masked multi-head self-attention, and the i-th attention head matrix Z i Calculated as follows:

[0029]

[0030] Among them, V i Represents a value matrix, and the values ​​need to be masked before softmax is performed;

[0031] The final attention matrix is ​​calculated by the following formula:

[0032] M=Contact(Z1,...,Z i ,..Z h )W O

[0033] After the final attention matrix is ​​layer-normalized, it is input into the encoder-decoder attention layer and the output of the encoder. After the steps in the decoder are repeated n times, the final output of the decoder is obtained. The final output of the decoder is passed through the fully connected layer to obtain the image context vector.

[0034] Furthermore, the bidirectional LSTM layer of the text attention layer contains two LSTM layers, one that processes the input sequence from left to right in time steps, and the other that processes the input sequence from right to left in time steps; the outputs of these two LSTM layers are concatenated to form the output of the bidirectional LSTM layer;

[0035] The LSTM calculation method is as follows:

[0036] I t =σ(X t W xi +H t-1 W hi +b i )

[0037] F t =σ(Xt W xf +H t-1 W hf +b f )

[0038] O t =σ(X t W xo +H t-1 W ho +b o )

[0039]

[0040]

[0041] H t =tanh(C t )⊙O t

[0042] Among them, the input gate is I t ∈R n×h , the forget gate is F t ∈R n×h , the output gate is O t ∈R n×h , n is the batch size, h is the number of hidden units; W xi , W xf , W xo All∈R d×h , are the d×h dimensional weight matrices of the input gate, forget gate, and output gate, respectively, W xc Represents a d×h dimensional weight matrix, d is the size of the input number; W hi , W hf , W ho are the weight parameters of the input gate, forget gate and output gate respectively, W hc represents the weight parameter, b i , b f and b o are the bias parameters of the input gate, forget gate, and output gate respectively; σ represents the sigmoid function, X t represents the input at time t; Represents the candidate memory state of the current time step; C t Represents the memory state of the current time step; C t-1 represents the memory state of the previous time step; ⊙ represents the multiplication of elements in the matrix; H t Represents the hidden layer output at the current time step;

[0043] For text features, LSTM is used to obtain the context vector of the document. The forward sequence and the reverse sequence are combined through a fully connected layer to obtain the output of the bidirectional LSTM, that is, the text context vector is obtained, as shown below:

[0044]

[0045] Compared with the prior art, the present invention has the following beneficial technical effects:

[0046] (1) Based on the solution of the present invention, a news summary can be automatically generated based on the input multimodal data (such as pictures and texts). Users can directly use the summary or use it to assist in writing relevant report content, thereby reducing workload and errors.

[0047] (2) During the training process, there is no need to convert sample images and texts into keywords separately. Instead, only the original images and text descriptions need to be obtained as training data. This reduces the workload and avoids the problem of information loss that is easily caused when manually extracting key information from multimodal data.

[0048] (3) The performance of the dual-tower structure and attention mechanism used is significantly better than that of traditional methods: First, the dual-tower structure and attention mechanism are more complex in network structure, and the network's semantic feature extraction capability is much stronger than traditional methods, making the final summary generated by the system more accurate; second, the attention mechanism can better integrate multimodal information and improve the performance of the model when processing multimodal data.

[0049] (4) The multimodal summary model of the present invention has strong versatility and can be widely used in news reporting, medical diagnosis, financial analysis and other fields, improving the efficiency and accuracy of information processing and providing a new solution for processing big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0051] Figure 1 It is a structural schematic diagram of the model training process of the present invention.

[0052] Figure 2 This is a schematic diagram of the model algorithm of the present invention.

[0053] Figure 3 Schematic diagram of the attention visual layer structure of the present invention.

[0054] Figure 4Schematic diagram of the text attention layer structure of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The present invention aims to propose a summary generation method for multimodal data based on deep learning, so as to solve the problems of low training efficiency, unbalanced modal contribution and inadequate modal fusion in the prior art of processing multimodal data to generate summaries. The method includes a training process and a prediction process. In the training process, news data is first obtained and preprocessed to obtain news text and picture illustrations of the corresponding news. Then, text features and picture features are extracted through the designed text embedding layer and picture embedding layer. Then, a visual attention layer based on a Transfomer module is used for picture features, and a text attention layer based on a bidirectional LSTM is used for text features. Two context vectors are obtained by processing the multimodal attention layer of the attention mechanism. A multimodal decoder based on the Transfomer module is used. Finally, the reference summary of the news is used as the prediction result, and the model parameters are trained using the gradient descent method to obtain the trained multimodal encoder and decoder. In the prediction process, after preprocessing the news data, the previously designed text and picture embedding layers are used to extract features, and the picture and text context vectors are obtained through the multimodal encoder that has been in the training process. The vectors processed by the attention layer are sent to the multimodal decoder to finally obtain the generated summary corresponding to the news.

[0057] The deep learning-based summary generation method for multimodal data in this embodiment includes two parts: a training process and a prediction process, which are described in detail below with reference to the accompanying drawings:

[0058] The training process of the deep learning-based summary generation method for multimodal data is as follows Figure 1-Figure 2 As shown, the following steps are included:

[0059] Step S1: Acquire news sample data and perform preprocessing to obtain news picture samples and matching news text descriptions.

[0060] Specifically, step S1 includes the following steps:

[0061] Step S11: Obtain news sample data; specifically, a python crawler may be used to crawl major electronic news websites to obtain news sample data, such as the Daily Mail;

[0062] Step S12: extracting picture samples from the news sample data, extracting news texts described in news texts, using the same ID for the extracted picture samples and news texts, and extracting reference summaries of the news sample data, which will be used as training data sets, verification data sets, and test data sets;

[0063] Step S13: scaling the news picture samples to ensure uniform size and uniform picture format; for example, all are adjusted to 224x224 pixels.

[0064] Step S2: Two parallel threads of the dual-tower structure model are used, one of which extracts image features from news image samples through a convolutional neural network based on a residual mechanism, and the other thread extracts text features from news text descriptions using a BERT-base model.

[0065] Specifically, step S2 includes the following steps:

[0066] Step S21: Use the pre-trained ResNet neural network to extract image features; convert the original image to a specified size (224x224 pixels) and normalize it, adjusting its pixel value from the range of 0-255 to between 0-1 to meet the input requirements of the ResNet model. Input the pre-processed image into the ResNet model for forward propagation calculation. The model processes image information layer by layer, extracting high-level abstract features through structures such as convolution, batch normalization, activation functions (such as ReLU), and residual blocks. Obtain the corresponding image features representing the image information.

[0067] Specifically, in the convolutional neural network ResNet based on the residual mechanism built in this embodiment, there are a total of 4 residual blocks, each block contains 2 convolutional layers, plus the input layer, the first convolutional layer, the average pooling layer and the last fully connected layer, a total of 12 weight layers. Each layer is introduced as follows:

[0068] 1) Input layer: The size of the input image is usually 224x224x3 (height x width x number of channels).

[0069] 2) First convolutional layer: The convolution kernel size is 7x7, the number of output channels is 64, the stride is 2, and SAME padding is used. Batch normalization is used. ReLU activation function is used. Max Pooling layer, the pooling kernel size is 3x3, and the stride is 2.

[0070] 3) Residual blocks (each block contains two convolutional layers):

[0071] The first residual block: 2 3x3 convolutional layers, 64 filters each.

[0072] The second residual block: 2 3x3 convolutional layers, 128 filters per layer, and a stride of 2.

[0073] The third residual block: 2 3x3 convolutional layers, 256 filters per layer, and a stride of 2.

[0074] The 4th residual block: 2 3x3 convolutional layers, 512 filters per layer, and stride 2.

[0075] The first convolutional layer in each residual block has a stride of 2 (except the first residual block) to reduce the size of the feature map. Each convolutional layer is followed by batch normalization and ReLU activation function. Residual connections (skip connections) add the input directly to the output of the convolutional layer.

[0076] 4) Average Pooling: A global average pooling layer that reduces the size of the feature map to 1x1.

[0077] 5) Fully connected layer: outputs image feature vector.

[0078] In the above deep convolutional neural network structure, the convolution layer and residual are used to extract the feature information of the image, the pooling layer is used to reduce the size of the image features and improve the calculation efficiency, and the fully connected layer outputs the image feature vector.

[0079] Step S22: Use the BERT-base model to extract text features. The original text is segmented according to predefined segmentation rules (such as character-based, word-based, subword-based, etc.) to form a word sequence. Use the Tokenizer corresponding to the BERT-base model (BERT's WordPiece Tokenizer) to convert the word sequence into a token sequence that the model can understand, including common tokens, special tokens (such as [CLS], [SEP], [PAD], etc.) The preprocessed token sequence and its corresponding segment embeddings and position embeddings are concatenated as the input of the BERT-base model for forward propagation calculation. Finally, a text context vector representing the text information is obtained.

[0080] The input formula of the BERT-base model encoder is as follows:

[0081]

[0082] e l =EncodeLayer(e l-1),l∈[1,12]

[0083] Among them, e 0 Input for the BERT-base model encoder; Embed img Represents the image embedding vector, Emed text represents the text embedding vector, represents position encoding; EncodeLayer(·) is the encoder layer, e l is the output of the lth encoder layer.

[0084] The BERT-BASE model used in the present invention includes a 12-layer Transformer encoder, each layer has 12 self-attention heads, the hidden layer size is 768, and the total number of parameters is 110M.

[0085] To extract text feature embedding using the BERT-base model, you need to follow the following steps, as explained below:

[0086] 1) Install the Transformers library, a specific library for BERT. It provides pre-trained BERT models and tools for processing text.

[0087] 2) Load the pre-trained BERT-base model and its tokenizer from the Transformers library.

[0088] 3) Use the tokenizer to convert the text into the format required by the BERT model, including adding special tags [CLS] and [SEP], and converting words into corresponding IDs.

[0089] 4) Input the encoded text into the BERT-base model and obtain the hidden layer state as the feature embedding of the text.

[0090] S3. Send the image features to the visual attention layer based on the Transformer module to obtain an image context vector; send the text features to the text attention layer based on the bidirectional LSTM to obtain a text context vector.

[0091] Specifically, step S3 includes the following steps:

[0092] Step S31: Transformer module. The encoder and decoder of Transformer are based on the superposition of self-attention modules. The embedding representation of the source (input) sequence and the target (output) sequence will be added with positional encoding and then input into the encoder and decoder respectively, as shown in Figure 3 shown.

[0093] The Transformer encoder is composed of n identical layers stacked together, each with two sublayers. The first sublayer is a multi-head self-attention pool; the second sublayer is a positionwise feed-forward network. Specifically, when calculating the encoder's self-attention, the query, key, and value all come from the output of the previous encoder layer. Each sublayer uses a residual connection. In the Transformer, for any input at any position in the sequence, is required to satisfy in order for the residual connection to satisfy . After the addition calculation of the residual connection, layer normalization is applied. Therefore, for each position corresponding to the input sequence, the Transformer encoder will output a one-dimensional representation vector.

[0094] The Transformer decoder is also composed of n identical layers stacked together, and residual connections and layer normalization are used in the layers. The first multi-head attention layer of the decoder is a masked multi-head self-attention, and the second multi-head attention of the decoder is an encoder-decoder attention layer. In encoder-decoder attention, the query comes from the output of the previous decoder layer, while the key and value come from the output of the entire encoder. In decoder self-attention, the query, key, and value all come from the output of the previous decoder layer. However, each position in the decoder can only consider all the positions before that position. This masked attention retains the auto-regressive property, ensuring that the prediction depends only on the output tokens that have been generated.

[0095] The encoder layer formula is as follows:

[0096] e mid =LayerNorm(e input +MultiHeadAttention(e input ))

[0097] e output =LayerNorm(e mid +FFN(e mid ))

[0098] In the formula, e input represents the input of the encoder layer, e output represents the output of the encoder layer, e midRepresents the output of the first sublayer of the encoder, MultiHeadAttention(·) is a multi-head attention mechanism, FFN is a bit-by-bit feedforward neural network, and LayerNorm(·) represents the layer normalization performed by the model in this field.

[0099] Input to the encoder layer Where N is the length of the input sequence, d model is the embedding dimension size; the query vector sequence Q, key vector sequence K and value vector sequence V are all set to e input .

[0100] MultiHeadAttention(e input ) can be represented as MultiHead(Q, K, V)

[0101] MultiHead(Q,K,V)=Contact(head1,...,head i ,..head h )W O

[0102] head i =Attention(QW i Q , KW i K , VW i V )

[0103] MultiHead(Q, K, V) represents the attention matrix of all head information, Contact represents concatenation; head i represents the i-th attention matrix, h represents the number of attention heads, are all weight matrices learned during training; d q represents the query vector dimension, d k represents the dimension of the key vector, d v Indicates the value vector dimension.

[0104] The self-attention formula used is as follows:

[0105]

[0106] Softmax represents the normalized exponential function; T represents transpose.

[0107] The sequence obtained after the multi-head attention model is added to the original sequence and then layer normalized. The obtained sequence is input into the bit-by-bit feedforward neural network and layer normalized again. This process is repeated n times to obtain the output of the encoder.

[0108] In the decoder, the sequence first inputs the masked multi-head self-attention, and the i-th attention head matrix Z i Calculated as follows:

[0109]

[0110] Among them, V i Represents a value matrix, and the values ​​need to be masked before softmax is performed;

[0111] The final attention matrix is ​​calculated by the following formula:

[0112] M=Contact(Z1,...,Z i ,..Z h )W O

[0113] After the final attention matrix is ​​layer-normalized, it is input into the encoder-decoder attention layer, and the output of the encoder is also input. The subsequent steps are the same as those of the encoder. After the steps in the decoder are repeated n times, the final output of the decoder is obtained; the final output of the decoder is passed through the fully connected layer to obtain the image context vector.

[0114] Step S32: The bidirectional LSTM layer includes two LSTM layers, one that processes the input sequence from left to right in time steps, and the other that processes the input sequence from right to left in time steps. The outputs of these two LSTM layers are concatenated to form the output of the bidirectional LSTM layer.

[0115] Specifically, the bidirectional LSTM contains two LSTM layers, which process data in the forward and reverse directions of the time series, respectively, so that both past and future contextual information can be taken into account. The forward LSTM layer processes the input sequence in the forward direction of the time series (from the beginning to the end). At each time step, it updates its hidden state and cell state based on the current input and the previous hidden state. The reverse LSTM layer processes the input sequence in the reverse direction of the time series (from the end to the beginning). It works similarly to the forward LSTM layer, but it starts processing from the end of the sequence and moves forward step by step. The hidden states of the forward and reverse LSTM layers are merged (by concatenation) to form the final output. In this way, the output contains the contextual information before and after each time step. After the bidirectional LSTM, a fully connected layer is added to further process the merged features.

[0116] The mathematical expression of LSTM is as follows, assuming that there are h hidden units, the batch size is n, and the input size is d. Therefore, the input is X t ∈R n×d , the hidden state of the previous time step is H t-1 ∈Rn×h Accordingly, the gates of the time step are defined as follows: The input gate is I t ∈R n×h , the forget gate is F t ∈R n×h , the output gate is O t ∈R n×h ,like Figure 4 The calculation method is as follows:

[0117] I t =σ(X t W xi +H t-1 W hi +b i )

[0118] F t =σ(X t W xf +H t-1 W hf +b f )

[0119] O t =σ(X t W xo +H t-1 W ho +b o )

[0120]

[0121]

[0122] H t =tanh(C t )⊙O t

[0123] Among them, W xi , W xf , W xo All∈R d×h , are the d×h dimensional weight matrices of the input gate, forget gate, and output gate, respectively, W xc represents a d×h dimensional weight matrix; W hi , W hf , W ho are the weight parameters of the input gate, forget gate and output gate respectively, W hc represents the weight parameter, b i , b f and b o are the bias parameters of the input gate, forget gate, and output gate respectively; σ represents the sigmoid function, X t represents the input at time t; Represents the candidate memory state of the current time step; C t Indicates the cell state at the current time step; C t-1 represents the cell state of the previous time step; ⊙ represents the multiplication of elements in the matrix; H t Represents the hidden layer output at the current time step.

[0124] For text features, LSTM is used to obtain the context vector of the document. The forward sequence and the reverse sequence are combined through a fully connected layer to obtain the output of the bidirectional LSTM, that is, the text context vector is obtained, as shown below:

[0125]

[0126] Step S4: Using the multimodal attention layer of the attention mechanism, by processing the image context vector and the text context vector, the attention relationship between the news image features and the semantic features of the corresponding news text description is learned to achieve the fusion of text and image features;

[0127] Step S5: Use the gradient descent method and the reference summary of the news as the prediction result to supervise the optimization of the parameters of the visual attention layer and the text attention layer, as well as the parameters of the attention mechanism of the multimodal attention layer; use the Adam optimizer, set the learning rate to 5e-05, and the momentum parameters to β1=0.9, β2=0.99

[0128] Step S6: The fused features are processed by a multimodal decoder to generate a news summary and optimize the parameters of the multimodal decoder. The multimodal decoder is also based on the Transformer module and uses four stacked Transformer modules.

[0129] Step S7: Repeat the above steps until the model parameters converge to obtain the trained visual attention layer, text attention layer, multimodal attention mechanism and multimodal decoder.

[0130] The prediction process of the summary generation method for multimodal data based on deep learning is as follows Figure 3 As shown, the following steps are included:

[0131] Step S1: Obtain news sample data and perform preprocessing to obtain news pictures and news text descriptions, and scale the news pictures to a size of 224x224 pixels.

[0132] Step S2: Extract image features from news images through the trained convolutional neural network based on the residual mechanism; extract text features from news text descriptions through the trained BERT-base model.

[0133] Step S3: Send the image features to the trained visual attention layer based on the Transformer module to obtain the image context vector; send the text features to the trained text attention layer based on the bidirectional LSTM to obtain the text context vector.

[0134] Step S4: Input the image context vector and the text context vector into the multimodal attention layer to achieve the fusion of text features and image features.

[0135] Step S5: Input the fused features into the multimodal decoder to generate a summary.

[0136] During the prediction process, L groups of news are preprocessed to obtain the corresponding visual feature embedding sequence and text feature embedded sequence, which are input into the two processing modules of the dual-tower structure respectively. The trained multimodal encoder and decoder are used to finally obtain the text summary corresponding to the multimodal news.

[0137] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A method for generating summaries for multimodal data based on deep learning, characterized in that: The steps include: Step S1: Obtain news sample data and perform preprocessing to obtain news pictures and news text descriptions, and scale the news pictures to a size of 224x224 pixels; Step S2: extract image features from news images through the trained convolutional neural network based on residual mechanism; extract text features from news text descriptions through the trained BERT-base model; Step S3: Send the image features to the trained visual attention layer based on the Transformer module to obtain the image context vector; send the text features to the trained text attention layer based on the bidirectional LSTM to obtain the text context vector; Step S4: Input the image context vector and the text context vector into the multimodal attention layer to achieve the fusion of text features and image features; Step S5: Input the fused features into the multimodal decoder to generate a summary.

2. The method for generating a summary for multimodal data based on deep learning according to claim 1, characterized in that: The convolutional neural network based on the residual mechanism in step S2 has a total of 4 residual blocks, each of which contains 2 convolutional layers, plus the input layer, the first convolutional layer, the average pooling layer and the last fully connected layer, a total of 12 weight layers; the convolutional layer and the residual block are used to extract the feature information of the image, the pooling layer is used to reduce the size of the image features and improve the calculation efficiency, and the fully connected layer outputs the image feature vector.

3. The method for generating a summary for multimodal data based on deep learning according to claim 2, characterized in that: The Transformer encoder of the visual attention layer is composed of n identical layers stacked together, each layer has two sub-layers; the first sub-layer is multi-head self-attention; the second sub-layer is a position-based bit-by-bit feedforward network; the Transformer decoder is also composed of n identical layers stacked together. The first multi-head attention layer of the decoder is a masked multi-head self-attention, and the second multi-head attention of the decoder is the encoder-decoder attention layer.

4. The method for generating a summary for multimodal data based on deep learning according to claim 3, characterized in that: The encoder layer formula is as follows: in mid =LayerNorm(e input +MultiHeadAttention(e input )) in output =LayerNorm(e mid +FFN(e mid )) Among them, e input represents the input of the encoder layer, e output represents the output of the encoder layer, e mid Represents the output of the first sublayer of the encoder, MultiHeadAttention(·) is a multi-head attention mechanism, FFN is a bit-by-bit feedforward neural network, and LayerNorm(·) represents the layer normalization performed by the model in this field; Input to the encoder layer Where h is the number of attention heads, N is the length of the input sequence, and d is model is the embedding dimension size; the query vector sequence Q, key vector sequence K and value vector sequence V are all set to e input ; MultiHeadAttention(e input ) is represented as MultiHead(Q,K,V) MultiHead(Q,K,V)=Contact(head1,…,head i ,..head h )W O MultiHead(Q,K,V) represents the attention matrix of all head information, Contact represents concatenation; head i represents the i-th attention matrix, are all weight matrices learned during training; d q represents the query vector dimension, d k represents the dimension of the key vector, d v Indicates the dimension of the value vector; The self-attention formula is as follows: sofrmax represents the normalized exponential function; T represents transpose; The sequence obtained after processing by the multi-head attention model is added to the original sequence and then layer normalized. The resulting sequence is input into the bit-by-bit feedforward neural network and layer normalized again. After repeating n times, the output of the encoder is obtained.

5. The method for generating a summary for multimodal data based on deep learning according to claim 4, characterized in that: In the decoder, the sequence first inputs the masked multi-head self-attention, and the i-th attention head matrix Z i Calculated as follows: Among them, V i Represents a value matrix, and the values ​​need to be masked before softmax is performed; The final attention matrix is ​​calculated by the following formula: M=Contact(Z1,…,Z i ,..Z h )W O After the final attention matrix is ​​layer-normalized, it is input into the encoder-decoder attention layer, and the output of the encoder is also input. After the steps in the decoder are repeated n times, the final output of the decoder is obtained; the final output of the decoder is passed through the fully connected layer to obtain the image context vector.

6. The method for generating a summary for multimodal data based on deep learning according to claim 5, characterized in that: The bidirectional LSTM layer of the text attention layer consists of two LSTM layers, one that processes the input sequence from left to right in time steps, and the other that processes the input sequence from right to left in time steps; the outputs of these two LSTM layers are concatenated to form the output of the bidirectional LSTM layer; The LSTM calculation method is as follows: I t =σ(X t W xi +H t-1 W hi +b i ) F t =σ(X t W xf +H t-1 W hf +b f ) O t =σ(X t W xo +H t-1 W ho +b o ) H t =tanh(C t )⊙O t Among them, the input gate is I t ∈R n×h , the forget gate is F t ∈R n×h , the output gate is O t ∈R n×h , n is the batch size, h is the number of hidden units; W xi ,W xf ,W xo All∈R d×h , respectively, are the d×h dimensional weight matrices of the input gate, forget gate, and output gate, W xc Represents a d×h dimensional weight matrix, d is the size of the input number; W hi ,W hf ,W ho are the weight parameters of the input gate, forget gate and output gate respectively, W hc represents the weight parameter, b i , b f and b o are the bias parameters of the input gate, forget gate, and output gate respectively; σ represents the sigmoid function, X t represents the input at time t; Represents the candidate memory state of the current time step; C t Represents the memory state of the current time step; C t-1 represents the memory state of the previous time step; ⊙ represents the multiplication of elements in the matrix; H t Represents the hidden layer output at the current time step; For text features, LSTM is used to obtain the context vector of the document. The forward sequence and the reverse sequence are combined through a fully connected layer to obtain the output of the bidirectional LSTM, that is, the text context vector is obtained, as shown below:

Images