Text summary generation method, training method and device based on multimodal information

Abstract

The present application discloses a method for generating text summaries based on multimodal information, a training method, and a device thereof, and relates to the field of text processing technology. A specific implementation scheme is as follows: obtaining the encoded hidden state vector of the input text and extracting the global image feature vector of the input image; obtaining the text attention vector of the current time step, the visual attention vector of the current time step, and the decoding hidden state vector of the current time step based on the cross-modal attention mechanism; obtaining the generation probability based on the decoding hidden state vector; obtaining the text copy probability based on the text attention vector; obtaining the visual copy probability based on the association probability between the visual attention vector and the input text and the global image feature vector; obtaining the decoding probability based on the generation probability, text copy probability, and visual copy probability, and filtering the vocabulary and input text to obtain the summary word of the current time step.

Description

Technical Field

[0001] The present application relates to the field of text processing technology, and in particular to a text summary generation method, training method and device based on multimodal information. Background Art

[0002] As the internet generates more and more text data, the problem of text information overload is becoming increasingly serious. It is becoming increasingly important to streamline various types of text and express the most important information. Text summarization is an important means of doing this, aiming to convert text into a brief summary containing key information. Multimodal text generation refers to the automatic generation of a text summary based on multimodal information (such as text, images, and other information about the same object). Summary of the Invention

[0003] The present application provides a text summary generation method, training method and device based on multimodal information.

[0004] According to a first aspect of the present application, a method for generating a text summary based on multimodal information is provided, comprising:

[0005] Obtaining an encoded hidden layer state vector of an input text and extracting a global image feature vector of an input image; the input text and the input image are different modal information of the same object;

[0006] According to the encoding hidden state vector and the global image feature vector, based on a cross-modal attention mechanism, a text attention vector of the current time step, a visual attention vector of the current time step, and a decoding hidden state vector of the current time step are obtained;

[0007] Obtaining a generation probability according to the decoded hidden layer state vector; the generation probability is the probability that the decoded word is generated from the vocabulary;

[0008] Obtaining a text copy probability according to the text attention vector, wherein the text copy probability is a probability that a word generated by decoding is copied from the input text;

[0009] Obtaining a visual copy probability according to the visual attention vector and an association probability between the input text and the global image feature vector;

[0010] A decoding probability is obtained according to the generation probability, the text copy probability, and the visual copy probability, and a summary word of the current time step is obtained by filtering the vocabulary and the input text based on the decoding probability.

[0011] According to a second aspect of the present application, a training method for a text summary generation model is provided. The text summary generation model is suitable for a text summary generation task based on multimodal information. The text summary generation model includes an image encoder, a text encoder, and a decoder, including:

[0012] Inputting an input image sample into the image encoder to obtain a global image feature vector of the input image sample;

[0013] Inputting an input text sample into the encoder to obtain an encoded hidden layer state vector of the input text sample; the input text sample and the input image sample are different modal information of the same object sample;

[0014] Inputting the encoded hidden state vector and the global image feature vector into the decoder, obtaining the text attention vector of the current time step, the visual attention vector of the current time step, and the decoded hidden state vector of the current time step;

[0015] The decoder is used to obtain a generation probability based on the decoded hidden state vector, and a text copy probability is obtained based on the text attention vector; the generation probability is the probability that the decoded word is generated from the vocabulary, and the text copy probability is the probability that the decoded word is copied from the input text sample;

[0016] Obtaining a visual copy probability using the decoder based on the visual attention vector and an association probability between the input text sample and the global image feature vector;

[0017] Using the decoder to obtain a decoding probability based on the generation probability, the text copy probability, and the visual copy probability, and calculating a prediction loss value based on the decoding probability;

[0018] Obtaining an association probability between the input text sample and the global image feature vector, and calculating a cross-modal alignment loss value based on the association probability;

[0019] A model training loss value is obtained according to the prediction loss value and the cross-modal alignment loss value, and model parameters of the text summary generation model are adjusted based on the model training loss value to obtain a trained text summary generation model.

[0020] According to a third aspect of the present application, a device for generating a text summary based on multimodal information is provided, comprising:

[0021] A first acquisition module is used to obtain the encoded hidden layer state vector of the input text and extract the global image feature vector of the input image; the input text and the input image are different modal information of the same object;

[0022] A second acquisition module is configured to obtain, based on the encoding hidden state vector and the global image feature vector, a text attention vector of the current time step, a visual attention vector of the current time step, and a decoding hidden state vector of the current time step based on a cross-modal attention mechanism;

[0023] A third acquisition module is configured to acquire a generation probability based on the decoded hidden layer state vector; the generation probability is a probability that a word generated by decoding is generated from a vocabulary;

[0024] a fourth acquisition module, configured to acquire a text copy probability according to the text attention vector, wherein the text copy probability is a probability that a word generated by decoding is copied from the input text;

[0025] a fifth acquisition module, configured to acquire a visual copy probability based on the visual attention vector and an association probability between the input text and the global image feature vector;

[0026] A sixth acquisition module is configured to acquire a decoding probability according to the generation probability, the text copy probability, and the visual copy probability, and to filter the vocabulary and the input text based on the decoding probability to obtain a summary word for the current time step.

[0027] According to a fourth aspect of the present application, a training device for a text summarization generation model is provided. The text summarization generation model is suitable for a text summarization generation task based on multimodal information. The text summarization generation model includes an image encoder, a text encoder, and a decoder. The device includes:

[0028] A first acquisition module, configured to input an input image sample into the image encoder to obtain a global image feature vector of the input image sample;

[0029] A second acquisition module is configured to input an input text sample into the encoder to obtain an encoded hidden layer state vector of the input text sample; the input text sample and the input image sample are different modal information of the same object sample;

[0030] A third acquisition module is used to input the encoding hidden layer state vector and the global image feature vector into the decoder to obtain the text attention vector of the current time step, the visual attention vector of the current time step and the decoding hidden layer state vector of the current time step;

[0031] a fourth acquisition module, configured to use the decoder to acquire a generation probability based on the decoded hidden state vector, and to acquire a text copy probability based on the text attention vector; the generation probability being the probability that a word generated by decoding is generated from a vocabulary, and the text copy probability being the probability that a word generated by decoding is copied from the input text sample;

[0032] a fifth acquisition module, configured to acquire a visual copy probability using the decoder according to the visual attention vector and an association probability between the input text sample and the global image feature vector;

[0033] a sixth acquisition module, configured to acquire a decoding probability using the decoder according to the generation probability, the text copy probability, and the visual copy probability, and calculate a prediction loss value according to the decoding probability;

[0034] a seventh acquisition module, configured to obtain an association probability between the input text sample and the global image feature vector, and calculate a cross-modal alignment loss value based on the association probability;

[0035] A generation module is used to obtain a model training loss value according to the prediction loss value and the cross-modal alignment loss value, and adjust the model parameters of the text summary generation model based on the model training loss value to obtain a trained text summary generation model.

[0036] According to a fifth aspect of the present application, a computer device is provided, comprising:

[0037] at least one processor; and

[0038] a memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the text summary generation method based on multimodal information described in the first aspect, or execute the text summary generation model training method described in the second aspect.

[0040] According to a fifth aspect of the present application, a computer-readable storage medium is provided, wherein the computer instructions are used to enable the computer to execute the text summary generation method based on multimodal information described in the first aspect, or to execute the text summary generation model training method described in the second aspect.

[0041] According to the technical solution of the present application, in the process of generating a text summary, both the text attention distribution of the input text and the visual attention distribution of the input image are taken into consideration, and the text summary is generated using multimodal information, thereby improving the quality of the text summary.

[0042] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present application.

[0044] Figure 1 A flowchart of a method for generating a text summary based on multimodal information provided in an embodiment of the present application;

[0045] Figure 2 A flowchart of a method for generating a text summary based on multimodal information provided in an embodiment of the present application;

[0046] Figure 3 A flowchart of a method for generating a text summary based on multimodal information provided in an embodiment of the present application;

[0047] Figure 4 A flowchart of a method for training a text summarization model provided in an embodiment of the present application;

[0048] Figure 5 A structural block diagram of a text summary generation device based on multimodal information provided in an embodiment of the present application;

[0049] Figure 6 A structural block diagram of a training device for a text summarization model provided in an embodiment of the present application;

[0050] Figure 7 is a schematic block diagram of an example computer device for implementing embodiments of the present application. DETAILED DESCRIPTION

[0051] The following description of exemplary embodiments of the present application is made in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0052] This application proposes a method for generating a text summary based on multimodal information, a training method, and a device thereof to improve the quality of text summaries. The following describes the method for generating a text summary based on multimodal information, a training method, and a device thereof according to an embodiment of the present application with reference to the accompanying drawings.

[0053] Figure 1 The flowchart of a method for generating a text summary based on multimodal information provided in the embodiment of the present application is provided. It should be noted that the method for generating a text summary based on multimodal information proposed in the embodiment of the present application is a multimodal text generation method based on a copy mechanism. The copy mechanism refers to directly copying certain texts in the input text when generating a text summary. Figure 1 As shown, the text summary generation method based on multimodal information may include the following steps:

[0054] Step 101: Obtain the encoded hidden layer state vector of the input text and extract the global image feature vector of the input image; the input text and the input image are different modal information of the same object.

[0055] As an example, an RNN (Recurrent Neural Network) encoder can be used to obtain the encoded hidden state vector of the input text, and a pre-trained visual object recognition model can be used to obtain the global image feature vector of the input image.

[0056] Step 102: According to the encoding hidden state vector and the global image feature vector, based on the cross-modal attention mechanism, obtain the text attention vector of the current time step, the visual attention vector of the current time step, and the decoding hidden state vector of the current time step.

[0057] As an example, the text attention vector for the current time step can be obtained using the tanh function based on the decoded hidden state vector and the encoded hidden state vector of the previous time step. The visual attention vector for the current time step can be obtained using the tanh function based on the decoded hidden state vector and the global image feature vector of the previous time step. The text attention vector and the visual attention vector of the current time step are fused to obtain the context vector of the current time step. The decoder can be used to obtain the decoded hidden state vector of the current time step based on the context vector of the current time step, the decoded hidden state vector of the previous time step, and the standard decoding answer of the previous time step.

[0058] Step 103: Obtain a generation probability based on the decoded hidden layer state vector; the generation probability is the probability that the decoded word is generated from the vocabulary.

[0059] As an example, according to the decoded hidden state vector, the generation probability P can be obtained by formula (1): vocab (w).

[0060] P vocab (w)=softmax(W a s t ) (1)

[0061] Among them, w is the predicted word of the current time step, softmax is the softmax function, W a is the word vector matrix, s t is the decoded hidden state vector.

[0062] Step 104: Obtain a text copy probability based on the text attention vector. The text copy probability is the probability that the decoded word is copied from the input text.

[0063] As an example, we can use the text attention vector The text copy probability is obtained by formula (2)

[0064]

[0065] Among them, w is the predicted word of the current time step, i:w i =w represents the i-th predicted word in the input text at the current time step, is the text attention vector of the current time step.

[0066] Step 105 : Obtain a visual copy probability based on the visual attention vector and the association probability between the input text and the global image feature vector.

[0067] Optionally, in some embodiments of the present application, based on the alignment information of the input text and the input image, an extended modal association model can be used to obtain the association probability between the input text and the global image feature vector. Based on the association probability and the visual attention vector, the visual copy probability is obtained.

[0068] Step 106 , obtaining a decoding probability based on the generation probability, the text copy probability, and the visual copy probability, and filtering the vocabulary and the input text based on the decoding probability to obtain a summary word for the current time step.

[0069] Optionally, in some embodiments of the present application, the generation probability, text copy probability, and visual copy probability may be weighted to obtain a decoding probability. Based on the decoding probability, the vocabulary and input text are screened to obtain the summary word for the current time step.

[0070] According to the text summary generation method based on multimodal information in an embodiment of the present application, in the process of generating the text summary, both the text attention distribution of the input text and the visual attention distribution of the input image are taken into consideration, and the text summary is generated by utilizing multimodal information, thereby improving the quality of the text summary.

[0071] Figure 2 The flowchart of a method for generating a text summary based on multimodal information provided in the embodiment of the present application is as follows. Figure 2As shown, the text summary generation method based on multimodal information may include the following steps:

[0072] Step 201: Obtain the encoded hidden layer state vector of the input text and extract the global image feature vector of the input image; the input text and the input image are different modal information of the same object.

[0073] Step 202: Obtain the text attention vector of the current time step based on the decoding hidden layer state vector and the encoding hidden layer state vector of the previous time step.

[0074] As an example, the text attention vector of the i-th word in the input text at the current time step can be obtained by formula (3) and formula (4):

[0075] e t，i =tanh(W1S t-1 +V1h i ) (3)

[0076]

[0077] Among them, tanh is the tanh function, W1 is the parameter matrix, s t-1 is the decoding hidden state vector of the previous time step, V1 is the parameter matrix, h i is the encoded hidden state vector of the i-th word in the input text, and softmax is the softmax function.

[0078] Step 203: Obtain the visual attention vector of the current time step based on the decoded hidden layer state vector and the global image feature vector of the previous time step.

[0079] As an example, the visual attention vector of the j-th image feature at the current time step can be obtained by formula (5) and formula (6):

[0080] b t，j =tanh(W2S t-1 +V2v j ) (5)

[0081]

[0082] Among them, tanh is the tanh function, W2 is the parameter matrix, s t-1 is the decoding hidden state vector of the previous time step, V2 is the parameter matrix, v j is the jth image feature in the global image feature v, and softmax is the softmax function.

[0083] Step 204: Obtain the context vector of the current time step based on the text attention vector of the current time step and the visual attention vector of the current time step.

[0084] As an example, the context vector c of the current time step can be obtained by formula (7) t .

[0085]

[0086] in, The text attention vector of the i-th word in the input text at the current time step, h i is the encoded hidden state vector of the i-th word in the input text, is the visual attention vector of the jth image feature at the current time step, v j is the jth image feature in the global image feature v.

[0087] Step 205 : Obtain the decoding hidden layer state vector of the current time step according to the context vector of the current time step, the decoding hidden layer state vector of the previous time step, and the standard decoding answer of the previous time step.

[0088] As an example, the decoder can be used to obtain the decoded hidden layer state vector of the current time step, as shown in formula (8).

[0089] s t =f dec (s t-1 , C t ,y t-1 ) (8)

[0090] Among them, s t is the decoded hidden state vector of the current time step, f dec is the decoder function, s t-1 is the decoded hidden state vector of the previous time step, c t is the context vector of the current time step, y t-1 is the standard decoding answer for the previous time step.

[0091] Step 206: Obtain a generation probability based on the decoded hidden layer state vector; the generation probability is the probability that the decoded word is generated from the vocabulary.

[0092] Step 207: Obtain a text copy probability based on the text attention vector. The text copy probability is the probability that the decoded word is copied from the input text.

[0093] Step 208 : Obtain a visual copy probability based on the visual attention vector and the association probability between the input text and the global image feature vector.

[0094] Step 209 : Obtain a decoding probability based on the generation probability, the text copy probability, and the visual copy probability, and filter the vocabulary and the input text based on the decoding probability to obtain a summary word for the current time step.

[0095] In the embodiment of the present application, step 201 and step 206 to step 209 can be implemented respectively using any of the methods in the embodiments of the present application. The present application does not make any specific limitations on this and will not elaborate on it.

[0096] According to the text summary generation method based on multimodal information of the embodiment of the present application, the text attention vector and the visual attention vector are obtained based on the encoded hidden state vector of the input text, the global image feature vector of the input image, and the decoded hidden state vector of the previous time step. According to the text attention vector of the current time step, the visual attention vector of the current time step, the decoded hidden state vector of the previous time step, and the standard decoding answer of the previous time step, the decoded hidden state vector of the current time step is obtained to obtain the generation probability. According to the generation probability, the text copy probability and the visual copy probability, the decoding probability is obtained, and based on the decoding probability, the vocabulary and the input text are screened to obtain the summary word of the current time step. In the process of generating the text summary, both the text attention distribution of the input text and the visual attention distribution of the input image are taken into account, and the text summary is generated by using multimodal information, thereby improving the quality of the text summary.

[0097] Figure 3 The flowchart of a method for generating a text summary based on multimodal information provided in the embodiment of the present application is as follows. Figure 3 As shown, the text summary generation method based on multimodal information may include the following steps:

[0098] Step 301: Obtain the encoded hidden layer state vector of the input text and extract the global image feature vector of the input image; the input text and input image are different modal information of the same object.

[0099] Step 302: According to the encoding hidden state vector and the global image feature vector, based on the cross-modal attention mechanism, obtain the text attention vector of the current time step, the visual attention vector of the current time step, and the decoding hidden state vector of the current time step.

[0100] Step 303: Obtain a generation probability based on the decoded hidden layer state vector; the generation probability is the probability that the decoded word is generated from the vocabulary.

[0101] Step 304: Obtain a text copy probability based on the text attention vector. The text copy probability is the probability that the decoded word is copied from the input text.

[0102] Step 305 : Obtain the association probability between the input text and the global image feature vector based on the alignment information of the input text and the input image.

[0103] As an example, the association probability A(v j ,x i ).

[0104] A(v j , x i )=σ(u5v j +u6x i ) (9)

[0105] Among them, σ is the sigmoid function, u5 is the model parameter matrix, v j is the jth image feature in the global image feature v, u6 is the model parameter matrix, x i is the i-th word in the input text.

[0106] Step 306: Obtain the visual copy probability based on the association probability and the visual attention vector.

[0107] Optionally, in some embodiments of the present application, the visual copy probability can be calculated by formula (10):

[0108]

[0109] Among them, w is the predicted word at the current time step, is the visual copy probability, x i is the i-th word in the input text, v j is the jth image feature in the global image feature vector v, is the visual attention distribution of the j-th image feature at the current time step, A(v j ,x i ) is the association probability between the j-th image feature and the i-th word.

[0110] Step 307 : Obtain a decoding probability based on the generation probability, the text copy probability, and the visual copy probability, and filter the vocabulary and the input text based on the decoding probability to obtain a summary word for the current time step.

[0111] Optionally, in some embodiments of the present application, the decoding probability P(w) may be calculated using formula (11):

[0112]

[0113] Among them, p gen =σ(u3s t-1), σ is the sigmoid function, u3 is the model parameter matrix, s t-1 is the decoded hidden state vector of the previous time step, w is the predicted word of the current time step, is the visual copy probability, is the probability of text copying, P vocab is the generation probability, p t =σ(u4s t-1 ), u4 is the model parameter matrix.

[0114] In the embodiment of the present application, steps 301 to 304 can be implemented in any of the embodiments of the present application, and the present application does not make any specific limitations on this and will not elaborate on it.

[0115] According to the text summary generation method based on multimodal information of an embodiment of the present application, the generation probability and text copy probability are obtained based on the encoded hidden state vector of the input text and the global image feature vector of the input image. Based on the alignment information of the input text and the input image, the association probability between the input text and the global image feature vector is obtained. According to the association probability and the visual attention vector, the visual copy probability is obtained. According to the generation probability, the text copy probability and the visual copy probability, the decoding probability is obtained, and based on the decoding probability, the vocabulary and the input text are screened to obtain the summary word of the current time step. In the process of generating the text summary, both the text attention distribution of the input text and the visual attention distribution of the input image are taken into account, and the text summary is generated by using multimodal information, thereby improving the quality of the text summary.

[0116] Figure 4 This is a flow chart of a training method for a text summary generation model provided in accordance with an embodiment of the present application. It should be noted that the text summary generation module is suitable for text summary generation tasks based on multimodal information, and the text summary generation model includes an image encoder, a text encoder, and a decoder. Figure 4 As shown, the training method of the text summary generation model may include the following steps:

[0117] Step 401: Input an input image sample to an image encoder to obtain a global image feature vector of the input image sample.

[0118] Step 402: Input the input text sample to the encoder to obtain the encoded hidden state vector of the input text sample; the input text sample and the input image sample are different modal information of the same object sample.

[0119] Step 403: Input the encoded hidden state vector and the global image feature vector into the decoder to obtain the text attention vector of the current time step, the visual attention vector of the current time step, and the decoded hidden state vector of the current time step.

[0120] In step 404, a decoder is used to obtain a generation probability based on the decoded hidden state vector, and a text copy probability is obtained based on the text attention vector; the generation probability is the probability that the decoded word is generated from the vocabulary, and the text copy probability is the probability that the decoded word is copied from the input text sample.

[0121] Step 405 : Using a decoder, a visual copy probability is obtained based on the visual attention vector and the association probability between the input text sample and the global image feature vector.

[0122] Optionally, in some embodiments of the present application, the visual copy probability can be calculated by formula (12):

[0123]

[0124] Among them, w is the predicted word at the current time step, is the visual copy probability, x i is the i-th word in the input text, v j is the jth image feature in the global image feature vector v, is the visual attention distribution of the j-th image feature at the current time step, A(v j ,x i ) is the association probability between the j-th image feature and the i-th word.

[0125] Step 406 : Using a decoder to obtain a decoding probability based on the generation probability, the text copy probability, and the visual copy probability, and calculating a prediction loss value based on the decoding probability.

[0126] It should be noted that after obtaining the decoding probability based on the generation probability, text copy probability and visual copy probability, the decoder is used to filter the vocabulary and input text based on the decoding probability to obtain the summary word of the current time step, that is, the standard decoding answer of the current time step. t Calculate the prediction loss value Please refer to formula (13).

[0127]

[0128] Step 407 : Obtain the association probability between the input text sample and the global image feature vector, and calculate the cross-modal alignment loss value based on the association probability.

[0129] Optionally, in some embodiments of the present application, the cross-modal alignment loss value can be calculated by formula (14):

[0130]

[0131] Among them, A(v j ,x i ) is the association probability between the j-th image feature and the i-th word, v j is the jth image feature in the global image feature vector v, l j v j The corresponding text label, x i is the i-th word in the input text.

[0132] In step 408 , a model training loss value is obtained according to the prediction loss value and the cross-modal alignment loss value, and model parameters of the text summary generation model are adjusted based on the model training loss value to obtain a trained text summary generation model.

[0133] Optionally, in the embodiment of the present application, the predicted loss value and cross-modal alignment loss Sum to get the model training loss value As an example, refer to formula (15).

[0134]

[0135] According to the training method of the text summary generation model disclosed in this application, the generation probability, text copy probability, and visual copy probability are obtained based on the global image feature vector of the input image sample and the encoded hidden state vector of the input text sample. Based on the generation probability, text copy probability, and visual copy probability, the model training loss value is obtained, and the model parameters of the text summary generation model are adjusted to obtain a trained text summary generation model, which can generate high-quality text summaries based on multimodal information.

[0136] Figure 5 This is a structural block diagram of a text summary generation device based on multimodal information provided in accordance with an embodiment of the present application. Figure 5 As shown, the apparatus may include a first acquisition module 501 , a second acquisition module 502 , a third acquisition module 503 , a fourth acquisition module 504 , a fifth acquisition module 505 and a sixth acquisition module 506 .

[0137] Specifically, the first acquisition module 501 is used to obtain the encoded hidden layer state vector of the input text and extract the global image feature vector of the input image; the input text and the input image are different modal information of the same object.

[0138] The second acquisition module 502 is used to obtain the text attention vector of the current time step, the visual attention vector of the current time step and the decoding hidden state vector of the current time step based on the encoding hidden state vector and the global image feature vector and based on the cross-modal attention mechanism.

[0139] The third acquisition module 503 is used to acquire a generation probability according to the decoded hidden layer state vector; the generation probability is the probability that the decoded word is generated from the vocabulary.

[0140] The fourth acquisition module 504 is used to obtain the text copy probability according to the text attention vector, where the text copy probability is the probability that the decoded word is copied from the input text.

[0141] The fifth acquisition module 505 is configured to acquire a visual copy probability based on the visual attention vector and the association probability between the input text and the global image feature vector.

[0142] The sixth acquisition module 506 is configured to acquire a decoding probability based on the generation probability, the text copy probability, and the visual copy probability, and to filter the vocabulary and the input text based on the decoding probability to obtain a summary word for the current time step.

[0143] In some embodiments of the present application, the second acquisition module 502 is specifically used to: obtain the text attention vector of the current time step based on the decoded hidden state vector and the encoded hidden state vector of the previous time step; obtain the visual attention vector of the current time step based on the decoded hidden state vector and the global image feature vector of the previous time step; obtain the context vector of the current time step based on the text attention vector of the current time step and the visual attention vector of the current time step; obtain the decoding hidden state vector of the current time step based on the context vector of the current time step, the decoding hidden state vector of the previous time step and the standard decoding answer of the previous time step.

[0144] In some embodiments of the present application, the fifth acquisition module 505 is specifically used to: obtain the association probability between the input text and the global image feature vector based on the alignment information of the input text and the input image; and obtain the visual copy probability based on the association probability and the visual attention vector.

[0145] In some embodiments of the present application, the visual copy probability The calculation formula can refer to formula (16):

[0146]

[0147] Among them, w is the predicted word at the current time step, is the visual copy probability, x i is the i-th word in the input text, v jis the jth image feature in the global image feature vector v, is the visual attention distribution of the j-th image feature at the current time step, A(v j ,x i ) is the association probability between the j-th image feature and the i-th word.

[0148] In some embodiments of the present application, the calculation formula of the decoding probability P(w) can refer to formula (17):

[0149]

[0150] Among them, p gen =σ(u3s t-1 ), σ is the sigmoid function, u3 is the model parameter matrix, s t-1 is the decoded hidden state vector of the previous time step, w is the predicted word of the current time step, is the visual copy probability, is the probability of text copying, P vocab is the generation probability, p t =σ(u4s t-1 ), u4 is the model parameter matrix.

[0151] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method and will not be elaborated on here.

[0152] According to the text summary generation device based on multimodal information in an embodiment of the present application, in the process of generating a text summary, both the text attention distribution of the input text and the visual attention distribution of the input image are taken into consideration, and the text summary is generated by utilizing multimodal information, thereby improving the quality of the text summary.

[0153] Figure 6 This is a structural block diagram of a training device for a text summary generation model provided in an embodiment of the present application. Figure 6 As shown, the apparatus may include a first acquisition module 601 , a second acquisition module 602 , a third acquisition module 603 , a fourth acquisition module 604 , a fifth acquisition module 605 , a sixth acquisition module 606 , a seventh acquisition module 607 and a generation module 608 .

[0154] Specifically, the first acquisition module 601 is configured to input an input image sample into an image encoder to obtain a global image feature vector of the input image sample.

[0155] The second acquisition module 602 is used to input the input text sample into the encoder to obtain the encoded hidden state vector of the input text sample; the input text sample and the input image sample are different modal information of the same object sample.

[0156] The third acquisition module 603 is used to input the encoded hidden state vector and the global image feature vector into the decoder to obtain the text attention vector of the current time step, the visual attention vector of the current time step and the decoded hidden state vector of the current time step.

[0157] The fourth acquisition module 604 is used to use the decoder to obtain the generation probability based on the decoded hidden state vector, and to obtain the text copy probability based on the text attention vector; the generation probability is the probability that the decoded word is generated from the vocabulary, and the text copy probability is the probability that the decoded word is copied from the input text sample.

[0158] The fifth acquisition module 605 is configured to acquire a visual copy probability using a decoder according to the visual attention vector and the association probability between the input text sample and the global image feature vector.

[0159] The sixth acquisition module 606 is configured to acquire a decoding probability using a decoder according to the generation probability, the text copy probability, and the visual copy probability, and calculate a prediction loss value according to the decoding probability.

[0160] The seventh acquisition module 607 is used to obtain the association probability between the input text sample and the global image feature vector, and calculate the cross-modal alignment loss value according to the association probability.

[0161] The generation module 608 is used to obtain the model training loss value according to the prediction loss value and the cross-modal alignment loss value, and adjust the model parameters of the text summary generation model based on the model training loss value to obtain a trained text summary generation model.

[0162] In some embodiments of the present application, the visual copy probability The calculation formula of can refer to formula (18):

[0163]

[0164] Among them, w is the predicted word at the current time step, is the visual copy probability, x i is the i-th word in the input text, v j is the jth image feature in the global image feature vector v, is the visual attention distribution of the j-th image feature at the current time step, A(v j ,x i) is the association probability between the j-th image feature and the i-th word.

[0165] In some embodiments of the present application, the calculation formula of the decoding probability P(w) can refer to formula (19):

[0166]

[0167] Among them, p gen =σ(u3s t-1 ), σ is the sigmoid function, u3 is the model parameter matrix, s t-1 is the decoded hidden state vector of the previous time step, w is the predicted word of the current time step, is the visual copy probability, is the probability of text duplication, P vocab is the generation probability, p t =σ(u4s t-1 ), u4 is the model parameter matrix.

[0168] In some embodiments of the present application, the generation module 608 is further used to: sum the prediction loss value and the cross-modal alignment loss value to obtain a model training loss value.

[0169] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method and will not be elaborated on here.

[0170] According to the training device for the text summary generation model of the embodiment of the present application, the generation probability, text copy probability, and visual copy probability are obtained based on the global image feature vector of the input image sample and the encoded hidden state vector of the input text sample. Based on the generation probability, text copy probability, and visual copy probability, the model training loss value is obtained, and the model parameters of the text summary generation model are adjusted to obtain a trained text summary generation model, which can generate high-quality text summaries based on multimodal information.

[0171] Based on the embodiments of the present application, the present application also provides a computer device, at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the text summary generation method based on multimodal information or the text summary generation model training method of any of the aforementioned embodiments.

[0172] Based on the embodiments of the present application, the present application also provides a computer-readable storage medium, wherein computer instructions are used to enable a computer to execute a text summary generation method based on multimodal information or a text summary generation model training method provided according to the embodiments of the present application.

[0173] Figure 7 A schematic block diagram of an example computer device that can be used to implement embodiments of the present application is shown. Computer device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present application as described and / or claimed herein.

[0174] like Figure 7 As shown, computer device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of device 700 can also be stored in RAM 703. Computing unit 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0175] Multiple components in computer device 700 are connected to I / O interface 705, including: input unit 706, such as a keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as a magnetic disk, optical disk, etc.; and communication unit 709, such as a network card, modem, wireless communication transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0176] The computing unit 701 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the method for generating a text summary based on multimodal information or the method for training a text summary generation model. For example, in some embodiments, the method for generating a text summary based on multimodal information or the method for training a text summary generation model can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method for generating a text summary based on multimodal information or the method for training a text summary generation model described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured in any other appropriate manner (for example, by means of firmware) to execute a text summary generation method based on multimodal information or a training method for a text summary generation model.

[0177] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0178] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0179] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0180] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0181] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0182] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0183] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved. This is not a limitation herein.

[0184] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A method for generating text summarization based on multimodal information, characterized in that: include: Obtaining an encoded hidden layer state vector of an input text and extracting a global image feature vector of an input image; the input text and the input image are different modal information of the same object; According to the encoding hidden state vector and the global image feature vector, based on a cross-modal attention mechanism, a text attention vector of the current time step, a visual attention vector of the current time step, and a decoding hidden state vector of the current time step are obtained; Obtaining a generation probability according to the decoded hidden layer state vector; the generation probability is the probability that the decoded word is generated from the vocabulary; Obtaining a text copy probability according to the text attention vector, wherein the text copy probability is a probability that a word generated by decoding is copied from the input text; Obtaining a visual copy probability according to the visual attention vector and an association probability between the input text and the global image feature vector; A decoding probability is obtained according to the generation probability, the text copy probability, and the visual copy probability, and a summary word of the current time step is obtained by filtering the vocabulary and the input text based on the decoding probability.

2. The method according to claim 1, wherein The method of obtaining a text attention vector at a current time step, a visual attention vector at a current time step, and a decoding hidden state vector at a current time step based on the encoding hidden state vector and the global image feature vector and a cross-modal attention mechanism includes: Obtaining a text attention vector for the current time step based on the decoded hidden layer state vector and the encoded hidden layer state vector of the previous time step; Obtaining a visual attention vector for a current time step based on the decoded hidden layer state vector of the previous time step and the global image feature vector; Obtaining a context vector for the current time step according to the text attention vector for the current time step and the visual attention vector for the current time step; The decoding hidden layer state vector of the current time step is obtained according to the context vector of the current time step, the decoding hidden layer state vector of the previous time step and the standard decoding answer of the previous time step.

3. The method according to claim 1, wherein The obtaining of a visual copy probability according to the visual attention vector and an association probability between the input text and the global image feature vector includes: Based on the alignment information of the input text and the input image, obtaining an association probability between the input text and the global image feature vector; A visual copy probability is obtained according to the association probability and the visual attention vector.

4. The method according to claim 3, wherein The calculation formula of the visual copy probability is: Among them, w is the predicted word at the current time step, is the visual copy probability, x i is the i-th word in the input text, v j is the jth image feature in the global image feature vector v, is the visual attention distribution of the j-th image feature at the current time step, A(v j ,x i ) is the association probability between the j-th image feature and the i-th word.

5. The method according to any one of claims 1 to 4, characterized in that The calculation formula of the decoding probability is: Among them, p gen =σ(u3s t-1 ), σ is the sigmoid function, u3 is the model parameter matrix, s t-1 is the decoded hidden state vector of the previous time step, w is the predicted word of the current time step, is the visual copy probability, is the probability of text copying, P vocab is the generation probability, p t =σ(u4s t-1 ), u4 is the model parameter matrix.

6. A training method for a text summarization model, characterized in that: The text summary generation model is suitable for a text summary generation task based on multimodal information. The text summary generation model includes an image encoder, a text encoder, and a decoder. The training method includes: Inputting an input image sample into the image encoder to obtain a global image feature vector of the input image sample; Inputting an input text sample into the encoder to obtain an encoded hidden layer state vector of the input text sample; the input text sample and the input image sample are different modal information of the same object sample; Inputting the encoded hidden state vector and the global image feature vector into the decoder, obtaining the text attention vector of the current time step, the visual attention vector of the current time step, and the decoded hidden state vector of the current time step; The decoder is used to obtain a generation probability based on the decoded hidden state vector, and a text copy probability is obtained based on the text attention vector; the generation probability is the probability that the decoded word is generated from the vocabulary, and the text copy probability is the probability that the decoded word is copied from the input text sample; Obtaining a visual copy probability using the decoder based on the visual attention vector and an association probability between the input text sample and the global image feature vector; Using the decoder to obtain a decoding probability based on the generation probability, the text copy probability, and the visual copy probability, and calculating a prediction loss value based on the decoding probability; Obtaining an association probability between the input text sample and the global image feature vector, and calculating a cross-modal alignment loss value based on the association probability; A model training loss value is obtained according to the prediction loss value and the cross-modal alignment loss value, and model parameters of the text summary generation model are adjusted based on the model training loss value to obtain a trained text summary generation model.

7. The method according to claim 6, wherein The calculation formula of the visual copy probability is: Among them, w is the predicted word at the current time step, is the visual copy probability, x i is the i-th word in the input text sample, v j is the jth image feature in the global image feature vector v, is the visual attention distribution of the j-th image feature at the current time step, A(v j ,x i ) is the association probability between the j-th image feature and the i-th word.

8. The method according to claim 6, wherein The calculation formula of the decoding probability is: Among them, p gen =σ(u3s t-1 ), σ is the sigmoid function, u3 is the model parameter matrix, s t-1 is the decoded hidden state vector of the previous time step, w is the predicted word of the current time step, is the visual copy probability, is the probability of text copying, P vocab is the generation probability, p t =σ(u4s t-1 ), u4 is the model parameter matrix.

9. The method according to any one of claims 6 to 8, characterized in that The obtaining of a model training loss value according to the prediction loss value and the cross-modal alignment loss value includes: The prediction loss value and the cross-modal alignment loss value are summed to obtain a model training loss value.

10. A text summary generation device based on multimodal information, characterized in that: include: A first acquisition module is used to obtain the encoded hidden layer state vector of the input text and extract the global image feature vector of the input image; the input text and the input image are different modal information of the same object; A second acquisition module is configured to obtain, based on the encoding hidden state vector and the global image feature vector, a text attention vector of the current time step, a visual attention vector of the current time step, and a decoding hidden state vector of the current time step based on a cross-modal attention mechanism; A third acquisition module is configured to acquire a generation probability based on the decoded hidden layer state vector; the generation probability is a probability that a word generated by decoding is generated from a vocabulary; a fourth acquisition module, configured to acquire a text copy probability according to the text attention vector, wherein the text copy probability is a probability that a word generated by decoding is copied from the input text; a fifth acquisition module, configured to acquire a visual copy probability based on the visual attention vector and an association probability between the input text and the global image feature vector; A sixth acquisition module is configured to acquire a decoding probability according to the generation probability, the text copy probability, and the visual copy probability, and to filter the vocabulary and the input text based on the decoding probability to obtain a summary word for the current time step.

11. The device according to claim 10, wherein The second acquisition module is specifically used for: Obtaining a text attention vector for the current time step based on the decoded hidden layer state vector and the encoded hidden layer state vector of the previous time step; Obtaining a visual attention vector for a current time step based on the decoded hidden layer state vector of the previous time step and the global image feature vector; Obtaining a context vector for the current time step according to the text attention vector for the current time step and the visual attention vector for the current time step; The decoding hidden layer state vector of the current time step is obtained according to the context vector of the current time step, the decoding hidden layer state vector of the previous time step and the standard decoding answer of the previous time step.

12. The device according to claim 10, wherein The fifth acquisition module is specifically used for: Based on the alignment information of the input text and the input image, obtaining an association probability between the input text and the global image feature vector; A visual copy probability is obtained according to the association probability and the visual attention vector.

13. A training device for a text summarization model, characterized in that: The text summary generation model is suitable for text summary generation tasks based on multimodal information, and the text summary generation model includes an image encoder, a text encoder, and a decoder; the device includes: A first acquisition module, configured to input an input image sample into the image encoder to obtain a global image feature vector of the input image sample; A second acquisition module is configured to input an input text sample into the encoder to obtain an encoded hidden layer state vector of the input text sample; the input text sample and the input image sample are different modal information of the same object sample; A third acquisition module is used to input the encoding hidden layer state vector and the global image feature vector into the decoder to obtain the text attention vector of the current time step, the visual attention vector of the current time step and the decoding hidden layer state vector of the current time step; a fourth acquisition module, configured to use the decoder to acquire a generation probability based on the decoded hidden state vector, and to acquire a text copy probability based on the text attention vector; the generation probability being the probability that a word generated by decoding is generated from a vocabulary, and the text copy probability being the probability that a word generated by decoding is copied from the input text sample; a fifth acquisition module, configured to acquire a visual copy probability using the decoder according to the visual attention vector and an association probability between the input text sample and the global image feature vector; a sixth acquisition module, configured to acquire a decoding probability using the decoder according to the generation probability, the text copy probability, and the visual copy probability, and calculate a prediction loss value according to the decoding probability; a seventh acquisition module, configured to obtain an association probability between the input text sample and the global image feature vector, and calculate a cross-modal alignment loss value based on the association probability; A generation module is used to obtain a model training loss value according to the prediction loss value and the cross-modal alignment loss value, and adjust the model parameters of the text summary generation model based on the model training loss value to obtain a trained text summary generation model.

14. The device according to claim 13, wherein The generation module is further configured to: The prediction loss value and the cross-modal alignment loss value are summed to obtain a model training loss value.

15. A computer device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

16. A computer-readable storage medium, characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

Images