Multi-modal dialogue abstract method based on multi-level visual guidance

Abstract

The invention discloses a multi-modal dialogue abstract method based on multi-level visual guidance, and relates to the technical field of internet and artificial intelligence. The method comprises the following steps: extracting global features and local features of visual information contained in a dialogue by using a pre-trained CLIP model, performing text feature extraction on a text of the dialogue by using a pre-trained model T5, and extracting a text feature of the dialogue by using a pre-trained model T5; visual features and text features rich in deep semantic information are obtained, global visual information and local visual information are fused and aligned with the text features through a local multi-modal attention crossing module and a global multi-modal attention crossing module, and a modal fusion module is used for fusing and aligning the local visual information and the text features. According to the method, text features guided by global vision and local visual features guided by semantics are fused and spliced, so that multi-modal dialogue information can complement each other, the context of the dialogue is concerned, and the quality and accuracy of generating the abstract are improved.

Description

Technical Field

[0001] The present application relates to the fields of Internet and artificial intelligence technology, and in particular to a multimodal dialogue summarization method based on multi-level visual guidance. Background Art

[0002] In recent years, with the rapid development of information technology and human society, massive amounts of information have brought unprecedented challenges to people. Dialogue, as an indispensable way of communication in human society, is the core way of information exchange, expression of opinions and emotional communication. Due to the explosive growth of information, it has gradually become difficult for users to process. Faced with multimodal and massive dialogue information, users need to efficiently and accurately obtain the core information of the dialogue to assist in relevant decision-making or understand the problems of related scenarios. This has become a common demand of users. In the field of artificial intelligence, the application of machine learning, deep learning, and large language model technologies in natural language processing has been continuously deepened, improving the ability to process dialogue information. Against this background, dialogue summary technology came into being. The development of dialogue summary technology is not only of great significance to solving the problem of a large amount of dialogue information that is difficult to capture key information, but also plays a key role in enabling users to improve the efficiency of obtaining information and the accuracy of capturing key information. It has shown great potential in the fields of conference records, customer service systems, and social media industries, and has broad application prospects.

[0003] The task of dialogue summarization aims to extract key content from dialogue data and generate concise summaries while maintaining the core information of the original text. It can summarize dialogues containing a lot of information in a short text. With the rapid development of artificial intelligence, the generation and understanding of dialogue summaries have received extensive attention and application in various application scenarios. This task has an important position and application value. This task requires not only that the method can identify and extract core information, but also that it needs to capture the semantic information implicit in the dialogue, and it also needs to have a certain language generation ability to ensure that the generated summary is easy to understand and can accurately convey the core information of the dialogue. Dialogue summarization is considered a major challenge in summary tasks, because dialogue summaries involve changes in speakers and constant changes in conversation topics, and it is necessary to capture the implicit logic in the dialogue, and the generated summary needs to be understandable to users. In the customer service system industry, conversation summarization technology can automatically generate conversation summaries between customer service personnel and users, providing quick references for customer service personnel, helping to shorten response time and improve service quality, thereby improving services and optimizing strategies. In terms of meeting records, conversation summarization technology can help generate meeting minutes, presenting discussion points, decisions, and to-do items in a concise manner. In the social media industry, the content of user interactions is rich and complex, and conversation summarization can automatically summarize the core ideas and user feedback of the discussion, helping the platform to better understand user needs and improve community management efficiency. Conversation summarization is not only a key technology for information retrieval and information extraction, but also provides important support for other text summarization tasks. It is one of the most important tasks in the summary task.

[0004] The dialogue summarization task is to extract features from the dialogue data to be processed, and use the designed algorithm or trained model to enable the computer to automatically summarize the data. Traditionally, there are two methods for dialogue summarization: extractive summarization and generative summarization. The former directly selects important sentences or fragments from the dialogue text and combines them to form a concise dialogue summary, while the latter automatically generates new sentences to describe the core content of the dialogue by understanding and reorganizing the dialogue content. However, traditional methods may cause information redundancy, and it is difficult to accurately capture the dialogue topic and context association for long or complex dialogues. In order to solve the above problems, some researchers have combined deep learning methods, such as sequence-to-sequence methods, methods based on RNN and LSTM architecture models, etc., to improve the quality and application breadth of text summarization.

[0005] With the development of multimedia technology and the popularity of social media platforms, the information generated by user conversations has gradually become multimodal. For example, user conversations on social media platforms are usually text conversations combined with videos or images. In such scenarios, only by combining multimodal information can the key information in the conversation be captured more efficiently. However, some existing conversation summary methods are text-based methods that do not take into account the contribution of visual information, resulting in the conversation summary results being somewhat one-sided and incomplete in multimodal scenarios. To this end, in recent years, some researchers have proposed multimodal conversation summary methods, aiming to more comprehensively and accurately reflect the core content of the conversation by comprehensively utilizing information from multiple modalities such as conversations, images, and videos. Multimodal conversation summaries are different from traditional conversation summaries. Multimodal conversation summaries need to combine text and other modal data, obtain their features, fuse the features of text and other modalities to obtain the final features, and then decode the features into conversation summaries to produce more comprehensive and accurate conversation summaries. Unlike plain text conversation summaries, multimodal conversation summaries face two challenges: first, how to capture the implicit semantics of different modalities to avoid neglecting conversation information due to the complexity of multimodal information; second, how to maintain conversation memory and contextual logic as well as information from other modalities in the case of multiple rounds of conversations to ensure contribution to conversation summary generation.

[0006] In order to meet the challenges of multimodal dialogue summarization mentioned above, some methods integrate image and text features by building complex network structures; other methods use pre-trained models to improve the quality of text summaries generated by the model.

[0007] like Figure 1 As shown in the article "Aspect-oriented Multimodal Summarization Model", one of the existing technologies, mentions the use of visual information combined with text information for text summarization, proposes a multimodal pointer generation network and aspect-enhanced maximum likelihood training, and combines the aspect coverage mechanism to achieve excellent text summarization results:

[0008] First, in terms of text embedding, a pre-trained word vector model is used to convert text data into an embedded representation. The text embedding is input into the encoder of the model and encoded together with the visual features into a contextual representation. Secondly, the visual features are extracted by ResNet and the hidden states of the encoder and decoder are initialized. At the same time, the local visual features extracted by Faster R-CNN are used together with the text embedding to generate a contextual representation for multimodal fusion of the hierarchical attention mechanism. Thirdly, an aspect-oriented reward-enhanced maximum likelihood mechanism is introduced in model training to ensure that the summary covers the important aspects of the text data. The model combines text embedding and visual information to generate summary content during training, and avoids repeated description of the same aspect through the aspect coverage mechanism. Finally, the aspect consistency strategy and constrained decoding are used to avoid the same aspect from appearing repeatedly. During decoding, the text embedding and visual features work together to ensure that the generated content is consistent with the text data, thereby improving the coherence and accuracy of the summary.

[0009] like Figure 2 As shown in the article "Multimodal summary generation task based on video", one of the existing technologies, mentions the use of self-attention mechanism and global attention mechanism of dual interactive modules to capture the global and local semantics of text and video and obtain high-quality text summaries:

[0010] First, the text data and video data are encoded separately. For text, bidirectional RNN is used to encode semantic information; for video, it is divided into multiple segments, frame features are extracted using ResNet, and the temporal dependencies between segments are encoded through bidirectional RNN. Secondly, a dual interaction module is used to achieve deep interaction. The module contains a conditional self-attention mechanism and a global attention mechanism. The conditional self-attention captures the local semantic information in the video segment and highlights the key content under the guidance of the text; the global attention processes the high-level semantic relationship between the text and the video to achieve a deep fusion of the two. Thirdly, the multimodal generator generates a text summary and selects the video cover frame. The text summary generation uses an editing gate mechanism to fuse the video-aware text representation with the original text representation, and avoids the missing of the generated vocabulary through the pointer network; the cover frame selection is based on the hierarchical video representation, and the candidate frames are matched and scored using hierarchical attention, and the most representative frame is selected as the cover. Finally, the entire model is optimized through a joint loss function, including the negative log-likelihood loss of the text summary and the pairing hinge loss of the cover selection, so that the multimodal generation and frame selection promote each other and improve the generation effect.

[0011] However, the existing technologies of "aspect-oriented multimodal summary model" and "video-based multimodal summary generation task":

[0012] 1. In the multimodal summarization task, the visual feature extraction method used plays a key role in the quality of the generated summary, but the extraction method used does not fully understand the internal semantics related to the text.

[0013] 2. Previous work mainly focused on feature filtering and modal fusion on the encoder side, but ignored the fusion on the decoder side.

[0014] Therefore, there are the following disadvantages:

[0015] 1. The visual feature extraction method is not sufficiently relevant to the internal semantics of the text data, making it difficult for the extracted visual features to complement the text features semantically. Adding visual features is not necessarily conducive to the generation of text summaries and may lead to information redundancy.

[0016] 2. The lack of contextual understanding of the conversation and the capture of the information content of multimodal conversations may lead to the loss of summary information and the loss of understanding of the emotional intention implicit in the conversation. Summary of the invention

[0017] The embodiment of the present application provides a multimodal conversation summarization method based on multi-level visual guidance, uses a pre-trained CLIP model to extract global and local features of visual information contained in the conversation, uses a pre-trained model T5 to extract text features of the text of the conversation, and fuses and splices the text features guided by global vision and the local visual features guided by semantics, so that the multimodal conversation information can complement each other and pay attention to the context of the conversation, thereby improving the quality and accuracy of the generated summary.

[0018] The present application embodiment provides a multimodal dialogue summarization method based on multi-level visual guidance, which performs the following steps in the model training stage:

[0019] The dialogue text and target summary in the input multimodal dialogue data are extracted using the encoder of the T5 model to extract the text features and summary text features of the multimodal dialogue; and the image of the input multimodal dialogue data is extracted using the visual encoder of the CLIP model to obtain global visual features and local visual features, and construct an index of text features and visual features;

[0020] Input text features and local visual features into the local multimodal attention cross module, and input text features into the long short-term memory network to obtain a hidden state containing global semantic information;

[0021] In the local multimodal attention cross module, the self-attention weight is calculated based on the conditional self-attention mechanism, the conditional weight of the local visual feature is calculated by combining the hidden state containing the global semantic information with the local visual feature, and the local visual feature is weighted by the conditional weight and the self-attention weight, and the local visual feature is passed through the Transformer encoder to obtain the semantically guided local visual feature;

[0022] The text features and global visual features are input into the global multimodal attention cross module. The text features are input into the long short-term memory network to obtain the text features that capture the context information. The global visual features are input into the long short-term memory network to obtain the visual features that capture the global information.

[0023] In the global multimodal attention cross module, the cross-modal multi-head attention mechanism and the cross-modal bilinear attention fusion mechanism are used to perform a weighted sum of the text features combined with visual information obtained by the two mechanisms, and then pass through the Transformer encoder to obtain the text features guided by global vision.

[0024] The semantically guided local visual features and the global visually guided text features are input into the modal fusion module. Through the cross-modal attention mechanism, the semantically guided local visual features are used to guide the global visually guided text features to obtain the visual information guided text features; the global visually guided text features are used to guide the semantically guided local visual features to obtain the text information guided visual features; and the visual information guided text features are spliced ​​with the text information guided visual features to obtain the bidirectional enhanced visual-text features;

[0025] The bidirectional enhanced visual-text features and summary text features are input into the Transformer decoder. The decoded text features are converted into vector dimensions that can be mapped by the vocabulary through the linear layer. The obtained vector is input into the Softmax layer and converted into a probability distribution using the beam search method.

[0026] During training, the cross entropy loss is calculated between the probability distribution predicted by the model and the target summary text.

[0027] Optionally, use the encoder of the T5 model to extract text features of the multimodal dialogue and summary text features to meet the following requirements:

[0028] T i =T5(t i )

[0029] S i =T5(s i )

[0030] Among them, t i is the input of the ith dialogue, si is the target summary of the ith dialogue, T5 is the T5 model, T i Enter t for the conversation i The embedding vector S i For summary text i The embedding vector of .

[0031] Optionally, use the visual encoder of the CLIP model to obtain global and local visual features that satisfy:

[0032] V fi =CLIP vis (V i )

[0033] V fi = {V a ,V1,…,V P}

[0034] Among them, V fi To use the CLIP model visual encoder CLIP vis The visual features obtained, D v is the visual feature dimension, V a is the global visual feature of the image, V i is the local visual feature of the image.

[0035] Optionally, in the local multimodal attention cross module, calculating the self-attention weight based on the conditional self-attention mechanism, calculating the conditional weight of the local visual feature by combining the hidden state containing the global semantic information with the local visual feature includes:

[0036] In the local multimodal attention cross module, based on the conditional self-attention mechanism, the query Q, key K and value V of the local visual feature are obtained through the mapping matrix;

[0037] Compute the self-attention weight using the query Q, key K, and value V;

[0038] The hidden state containing global semantic information and the local visual features are combined to calculate the conditional weight of the local visual features.

[0039] Optionally, the text features are fed into a long short-term memory network to satisfy:

[0040]

[0041] in, is the hidden state at time step t, is the cell state at time step t, and the hidden state at the last step is The output is T s .

[0042] Optionally, in the local multimodal attention cross module, calculating the self-attention weight based on the conditional self-attention mechanism, calculating the conditional weight of the local visual feature by combining the hidden state containing the global semantic information with the local visual feature includes:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] Where W∈R d×d is the weight matrix, α i,j is the self-attention weight, β i is the conditional weight, V i ′ The semantically guided local visual feature X is obtained through the Transformer encoder v .

[0050] Optionally, the global visual features are input into the long short-term memory network, and the visual features that capture the global information include:

[0051]

[0052] in, is the hidden state at time step t, is the cell state at time step t, and the output is V s .

[0053] Optionally, in the global multimodal attention cross module, the cross-modal multi-head attention mechanism and the cross-modal bilinear attention fusion mechanism are used to perform a weighted sum of the text features combined with visual information obtained by the two mechanisms, and pass through the Transformer encoder to obtain the global visually guided text features including:

[0054]

[0055]

[0056]

[0057]

[0058] A=V s BTs

[0059] A ′ =softmax(A)

[0060] T2=A ′ T s

[0061] T=αT1+(1-α)T2

[0062] Among them, CMA stands for cross-modal multi-head attention, W is the weight matrix, B is the bilinear weight matrix, α is the trainable weight, and the output obtained after the Transformer encoder layer is X t .

[0063] Optionally, through a cross-modal attention mechanism, semantically guided local visual features are used to guide global visually guided text features to obtain visual information guided text features; global visually guided text features are used to guide semantically guided local visual features to obtain text information guided visual features; and the visual information guided text features are concatenated with the text information guided visual features to obtain bidirectional enhanced visual-text features including:

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072] O enc =concat(T ′ ,V ′ )

[0073] Among them, T ′ is the text feature guided by visual information, V ′ is the visual feature guided by text information, concat is the concatenation operation, O enc It is a bidirectional enhanced visual-text feature.

[0074] Optionally, during training, the cross entropy loss between the probability distribution predicted by the model and the target summary text is calculated by:

[0075] During training, the cross entropy loss between the probability distribution predicted by the model and the target summary text is calculated;

[0076] The cross entropy losses of all time steps are averaged as the overall average loss. The calculated average loss is back-propagated and the model parameters are updated using the gradient descent method to minimize the loss.

[0077] The cross entropy loss between the probability distribution predicted by the calculation model and the target summary text satisfies:

[0078]

[0079] Among them, y i is the true distribution of the target summary text, p i is the probability distribution predicted by the model and θ is the model parameter.

[0080] The embodiment of the present application uses the pre-trained CLIP model to extract the global features and local features of the visual information contained in the conversation, uses the pre-trained model T5 to extract text features of the text of the conversation, and fuses and splices the text features guided by global vision and the local visual features guided by semantics, so that the multimodal conversation information can complement each other and pay attention to the context of the conversation, thereby improving the quality and accuracy of the generated summary.

[0081] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0083] Figure 1 The technical process of aspect-oriented multimodal summary model for existing technologies;

[0084] Figure 2 The technical process of the existing video-based multimodal summary generation task;

[0085] Figure 3The basic process diagram of the multimodal dialogue summarization method based on multi-level visual guidance according to an embodiment of the present application is shown;

[0086] Figure 4 The following is a schematic diagram of the application process of the multimodal dialogue summarization method based on multi-level visual guidance in GPLM according to an embodiment of the present application. DETAILED DESCRIPTION

[0087] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0088] The abbreviations and key terms that appear in this embodiment are defined as follows:

[0089] BP: Back Propagation;

[0090] CLIP: Contrastive Language-Image Pre-training Contrastive Language-Image Pre-training model;

[0091] T5: Text-To-Text Transfer Transformer;

[0092] CMA: Cross-modal Multi-modal Attention cross-modal multi-head attention mechanism;

[0093] LSTM: Long Short-Term Memory;

[0094] ViT:Vision Transformer;

[0095] GPLM:Generative Pre-trained Language Model Generative pre-trained language model;

[0096] The present application embodiment provides a multi-modal dialogue summarization method based on multi-level visual guidance, such as Figure 3 , Figure 4 As shown, the following steps are performed during the model training phase:

[0097] The dialogue text and target summary in the input multimodal dialogue data are extracted using the encoder of the T5 model to extract the text features and summary text features of the multimodal dialogue; in some embodiments, the encoder of the T5 model is used to extract the text features and summary text features of the multimodal dialogue to meet the following requirements:

[0098] T i =T5(t i )

[0099] S i =T5(s i )

[0100] Among them, t i is the input of the ith dialogue, s i is the target summary of the ith dialogue, T5 is the T5 model, T i Enter t for the conversation i The embedding vector S i For summary text i In this embodiment, the text features are also encoded into a vector of 768 dimensions.

[0101] The image of the input multimodal dialogue data is used to obtain global visual features and local visual features using the visual encoder of the CLIP model, and an index of text features and visual features is constructed.

[0102] In some embodiments, the visual encoder of the CLIP model is used to obtain global visual features and local visual features that satisfy:

[0103] V fi =CLIP vis (V i )

[0104] V fi = {V a ,V1,…,V P}

[0105] Among them, V fi To use the CLIP model visual encoder CLIP vis The visual features obtained, D v is the visual feature dimension, V a is the global visual feature of the image, V i is the local visual feature of the image.

[0106] In this embodiment, the pre-trained multimodal language model CLIP is pre-trained on a dataset of 400 million image-text pairs constructed by crawling public images and their corresponding description texts on the Internet, and can capture the deep semantic information of the data. The output of the visual encoder is set to 7×7, and the selected architecture is the CLIP-vit-base-32 model. The image is converted to 224×224, and each 7×7 is encoded as a 768-dimensional vector.

[0107] The text features and local visual features are input into the local multimodal attention cross module, and the text features are input into the long short-term memory network to obtain the final hidden state containing global semantic information.

[0108] In some embodiments, inputting text features into a long short-term memory network satisfies:

[0109]

[0110] in, is the hidden state at time step t, is the cell state at time step t, and the hidden state at the last step is The output is T s .

[0111] In the local multimodal attention cross module, the self-attention weight is calculated based on the conditional self-attention mechanism, and the conditional weight of the local visual feature is calculated by combining the hidden state containing the global semantic information with the local visual feature. The local visual feature is weighted by the conditional weight and the self-attention weight, and passes through a Transformer encoder layer to obtain the semantically guided local visual feature.

[0112] In some embodiments, in the local multimodal attention cross module, calculating the self-attention weight based on the conditional self-attention mechanism, calculating the conditional weight of the local visual feature by combining the hidden state containing the global semantic information with the local visual feature includes:

[0113] In the local multimodal attention cross module, based on the conditional self-attention mechanism, the query Q, key K and value V of the local visual feature are obtained through the mapping matrix;

[0114] Compute the self-attention weight using the query Q, key K, and value V;

[0115] The conditional weights of local visual features are calculated by combining the hidden state containing global semantic information with the local visual features, and then the conditional weights of local visual features are calculated by combining the last hidden state containing global semantic information with the local visual features; finally, weighted calculation is performed through the conditional weights, self-attention weights and local visual features, and a Transformer encoder layer is passed to obtain semantically guided local visual features.

[0116] In some embodiments, in the local multimodal attention cross module, calculating the self-attention weight based on the conditional self-attention mechanism, calculating the conditional weight of the local visual feature by combining the hidden state containing the global semantic information with the local visual feature includes:

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123] Where W∈R d×d is the weight matrix, α i,j is the self-attention weight, β i is the conditional weight, V i ′ The semantically guided local visual feature X is obtained through the Transformer encoder v .

[0124] The text features and global visual features are input into the global multimodal attention cross module, and the text features are input into the long short-term memory network to obtain text features that capture contextual information; the global visual features are input into the long short-term memory network to obtain visual features that capture global information.

[0125] In some embodiments, the global visual features are input into the long short-term memory network, and obtaining the visual features that capture the global information includes:

[0126]

[0127] in, is the hidden state at time step t, is the cell state at time step t, and the output is V s .

[0128] In the global multimodal attention cross module, the cross-modal multi-head attention mechanism and the cross-modal bilinear attention fusion mechanism are used to weight the text features combined with visual information obtained by the two mechanisms, and then pass through the Transformer encoder to obtain the global visually guided text features.

[0129] In some embodiments, in the global multimodal attention cross module, a cross-modal multi-head attention mechanism and a cross-modal bilinear attention fusion mechanism are used to weight the text features combined with visual information obtained by the two mechanisms, and the global visually guided text features are obtained through a Transformer encoder, including:

[0130]

[0131]

[0132]

[0133]

[0134] A=V s BT s

[0135] A ′ =softmax(A)

[0136] T2=A ′ T s

[0137] T=αT1+(1-α)T2

[0138] Among them, CMA stands for cross-modal multi-head attention, W is the weight matrix, B is the bilinear weight matrix, α is the trainable weight, and the output obtained after the Transformer encoder layer is X t .

[0139] In the specific example, the number of global multimodal attention cross-module heads is set to 8.

[0140] The semantically guided local visual features and the global visually guided text features are input into the modal fusion module. Through the cross-modal attention mechanism, the semantically guided local visual features are used to guide the global visually guided text features to obtain the visual information guided text features; the global visually guided text features are used to guide the semantically guided local visual features to obtain the text information guided visual features; and the visual information guided text features and the text information guided visual features are spliced ​​to obtain the bidirectionally enhanced visual-text features.

[0141] In some embodiments, through a cross-modal attention mechanism, semantically guided local visual features are used to guide global visually guided text features to obtain visual information guided text features; global visually guided text features are used to guide semantically guided local visual features to obtain text information guided visual features; and the visual information guided text features are spliced ​​with the text information guided visual features to obtain bidirectional enhanced visual-text features, including:

[0142]

[0143]

[0144]

[0145]

[0146]

[0147]

[0148]

[0149]

[0150] O enc =concat(T ′ ,V ′ )

[0151] Among them, T ′ is the text feature guided by visual information, V ′ is the visual feature guided by text information, concat is the concatenation operation, O enc It is a bidirectional enhanced visual-text feature.

[0152] The bidirectional enhanced visual-text features and summary text features are input into the Transformer decoder, and the decoded text features are converted into vector dimensions that can be mapped by the vocabulary through the linear layer. The obtained vector is input into the Softmax layer, converted into a probability distribution, and the final output probability distribution is obtained by beam search. In this embodiment, the number of beam searches is set to 5.

[0153] During the training process, the cross entropy loss between the probability distribution predicted by the model and the target summary text is calculated. After training, the trained model is deployed and applied. During the training process, the cross entropy loss between the probability distribution predicted by the model and the target summary text is calculated; the cross entropy loss of all time steps is averaged as the overall average loss, the calculated average loss is back-propagated, and the model parameters are updated using the gradient descent method to minimize the loss.

[0154] In this embodiment, the learning rate is set to 1e-5 and the batch size is set to 4.

[0155] In some embodiments, during the training process, calculating the cross entropy loss between the probability distribution predicted by the model and the target summary text includes:

[0156] During the training process,

[0157] The cross entropy loss between the probability distribution predicted by the calculation model and the target summary text satisfies:

[0158]

[0159] Among them, y i is the true distribution of the target summary text, p i is the probability distribution predicted by the model and θ is the model parameter.

[0160] like Figure 4 As shown, the application example of the present application first uses the T5 and CLIP models to extract the text features of the conversation and the global and local visual features of the image respectively; then, the model sends the text features and local visual features to the local multimodal attention module, uses the conditional self-attention mechanism to calculate the weights, and generates semantically guided local visual features; at the same time, the text features and global visual features are input into the global multimodal attention module, and are fused through multi-head and bilinear attention mechanisms to generate global visually guided text features; then, these features guide each other in the modal fusion module and perform bidirectional enhancement to obtain fused visual-text features; finally, the model inputs the fused features into the Transformer decoder to obtain the probability distribution of the generated summary text. In the training case, the cross entropy loss is used to calculate the difference between the generated summary and the target summary, and the gradient descent method is used for optimization to make the generated result closer to the target summary; in the non-training case, the beam search strategy is used to select the conversation summary text with the highest probability as the result output.

[0161] The method of the present application is aimed at the task of summarizing a conversation containing visual information. The pre-trained model CLIP is used to extract the global and local features of the visual information contained in the conversation, and the pre-trained model T5 is used to extract text features of the text of the conversation, thereby obtaining a more comprehensive and deeper visual feature representation and a more semantically richer text feature representation; through a local multimodal attention cross module, a long short-term memory network and a conditional self-attention mechanism are used to capture the deep semantic association between local visual information features and text features, obtain semantically guided local visual features, and enhance the deep semantic expression of the model on visual information; through a global multimodal attention cross module, a long short-term memory network and a multimodal fusion mechanism that parallels a cross-modal multi-head attention mechanism and a cross-modal bilinear attention mechanism are used to obtain text features that capture global visual guidance, thereby ensuring the integrity and effectiveness of the information and improving the quality of the summary; in the modal fusion module, a cross-modal attention mechanism is used to perform modal fusion, which is input into the multi-head encoder-decoder attention layer of the Transformer decoder for decoding, and a beam search strategy is adopted at the end to improve the effect of modal fusion, supplement the dialogue summary information, improve the quality of the text summary, and provide a new method for the dialogue summary task.

[0162] The embodiment of the present application uses the pre-trained CLIP model to extract the global features and local features of the visual information contained in the conversation, uses the pre-trained model T5 to extract text features of the text of the conversation, and fuses and splices the text features guided by global vision and the local visual features guided by semantics, so that the multimodal conversation information can complement each other and pay attention to the context of the conversation, thereby improving the quality and accuracy of the generated summary.

[0163] It should be noted that in the various embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0164] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0165] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0166] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A multimodal dialogue summarization method based on multi-level visual guidance, characterized in that: The following steps are performed during the model training phase: The dialogue text and target summary in the input multimodal dialogue data are extracted using the encoder of the T5 model to extract the text features and summary text features of the multimodal dialogue; and the image of the input multimodal dialogue data is extracted using the visual encoder of the CLIP model to obtain global visual features and local visual features, and construct an index of text features and visual features; Input text features and local visual features into the local multimodal attention cross module, and input text features into the long short-term memory network to obtain a hidden state containing global semantic information; In the local multimodal attention cross module, the self-attention weight is calculated based on the conditional self-attention mechanism, the conditional weight of the local visual feature is calculated by combining the hidden state containing the global semantic information with the local visual feature, and the local visual feature is weighted by the conditional weight and the self-attention weight, and the local visual feature is passed through the Transformer encoder to obtain the semantically guided local visual feature; The text features and global visual features are input into the global multimodal attention cross module. The text features are input into the long short-term memory network to obtain the text features that capture the context information. The global visual features are input into the long short-term memory network to obtain the visual features that capture the global information. In the global multimodal attention cross module, the cross-modal multi-head attention mechanism and the cross-modal bilinear attention fusion mechanism are used to perform a weighted sum of the text features combined with visual information obtained by the two mechanisms, and then pass through the Transformer encoder to obtain the text features guided by global vision. The semantically guided local visual features and the global visually guided text features are input into the modal fusion module. Through the cross-modal attention mechanism, the semantically guided local visual features are used to guide the global visually guided text features to obtain the visual information guided text features; the global visually guided text features are used to guide the semantically guided local visual features to obtain the text information guided visual features; and the visual information guided text features are spliced ​​with the text information guided visual features to obtain the bidirectional enhanced visual-text features; The bidirectional enhanced visual-text features and summary text features are input into the Transformer decoder. The decoded text features are converted into vector dimensions that can be mapped by the vocabulary through the linear layer. The obtained vector is input into the Softmax layer and converted into a probability distribution using the beam search method. During training, the cross entropy loss is calculated between the probability distribution predicted by the model and the target summary text.

2. The multimodal dialogue summarization method based on multi-level visual guidance as claimed in claim 1, characterized in that: Using the encoder of the T5 model, the text features of the multimodal dialogue and the summary text features are extracted to meet the following requirements: T i =T5(t i ) S i =T5(s i ) Among them, t i is the input of the ith dialogue, s i is the target summary of the ith dialogue, T5 is the T5 model, T i Enter t for the conversation i The embedding vector S i For summary text i The embedding vector of .

3. The multimodal dialogue summarization method based on multi-level visual guidance as claimed in claim 1, characterized in that: The visual encoder of the CLIP model is used to obtain global and local visual features to meet the following requirements: In fi =CLIP vis (In i ) V fi ={V a ,V1,…,V P } Among them, V fi To use the CLIP model visual encoder CLIP vis The visual features obtained, D v is the visual feature dimension, V a is the global visual feature of the image, V i is the local visual feature of the image.

4. The multimodal dialogue summarization method based on multi-level visual guidance as claimed in claim 1, characterized in that: In the local multimodal attention cross module, the self-attention weight is calculated based on the conditional self-attention mechanism, and the conditional weight of the local visual feature is calculated by combining the hidden state containing the global semantic information with the local visual feature, including: In the local multimodal attention cross module, based on the conditional self-attention mechanism, the query Q, key K and value V of the local visual feature are obtained through the mapping matrix; Compute the self-attention weight using the query Q, key K, and value V; The hidden state containing global semantic information and the local visual features are combined to calculate the conditional weight of the local visual features.

5. The multimodal dialogue summarization method based on multi-level visual guidance as claimed in claim 4, characterized in that: Inputting text features into the long short-term memory network satisfies: in, is the hidden state at time step t, is the cell state at time step t, and the hidden state at the last step is The output is T s .

6. The multimodal dialogue summarization method based on multi-level visual guidance as claimed in claim 5, characterized in that: In the local multimodal attention cross module, the self-attention weight is calculated based on the conditional self-attention mechanism, and the conditional weight of the local visual feature is calculated by combining the hidden state containing the global semantic information with the local visual feature, including: Where W∈R d×d is the weight matrix, α i,j is the self-attention weight, β i is the conditional weight, V i ′ The semantically guided local visual feature X is obtained through the Transformer encoder v .

7. The multimodal dialogue summarization method based on multi-level visual guidance as claimed in claim 6, characterized in that: The global visual features are input into the long short-term memory network, and the visual features that capture the global information include: in, is the hidden state at time step t, is the cell state at time step t, and the output is V s .

8. The multimodal dialogue summarization method based on multi-level visual guidance as claimed in claim 7, characterized in that: In the global multimodal attention cross module, the cross-modal multi-head attention mechanism and the cross-modal bilinear attention fusion mechanism are used to weight the text features combined with visual information obtained by the two mechanisms. After passing through the Transformer encoder, the global visually guided text features are obtained, including: A=V s BT s A ′ =softmax(A) T2=A ′ T s T=αT1+(1-α)T2 Among them, CMA stands for cross-modal multi-head attention, W is the weight matrix, B is the bilinear weight matrix, α is the trainable weight, and the output obtained after the Transformer encoder layer is X t .

9. The multimodal dialogue summarization method based on multi-level visual guidance as claimed in claim 8, characterized in that: Through the cross-modal attention mechanism, semantically guided local visual features are used to guide global visually guided text features to obtain visual information-guided text features; global visually guided text features are used to guide semantically guided local visual features to obtain text information-guided visual features; The text features guided by visual information are concatenated with the visual features guided by text information to obtain bidirectional enhanced visual-text features including: O enc =concat(T ′ ,V ′ ) Among them, T ′ is the text feature guided by visual information, V ′ is the visual feature guided by text information, concat is the concatenation operation, O enc It is a bidirectional enhanced visual-text feature.

10. The multimodal dialogue summarization method based on multi-level visual guidance according to claim 8, characterized in that: During training, the cross entropy loss between the probability distribution predicted by the model and the target summary text is calculated by: During training, the cross entropy loss between the probability distribution predicted by the model and the target summary text is calculated; The cross entropy losses of all time steps are averaged as the overall average loss. The calculated average loss is back-propagated and the model parameters are updated using the gradient descent method to minimize the loss. The cross entropy loss between the probability distribution predicted by the calculation model and the target summary text satisfies: Among them, y i is the true distribution of the target summary text, p i is the probability distribution predicted by the model and θ is the model parameter.

Images