A multi-modal dialogue summarization method based on multi-level visual guidance

Abstract

The application discloses a multi-level visual guidance multi-modal dialogue summary method, relates to the technical field of Internet and artificial intelligence, and uses a pre-trained CLIP model to extract global features and local features of visual information contained in a dialogue, uses a pre-trained model T5 to perform text feature extraction on text of the dialogue, obtains visual features and text features rich in deep semantic information, fuses and aligns global visual information and local visual information with text features through a local multi-modal attention cross module and a global multi-modal attention cross module, and fuses and splices text features guided by global vision and local visual features guided by semantics through a modal fusion module, so that multi-modal dialogue information can be complementary to each other, and the context of the dialogue is paid attention to, thereby improving the quality and accuracy of generated summaries.

Description

Technical Field

[0001] This application relates to the fields of Internet and artificial intelligence technologies, and in particular to a multimodal dialogue summarization method based on multi-level visual guidance. Background Technology

[0002] In recent years, with the rapid development of information technology and human society, the massive amount of information has brought unprecedented challenges to people. Dialogue, as an indispensable mode of communication in human society, is a core method for information exchange, expression of viewpoints, and emotional communication. However, due to the explosive growth of information, it has gradually become difficult for users to process. Faced with multimodal and massive amounts of dialogue information, users need to efficiently and accurately extract the core information of the dialogue to assist in relevant decision-making or understand relevant scenarios. This has become a common user need. Meanwhile, the application of machine learning, deep learning, and large language model technology in natural language processing in the field of artificial intelligence is constantly deepening, improving the ability to process dialogue information. Against this backdrop, dialogue summarization technology has emerged. The development of dialogue summarization technology is not only significant in solving the problem of large amounts of dialogue information where key information is difficult to capture, but also plays a crucial role in improving users' efficiency in obtaining information and the accuracy of capturing key information. It has shown great potential in fields such as meeting recording, customer service systems, and the social media industry, and has broad application prospects.

[0003] Dialogue summarization aims to extract key content from dialogue data and generate concise summaries while preserving the core information of the original text. These summaries can summarize a dialogue containing a wealth of information in a short text. With the rapid development of artificial intelligence, dialogue summarization generation and understanding have received widespread attention and application in various scenarios, highlighting its significant importance and value. This task requires methods not only to identify and extract core information but also to capture the implicit semantic information within the dialogue and possess a certain level of language generation capability to ensure that the generated summary is easy to understand and accurately conveys the core information of the dialogue. Dialogue summarization is considered a major challenge in summarization tasks because it involves changes in speakers and constantly shifting topics, requiring the capture of implicit logic within the dialogue, and ensuring that the generated summary is understandable to the user. In the customer service industry, conversation summarization technology can automatically generate summaries of conversations between customer service personnel and users, providing staff with quick references to help shorten response times and improve service quality, thereby enabling service improvement and strategy optimization. In meeting minutes, conversation summarization technology can help generate meeting minutes, presenting key discussion points, decisions, and to-do items in a concise manner. In the social media industry, where user interactions are rich and complex, conversation summarization can automatically summarize core viewpoints and user feedback, helping platforms better understand user needs and improve community management efficiency. Conversation summarization is not only a key technology for information retrieval and extraction but also provides important support for other text summarization tasks, making it one of the most important summarization tasks.

[0004] Dialogue summarization involves extracting features from dialogue data and using designed algorithms or trained models to enable computers to automatically summarize the data. Traditionally, dialogue summarization employs two main methods: extractive summarization and generative summarization. Extractive summarization directly selects important sentences or fragments from the dialogue text and combines them to form a concise summary, while generative summarization automatically generates new sentences to describe the core content of the dialogue by understanding and reorganizing the dialogue content. However, traditional methods can lead to information redundancy and struggle to accurately capture the dialogue's topic and contextual relationships in long or complex dialogues. To address these issues, some researchers have combined deep learning methods, such as sequence-to-sequence methods and methods based on RNN and LSTM architectures, to improve the quality and breadth of applications of text summarization.

[0005] With the development of multimedia technology and the popularization of social media platforms, the information generated by user dialogues is becoming increasingly multimodal. For example, dialogues on social media platforms are often text-based conversations combined with videos or images. In such scenarios, only by combining multimodal information can key information in the dialogue be captured more efficiently. However, some existing dialogue summarization methods are text-based and do not consider the contribution of visual information, resulting in somewhat one-sided and incomplete dialogue summarization results in multimodal scenarios. To address this, in recent years, some researchers have proposed multimodal dialogue summarization methods, aiming to comprehensively and accurately reflect the core content of the dialogue by integrating information from multiple modalities such as dialogue, images, and videos. Multimodal dialogue summarization differs from traditional dialogue summarization. It requires combining text and other modal data, obtaining their features, fusing the features of the text and other modalities to obtain the final features, and then decoding the features to generate a more comprehensive and accurate dialogue summary. Unlike plain text dialogue summarization, multimodal dialogue summarization faces two challenges: first, how to capture the implicit semantics of different modalities to avoid the dialogue information being ignored due to the complexity of multimodal information; second, how to maintain the memory of the dialogue, the logic of the context, and the information of other modalities in the case of multi-turn dialogues to ensure their contribution to the generation of dialogue summarization.

[0006] To address the challenges of multimodal dialogue summarization mentioned above, some methods fuse image and text features by constructing complex network structures; others improve the quality of text summarization generated by the model by using pre-trained models.

[0007] like Figure 1 As shown in the paper "Aspect-Oriented Multimodal Summarizing Model," one of the existing technologies mentions using visual information combined with textual information for text summarization. It proposes a multimodal pointer generation network and maximum likelihood training for aspect enhancement, and combines this with an aspect coverage mechanism to achieve excellent text summarization results.

[0008] First, for text embedding, a pre-trained word vector model is used to transform text data into embedded representations. These text embeddings are input into the model's encoder and, together with visual features, are encoded into a contextual representation. Second, visual features are extracted by ResNet and used to initialize the hidden states of the encoder and decoder. Simultaneously, local visual features extracted by Faster R-CNN, along with the text embeddings, generate a contextual representation for multimodal fusion using a hierarchical attention mechanism. Third, an aspect-oriented reward-enhanced maximum likelihood mechanism is introduced during model training to ensure that the summary covers important aspects of the text data. During training, the model combines text embeddings and visual information to generate summary content, while an aspect coverage mechanism avoids repetitive descriptions of the same aspects. Finally, aspect consistency strategies and constrained decoding prevent the same aspect from appearing repeatedly. During decoding, text embeddings and visual features work together to ensure that the generated content remains consistent with the text data, thereby improving the coherence and accuracy of the summary.

[0009] like Figure 2 As shown in the paper "Video-based Multimodal Summarization Generation Task," one of the existing technologies mentions using a dual-interaction module's self-attention mechanism and global attention mechanism to capture the global and local semantics of text and video, thereby obtaining high-quality text summarization.

[0010] First, text and video data are encoded separately. For text, a bidirectional RNN is used to encode semantic information; for video, it is divided into multiple segments, and ResNet is used to extract frame features, with a bidirectional RNN used to encode the temporal dependencies between segments. Second, a dual-interaction module is employed to achieve deep interaction. This module includes a conditional self-attention mechanism and a global attention mechanism. Conditional self-attention captures local semantic information in video segments and highlights key content under text guidance; global attention handles high-level semantic relationships between text and video, achieving deep fusion between the two. Third, a multimodal generator generates text summaries and selects video cover frames. Text summarization uses an edit gate mechanism to fuse the video-aware text representation with the original text representation, and a pointer network is used to avoid missing words in the generated content. Cover frame selection is based on hierarchical video representation, using hierarchical attention to match and score candidate frames, selecting the most representative frame as the cover. Finally, the entire model is optimized through a joint loss function, including negative log-likelihood loss for text summarization and paired hinge loss for cover selection, so that multimodal generation and frame selection mutually promote each other, improving the generation effect.

[0011] However, existing technologies such as "aspect-oriented multimodal summarization models" and "video-based multimodal summarization generation tasks" are as follows:

[0012] 1. In multimodal summarization tasks, the visual feature extraction methods used play a crucial role in the quality of the generated summaries, but the extraction methods used do not fully understand the internal semantics related to the text.

[0013] 2. Previous work mainly focused on feature filtering and modal fusion at the encoder end, but neglected fusion at the decoder end.

[0014] Therefore, it has the following drawbacks:

[0015] 1. Visual feature extraction methods are not sufficiently correlated with the internal semantics of text data, making it difficult for the extracted visual features to complement the text features semantically. Adding visual features may not necessarily be beneficial for the generation of text summaries and may lead to information redundancy.

[0016] 2. A lack of contextual understanding of the dialogue and the capture of information content in multimodal dialogue may lead to the loss of summary information and the loss of understanding of the implied emotional intentions in the dialogue. Summary of the Invention

[0017] This application provides a multimodal dialogue summarization method based on multi-level visual guidance. It uses a pre-trained CLIP model to extract global and local features of the visual information contained in the dialogue, and uses a pre-trained T5 model to extract text features of the dialogue text. The global visual guidance text features and semantic guidance local visual features are fused and spliced ​​together, so that the multimodal dialogue information can complement each other and pay attention to the context of the dialogue, thereby improving the quality and accuracy of the generated summary.

[0018] This application provides a multimodal dialogue summarization method based on multi-level visual guidance, which performs the following steps during the model training phase:

[0019] The T5 model encoder extracts the text features of the multimodal dialogue and the text features of the target summary from the input multimodal dialogue data; and the CLIP model visual encoder is used to obtain global visual features and local visual features from the images of the input multimodal dialogue data, and an index of text features and visual features is constructed.

[0020] Text features and local visual features are input into a local multimodal attention cross module, and text features are input into a long short-term memory network to obtain hidden states containing global semantic information;

[0021] In the local multimodal attention cross module, self-attention weights are calculated based on the conditional self-attention mechanism. The hidden state containing global semantic information and the local visual features are used to calculate the conditional weights of the local visual features. The local visual features are weighted by the conditional weights and self-attention weights and then passed through the Transformer encoder to obtain semantically guided local visual features.

[0022] Text features and global visual features are input into the global multimodal attention cross module. Text features are input into the long short-term memory network to obtain text features that capture contextual information; global visual features are input into the long short-term memory network to obtain visual features that capture global information.

[0023] In the global multimodal attention cross module, a cross-modal multi-head attention mechanism and a cross-modal bilinear attention fusion mechanism are used to weight and sum the text features obtained by the two mechanisms that combine visual information. After passing through the Transformer encoder, global visual-guided text features are obtained.

[0024] Semantically guided local visual features and globally visually guided text features are input into the modality fusion module. Through a cross-modal attention mechanism, the semantically guided local visual features are used to guide the globally visually guided text features to obtain visual information guided text features; the globally visually guided text features are used to guide the semantically guided local visual features to obtain text information guided visual features; and the visual information guided text features are concatenated with the text information guided visual features to obtain bidirectionally enhanced visual-text features.

[0025] The bidirectional enhanced visual-text features and the summary text features are input into the Transformer decoder. The decoded text features are passed through a linear layer to convert the high-dimensional vector dimension into a vector dimension that can be mapped to the vocabulary. The resulting vector is then input into the Softmax layer and converted into a probability distribution using the beam search method.

[0026] During training, the cross-entropy loss between the probability distribution predicted by the model and the target summary text is calculated.

[0027] Optionally, using the encoder of the T5 model, the text features and summary text features of the multimodal dialogue are extracted to satisfy:

[0028] T i =T5(t) i )

[0029] S i =T5(s) i )

[0030] Among them, t i For the i-th dialogue input, si Let T5 be the target summary of the i-th dialogue, and T be the T5 model. i Enter t for the dialogue i The embedding vector, S i For the abstract text s i The embedding vector.

[0031] Optionally, the visual encoder of the CLIP model can be used to obtain global and local visual features that satisfy:

[0032] V fi =CLIP vis (V i )

[0033] V fi ={V a ,V1,…,V P}

[0034] Among them, V fi For using CLIP model visual encoder CLIP vis The acquired visual features D v V is the visual feature dimension. a V represents the global visual features of the image. i These are the local visual features of the image.

[0035] Optionally, in the local multimodal attention cross-module, the self-attention weights are calculated based on the conditional self-attention mechanism. The conditional weights for calculating local visual features by combining the hidden state containing global semantic information with local visual features include:

[0036] In the local multimodal attention cross module, based on the conditional self-attention mechanism, the query Q, key K, and value V of local visual features are obtained through the mapping matrix;

[0037] Calculate the self-attention weights using query Q, key K, and value V;

[0038] The conditional weights of the local visual features are calculated by combining the hidden state containing global semantic information with the local visual features.

[0039] Optionally, the text features are input into the Long Short-Term Memory network to satisfy:

[0040]

[0041] in, The hidden state at time step t, Let be the cell state at time step t, and let be the hidden state at the last step. The output is T s .

[0042] Optionally, in the local multimodal attention cross-module, the self-attention weights are calculated based on the conditional self-attention mechanism. The conditional weights for calculating local visual features by combining the hidden state containing global semantic information with local visual features include:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] Where W∈R d×d Let α be the weight matrix. i,j For self-attention weights, β i For conditional weights, V i ′ The semantically guided local visual features X are obtained through the Transformer encoder. v .

[0050] Optionally, global visual features are input into the Long Short-Term Memory network to obtain visual features that capture global information, including:

[0051]

[0052] in, The hidden state at time step t, The cell state at time step t is represented by V. s .

[0053] Optionally, in the global multimodal attention cross-module, a cross-modal multi-head attention mechanism and a cross-modal bilinear attention fusion mechanism are used to weight and sum the text features obtained by the two mechanisms that combine visual information. After passing through the Transformer encoder, the globally visually guided text features are obtained, including:

[0054]

[0055]

[0056]

[0057]

[0058] A = V s BTs

[0059] A ′ =softmax(A)

[0060] T2 = A ′ T s

[0061] T = αT1 + (1-α)T2

[0062] Where CMA stands for cross-modal multi-head attention, W is the weight matrix, B is the bilinear weight matrix, α is the trainable weights, and the output obtained after passing through the Transformer encoder layer is X. t .

[0063] Optionally, through a cross-modal attention mechanism, semantically guided local visual features are used to guide globally visually guided text features to obtain visually guided text features; globally visually guided text features are used to guide semantically guided local visual features to obtain text-information guided visual features; and the visually guided text features are concatenated with the text-information guided visual features to obtain bidirectionally enhanced visual-text features, including:

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072] O enc =concat(T ′ V ′ )

[0073] Among them, T ′ V is a text feature guided by visual information. ′ Visual features guided by text information; concat is a concatenation operation; O enc It is a two-way enhanced visual-text feature.

[0074] Optionally, during training, the cross-entropy loss between the probability distribution predicted by the model and the target summary text is calculated, including:

[0075] During training, the cross-entropy loss between the probability distribution predicted by the model and the target summary text is calculated.

[0076] The cross-entropy loss of all time steps is averaged to obtain the overall average loss. The calculated average loss is then backpropagated, and the model parameters are updated using gradient descent to minimize the loss.

[0077] The cross-entropy loss between the probability distribution predicted by the computational model and the target summary text satisfies:

[0078]

[0079] Among them, y i It is the true distribution of the target summary text, p i θ is the probability distribution predicted by the model, and θ is the model parameter.

[0080] This application uses a pre-trained CLIP model to extract global and local features of the visual information contained in the dialogue, and uses a pre-trained T5 model to extract text features from the dialogue text. The global visual-guided text features and semantically guided local visual features are fused and spliced ​​together, so that the multimodal dialogue information can complement each other and pay attention to the context of the dialogue, thereby improving the quality and accuracy of the generated summary.

[0081] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0082] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0083] Figure 1 The technical process for existing technology-oriented multimodal summarization models;

[0084] Figure 2 The technical process for existing video-based multimodal summarization generation tasks;

[0085] Figure 3This is a basic flowchart illustrating the multimodal dialogue summarization method based on multi-level visual guidance, as described in an embodiment of this application.

[0086] Figure 4 This is a schematic diagram of the application process of the multi-modal dialogue summarization method based on multi-level visual guidance in GPLM according to an embodiment of this application. Detailed Implementation

[0087] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0088] The abbreviations and key terms used in this embodiment are defined as follows:

[0089] BP: Back Propagation;

[0090] CLIP: Contrastive Language-Image Pre-training;

[0091] T5: Text-to-Text Transfer Transformer;

[0092] CMA: Cross-modal Multi-modal Attention.

[0093] LSTM: Long Short-Term Memory network;

[0094] ViT: Vision Transformer;

[0095] GPLM: Generative Pre-trained Language Model;

[0096] This application provides a multimodal dialogue summarization method based on multi-level visual guidance, such as... Figure 3 , Figure 4 As shown, the following steps are performed during the model training phase:

[0097] The input multimodal dialogue data, including the dialogue text and target summary, is processed using the encoder of a T5 model to extract text features of the multimodal dialogue and text features of the summary text. In some embodiments, the extracted text features and text features of the multimodal dialogue, obtained using the encoder of the T5 model, satisfy the following conditions:

[0098] T i =T5(t) i )

[0099] S i =T5(s) i )

[0100] Among them, t i For the i-th dialogue input, s i Let T5 be the target summary of the i-th dialogue, and T be the T5 model. i Enter t for the dialogue i The embedding vector, S i For the abstract text s i The embedding vector. In this embodiment, the text features are also encoded as 768-dimensional vectors.

[0101] The images of the input multimodal dialogue data are processed using the visual encoder of the CLIP model to obtain global and local visual features, and an index of text features and visual features is constructed.

[0102] In some embodiments, a visual encoder based on the CLIP model is used to obtain global and local visual features that satisfy:

[0103] V fi =CLIP vis (V i )

[0104] V fi ={V a ,V1,…,V P}

[0105] Among them, V fi For using CLIP model visual encoder CLIP vis The acquired visual features D v V is the visual feature dimension. a V represents the global visual features of the image. i These are the local visual features of the image.

[0106] In this embodiment, the pre-trained multimodal language model CLIP was pre-trained on a dataset of 400 million image-text pairs constructed by crawling public images and their corresponding descriptive texts from the internet, enabling it to capture deep semantic information of the data. The visual encoder output is set to 7×7, and the selected architecture is the CLIP-vit-base-32 model. Images are converted to 224×224, and each 7×7 is encoded as a 768-dimensional vector.

[0107] Text features and local visual features are input into a local multimodal attention cross module, and text features are input into a long short-term memory network to obtain the final hidden state containing global semantic information.

[0108] In some embodiments, inputting text features into a long short-term memory network satisfies:

[0109]

[0110] in, The hidden state at time step t, Let be the cell state at time step t, and let be the hidden state at the last step. The output is T s .

[0111] In the local multimodal attention cross module, self-attention weights are calculated based on the conditional self-attention mechanism. The hidden state containing global semantic information and the local visual features are used to calculate the conditional weights of the local visual features. The local visual features are weighted by the conditional weights and self-attention weights and passed through a Transformer encoder layer to obtain semantically guided local visual features.

[0112] In some embodiments, in the local multimodal attention cross module, calculating self-attention weights based on a conditional self-attention mechanism, and calculating the conditional weights of local visual features by combining the hidden state containing global semantic information with local visual features, includes:

[0113] In the local multimodal attention cross module, based on the conditional self-attention mechanism, the query Q, key K, and value V of local visual features are obtained through the mapping matrix;

[0114] Calculate the self-attention weights using query Q, key K, and value V;

[0115] The conditional weights of the local visual features are calculated by combining the hidden state containing global semantic information with the local visual features. Then, the conditional weights of the local visual features are calculated by combining the last hidden state containing global semantic information with the local visual features. Finally, the conditional weights, self-attention weights, and local visual features are weighted and calculated, and then passed through a Transformer encoder layer to obtain the semantically guided local visual features.

[0116] In some embodiments, in the local multimodal attention cross module, calculating self-attention weights based on a conditional self-attention mechanism, and calculating the conditional weights of local visual features by combining the hidden state containing global semantic information with local visual features, includes:

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123] Where W∈R d×d Let α be the weight matrix. i,j For self-attention weights, β i For conditional weights, V i ′ The semantically guided local visual features X are obtained through the Transformer encoder. v .

[0124] Text features and global visual features are input into the global multimodal attention cross-module. Text features are input into the long short-term memory network to obtain text features that capture contextual information; global visual features are input into the long short-term memory network to obtain visual features that capture global information.

[0125] In some embodiments, the global visual features are input into a long short-term memory network to obtain visual features that capture global information, including:

[0126]

[0127] in, The hidden state at time step t, The cell state at time step t is represented by V. s .

[0128] In the global multimodal attention cross module, a cross-modal multi-head attention mechanism and a cross-modal bilinear attention fusion mechanism are used to weight and sum the text features obtained by the two mechanisms that combine visual information. After passing through the Transformer encoder, global visual-guided text features are obtained.

[0129] In some embodiments, in the global multimodal attention cross-module, a cross-modal multi-head attention mechanism and a cross-modal bilinear attention fusion mechanism are used to weight and sum the text features obtained by the two mechanisms that combine visual information. After passing through the Transformer encoder, the globally visually guided text features are obtained, including:

[0130]

[0131]

[0132]

[0133]

[0134] A = V s BT s

[0135] A ′ =softmax(A)

[0136] T2 = A ′ T s

[0137] T = αT1 + (1-α)T2

[0138] Where CMA stands for cross-modal multi-head attention, W is the weight matrix, B is the bilinear weight matrix, α is the trainable weights, and the output obtained after passing through the Transformer encoder layer is X. t .

[0139] In the specific example, the number of cross-module heads for global multimodal attention is set to 8.

[0140] The semantically guided local visual features and the globally guided textual features are input into the modality fusion module. Through a cross-modal attention mechanism, the semantically guided local visual features are used to guide the globally guided textual features to obtain visual information-guided textual features; the globally guided textual features are used to guide the semantically guided local visual features to obtain textual information-guided visual features; and the visual information-guided textual features and the textual information-guided visual features are concatenated to obtain bidirectionally enhanced visual-textual features.

[0141] In some embodiments, a cross-modal attention mechanism is used to guide globally visually guided text features with semantically guided local visual features to obtain visually guided text features; globally visually guided text features are used to guide semantically guided local visual features to obtain text-information guided visual features; and the visually guided text features are concatenated with the text-information guided visual features to obtain bidirectionally enhanced visual-text features, including:

[0142]

[0143]

[0144]

[0145]

[0146]

[0147]

[0148]

[0149]

[0150] O enc =concat(T ′ V ′ )

[0151] Among them, T ′ V is a text feature guided by visual information. ′ Visual features guided by text information; concat is a concatenation operation; O enc It is a two-way enhanced visual-text feature.

[0152] The bidirectional enhanced visual-text features and the summarized text features are input into the Transformer decoder. The decoded text features are passed through a linear layer to convert the high-dimensional vector dimensions into vector dimensions that can be mapped to the vocabulary. The resulting vectors are then input into a Softmax layer to convert them into probability distributions, and beam search is used to obtain the final output probability distribution. In this embodiment, the beam search number is set to 5.

[0153] During training, the cross-entropy loss between the probability distribution predicted by the model and the target summary text is calculated. After training, the trained model is deployed and applied. During training, the cross-entropy loss between the model's predicted probability distribution and the target summary text is calculated; the cross-entropy loss at all time steps is averaged as the overall average loss. This average loss is then used for backpropagation, and gradient descent is used to update the model parameters to minimize the loss.

[0154] In this embodiment, the learning rate is set to 1e-5 and the batch size is set to 4.

[0155] In some embodiments, during training, calculating the cross-entropy loss between the probability distribution predicted by the model and the target summary text includes:

[0156] During the training process,

[0157] The cross-entropy loss between the probability distribution predicted by the computational model and the target summary text satisfies:

[0158]

[0159] Among them, y i It is the true distribution of the target summary text, p i θ is the probability distribution predicted by the model, and θ is the model parameter.

[0160] like Figure 4 As shown, the application example of this application first uses T5 and CLIP models to extract text features of the dialogue and global and local visual features of the image, respectively. Then, the model feeds the text features and local visual features into the local multimodal attention module, calculates the weights using the conditional self-attention mechanism, and generates semantically guided local visual features. At the same time, the text features and global visual features are input into the global multimodal attention module, and fused through multi-head and bilinear attention mechanisms to generate globally visually guided text features. After that, these features guide each other and are bidirectionally enhanced in the modality fusion module to obtain fused visual-text features. Finally, the model inputs the fused features into the Transformer decoder to obtain the probability distribution of the generated summary text. In the training case, the difference between the generated summary and the target summary is calculated using cross-entropy loss, and gradient descent is used to optimize the result to make the generated result closer to the target summary. In the non-training case, a bundle search strategy is used to select the dialogue summary text with the highest probability as the output result.

[0161] This application's method targets dialogue summarization tasks involving visual information. It uses the pre-trained CLIP model to extract global and local features of the visual information contained in the dialogue, and the pre-trained T5 model to extract textual features from the dialogue text, obtaining more comprehensive and deeper visual feature representations and semantically richer textual feature representations. Through a local multimodal attention cross-module, using a long short-term memory network and a conditional self-attention mechanism, it captures the deep semantic association between local visual information features and textual features, obtaining semantically guided local visual features, enhancing the model's deep semantic expression of visual information. Through a global multimodal attention cross-module, using a long short-term memory network and a multimodal fusion mechanism that combines cross-modal multi-head attention and cross-modal bilinear attention mechanisms, it obtains globally visually guided textual features, ensuring the integrity and effectiveness of information and improving the quality of the summary. In the modality fusion module, a cross-modal attention mechanism is used for modality fusion, decoding the input into the Transformer decoder's multi-head encoder-decoder attention layer, and finally employing a bundle search strategy to improve the modality fusion effect, supplementing the dialogue summary information, improving the quality of text summarization, and providing a new method for dialogue summarization tasks.

[0162] This application uses a pre-trained CLIP model to extract global and local features of the visual information contained in the dialogue, and uses a pre-trained T5 model to extract text features from the dialogue text. The global visual-guided text features and semantically guided local visual features are fused and spliced ​​together, so that the multimodal dialogue information can complement each other and pay attention to the context of the dialogue, thereby improving the quality and accuracy of the generated summary.

[0163] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0164] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0166] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A multimodal dialogue summarization method based on multi-level visual guidance, characterized in that, The following steps are performed during the model training phase: The T5 model encoder extracts the text features of the multimodal dialogue and the text features of the target summary from the input multimodal dialogue data; and the CLIP model visual encoder is used to obtain global visual features and local visual features from the images of the input multimodal dialogue data, and an index of text features and visual features is constructed. Text features and local visual features are input into a local multimodal attention cross module, and text features are input into a long short-term memory network to obtain hidden states containing global semantic information; In the local multimodal attention cross module, self-attention weights are calculated based on the conditional self-attention mechanism. The hidden state containing global semantic information and the local visual features are used to calculate the conditional weights of the local visual features. The local visual features are weighted by the conditional weights and self-attention weights and then passed through the Transformer encoder to obtain semantically guided local visual features. Text features and global visual features are input into the global multimodal attention cross module. Text features are input into the long short-term memory network to obtain text features that capture contextual information; global visual features are input into the long short-term memory network to obtain visual features that capture global information. In the global multimodal attention cross module, a cross-modal multi-head attention mechanism and a cross-modal bilinear attention fusion mechanism are used to weight and sum the text features obtained by the two mechanisms that combine visual information. After passing through the Transformer encoder, global visual-guided text features are obtained. Semantically guided local visual features and globally visually guided text features are input into the modality fusion module. Through a cross-modal attention mechanism, the semantically guided local visual features are used to guide the globally visually guided text features to obtain visual information guided text features; the globally visually guided text features are used to guide the semantically guided local visual features to obtain text information guided visual features; and the visual information guided text features are concatenated with the text information guided visual features to obtain bidirectionally enhanced visual-text features. The bidirectional enhanced visual-text features and the summary text features are input into the Transformer decoder. The decoded text features are passed through a linear layer to convert the high-dimensional vector dimension into a vector dimension that can be mapped to the vocabulary. The resulting vector is then input into the Softmax layer and converted into a probability distribution using the beam search method. During training, the cross-entropy loss between the probability distribution predicted by the model and the target summary text is calculated.

2. The multimodal dialogue summarization method based on multi-level visual guidance as described in claim 1, characterized in that, Using the encoder of the T5 model, the text features and summary text features extracted from the multimodal dialogue satisfy the following: T i =T5(t i ) S i =T5(s i ) Among them, t i For the i-th dialogue input, s i Let T5 be the target summary of the i-th dialogue, and T be the T5 model. i Enter t for the dialogue i The embedding vector, S i For the abstract text s i The embedding vector.

3. The multimodal dialogue summarization method based on multi-level visual guidance as described in claim 1, characterized in that, The visual encoder using the CLIP model is used to obtain global and local visual features that satisfy the following: In fi =CLIP vis (In i ) V fi ={V a ,V1,…,V P } Among them, V fi For using CLIP model visual encoder CLIP vis The acquired visual features D v V is the visual feature dimension. a V represents the global visual features of the image. i These are the local visual features of the image.

4. The multimodal dialogue summarization method based on multi-level visual guidance as described in claim 1, characterized in that, In the local multimodal attention cross-module, self-attention weights are calculated based on a conditional self-attention mechanism. The conditional weights for calculating local visual features by combining the hidden state containing global semantic information with local visual features include: In the local multimodal attention cross module, based on the conditional self-attention mechanism, the query Q, key K, and value V of local visual features are obtained through the mapping matrix; Calculate the self-attention weights using query Q, key K, and value V; The conditional weights of the local visual features are calculated by combining the hidden state containing global semantic information with the local visual features.

5. The multimodal dialogue summarization method based on multi-level visual guidance as described in claim 4, characterized in that, Inputting text features into a Long Short-Term Memory network satisfies: in, The hidden state at time step t, Let be the cell state at time step t, and let be the hidden state at the last step. The output is T s .

6. The multimodal dialogue summarization method based on multi-level visual guidance as described in claim 5, characterized in that, In the local multimodal attention cross-module, self-attention weights are calculated based on a conditional self-attention mechanism. The conditional weights for calculating local visual features by combining the hidden state containing global semantic information with local visual features include: Where W∈R d×d Let α be the weight matrix. i,j For self-attention weights, β i For conditional weights, V i ′ The semantically guided local visual features X are obtained through the Transformer encoder. v .

7. The multimodal dialogue summarization method based on multi-level visual guidance as described in claim 6, characterized in that, Global visual features are input into the Long Short-Term Memory network to obtain visual features that capture global information, including: in, The hidden state at time step t, The cell state at time step t is represented by V. s .

8. The multimodal dialogue summarization method based on multi-level visual guidance as described in claim 7, characterized in that, In the global multimodal attention cross-module, a cross-modal multi-head attention mechanism and a cross-modal bilinear attention fusion mechanism are used to weight and sum the text features obtained by the two mechanisms that combine visual information. After passing through the Transformer encoder, the globally visually guided text features are obtained, including: A=V s BT s A ′ =softmax(A) T2=A ′ T s T = αT1 + (1-α)T2 Where CMA stands for cross-modal multi-head attention, W is the weight matrix, B is the bilinear weight matrix, α is the trainable weights, and the output obtained after passing through the Transformer encoder layer is X. t .

9. The multimodal dialogue summarization method based on multi-level visual guidance as described in claim 8, characterized in that, By using a cross-modal attention mechanism, semantically guided local visual features are used to guide globally guided text features to obtain visual information guided text features; and globally guided text features are used to guide semantically guided local visual features to obtain text information guided visual features. Then, visual information-guided text features are concatenated with text information-guided visual features to obtain bidirectionally enhanced visual-text features, including: O enc =concat(T ′ ,V ′ ) Among them, T ′ V is a text feature guided by visual information. ′ Visual features guided by text information; concat is a concatenation operation; O enc It is a two-way enhanced visual-text feature.

10. The multimodal dialogue summarization method based on multi-level visual guidance as described in claim 8, characterized in that, During training, the cross-entropy loss between the probability distribution predicted by the model and the target summary text is calculated, including: During training, the cross-entropy loss between the probability distribution predicted by the model and the target summary text is calculated. The cross-entropy loss of all time steps is averaged to obtain the overall average loss. The calculated average loss is then backpropagated, and the model parameters are updated using gradient descent to minimize the loss. The cross-entropy loss between the probability distribution predicted by the computational model and the target summary text satisfies: Among them, y i It is the true distribution of the target summary text, p i θ is the probability distribution predicted by the model, and θ is the model parameter.

Images