A multi-modal summarization method and system based on visual information fusion

Abstract

The application discloses a multi-modal summarization method and system based on visual information fusion, relates to data processing and artificial intelligence technology, and comprises the following steps: splicing extracted visual features and text features, and inputting the spliced visual-text features into a multi-head attention module and a feedforward network module of an encoder; taking the extracted visual features and the output of the feedforward network module as inputs of a modal fusion layer of the encoder to obtain text features combined with visual information; inputting the obtained text features combined with visual information and summary text features into a decoder; taking the extracted visual features and the output of a decoder feedforward network module as inputs of a decoder modal fusion layer to obtain text features output by the decoder; inputting the output vector of the text features output by the decoder into a Softmax layer through a linear layer, and obtaining required summary text according to a probability distribution and a vocabulary table. The method disclosed by the application can improve the quality and accuracy of generated summaries.

Description

Technical Field

[0001] This application relates to the fields of data processing and artificial intelligence, and in particular to a multimodal summarization method and system based on visual information fusion. Background Technology

[0002] With the rapid development of information technology, the explosive growth of information has brought unprecedented challenges. On social media and online platforms, users face a massive amount of information, and how to efficiently and accurately obtain the content they need has become a pressing issue, placing higher demands on automated text processing technologies. Meanwhile, the application of technologies such as machine learning and deep learning in natural language processing within the field of artificial intelligence is deepening, significantly improving the ability to process large amounts of data. Against this backdrop, text summarization technology has emerged. The development of text summarization technology is not only significant in solving the problem of information overload caused by information explosion, but also plays a crucial role in enabling users to efficiently capture key information and improve information acquisition efficiency. It has shown great potential and broad application prospects in fields such as news media, education and academic research, social media, and intelligent assistants.

[0003] Text summarization aims to simplify and compress text content into shorter segments while preserving the core information and semantics of the original text. This task holds significant importance and application value in Natural Language Processing (NLP). It requires models not only to identify and extract key information but also to possess language generation capabilities to ensure the generated summary is fluent, coherent, and accurately conveys the main idea of ​​the original text. Text summarization is considered a major challenge in NLP because its quality requires distinguishing between primary and secondary information within the text, and the generated summary must also be coherent and consistent. In the news media industry, text summarization technology can automatically generate concise summaries of news reports, allowing readers to quickly grasp the main content. In academic research, automatic summarization technology helps researchers obtain key information. In social media, text summarization technology can be used to generate short summaries of posts, helping users quickly browse and understand information. Text summarization is not only a key technology for information retrieval and extraction but also provides crucial support for other NLP tasks, making it one of the most important tasks in NLP.

[0004] Text summarization involves extracting features from text data and using designed algorithms or trained models to enable computers to automatically summarize the data. Traditionally, text summarization employs two main methods: extractive summarization and generative summarization. Extractive summarization directly selects key sentences or paragraphs from the original text to form a summary, while generative summarization understands and reconstructs the text content to generate new, concise summaries. However, traditional methods may result in information loss, low-quality results, and are limited by domain-specific rules.

[0005] With the development of multimedia technology and the popularization of social media platforms, the information users receive is becoming increasingly multimodal. For example, news received by users on news platforms often combines images and text to describe events. In such scenarios, only by combining multimodal information can key information in the text be captured more efficiently. However, most existing text summarization methods are based on pure text and cannot consider the contribution of visual information, resulting in somewhat one-sided and incomplete text summarization results in multimodal scenarios. To address this, in recent years, some researchers have proposed multimodal summarization methods, aiming to comprehensively and accurately reflect the core content of information by utilizing information from multiple modalities such as text, images, audio, and video. Multimodal summarization differs from traditional text summarization. It requires combining text and other modal data, obtaining their features, fusing the features of text and other modalities to obtain multimodal text features, and then decoding the text to produce a more comprehensive and accurate text summary. Unlike plain text summarization, multimodal summarization faces two challenges: first, how to effectively integrate features from different modalities to avoid feature complexity caused by differences in multimodal information; and second, how to effectively extract semantically rich features from other modalities to ensure their contribution to text summarization generation.

[0006] like Figure 1 As shown in the paper "Multimodal Sentence Summarization Based on Modal Attention and Image Filtering," one of the existing technologies mentions using visual information combined with textual information for text summarization, using bidirectional gated recurrent units to encode data, and combining image attention filters and image context filters for image denoising and modal fusion, achieving excellent text summarization results.

[0007] First, the input text is embedded using a pre-trained or randomly initialized word embedding matrix, followed by bidirectional gated recurrent unit (GRU) encoding to obtain the sentence's feature representation. Second, a pre-trained VGG19 model is used to extract image features, obtaining the image's feature representation. Third, a modal attention mechanism is used to calculate the attention weights for the text and image separately. The text's context vector is calculated by weighted summation of the source text's annotation vectors, while the image's context vector is calculated by weighted summation of image patch representations. Fourth, an image filter is used to selectively utilize visual information to enhance the sentence's semantics and avoid introducing image noise. In the image attention filter, attention weights are first calculated based on the decoder state and the image's global features from the previous time step, and then adjusted based on these weights during modal fusion. In the image context filter, weights are calculated based on the decoder state and the image's global features, and these weights are used to adjust the image features. Finally, the fused features are fed into the unidirectional gated recurrent unit sentence decoder, where the decoder initializes and generates a summary based on different visual features.

[0008] like Figure 2 As shown in the paper "Visual Information-Guided Generative Pre-trained Language Model for Multimodal Abstract Summarization," one of the existing technologies mentions a method using a pre-trained language model to fuse visual information using an attention mechanism without compromising the model's generative capabilities, thereby obtaining high-quality text summarization.

[0009] First, the text and video data are preprocessed, including word segmentation and serialization of the text data, and feature representation extraction and serialization of the video data from video segments using a pre-trained 3D ResNeXt-101 model. Second, the preprocessed text sequence is input into a pre-trained language model to extract text features, and the extracted visual features are input into a VTF to enhance the temporal information of the visual features. Third, the text and video features are input into a multimodal fusion layer in the model, utilizing cross-modal dot product attention and cross-modal multi-head attention for modal fusion to obtain visually guided text features. Finally, the decoder decodes the text features and outputs them, which are then transformed into a probability distribution through linear and softmax layers to generate the final text summary.

[0010] The existing technologies, namely "multimodal sentence summarization based on modal attention and image filtering" and "visual information-guided generative pre-trained language models for multimodal abstract summarization," have the following technical problems:

[0011] 1. In multimodal summarization tasks, the visual feature extraction methods used play a crucial role in the quality of the generated summaries, but the extraction methods used do not fully understand the internal semantics related to the text.

[0012] 2. Previous work mainly focused on feature filtering and modal fusion at the encoder end, but neglected fusion at the decoder end.

[0013] The aforementioned technical problems result in the following drawbacks in the existing technology:

[0014] 1. Using visual feature extraction methods that do not understand the internal semantics results in extracted visual features that are not closely related to text features. Adding visual features may not necessarily help generate text summaries, thus affecting the quality of the final generated results.

[0015] 2. The shallow modality fusion depth prevents the full utilization of the correlation between modalities, resulting in the fused text features not containing all semantic information and affecting the quality of the final generated summary. Summary of the Invention

[0016] This application provides a multimodal summarization method and system based on visual information fusion. It uses the pre-trained language model FLAN-T5 as the core architecture and uses the visual encoder of the pre-trained CLIP model to add multi-dimensional visual feature transformation to obtain visual feature mapping from different angles. Finally, it fully captures the correlation between visual information and text information, thereby improving the quality and accuracy of the generated summary.

[0017] This application provides a multimodal summarization method based on visual information fusion, including:

[0018] Extract visual features from the image in the input text-image pair, and extract text features from the text in the input text-image pair;

[0019] The extracted visual features are concatenated with the text features, and the concatenated visual-text features are input into the multi-head attention module and feedforward network module of the encoder.

[0020] The extracted visual features and the output of the feedforward network module are used as inputs to the modality fusion layer of the encoder to obtain text features that combine visual information.

[0021] The obtained text features combining visual information and the summary text features are input into the decoder, wherein the summary text features are obtained based on the extraction of start elements with no actual content added to the beginning of the text in the input text-image pair;

[0022] The extracted visual features and the output of the decoder feedforward network module are used as inputs to the decoder modality fusion layer to obtain the text features output by the decoder.

[0023] The text features output by the decoder are passed through a linear layer to be converted into vectors of a specified dimension, and the output vectors are then fed into a Softmax layer to be converted into a probability distribution.

[0024] Optionally, extracting visual features from the input text-image pair includes:

[0025] The input text-image pair image is fed into the visual encoder of the CLIP model to extract visual features. The image embedding representation obtained using the CLIP image encoder is as follows:

[0026] Emb v (i) = CLIP vis (v i )

[0027] Among them, v i It is the i-th image input, CLIP vis It is the visual encoder of the CLIP model, Emb v (i) Input image v i The embedding vector;

[0028] The extracted visual features are subjected to nonlinear transformations using different functions, and the transformed features are then spliced ​​together to obtain the visual features.

[0029] Optionally, the extracted visual features are subjected to nonlinear transformations using different functions, and the linearly transformed features are spliced ​​together to obtain the visual features, including:

[0030] Nonlinear transformations are performed using the hyperbolic tangent, cosine, and sigmoid functions respectively, and the resulting visual features are then concatenated to satisfy the following condition:

[0031] V1 = tanh(Emb) v (i))

[0032] V2=cos(Emb v (i))

[0033] V3=σ(Emb v (i))

[0034] V = Concat(V1, V2, V3)

[0035] Where V1, V2, and V3 are visual features transformed using the hyperbolic tangent, cosine, and sigmoid functions, respectively; Concat is the concatenation operation; and V is the final visual feature obtained.

[0036] Optionally, extracting text features from the input text-image pair of Chinese text includes:

[0037] Based on the text in the input text-image pair, an index of text and visual features is constructed in the text;

[0038] The text in the input text-image pair, the index, and the summary text features are input into the FLAN-T5 model so that the text in the input text-image pair enters the encoder to segment the text and convert it into vector form.

[0039] Optionally, concatenating the extracted visual features with text features includes:

[0040] A set of learnable vectors is initialized, and these vectors, along with the visual features, are input into the visual feature mapping module. An attention mechanism is then used to obtain the visual features to be concatenated, wherein the visual feature attention mechanism of the visual feature mapping module satisfies the following:

[0041] Q = [q1, q2, q3, ..., q M ]

[0042] K = V·W v

[0043]

[0044] v = A·V

[0045] Where Q is an initialized learnable vector, Q∈R M×d W v Let K be the weight matrix and V be the extracted visual features;

[0046] The visual features to be concatenated are concatenated with the text features input from the encoder; and,

[0047] Positional encoding is added to the concatenated feature vector to capture the relative positional information of words in the sequence.

[0048] Optionally, the extracted visual features and the output of the feedforward network module are used as inputs to the modality fusion layer of the encoder to obtain text features that combine visual information, including:

[0049] Visual features and input text features are input into the modality fusion layer of the encoder. The text features combining visual information obtained from the two mechanisms are weighted and summed using a cross-modal multi-head attention fusion mechanism and a cross-modal bilinear attention fusion mechanism to obtain the text features incorporating visual information. The cross-modal multi-head attention fusion mechanism satisfies the following:

[0050] Q = T0W q

[0051] K = VW k

[0052] V = VW v

[0053] O = CMA(Q,K,V)

[0054] T1 = Concat(T0, O)W

[0055] Where T0 is the processed input text feature, CMA is cross-modal multi-head attention, and W is the weight matrix;

[0056] The cross-modal bilinear attention fusion mechanism satisfies:

[0057] A = VBT0

[0058] A' = Softmax(A)

[0059] T2=A'T0

[0060] Where B is the bilinear weight matrix;

[0061] The text features obtained by the two mechanisms, which combine visual information, are weighted and summed to satisfy the following:

[0062] T = αT1 + (1-α)T2

[0063] Here, α is the trainable weight.

[0064] Optionally, the obtained text features combining visual information and the summarized text features are input into the decoder, including:

[0065] The summary text features are passed through a multi-head attention module, and the text features combined with visual information are input into the encoder-decoder multi-head attention module, and then through a feedforward network module to obtain text features containing previous text information and visual information;

[0066] The obtained text features, which contain previous textual and visual information, and the extracted visual features are input into the modality fusion layer of the decoder to obtain the final text features.

[0067] Optionally, it also includes calculating the cross-entropy loss between the probability distribution predicted by the model and the target summary text, averaging the cross-entropy losses at all time steps as the overall average loss, performing backpropagation on the calculated average loss, updating the model parameters using gradient descent, and minimizing the loss. The cross-entropy loss between the probability distribution predicted by the model and the target summary text satisfies the following condition:

[0068]

[0069] Among them, y i It is the true distribution of the target text, p i It is the probability distribution predicted by the model.

[0070] This application also proposes a multimodal summarization system based on visual information fusion, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps of the aforementioned multimodal summarization method based on visual information fusion.

[0071] The method in this application uses the pre-trained language model FLAN-T5 as the core architecture and adds multi-dimensional visual feature transformation to the visual encoder of the pre-trained CLIP model to capture visual features rich in deep semantics and obtain visual feature mappings from different angles. A visual feature mapping module is added to the text input so that the text input carries visual information from the beginning. Modality fusion layers are designed in the encoder and decoder respectively. The fusion layer uses cross-modal bilinear attention mechanism and cross-modal multi-head attention mechanism in parallel, and performs deep modality fusion through residual connections to fully capture the correlation between visual information and text information, thereby improving the quality and accuracy of the generated summary.

[0072] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0073] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0074] Figure 1 This is a schematic diagram of the technical architecture of the existing technology "multimodal sentence summarization based on modal attention and image filtering";

[0075] Figure 2 This is a schematic diagram of the technical architecture of the existing technology "Visual Information Guided Generative Pre-trained Language Model for Multimodal Abstract Summarization";

[0076] Figure 3 This is a basic flowchart illustrating the multimodal summarization method according to an embodiment of this application;

[0077] Figure 4 This is a schematic flowchart illustrating an implementation example of the multimodal summarization method according to an embodiment of this application;

[0078] Figure 5 This is a schematic diagram of the overall process of the multimodal summarization method according to an embodiment of this application. Detailed Implementation

[0079] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0080] The abbreviations and key terms used in this embodiment are defined as follows:

[0081] BP: Back Propagation;

[0082] CLIP: Contrastive Language-Image Pre-training;

[0083] GPLM: Generative Pre-training Language Model;

[0084] CMA: Cross-modal Multi-head Attention;

[0085] FLAN: Finetuned Language Models are Zero-Shot Learners;

[0086] ViT: Vision Transformer;

[0087] NLP: Natural Language Processing;

[0088] MLP: Multilayer Perceptron.

[0089] This application provides a multimodal summarization method based on visual information fusion, such as... Figure 3 As shown, it includes the following steps:

[0090] In step S301, visual features of the input text-image pair and text features of the input text-image pair are extracted. In a specific example, extracting visual features of the input text-image pair includes: inputting the input text-image pair into the visual encoder of the CLIP model to extract visual features. The extracted visual features are then subjected to nonlinear transformations using the hyperbolic tangent, cosine, and sigmoid functions, respectively. Finally, the three visual features are concatenated to obtain the final visual features.

[0091] For the text in the input text-image pair, the data is first cleaned, and an index of text and visual features is constructed in the text before being input into the FLAN-T5 model.

[0092] In step S302, the extracted visual features are concatenated with the text features, and the text features are input into the multi-head attention module and feedforward network module of the encoder. The multi-head attention module is used to capture the interrelationships between different positions in the text sequence and generate rich contextual representations. The feedforward network module can independently perform nonlinear transformations on the input at each position, enhancing the model's representational ability. Finally, the feature vector is input into the modality fusion layer of the encoder.

[0093] In step S303, the extracted visual features and the output of the feedforward network module are used as inputs to the modality fusion layer of the encoder to obtain text features incorporating visual information. The final text features incorporating visual information, obtained through cross-modal multi-head attention fusion and cross-modal bilinear attention fusion mechanisms, are then input to the decoder.

[0094] In step S304, the obtained text features combining visual information and the summary text features are input into the decoder, wherein the summary text features are obtained based on the extraction of start elements with no actual content added to the beginning of the text in the input text-image pair.

[0095] In step S305, the extracted visual features and the output of the decoder feedforward network module are used as inputs to the decoder modality fusion layer to obtain the text features output by the decoder. Specifically, the summary text features are input to the encoder-decoder multi-head attention module through the multi-head attention module and the text features combined with visual information, and then through the feedforward network module to obtain text features containing previous text information and visual information. The visual features are then fed into the modality fusion layer of the decoder to obtain the final text features.

[0096] In step S306, the text features output by the decoder are passed through a linear layer to convert them into vectors of a specified dimension. That is, the text features obtained by the decoder are passed through a linear layer to convert the dimension of the high-dimensional vector into a vector dimension that can be mapped by the vocabulary. The output vector is then input into a Softmax layer to convert it into a probability distribution. Based on the probability distribution and the vocabulary, the required summary text is obtained.

[0097] The method in this application uses the pre-trained language model FLAN-T5 as the core architecture and adds multi-dimensional visual feature transformation to the visual encoder of the pre-trained CLIP model to capture visual features rich in deep semantics and obtain visual feature mappings from different angles. A visual feature mapping module is added to the text input so that the text input carries visual information from the beginning. Modality fusion layers are designed in the encoder and decoder respectively. The fusion layer uses cross-modal bilinear attention mechanism and cross-modal multi-head attention mechanism in parallel, and performs deep modality fusion through residual connections to fully capture the correlation between visual information and text information, thereby improving the quality and accuracy of the generated summary.

[0098] In some embodiments, extracting visual features of the image in the input text-image pair includes: obtaining the image's embedded representation using a CLIP image encoder as follows:

[0099] Emb v (i) = CLIP vis (v i )

[0100] Among them, v i It is the i-th image input, CLIP vis It is the visual encoder of the CLIP model, Emb v (i) Input image v i The embedding vector;

[0101] The extracted visual features are subjected to nonlinear transformations using different functions, and the transformed features are then spliced ​​together to obtain the visual features.

[0102] In some embodiments, the extracted visual features are subjected to nonlinear transformations using different functions, and the linearly transformed features are spliced ​​together to obtain the visual features, including:

[0103] Nonlinear transformations are performed using the hyperbolic tangent, cosine, and sigmoid functions respectively, and the resulting visual features are then concatenated to satisfy the following condition:

[0104] V1 = tanh(Emb) v (i))

[0105] V2=cos(Emb v(i))

[0106] V3=σ(Emb v (i))

[0107] V = Concat(V1, V2, V3)

[0108] Where V1, V2, and V3 are visual features transformed using the hyperbolic tangent, cosine, and sigmoid functions, respectively; Concat is the concatenation operation; and V is the final visual feature obtained.

[0109] In some embodiments, extracting text features from the input text-image pair includes:

[0110] Based on the text in the input text-image pair, an index of text and visual features is constructed in the text;

[0111] Based on the text, index, and summary text features in the input text-image pair, the data is fed into the FLAN-T5 model so that the text in the input text-image pair enters the encoder to segment the text and convert it into vector form.

[0112] Specifically, the input text-image pair consists of the text itself and a summary text sequence with an extra start element (without actual content) added to the beginning. The former goes to the encoder, and the latter to the decoder. This allows the model to use the previously accurate summary text as input, accelerating training. First, the index portion of the encoder's input text is saved for subsequent fusion layers to input corresponding visual features. Then, the text inputs to both the encoder and decoder is segmented, and insufficient length is padded and masks are constructed. Next, the input text is converted into an index based on the vocabulary. Finally, word embedding operations are performed to convert the text into vector form.

[0113] In some embodiments, concatenating the extracted visual features with text features includes:

[0114] A set of learnable vectors is initialized, and these vectors, along with the visual features, are input into the visual feature mapping module. An attention mechanism is then used to obtain the visual features to be concatenated, wherein the visual feature attention mechanism of the visual feature mapping module satisfies the following:

[0115] Q = [q1, q2, q3, ..., q M ]

[0116] K = V·W v

[0117]

[0118] v = A·V

[0119] Where Q is an initialized learnable vector, Q∈R M×d W v Let K be the weight matrix and V be the extracted visual features;

[0120] The visual features to be concatenated are concatenated with the text features input from the encoder; and,

[0121] Positional encoding is added to the concatenated feature vector to capture the relative positional information of words in the sequence.

[0122] In some embodiments, the extracted visual features and the output of the feedforward network module are used as inputs to the modality fusion layer of the encoder to obtain text features incorporating visual information, including:

[0123] Visual features and input text features are input into the modality fusion layer of the encoder. The text features combining visual information obtained from the two mechanisms are weighted and summed using a cross-modal multi-head attention fusion mechanism and a cross-modal bilinear attention fusion mechanism to obtain the text features incorporating visual information. The cross-modal multi-head attention fusion mechanism satisfies the following:

[0124] Q = T0W q

[0125] K = VW k

[0126] V = VW v

[0127] O = CMA(Q,K,V)

[0128] T1 = Concat(T0, O)W

[0129] Where T0 is the processed input text feature, CMA is cross-modal multi-head attention, and W is the weight matrix;

[0130] The cross-modal bilinear attention fusion mechanism satisfies:

[0131] A = VBT0

[0132] A' = Softmax(A)

[0133] T2=A'T0

[0134] Where B is the bilinear weight matrix;

[0135] The text features obtained by the two mechanisms, which combine visual information, are weighted and summed to satisfy the following:

[0136] T = αT1 + (1-α)T2

[0137] Here, α is the trainable weight.

[0138] In some embodiments, the obtained text features combining visual information and the summarized text features are input into the decoder, including:

[0139] The abstract text features are passed through a multi-head attention module, and the text features combined with visual information are input into the encoder-decoder multi-head attention module, and then through a feedforward network module to obtain text features containing previous text information and visual information;

[0140] The obtained text features, which contain previous textual and visual information, and the visual features are input into the modality fusion layer of the decoder to obtain the final text features.

[0141] In some embodiments, the method further includes calculating the cross-entropy loss between the probability distribution predicted by the model and the target summary text, averaging the cross-entropy losses at all time steps as the overall average loss, performing backpropagation on the calculated average loss, updating the model parameters using gradient descent, and minimizing the loss. The cross-entropy loss between the probability distribution predicted by the model and the target summary text satisfies the following condition:

[0142]

[0143] Among them, y i It is the true distribution of the target text, p i It is the probability distribution predicted by the model.

[0144] In the non-training case, the input text and the summary text generated in the previous time step are input into the FLAN-T5 model. The former enters the encoder and the latter enters the decoder. First, the index part of the former is saved so that the corresponding visual features can be input into the subsequent fusion layer. Then, both are segmented into words, and those that are not long enough are padded and masks are constructed. Next, the input text is converted into an index according to the vocabulary. Finally, the text is converted into vector form by word embedding operation.

[0145] Furthermore, a set of learnable vectors is initialized and fed into the visual feature mapping module along with the visual features. The visual features are obtained using an attention mechanism and concatenated with the text features input from the encoder. Positional encoding is added to the concatenated text feature vector to capture the relative positional information of words in the sequence.

[0146] In the decoder, text features combining visual information and summary text features generated in the previous time step are fed into the decoder. The summary text features are fed into the encoder-decoder multi-head attention module through the multi-head attention module and the text features combining visual information. Then, they are fed into the feedforward network module to obtain text features containing previous text information and visual information. The visual features are fed into the modality fusion layer of the decoder to obtain the final text features.

[0147] Based on the probability distribution and vocabulary, the final summary text is obtained.

[0148] The method in this application uses the pre-trained model FLAN-T5 as its core architecture, which improves the text summarization generation capability. It uses the CLIP model's visual encoder for visual feature extraction and enriches the visual features through multiple nonlinear transformations and a visual feature mapping module, incorporating deeper semantics and enhancing the correlation between visual and text features. This ensures the integrity and effectiveness of the information, improves the fluency of the text summary, and broadens the application of multimodal summarization in the field of natural language processing. In the FLAN-T5 encoder, cross-modal multi-head attention and cross-modal bilinear attention are used in parallel for modality fusion. In the decoder, cross-modal multi-head attention is used for modality fusion, and a bundle search strategy is employed at the end to improve the modality fusion effect, thereby enhancing the quality of the text summary and providing a new method for multimodal technology.

[0149] This application also proposes implementation examples of a multimodal summarization method based on visual information fusion, such as... Figure 4 , Figure 5 As shown, this example provides a method for multimodal summarization based on visual information on GPLM. During model training, the method includes:

[0150] Step 1: Input the image of the input text-image pair into the visual encoder of the CLIP model to extract visual features. Then, apply the hyperbolic tangent function, cosine function and sigmoid function to the extracted visual features to perform nonlinear transformations. Finally, concatenate the three visual features to obtain the final visual features.

[0151] In this embodiment, the pre-trained multimodal language model CLIP was pre-trained on a dataset of 400 million image-text pairs constructed by crawling public images and their corresponding descriptive texts from the internet, enabling it to capture deep semantic information of the data. The output of the visual encoder is set to 7×7, and the selected architecture is the ViT model.

[0152] Furthermore, in step one, the image embedding representation obtained using the CLIP image encoder is calculated as follows:

[0153] Embv (i) = CLIP vis (v i )

[0154] Among them, v i It is the i-th image input, CLIP vis It is the visual encoder of the CLIP model, Emb v (i) is the image input v i The embedding vector.

[0155] Furthermore, in step one, the extracted visual features are subjected to nonlinear transformations using the hyperbolic tangent function, cosine function, and sigmoid function, respectively, and the three visual features are then concatenated to obtain the final visual features, calculated as follows:

[0156] V1 = tanh(Emb) v (i))

[0157] V2 = cos(Emb) v (i))

[0158] V3=σ(Emb v (i))

[0159] V = Concat(V1, V2, V3)

[0160] Where V1, V2, and V3 are visual features transformed using the hyperbolic tangent, cosine, and sigmoid functions, respectively; Concat is the concatenation operation; and V is the final visual feature obtained.

[0161] Step 2: Clean the text of the input text-image pair, construct an index of text and visual features in the text, and input the text into the FLAN-T5 model;

[0162] Step 3: The input text and a summary text sequence with an extra start element (without actual content) are fed into the FLAN-T5 model. The former goes to the encoder, and the latter to the decoder. This allows the model to use the previously accurate summary text as input, speeding up training. First, the index portion of the encoder's input text is saved for the corresponding visual features input to the subsequent fusion layer. Then, both input and decoder segments the text into words, padding and masking for insufficient length. The input text is then converted into an index based on the vocabulary. Finally, word embedding is performed to convert the text into vector form. A set of learnable vectors is initialized and fed into the visual feature mapping module along with the visual features. The visual features are obtained using an attention mechanism and concatenated with the text features input from the encoder. Positional encoding is added to the concatenated text feature vector to capture the relative position information of words in the sequence.

[0163] Furthermore, in step three, the visual feature attention mechanism of the visual feature mapping module is calculated as follows:

[0164] Q = [q1, q2, q3, ..., q M ]

[0165] K = V·W v

[0166]

[0167] v = A·V

[0168] Where Q is an initialized learnable vector, Q∈R M×d W v Let K be the weight matrix and V be the extracted visual features.

[0169] In this embodiment, the input text length is set to 256, the summary text length is set to 64, and the initial vector dimension is set to 64.

[0170] Step 4: In the encoder, the input text feature vector enters the multi-head attention module and feedforward network module of the model. The multi-head attention module can capture the interrelationships between different positions in the text sequence and generate rich contextual representations. The feedforward network module can independently perform nonlinear transformations on the input at each position to enhance the model's representational ability. Finally, the feature vector is obtained and enters the modality fusion layer of the encoder. In this embodiment, the number of multi-head attention heads is set to 8.

[0171] Step 5: Visual features and input text features enter the modality fusion layer of the encoder. Through cross-modal multi-head attention fusion mechanism and cross-modal bilinear attention fusion mechanism, the text features obtained by the two mechanisms combined with visual information are weighted and summed to obtain the final text features combined with visual information, which then enter the decoder.

[0172] Furthermore, in step five, a cross-modal multi-head attention fusion mechanism and a cross-modal bilinear attention fusion mechanism are used, and the text features obtained by the two mechanisms, which combine visual information, are weighted and summed. The cross-modal multi-head attention fusion mechanism is calculated as follows:

[0173] Q = T0W q

[0174] K = VW k

[0175] V = VW v

[0176] O = CMA(Q,K,V)

[0177] T1 = Concat(T0, O)W

[0178] Where T0 is the processed input text feature, and W is the weight matrix.

[0179] Furthermore, in step five, the cross-modal bilinear attention fusion mechanism is calculated as follows:

[0180] A = VBT0

[0181] A' = Softmax(A)

[0182] T2=A'T0

[0183] Where B is the bilinear weight matrix.

[0184] Furthermore, in step five, the text features obtained from the two mechanisms, combined with visual information, are weighted and summed as follows:

[0185] T = αT1 + (1-α)T2

[0186] Here, α is the trainable weight.

[0187] Step 6: In the decoder, the text features combined with visual information and the summary text features enter the decoder. The summary text features pass through the multi-head attention module and the text features combined with visual information enter the encoder-decoder multi-head attention module, and then pass through the feedforward network module to obtain text features containing previous text information and visual information. The visual features and the text features enter the modality fusion layer of the decoder to obtain the final text features.

[0188] Step 7: The text features obtained by the decoder are processed by an MLP to convert the high-dimensional vector into a vector dimension that can be mapped by the vocabulary. The resulting vector is then input into a Softmax layer to convert it into a probability distribution, and a beam search is used to obtain the final output probability distribution.

[0189] Step 8: Calculate the cross-entropy loss between the probability distribution predicted by the model and the target summary text, and average the cross-entropy losses at all time steps as the overall average loss. Perform backpropagation (BP) on the calculated average loss and use gradient descent to update the model parameters to minimize the loss. In this embodiment, the learning rate is set to 4e. -3 Set the batch size to 256.

[0190] In step eight, the cross-entropy loss between the probability distribution predicted by the model and the target summary text is calculated as follows:

[0191]

[0192] Among them, y i It is the true distribution of the target text, pi It is the probability distribution predicted by the model.

[0193] In the absence of training, multimodal summarization is performed by adjusting steps three, six, and eight, specifically including:

[0194] Step 3: The input text and the summary text generated in the previous time step are fed into the FLAN-T5 model. The former goes into the encoder, and the latter goes into the decoder. First, the index part of the former is saved so that the corresponding visual features can be input into the subsequent fusion layer. Then, both are segmented into words, and those that are not long enough are padded and masks are constructed. Then, the input text is converted into an index according to the vocabulary. Finally, the text is converted into vector form by word embedding operation. A set of learnable vectors is initialized and fed into the visual feature mapping module along with the visual features. The visual features are obtained using the attention mechanism and concatenated with the text features input from the encoder. Position encoding is added to the text feature vector to capture the relative position information of words in the sequence.

[0195] Step 6: In the decoder, the text features combining visual information and the summary text features generated in the previous time step are fed into the decoder. The summary text features are fed into the encoder-decoder multi-head attention module through the multi-head attention module and the text features combining visual information. Then, they are fed into the feedforward network module to obtain text features containing previous text information and visual information. The visual features are fed into the modality fusion layer of the decoder to obtain the final text features.

[0196] Step 8: Based on the probability distribution and vocabulary, obtain the final summary text.

[0197] A preferred embodiment, such as Figure 4As shown, the input image data is first processed by the pre-trained CLIP image encoder, and then subjected to multi-dimensional activation to obtain visual features. The input text data is embedded and positionally encoded, then concatenated with the visually mapped features to obtain text features, which are then fed into the encoder of the FLAN-T5 model. The text features pass through a multi-head attention module and a feedforward network module before entering the modality fusion layer of the encoder, where they are fused with the visual features. The text and visual features are then fused using a cross-modal multi-head attention mechanism and a cross-modal bilinear attention mechanism. Finally, the results of the two fusion methods are combined to obtain text features incorporating visual information, which are then input into the encoder. The encoder receives the text summary from the previous time step. The input is processed through a multi-head attention module, then enters the encoder-decoder multi-head attention module, where it undergoes multi-head attention operations with the encoder's output. The encoder-decoder text features output by the module are fused with visual features across modality in the decoder's modality fusion layer, and then passed through a feedforward network module before being output from the decoder. Finally, the probability distribution of the summarized text is obtained through linear and softmax layers. Under training conditions, the difference between the generated text summary and the target text is calculated using the cross-entropy loss function, and backpropagation is performed, with gradient descent used to optimize the parameters. Under non-training conditions, a beam search method is used to select the text summary with the highest probability as the output based on the vocabulary.

[0198] The method in this application enriches visual features and incorporates deeper semantics through the connection of multiple nonlinear transformations and a visual feature mapping module. This enhances the correlation between visual and text features, ensures the integrity and effectiveness of information, improves the fluency of text summarization, and broadens the application of multimodal summarization in the field of natural language processing. In the FLAN-T5 encoder, cross-modal multi-head attention and cross-modal bilinear attention are used in parallel for modality fusion. In the decoder, cross-modal multi-head attention is used for modality fusion, and a bundle search strategy is adopted at the end to improve the effect of modality fusion, improve the quality of text summarization, and provide a new method for multimodal technology.

[0199] This application also proposes a multimodal summarization system based on visual information fusion, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps of the aforementioned multimodal summarization method based on visual information fusion.

[0200] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0201] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0202] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0203] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A multimodal summarization method based on visual information fusion, characterized in that, include: Extract visual features from the image in the input text-image pair, and extract text features from the text in the input text-image pair; The extracted visual features are concatenated with the text features, and the concatenated visual-text features are input into the multi-head attention module and feedforward network module of the encoder. The extracted visual features and the output of the feedforward network module are used as inputs to the modality fusion layer of the encoder to obtain text features that combine visual information. The obtained text features combining visual information and the summary text features are input into the decoder, wherein the summary text features are extracted based on the addition of a start element with no actual content to the beginning of the text in the input text-image pair. The extracted visual features and the output of the decoder feedforward network module are used as inputs to the decoder modality fusion layer to obtain the text features output by the decoder. The text features output by the decoder are passed through a linear layer to be converted into vectors of a specified dimension, and the output vectors are then input into a Softmax layer to be converted into a probability distribution. Extracting visual features from the input text-image pair includes: The input text-image pair image is fed into the visual encoder of the CLIP model to extract visual features. The image embedding representation obtained using the CLIP image encoder is as follows: in, It is the i-th image input. It is the visual encoder of the CLIP model. Input image The embedding vector; The extracted visual features are subjected to nonlinear transformations using different functions, and the nonlinearly transformed features are then concatenated to obtain the visual features. The extracted visual features are subjected to nonlinear transformations using different functions, and the transformed features are concatenated to obtain the visual features, including: Nonlinear transformations are performed using the hyperbolic tangent, cosine, and sigmoid functions respectively, and the resulting visual features are then concatenated to satisfy the following condition: in, , , These are visual features that have been varied using the hyperbolic tangent function, cosine function, and sigmoid function, respectively. For splicing operations, The final visual features obtained; Extracting text features from the input text-image pair of Chinese text includes: Based on the text in the input text-image pair, an index of text and visual features is constructed in the text; The text and corresponding visual features, the index, and the summary text features of the input text-image pair are input into the FLAN-T5 model so that the text in the input text-image pair enters the encoder to segment the text and convert it into vector form.

2. The multimodal summarization method based on visual information fusion as described in claim 1, characterized in that, The process of concatenating extracted visual features with textual features includes: A set of learnable vectors is initialized, and these vectors, along with the visual features, are input into the visual feature mapping module. An attention mechanism is then used to obtain the visual features to be concatenated, wherein the visual feature attention mechanism of the visual feature mapping module satisfies the following: in, For the initial learnable vector, , Let K be the weight matrix. For extracted visual features; The visual features to be concatenated are concatenated with the text features input from the encoder; and, Positional encoding is added to the concatenated feature vector to capture the relative positional information of words in the sequence.

3. The multimodal summarization method based on visual information fusion as described in claim 2, characterized in that, The extracted visual features and the output of the feedforward network module are used as inputs to the modality fusion layer of the encoder to obtain text features that combine visual information, including: The visual features and the output of the feedforward network module are input into the modality fusion layer of the encoder. The text features combining visual information obtained by the two mechanisms are weighted and summed through a cross-modal multi-head attention fusion mechanism and a cross-modal bilinear attention fusion mechanism to obtain text features combining visual information. The cross-modal multi-head attention fusion mechanism satisfies the following: in, These are the processed input text features; CMA stands for cross-modal multi-head attention. It is a weight matrix; The cross-modal bilinear attention fusion mechanism satisfies: in, It is a bilinear weight matrix; The text features obtained by the two mechanisms, which combine visual information, are weighted and summed to satisfy the following: in, These are trainable weights.

4. The multimodal summarization method based on visual information fusion as described in claim 1, characterized in that, The obtained text features combining visual information and the summary text features are input into the decoder, including: The abstract text features are passed through a multi-head attention module, and the text features combined with visual information are input into the encoder-decoder multi-head attention module, and then through a feedforward network module to obtain text features containing previous text information and visual information; The obtained text features, which contain previous textual and visual information, and the visual features are input into the modality fusion layer of the decoder to obtain the final text features.

5. The multimodal summarization method based on visual information fusion as described in claim 3, characterized in that, This also includes calculating the cross-entropy loss between the probability distribution predicted by the model and the target summary text, averaging the cross-entropy loss across all time steps to obtain the overall average loss, performing backpropagation on the calculated average loss, updating the model parameters using gradient descent, and minimizing the loss. The cross-entropy loss between the probability distribution predicted by the model and the target summary text satisfies the following condition: in, It is the true distribution of the target text. It is the probability distribution predicted by the model.

6. A multimodal summarization system based on visual information fusion, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the steps of the multimodal summarization method based on visual information fusion as described in any one of claims 1 to 5.

Images