Multimodal abstract generation method and system based on keyword prediction enhancement

Abstract

The present application relates to the technical field of multi-modal abstract generation, in particular to a multi-modal abstract generation method and system based on keyword prediction enhancement, comprising the following steps: 1) obtaining multi-modal abstract data, wherein the multi-modal abstract data comprises text, images corresponding to the text, and text keywords, the text keywords are obtained by calculating entities in the intersection of the original text and the reference abstract, and provide reliable training data for subsequent visual key information extraction; the present application effectively uses a small model as an auxiliary model to extract keywords to assist a large model in performing a multi-modal abstract generation task, more effectively guides the large model to generate a multi-modal abstract by using this framework, can lock core information in the text, thereby obtaining more robust and stronger fact consistency results, effectively solves problems such as focus shift, visual redundancy, and semantic gap between different modalities, and improves the quality of abstract generation.

Description

Technical Field

[0001] This invention relates to the field of multimodal summarization technology, and in particular to a multimodal summarization method and system based on keyword prediction enhancement. Background Technology

[0002] Multimodal summarization aims to extract key information from input data containing multiple modalities (such as text, images, and audio) and generate concise and coherent summaries. Its applications are wide-ranging, including news summarization, video content analysis, multimodal poster generation, and multimodal document summarization. However, multimodal summarization remains a challenging task due to the susceptibility of multimodal data to factors such as inconsistencies between modalities, semantic gaps, noise interference, and data heterogeneity. Existing multimodal summarization methods generally employ deep learning models (such as Transformer or cross-modal attention mechanisms) to learn joint representations of multimodal data and improve the accuracy and coherence of summarization by having the model capture complementary information between modalities or by introducing modality alignment and fusion strategies.

[0003] Current cutting-edge multimodal sentence summarization methods can be divided into two categories: information compression-based methods and information preservation-based methods. Information compression-based methods aim to help the model focus on key content by filtering out task-irrelevant visual information, thereby generating better summaries. Information preservation-based methods, on the other hand, focus on selecting task-relevant visual information to ensure the model can fully utilize visual cues to enhance the accuracy and richness of the summaries.

[0004] Min Xiao et al. (Min Xiao, Junnan Zhu, Haitao Lin, Yu Zhou, and Chengqing Zong. 2023. CFSum Coarse-to-Fine Contribution Network for Multimodal Summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 8538-8553.) proposed a coarse-to-fine multimodal summarization contribution network, CFSum. This network removes useless images through a pre-filtering module and designs two visual supplementary modules at the word and phrase levels. Within a unified framework, it models the different contributions of images to text summarization, thereby improving the quality of multimodal summarization generation.

[0005] Dengtian Lin et al. (Dengtian Lin, Liqiang Jing, Xuemeng Song, Meng Liu, Teng Sun, and Liqiang Nie. 2023. Adapting Generative Pretrained Language Model for Open-domain Multimodal Sentence Summarization. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 195-204.) proposed adapting the generative pretrained language model BART to open-domain multimodal sentence summarization tasks. They designed a prompt-guided image encoding module and an explicit key token learning module to achieve visual information injection and key content modeling within a unified framework, thereby improving summarization performance.

[0006] Minghuan Yuan et al. (Minghuan Yuan, Shiyao Cui, Xinghua Zhang, Shicheng Wang, Hongbo Xu, and Liu, Tingwen. 2024. Exploring the Trade-Off within Visual Information for MultiModal Sentence Summarization. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2006-2017.) proposed T3, a multimodal sentence summarization method based on the information bottleneck. Through variational inference, it maximizes the preservation of task-relevant visual information while compressing task-irrelevant visual information, achieving balanced modeling of visual information within a unified framework, thus improving the accuracy and conciseness of the summarization.

[0007] However, these multimodal summarization methods either employ fixed modality fusion methods (such as splicing and attention mechanisms) or rely on manually designed cue vectors. The relationships between the text and image information obtained through these methods cannot be well modeled or fully utilized. Furthermore, the model framework designed in this way cannot fully express the complex interactions between text and images, resulting in insufficient learning of these relationships when generating summaries. Based on the Bart model and the Lava 1.5-7b large model, this invention combines task-related key information from the visual modality with information from the text modality. This allows the model to obtain corresponding text keywords from the input text and images and fully utilize them, thereby optimizing the quality of summaries generated by the multimodal large model. Summary of the Invention

[0008] The purpose of this invention is to provide a multimodal summarization method and system based on keyword prediction enhancement, which is beneficial for obtaining more stable, robust and accurate multimodal summarization results.

[0009] The technical solution of this invention: The multimodal summarization generation method based on keyword prediction enhancement includes the following steps: 1) Obtain multimodal summary data, which includes text, images corresponding to the text, and text keywords. The text keywords are obtained by calculating entities in the intersection of the original text and the reference summary, providing reliable training data for subsequent visual key information extraction. 2) Construct a multimodal summarization generation model based on keyword prediction enhancement. This model comprises multiple components: a visual key information extraction module, an auxiliary model BART, a text keyword prediction module, and a summarization generator. These components collaboratively execute the multimodal summarization and keyword prediction tasks. The visual key information extraction module filters information from the visual modality to obtain key features. The encoder of the auxiliary model BART encodes information from the text modality to obtain text information encoding, and the decoder of the auxiliary model BART decodes the keyword tokens from the text keyword prediction module to obtain predicted text keywords. The text keyword prediction module fuses visual modality key features with text features, filters the result to obtain keyword tokens, and the decoder of the auxiliary model BART decodes the predicted keyword tokens to obtain predicted text keywords. The summarization generator generates a summary based on the input text, image, and predicted text keywords, enhancing the factual consistency of the multimodal summarization generation model's generated summaries. The constructed multimodal summarization generation model is trained using a multimodal summarization dataset. 3) Use the trained multimodal summarization generation model for multimodal summarization generation tasks.

[0010] Further, in step 1), a multimodal summary generation dataset is obtained from public datasets including MMSS and MSMO, and the text keywords are obtained by calculating the intersection of the text and the reference summary.

[0011] Further, in step 2), the implementation method of the multimodal summarization generation model is as follows: A) Load the weights of the pre-trained auxiliary model BART to initialize the auxiliary model BART, and use the encoder of the initialized auxiliary model BART to extract text features from the multimodal summary generation dataset. B) The image features are fed into the visual key information extraction module. The visual key information extraction module first calculates the image-text consistency through the pre-trained CLIP model and filters out irrelevant images through a pre-set threshold to avoid the influence of irrelevant images on the prediction of text keywords. Then, the filtered images are segmented to obtain image patches, and then the features of each image patch are obtained through the image encoder. After assigning visual perceptrons to them, the visual key features are obtained. C) Visual key features and text features are fed into the text key token filtering module; the text key token filtering module first fuses the two modal information through multi-head attention, and the visual key features will enhance the keyword weight in the text features; then the key token filter filters out non-key tokens in the text features to obtain keyword tokens, and then the decoder of the auxiliary model BART decodes the keyword tokens to obtain the predicted keywords. D) Input the text, image, and predicted keywords into the summary generator. Based on the predicted keywords, the summary generator will lock in the core information in the text, generate a summary, and obtain the final summary. E) During training, the summarizing generator in the multimodal summarizing generation model performs the summarizing task, the auxiliary model BART performs the keyword prediction task, and the parameters of the multiple components are continuously optimized; the auxiliary model optimizes the loss of the visual key information prediction task and the loss of the keyword prediction task by combining them to enhance the model's sensitivity to visual and textual key information; after LoRA fine-tuning, the factual consistency and task adaptability of the large model's generated summaries are enhanced; the summarizing task and the keyword prediction task jointly affect the update of the parameters of the multiple components during training.

[0012] Further, in step A), the encoder of the auxiliary model BART is a BART model encoder pre-trained on a massive English dataset. The BART model encoder encodes the text in the multimodal summary dataset to obtain text features.

[0013] Furthermore, in step B), the specific implementation method of the visual key information extraction module is as follows: First, the image-text consistency is calculated using a pre-trained CLIP model, and then irrelevant images are filtered out using a pre-set threshold to avoid the influence of irrelevant images on the prediction of text keywords.

[0014] ; in and These represent the image and the corresponding text, respectively. Represents the relevance between the image and the text; the higher the relevance, the closer it is to 1. Then retain the image with a size of 224*224. The image was divided into blocks of 32x32 pixels, each sized to fit the input of the VIT encoder, and then processed by a visual encoder. After encoding, visual perceptual elements are assigned, and then concatenated to obtain key visual features; ; ; ; in This represents the j-th image block after image segmentation. The total number of blocks into which each image is segmented. Represents the visual perception metavector. This represents the visual feature vector of the j-th image after the visual perceptrons have been assigned. Represents key visual features. Represents splicing operation, splitter Perform on the image The division.

[0015] Furthermore, in step C), the specific implementation method of the text key token filtering module is as follows: Key visual features Text features Modality fusion is performed through cross-modal multi-head attention (MHA), and text features enhanced with key features are obtained after normalization. ; ; ; ; in, , and Representing the first Each attention head represents a query, key, and value, where T represents matrix transpose. for The transpose, and , and It is a weight matrix used to map features to the query, key, and value spaces. Indicates the first The fused features output by each attention head are used for key feature enhancement; U is the number of attention heads. This indicates the dimensional size of each head; It is layer normalization; Then, the key token filter removes non-key tokens from the text features, resulting in the keyword token, which is then decoded. After decoding the keyword token, the predicted keyword is obtained. ; ; ; ; in, It is the sigmoid function. It is used to generate the filter matrix. The weight matrix; This indicates element-wise multiplication. This represents the keyword "token".

[0016] Further, in step D), the summarization generator is a multimodal large language model LLaVA-1.5-7B-hf pre-trained on a large-scale image and text instruction dataset; when the summarization generator encodes text information based on predicted keywords and images, its internal attention mechanism will enhance the weight of keywords in the text, thereby locking in the core information in the text, generating a summary based on the key information, and obtaining a summary with stronger factual consistency.

[0017] Furthermore, in step E), during the training process, the summarizing generator in the multimodal summarizing generation model performs the LoRA controlled summarizing task and the preference comparison optimization task, the auxiliary model BART performs the keyword prediction task, and the parameters of each component are continuously optimized. ; ; ; in For reference abstract (positive sample), Generate summaries (negative samples) for large, unconstrained models. For the current strategy model, For reference strategy model, For temperature parameters, To optimize the weighting coefficients for preferences, For the Sigmoid function; The first in the true summary One word, For the first part of the true abstract One word, The number of summary words that the model needs to generate in the summary generation task; The first in the real keyword sequence One word, Indicates the number of keywords; The keyword sequence extracted by the auxiliary model serves as a cue. Guide the large model generation process; , represent the trainable parameters of the large model and the auxiliary model, respectively. To generate task loss for summarizing, Predict task loss for keywords; The large model is optimized end-to-end by combining LoRA controlled generation loss and preference contrast loss: ; The auxiliary model is optimized by combining the visual key information prediction task loss and the keyword prediction task loss to enhance the model's sensitivity to visual and textual key information. ; ; ; ; in For the true value of the key token, For Bart encoders, The first prediction of the auxiliary model One keyword token, For linear layers, for Activation layer, For the first The predicted probability of occurrence of each keyword. and These are the visual key information prediction loss and its weight parameters. The total loss of the auxiliary model; LoRA fine-tuning enhances the factual consistency and task adaptability of the large model's generated summaries; the two tasks jointly influence the update of the parameters of each component during training. ; in For the original pre-trained weights, and express The corresponding layer's input feature dimensions and output feature dimensions, The trainable increment introduced for LoRA and This represents the low-rank decomposition matrix in the LoRA method. This represents the rank in rank decomposition. Scaling factor This is the output predicted by the large model.

[0018] Furthermore, it includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the above-described method; the system is deployed on a server, personal computer, or mobile device for generating high-quality multimodal summaries in real time.

[0019] The beneficial effects of this invention are: This invention effectively utilizes a small model as an auxiliary model to extract keywords and assist the large model in multimodal summarization tasks. By using this framework to more effectively guide the large model in generating multimodal summaries, the core information in the text can be locked in, thereby obtaining more robust and factually consistent results. It effectively solves problems such as focus shift, visual redundancy, and semantic gaps between different modalities, and improves the quality of summarization generation. Attached Figure Description

[0020] Figure 1 This is an architecture diagram of the multimodal summarization generation model based on keyword prediction enhancement in this embodiment of the invention; Figure 2 This is a structural diagram of the visual key information extraction module in an embodiment of the present invention; Figure 3 This is a flowchart of the training method for the summary generator module in an embodiment of the present invention. Detailed Implementation

[0021] The invention will now be further described with reference to the accompanying drawings.

[0022] like Figures 1-3 As shown, this invention provides a multimodal summarization method based on keyword prediction enhancement, comprising the following steps: 1) Obtain multimodal summary data, which includes text, images corresponding to the text, and text keywords. The text keywords are obtained by calculating entities in the intersection of the original text and the reference summary, providing reliable training data for subsequent visual key information extraction. 2) Construct a multimodal summarization generation model based on keyword prediction enhancement. This model comprises multiple components: a visual key information extraction module, an auxiliary model BART, a text keyword prediction module, and a summarization generator. These components collaboratively execute the multimodal summarization and keyword prediction tasks. The visual key information extraction module filters information from the visual modality to obtain key features. The encoder of the auxiliary model BART encodes information from the text modality to obtain text information encoding, and the decoder of the auxiliary model BART decodes the keyword tokens from the text keyword prediction module to obtain predicted text keywords. The text keyword prediction module fuses visual modality key features with text features, filters the result to obtain keyword tokens, and the decoder of the auxiliary model BART decodes the predicted keyword tokens to obtain predicted text keywords. The summarization generator generates a summary based on the input text, image, and predicted text keywords, enhancing the factual consistency of the multimodal summarization generation model's generated summaries. The constructed multimodal summarization generation model is trained using a multimodal summarization dataset. 3) Use the trained multimodal summarization generation model for multimodal summarization generation tasks.

[0023] In step 1), a multimodal summary generation dataset is obtained from public datasets including MMSS and MSMO, and the text keywords are obtained by calculating the intersection of the text and the reference summary.

[0024] In step 2), the implementation method of the multimodal summarization generation model is as follows: A) Load the weights of the pre-trained auxiliary model BART to initialize the auxiliary model BART, and use the encoder of the initialized auxiliary model BART to extract text features from the multimodal summary generation dataset. B) The image features are fed into the visual key information extraction module. The visual key information extraction module first calculates the image-text consistency through the pre-trained CLIP model and filters out irrelevant images through a pre-set threshold to avoid the influence of irrelevant images on the prediction of text keywords. Then, the filtered images are segmented to obtain image patches, and then the features of each image patch are obtained through the image encoder. After assigning visual perceptrons to them, the visual key features are obtained. C) Visual key features and text features are fed into the text key token filtering module; the text key token filtering module first fuses the two modal information through multi-head attention, and the visual key features will enhance the keyword weight in the text features; then the key token filter filters out non-key tokens in the text features to obtain keyword tokens, and then the decoder of the auxiliary model BART decodes the keyword tokens to obtain the predicted keywords. D) Input the text, image, and predicted keywords into the summary generator. Based on the predicted keywords, the summary generator will lock in the core information in the text, generate a summary, and obtain the final summary. E) During training, the summarizing generator in the multimodal summarizing generation model performs the summarizing task, the auxiliary model BART performs the keyword prediction task, and the parameters of the multiple components are continuously optimized; the auxiliary model optimizes the loss of the visual key information prediction task and the loss of the keyword prediction task by combining them to enhance the model's sensitivity to visual and textual key information; after LoRA fine-tuning, the factual consistency and task adaptability of the large model's generated summaries are enhanced; the summarizing task and the keyword prediction task jointly affect the update of the parameters of the multiple components during training.

[0025] In step A), the encoder of the auxiliary model BART is a BART model encoder pre-trained on a massive English dataset. The BART model encoder encodes the text in the multimodal summary dataset to obtain text features.

[0026] In step B), the specific implementation method of the visual key information extraction module is as follows: First, the image-text consistency is calculated using a pre-trained CLIP model, and then irrelevant images are filtered out using a pre-set threshold to avoid the influence of irrelevant images on the prediction of text keywords.

[0027] ; in and These represent the image and the corresponding text, respectively. Represents the relevance between the image and the text; the higher the relevance, the closer it is to 1. Then retain the image with a size of 224*224. The image was divided into blocks of 32x32 pixels, each sized to fit the input of the VIT encoder, and then processed by a visual encoder. After encoding, visual perceptual elements are assigned, and then concatenated to obtain key visual features; ; ; ; in This represents the j-th image block after image segmentation. The total number of blocks into which each image is segmented. Represents the visual perception metavector. This represents the visual feature vector of the j-th image after the visual perceptrons have been assigned. Represents key visual features. Represents splicing operation, splitter Perform on the image The division.

[0028] In step C), the specific implementation method of the text key token filtering module is as follows: Key visual features Text features Modality fusion is performed through cross-modal multi-head attention (MHA), and text features enhanced with key features are obtained after normalization. ; The reason why text features enhanced with key features are described using "fuse" instead of words like "enforce" is to emphasize... It is obtained through modality fusion. The key feature enhancement of text features in our small model is done by fusing features.

[0029] ; ; ; in, , and Representing the first Each attention head represents a query, key, and value, where T represents matrix transpose. for The transpose, and , and It is a weight matrix used to map features to the query, key, and value spaces. Indicates the first The fused features output by each attention head are used for key feature enhancement; U is the number of attention heads. This indicates the dimensional size of each head; It is layer normalization; Then, the key token filter removes non-key tokens from the text features, resulting in the keyword token, which is then decoded. After decoding the keyword token, the predicted keyword is obtained. ; ; ; ; in, It is the sigmoid function. It is used to generate the filter matrix. The weight matrix; This indicates element-wise multiplication. This represents the keyword "token". FilterMatrix is ​​a filtering matrix used to filter redundant information in the "Hfuse fusion feature enhanced by key features".

[0030] In step D), the summarization generator is a multimodal large language model LLaVA-1.5-7B-hf pre-trained on a large-scale image and text instruction dataset. When the summarization generator encodes text information based on predicted keywords and images, its internal attention mechanism will enhance the weight of keywords in the text, thereby locking in the core information in the text and generating a summary based on the key information to obtain a summary with stronger factual consistency.

[0031] In step E), during the training process, the summarizing generator in the multimodal summarizing generation model performs the LoRA controlled summarizing task and the preference comparison optimization task, the auxiliary model BART performs the keyword prediction task, and the parameters of each component are continuously optimized. ; ; ; in For reference abstract (positive sample), Generate summaries (negative samples) for large, unconstrained models. For the current strategy model, For reference strategy model, For temperature parameters, To optimize the weighting coefficients for preferences, For the Sigmoid function; The first in the true summary One word, For the first part of the true abstract One word, The number of summary words that the model needs to generate in the summary generation task; The first in the real keyword sequence One word, Indicates the number of keywords; The keyword sequence extracted by the auxiliary model serves as a cue. Guide the large model generation process; , represent the trainable parameters of the large model and the auxiliary model, respectively. To generate task loss for summarizing, Predict task loss for keywords; The large model is optimized end-to-end by combining LoRA controlled generation loss and preference contrast loss: ; The auxiliary model is optimized by combining the visual key information prediction task loss and the keyword prediction task loss to enhance the model's sensitivity to visual and textual key information. ; ; ; ; in For the true value of the key token, For Bart encoders, The first prediction of the auxiliary model One keyword token, For linear layers, for Activation layer, For the first The predicted probability of occurrence of each keyword. and These are the visual key information prediction loss and its weight parameters. The total loss of the auxiliary model; LoRA fine-tuning enhances the factual consistency and task adaptability of the large model's generated summaries; the two tasks jointly influence the update of the parameters of each component during training. ; in For the original pre-trained weights, and express The corresponding layer's input feature dimensions and output feature dimensions, The trainable increment introduced for LoRA and This represents the low-rank decomposition matrix in the LoRA method. This represents the rank in rank decomposition. Scaling factor This is the output predicted by the large model.

[0032] A multimodal summarization generation system based on keyword prediction enhancement includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method. The system is deployed on a server, personal computer, or mobile device for generating high-quality multimodal summaries in real time.

[0033] The present invention will be further described below with reference to a specific embodiment: A multimodal summarization method based on keyword prediction enhancement includes the following steps: 1) Obtain multimodal summary data, which includes text, images corresponding to the text, and text keywords. The text keywords are obtained by calculating entities in the intersection of the original text and the reference summary, providing reliable training data for subsequent visual key information extraction. 2) Construct a multimodal summarization generation model based on keyword prediction enhancement. This model comprises multiple components: a visual key information extraction module, an auxiliary model BART, a text keyword prediction module, and a summarization generator. These components collaboratively execute the multimodal summarization and keyword prediction tasks. The visual key information extraction module filters information from the visual modality to obtain key features. The encoder of the auxiliary model BART encodes information from the text modality to obtain text information encoding, and the decoder of the auxiliary model BART decodes the keyword tokens in the text keyword prediction module to obtain predicted text keywords. The text keyword prediction module fuses visual modality key features with text features, filters the result to obtain keyword tokens, and the decoder of the auxiliary model BART decodes the predicted keyword tokens to obtain predicted text keywords. The summarization generator generates summaries based on the input text, image, and predicted text keywords, enhancing the factual consistency of the model's generated summaries. The constructed multimodal summarization generation model is trained using a multimodal summarization dataset. 3) Use the trained multimodal summarization generation model for multimodal summarization generation tasks.

[0034] In this embodiment, a multimodal summarization dataset is obtained from public datasets including MMSS and MSMO, and the text keywords are obtained by calculating the intersection of the text and the reference summary.

[0035] Figure 1 This is an architecture diagram of the multimodal summarization generation model based on keyword prediction enhancement in this embodiment. For example... Figure 1 As shown, the implementation method of the multimodal summarization generation model is as follows.

[0036] A) Load the weights of the pre-trained auxiliary model BART to initialize the auxiliary model BART, and use the encoder of the initialized auxiliary model BART to extract text features from the multimodal summary generation dataset.

[0037] In this embodiment, the encoder of the auxiliary model BART is a BART model encoder pre-trained on massive English datasets such as English Wikipedia and CC-News. The BART model encoder encodes the text in the multimodal summary dataset to obtain text features.

[0038] B) The image features are fed into the visual key information extraction module. The visual key information extraction module first calculates the image-text consistency through the pre-trained CLIP model and filters out irrelevant images through a pre-set threshold to avoid the influence of irrelevant images on the prediction of text keywords. Then, the filtered images are segmented to obtain image patches, and then the features of each image patch are obtained through the image encoder. After assigning visual perceptrons to them, the visual key features are obtained.

[0039] Figure 2 This is a structural diagram of the visual key information extraction module in this embodiment. For example... Figure 2 As shown, the specific implementation method of the visual key information extraction module is as follows: First, the image-text consistency is calculated using a pre-trained CLIP model, and then irrelevant images are filtered out using a pre-set threshold to avoid the influence of irrelevant images on the prediction of text keywords.

[0040]

[0041] in and These represent the image and the corresponding text, respectively. This represents the relevance of the image to the text; the higher the relevance, the closer it is to 1.

[0042] Then retain the image with a size of 224*224. The image was divided into blocks of 32x32 pixels, each sized to fit the input of the VIT encoder, and then processed by a visual encoder. After encoding, visual perceptual elements are assigned, and then concatenated to obtain key visual features.

[0043] ; ; ; in This represents the j-th image block after image segmentation. The total number of blocks into which each image is segmented. Represents the visual perception metavector. This represents the visual feature vector of the j-th image after the visual perceptrons have been assigned. Represents key visual features. Represents splicing operation, splitter Perform on the image The division.

[0044] Visual key features and text features are fed into the text key token filtering module. The text key token filtering module first fuses the two modal information through multi-head attention, and the visual key features will enhance the keyword weight in the text features. Then, the key token filter filters out non-key tokens in the text features to obtain keyword tokens. Then, the decoder of the auxiliary model BART decodes the keyword tokens to obtain the predicted keywords.

[0045] Figure 3 This is a structural diagram of the text key token filtering module in this embodiment. For example... Figure 3 As shown, the specific implementation method of the text key token filtering module is as follows: Key visual features Text features Modality fusion is performed through cross-modal multi-head attention (MHA), and text features enhanced with key features are obtained after normalization. .

[0046] ; ; ; in, , and Representing the first Each attention head has a query, key, and value. for The transpose, and , and It is a weight matrix used to map features to the query, key, and value spaces. Indicates the first The fused features output by each attention head are used for key feature enhancement. U is the number of attention heads. This indicates the dimensional size of each head. It is layer normalization.

[0047] Then, the key token filter removes non-key tokens from the text features, resulting in the keyword token, which is then decoded. After decoding the keyword token, the predicted keyword is obtained. .

[0048] ; ; ; in, It is the sigmoid function. It is used to generate the filter matrix. The weight matrix. This indicates element-wise multiplication. This represents the keyword "token".

[0049] Text, images, and predicted keywords are fed into the summarizing generator. Based on the predicted keywords, the summarizing generator will identify the core information in the text and generate a summary, resulting in the final summary.

[0050] In this embodiment, the summarization generator is a multimodal large language model LLaVA-1.5-7B-hf pre-trained on a large-scale image and text instruction dataset. When the summarization generator encodes text information based on predicted keywords and images, its internal attention mechanism enhances the weight of keywords in the text, thereby locking in the core information in the text and generating a summary based on the key information to obtain a summary with stronger factual consistency.

[0051] In this method, during the training process, the summarizing generator in the multimodal summarizing generation model performs the summarizing task, while the auxiliary model BART performs the keyword prediction task, and the parameters of each component are continuously optimized. The auxiliary model is optimized by combining the visual key information prediction task loss and the keyword prediction task loss to model the task loss, in order to enhance the model's sensitivity to visual and textual key information. After LoRA fine-tuning, the factual consistency and task adaptability of the large model's generated summaries are enhanced. The two tasks jointly affect the update of the parameters of each component during the training process.

[0052] In the framework of this invention, during the training process, the summarizing generator in the multimodal summarizing generation model performs the LoRA controlled summarizing task and the preference comparison optimization task, the auxiliary model BART performs the keyword prediction task, and the parameters of each component are continuously optimized.

[0053] ; ; ; in For reference abstract (positive sample), Generate summaries (negative samples) for large, unconstrained models. For the current strategy model, For reference strategy model, For temperature parameters, To optimize the weighting coefficients for preferences, For the Sigmoid function; The first in the true summary One word, For the first part of the true abstract One word, The number of summary words that the model needs to generate in the summary generation task. The first in the real keyword sequence One word, Indicates the number of keywords. Keyword sequences extracted for the auxiliary model, serving as prompts ; , represent the trainable parameters of the large model and the auxiliary model, respectively. To generate task loss for summarizing, Predict task loss based on keywords.

[0054] The large model is optimized end-to-end by combining LoRA controlled generation loss and preference contrast loss: ; The auxiliary model is optimized by combining the visual key information prediction task loss and the keyword prediction task loss to enhance the model's sensitivity to visual and textual key information.

[0055] ; ; ; ; in For the true value of the key token, For Bart encoders, The first prediction of the auxiliary model One keyword token, For linear layers, for Activation layer, For the first The predicted probability of occurrence of each keyword. and These are the visual key information prediction loss and its weight parameters. This is the total loss of the auxiliary model.

[0056] LoRA fine-tuning enhances the factual consistency and task adaptability of the large model's generated summaries; the two tasks jointly influence the update of the parameters of each component during training.

[0057] ; in For the original pre-trained weights, and express The corresponding layer's input feature dimensions and output feature dimensions, The trainable increment introduced for LoRA and This represents the low-rank decomposition matrix in the LoRA method. This represents the rank in rank decomposition. Scaling factor This is the output predicted by the large model.

[0058] This embodiment also provides a multimodal summarization generation system based on keyword prediction enhancement, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.

[0059] In this embodiment, validation sets of the MMSS and MSMO datasets are used for comparative validation. Table 1 shows the comparison results of the proposed method with other multimodal summarization methods on the MMSS and MSMO datasets. As can be seen from Table 1, the proposed method has higher accuracy and robustness compared to other multimodal summarization methods, specifically achieving the best Rouge score.

[0060] Table 1

[0061] In Table 1, CFSum corresponds to the method proposed by Min Xiao et al. (Min Xiao, Junnan Zhu, Haitao Lin, Yu Zhou, and Chengqing Zong. 2023. CFSum Coarse-to-Fine Contribution Network for Multimodal Summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 8538-8553.) BART-MMSS corresponds to the method proposed by Dengtian Lin et al. (Ning Zhang, Liqiang Jing, Xuemeng Song, Meng Liu, Teng Sun, and Liqiang Nie. 2023. Adapting GenerativePretrained Language Model for Open-domain Multimodal Sentence Summarization. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 195-204.) T3 corresponds to the method proposed by Minghuan Yuan et al. (Minghuan Yuan, Shiyao Cui, Xinghua Zhang, Shicheng Wang, Hongbo Xu, and Liu, Tingwen. 2024. Exploring the Trade-Off within Visual Information for MultiModal SentenceSummarization. Proceedings of the 47th International ACM SIGIR Conference onResearch and Development in Information Retrieval. 2006-2017.) Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0062] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0064] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0065] The above description is only a preferred embodiment of the present invention and should not be construed as a limitation of this application. All equivalent changes and modifications made in accordance with the scope of the patent application of the present invention should be covered by the present invention.

Claims

1. A method for multi-modal summarization generation based on keyword prediction enhancement, characterized in that, Includes the following steps: 1) Obtain multimodal summary data, which includes text, images corresponding to the text, and text keywords. The text keywords are obtained by calculating entities in the intersection of the original text and the reference summary, providing reliable training data for subsequent visual key information extraction. 2) Construct a multimodal summarization generation model based on keyword prediction enhancement. This model comprises multiple components: a visual key information extraction module, an auxiliary model BART, a text keyword prediction module, and a summarization generator. These components collaboratively execute the multimodal summarization and keyword prediction tasks. The visual key information extraction module filters information from the visual modality to obtain key features. The encoder of the auxiliary model BART encodes information from the text modality to obtain text information encoding, and the decoder of the auxiliary model BART decodes the keyword tokens from the text keyword prediction module to obtain predicted text keywords. The text keyword prediction module fuses visual modality key features with text features, filters the result to obtain keyword tokens, and the decoder of the auxiliary model BART decodes the predicted keyword tokens to obtain predicted text keywords. The summarization generator generates a summary based on the input text, image, and predicted text keywords, enhancing the factual consistency of the multimodal summarization generation model's generated summaries. The constructed multimodal summarization generation model is trained using a multimodal summarization dataset. 3) Use the trained multimodal summarization generation model for multimodal summarization generation tasks. 2.The method of claim 1, wherein, In step 1), a multimodal summary generation dataset is obtained from public datasets including MMSS and MSMO, and the text keywords are obtained by calculating the intersection of the text and the reference summary. 3.The method of claim 1, wherein, In step 2), the implementation method of the multimodal summarization generation model is as follows: A) Load the weights of the pre-trained auxiliary model BART to initialize the auxiliary model BART, and use the encoder of the initialized auxiliary model BART to extract text features from the multimodal summary generation dataset. B) The image features are fed into the visual key information extraction module. The visual key information extraction module first calculates the image-text consistency through the pre-trained CLIP model and filters out irrelevant images through a pre-set threshold to avoid the influence of irrelevant images on the prediction of text keywords. Then, the filtered images are segmented to obtain image patches, and then the features of each image patch are obtained through the image encoder. After assigning visual perceptrons to them, the visual key features are obtained. C) Visual key features and text features are fed into the text key token filtering module; the text key token filtering module first fuses the two modal information through multi-head attention, and the visual key features will enhance the keyword weight in the text features; then the key token filter filters out non-key tokens in the text features to obtain keyword tokens, and then the decoder of the auxiliary model BART decodes the keyword tokens to obtain the predicted keywords. D) Input the text, image, and predicted keywords into the summary generator. Based on the predicted keywords, the summary generator will lock in the core information in the text, generate a summary, and obtain the final summary. E) During training, the summarizing generator in the multimodal summarizing generation model performs the summarizing task, the auxiliary model BART performs the keyword prediction task, and the parameters of the multiple components are continuously optimized; the auxiliary model optimizes the loss of the visual key information prediction task and the loss of the keyword prediction task by combining them, in order to enhance the model's sensitivity to visual and textual key information. LoRA fine-tuning enhances the factual consistency and task adaptability of the large model's summary generation; the summary generation task and the keyword prediction task jointly influence the update of the parameters of the multiple components during training. 4.The method of claim 3, wherein, In step A), the encoder of the auxiliary model BART is a BART model encoder pre-trained on a massive English dataset. The BART model encoder encodes the text in the multimodal summary dataset to obtain text features.

5. The multimodal summarization method based on keyword prediction enhancement according to claim 3, characterized in that, In step B), the specific implementation method of the visual key information extraction module is as follows: First, the image-text consistency is calculated using a pre-trained CLIP model, and irrelevant images are filtered out using a pre-set threshold to avoid the influence of irrelevant images on the prediction of text keywords.   ； in and These represent the image and the corresponding text, respectively. Represents the relevance between the image and the text; the higher the relevance, the closer it is to 1. Then retain the image with a size of 224*224. The image was divided into blocks of 32x32 pixels, each sized to fit the input of the VIT encoder, and then processed by a visual encoder. After encoding, visual perceptual elements are assigned, and then concatenated to obtain key visual features; ； ； ； in This represents the j-th image block after image segmentation. The total number of blocks into which each image is segmented. Represents the visual perception metavector. This represents the visual feature vector of the j-th image after the visual perceptrons have been assigned. Represents key visual features. Represents splicing operation, splitter Perform on the image The division.

6. The multimodal summarization method based on keyword prediction enhancement according to claim 3, characterized in that, In step C), the specific implementation method of the text key token filtering module is as follows: Key visual features Text features Modality fusion is performed through cross-modal multi-head attention (MHA), and text features enhanced with key features are obtained after normalization. ; ； ； ； in, , and Representing the first Each attention head represents a query, key, and value, where T represents matrix transpose. for The transpose, and , and It is a weight matrix used to map features to the query, key, and value spaces. Indicates the first The fused features output by each attention head are used for key feature enhancement; U is the number of attention heads. This indicates the dimensional size of each head; It is layer normalization; Then, the key token filter removes non-key tokens from the text features, resulting in the keyword token, which is then decoded. After decoding the keyword token, the predicted keyword is obtained. ; ； ； ； in, It is the sigmoid function. It is used to generate the filter matrix. The weight matrix; This indicates element-wise multiplication. This represents the keyword "token".

7. The multimodal summarization method based on keyword prediction enhancement according to claim 3, characterized in that, In step D), the summarization generator is a multimodal large language model LLaVA-1.5-7B-hf pre-trained on a large-scale image and text instruction dataset. When the summarization generator encodes text information based on predicted keywords and images, its internal attention mechanism will enhance the weight of keywords in the text, thereby locking in the core information in the text and generating a summary based on the key information to obtain a summary with stronger factual consistency.

8. The multimodal summarization method based on keyword prediction enhancement according to claim 3, characterized in that, In step E), during the training process, the summarizing generator in the multimodal summarizing generation model performs the LoRA controlled summarizing task and the preference comparison optimization task, the auxiliary model BART performs the keyword prediction task, and the parameters of each component are continuously optimized. ； ； ； in For reference, this is a summary. Generate summaries for large, unconstrained models. For the current strategy model, For reference strategy model, For temperature parameters, To optimize the weighting coefficients for preferences, For the Sigmoid function; The first in the true summary One word, For the first part of the true abstract One word, The number of summary words that the model needs to generate in the summary generation task; The first in the real keyword sequence One word, Indicates the number of keywords; The keyword sequences extracted by the auxiliary model serve as prompts to guide the large model generation process; , represent the trainable parameters of the large model and the auxiliary model, respectively. To generate task loss for summarizing, Predict task loss for keywords; The large model is optimized end-to-end by combining LoRA controlled generation loss and preference contrast loss: ； The auxiliary model is optimized by combining the visual key information prediction task loss and the keyword prediction task loss to enhance the model's sensitivity to visual and textual key information. ； ； ； ； in For the true value of the key token, For Bart encoders, The first prediction of the auxiliary model One keyword token, For linear layers, for Activation layer, For the first The predicted probability of occurrence of each keyword. and These are the visual key information prediction loss and its weight parameters. The total loss of the auxiliary model; After fine-tuning with LoRA, the factual consistency and task adaptability of the large model's generated summaries are enhanced. Both tasks influence the update of the parameters of each component during training; ； in For the original pre-trained weights, and express The corresponding layer's input feature dimensions and output feature dimensions, The trainable increment introduced for LoRA and This represents the low-rank decomposition matrix in the LoRA method. This represents the rank in rank decomposition. Scaling factor This is the output predicted by the large model.

9. A multimodal summarization generation system based on keyword prediction enhancement, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, wherein when the processor executes the computer program instructions, it can implement the method as described in any one of claims 1-8; The system is deployed on servers, personal computers, or mobile devices to generate high-quality multimodal summaries in real time.

Images