Bidirectional mask-based multi-modal abstract generation method
A multimodal summarization generation method based on bidirectional masking and feature fusion solves the problem of model dependence on text in multimodal information processing, and generates high-quality summaries containing key image information.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-04-14
AI Technical Summary
Traditional multimodal processing methods, due to the significant differences between text and images in semantic structure and abstraction level, tend to lead to models that over-rely on text and ignore key visual cues, resulting in information compression imbalance and semantic fragmentation.
A multimodal summarization method based on bidirectional masking is adopted. By randomly masking text and images, the model's understanding of text semantics and image structural semantics is enhanced. An attention mechanism is used for feature fusion to strengthen the image semantic independent modeling capability. Finally, an autoregressive method is used to generate a summary.
It achieves a more balanced fusion of multimodal features, and the generated summary contains key information, improving semantic coherence and key information coverage, and avoiding information bias dominated by text.
Smart Images
Figure CN121858735A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to text summarization tasks in the field of natural language processing, and specifically to a multimodal summarization generation method based on bidirectional masking. Background Technology
[0002] In today's era of explosive growth in multimodal information, users struggle to efficiently capture core content and cross-modal relationships when faced with massive amounts of text and image data. Traditional multimodal processing methods, due to the significant differences in semantic structure and abstraction levels between text and images, tend to lead to models that over-rely on text and ignore key visual cues, resulting in information compression imbalances and semantic fragmentation. To address these challenges, a method is needed that can balance the processing of text and image information and strengthen independent modeling of visual semantics to improve the accuracy and completeness of multimodal summarization. Therefore, it is necessary to design a multimodal summarization generation method based on bidirectional masks. Summary of the Invention
[0003] To improve the model's ability to balance multimodal information processing and utilize visual information, this invention proposes a multimodal summarization method based on bidirectional masks, comprising: A method for generating multimodal digests based on bidirectional masks, comprising the following steps: Step 1: Obtain multimodal data containing text and images as input to the model; Step 2: Using corresponding techniques, extract feature vectors from text and images respectively, capturing key information such as text semantics and syntax, and image shape and color; Step 3: Randomly mask words in the text according to a certain proportion, and randomly mask image patches according to a certain probability in the image to enhance the model's understanding of text semantics and image structural semantics; Step 4: Use an attention mechanism to merge text and image features to form a unified multimodal feature representation; Step 5: Based on the fusion features, the model generates text descriptions solely based on the images, enhancing image semantic understanding and independent modeling capabilities; Step 6: Input the fused and visually trained features into the encoder for further deep encoding to uncover complex semantic relationships between features; Step 7: The features processed by the encoder are used for autoregressive prediction, and the output is a summary text that summarizes the key content of multimodal information.
[0004] According to a preferred embodiment, step 1 further includes: This is the starting point for model processing, where multimodal data, including text and images, will be input into the system.
[0005] According to a preferred embodiment, step 2 further includes: Text Feature Extraction: Text contains rich semantic information, but computers cannot directly understand raw text. Utilizing the BART advanced natural language processing model based on the Transformer architecture, elements such as words and sentences in the text are transformed into numerical feature vectors. In this process, the model captures the semantic information of the text through multi-layered encoding, such as semantic relationships between words and the core meaning of sentences; it can also learn grammatical information, such as parts of speech and sentence structure.
[0006] Image Feature Extraction: An image is a collection of visual information composed of a large number of pixels. The VisionTransformer (ViT) computer vision model is used to process the input image. ViT divides the image into multiple image patches, treating them as a sequence input. Within the Transformer architecture, a self-attention mechanism is used to capture the relationships between image patches and extract key features from the image.
[0007] According to a preferred embodiment, step 3 further includes: Text masking: To help the model better understand the contextual semantics of the text, 15% of the tokens in the text are randomly selected for masking. Multiple strategies are combined for the masking method. 80% of the masked tokens are replaced with special [MASK] markers, so that the model needs to infer the original word at the [MASK] position based on the surrounding unmasked tokens during subsequent predictions; 10% of the tokens are replaced with other random words, which increases the difficulty and generalization of the model's predictions and prevents the model from over-relying on fixed patterns; the remaining 10% of the tokens remain unchanged, allowing the model to utilize the original text information during processing.
[0008] Image masking: Images consist of numerous image patches. To improve the model's understanding of image structure and semantics, 60% of these image patches are randomly selected and masked, effectively hiding their original information. In subsequent steps, the model needs to use information from the surrounding visible image patches to reconstruct the masked patches.
[0009] According to a preferred embodiment, step 4 further includes: The preceding steps yielded textual and image features, but these are independent pieces of information from different modalities. To enable the model to comprehensively utilize information from both modalities, they need to be fused. An attention mechanism can be used, allowing the model to automatically learn the correlation between textual and image features. Different weights are assigned to different features based on the strength of the correlation, and then a weighted fusion is performed. This results in a fused feature that more effectively contains key information from both text and image, forming a unified multimodal feature representation and providing a more comprehensive information foundation for subsequent processing.
[0010] According to a preferred embodiment, step 5 further includes: After obtaining the fused multimodal features, additional visual task training is performed specifically for images. The purpose of this step is to enhance the model's understanding of image semantics and its ability to independently model them, avoiding over-reliance on textual information when processing multimodal information. Specifically, the model is required to generate corresponding text descriptions (summaries) based solely on image information, without relying on previous text input. Through repeated training in this manner, the model can establish a direct mapping relationship between image information and text summaries, improving its understanding and abstraction of image content, thereby better utilizing the information contained in images in multimodal summarization generation.
[0011] According to a preferred embodiment, step 6 further includes: The fused multimodal features, along with the features trained and adjusted for the visual task, are input into the BART decoder. The decoder plays a crucial role in the overall model architecture, further processing and transforming the input features. Based on the encoder's pre-defined architecture and parameters, it performs deep encoding of the multimodal features. During this process, the encoder uncovers more complex semantic relationships and patterns between features. Through multiple layers of encoding operations, the input features are transformed into a more suitable feature representation for generating summaries, preparing the model for subsequent summarization and enabling it to accurately generate compliant summary content based on these processed features.
[0012] According to a preferred embodiment, step 7 further includes: The features processed by the encoder contain rich multimodal information and have been transformed into a form suitable for summarizing. At this point, the model generates a summary using an autoregressive approach. In this process, the model generates summary text that accurately summarizes the key content of the input multimodal information based on language patterns, semantic relationships, and a comprehensive understanding of multimodal information learned during training. The final output summary concisely and accurately reflects the main information contained in the original text and image, meeting the user's need for rapid understanding and retrieval of key content from multimodal information.
[0013] This invention relates to electronic devices, characterized in that they comprise: At least one processor and at least one memory communicatively connected to the processor, wherein the processor is capable of performing at least the following: Preprocessing multimodal data: cleaning text and images, extracting structured information to form samples.
[0014] Bidirectional masking modeling: randomly masking words in text and predicting and reconstructing them, and randomly masking image blocks in images and restoring pixels, thereby enhancing the semantic understanding of text and images.
[0015] Feature fusion and visual training: Integrate text and image features to generate summaries using only images to enhance independent modeling of visual semantics.
[0016] Generate multimodal summaries: Generate summaries through autoregression using a decoder, and output the results of condensing key information.
[0017] The beneficial effects of this invention are: By using a bidirectional masking training mechanism for text and images, the model's ability to capture cross-modal semantic associations is enhanced, making the fusion of multimodal features more balanced and helping to generate accurate summaries containing key information from images and text. Independent visual training tasks construct a direct mapping from visual information to summary semantics, avoiding information bias dominated by text, and improving the model's efficiency in independently abstracting and utilizing image content, especially suitable for scenarios where images contain key clues; The multi-task joint optimization strategy enhances the model's ability to focus on task-related features in the input information, and the generated summary is significantly improved in terms of semantic coherence and key information coverage. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating the multimodal digest generation method based on bidirectional masking of the present invention. Figure 2 This is a schematic diagram of the text masking process in the bidirectional masking of this invention. Detailed Implementation
[0019] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.
[0020] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, it should be pointed out that unless otherwise specified, the structures, connections, positional relationships, power source relationships, etc., involved in this invention are all things that those skilled in the art can discover without creative effort based on existing technology. Example
[0021] refer to Figure 1 As shown in the figure, a multimodal digest generation method based on bidirectional masking is illustrated. The multimodal digest generation method based on bidirectional masking includes the following steps.
[0022] Step 1: Obtain multimodal data containing text and images as input to the model.
[0023] This is the starting point for model processing, where multimodal data, including text and images, will be input into the system.
[0024] Step 2: Using corresponding techniques, extract feature vectors from text and images respectively, and capture key information such as text semantics and syntax and image shape and color.
[0025] Text Feature Extraction: Text contains rich semantic information, but computers cannot directly understand raw text. Utilizing the BART advanced natural language processing model based on the Transformer architecture, elements such as words and sentences in the text are transformed into numerical feature vectors. In this process, the model captures the semantic information of the text through multi-layered encoding, such as semantic relationships between words and the core meaning of sentences; it can also learn grammatical information, such as parts of speech and sentence structure.
[0026] Image Feature Extraction: An image is a collection of visual information composed of a large number of pixels. The VisionTransformer (ViT) computer vision model is used to process the input image. ViT divides the image into multiple image patches, treating them as a sequence input. Within the Transformer architecture, a self-attention mechanism is used to capture the relationships between image patches and extract key features from the image.
[0027] Step 3: Randomly mask words in the text according to a certain proportion, and randomly mask image patches according to a certain probability in the image to enhance the model's understanding of text semantics and image structural semantics.
[0028] In the masked language modeling task, we combine textual and visual information to predict the masked tokens in the text. Similarly, in the masked image modeling (MIM) task, we utilize visual and textual information to predict the original RGB values of the image.
[0029] We model the MLM task of text by randomly masking 15% of the tokens in the input text. Of these 15% of tokens, there is an 80% probability that the token will be replaced with a [MASK] token, a 10% probability that it will be replaced with a random token, and the remaining 10% will remain unchanged.
[0030] We use This represents the index of the masked token, where M is the number of masked tokens. The loss function of MLM is defined as shown in the formula: ; In the formula Represents the loss function; This represents the masked token; This represents the remaining unmasked tokens.
[0031] For MIM modeling, we employ a random masking strategy based on image patch alignment, where the masking operation is performed at the image patch level; that is, an image patch is either fully visible or completely masked. We use a 32×32 image patch size, with a random masking probability of 60%.
[0032] Next, we use regression to predict the original pixel values of the masked region. Specifically, we will use visual features... Each feature vector in the algorithm is mapped back to the original resolution, and this vector is used to predict the corresponding original pixel as shown in the formula: ; In the formula These are the predicted pixel values.
[0033] Step 4: Use an attention mechanism to merge text and image features to form a unified multimodal feature representation.
[0034] The preceding steps yielded textual and image features, but these are independent pieces of information from different modalities. To enable the model to comprehensively utilize information from both modalities, they need to be fused. An attention mechanism can be used, allowing the model to automatically learn the correlation between textual and image features. Different weights are assigned to different features based on the strength of the correlation, and then a weighted fusion is performed. This results in a fused feature that more effectively contains key information from both text and image, forming a unified multimodal feature representation and providing a more comprehensive information foundation for subsequent processing.
[0035] Step 5: Based on the fusion features, the model generates text descriptions (summaries) solely from the images, enhancing image semantic understanding and independent modeling capabilities.
[0036] After obtaining the fused multimodal features, additional visual task training is performed specifically for images. The purpose of this step is to enhance the model's understanding of image semantics and its ability to independently model them, avoiding over-reliance on textual information when processing multimodal information. Specifically, the model is required to generate corresponding text descriptions (summaries) based solely on image information, without relying on previous text input. Through repeated training in this manner, the model can establish a direct mapping relationship between image information and text summaries, improving its understanding and abstraction of image content, thereby better utilizing the information contained in images in multimodal summarization generation.
[0037] Step 6: Input the fused and visually trained features into the encoder for further deep encoding to uncover complex semantic relationships between features.
[0038] The fused multimodal features, along with the features trained and adjusted for the visual task, are input into the BART decoder. The decoder plays a crucial role in the overall model architecture, further processing and transforming the input features. Based on the encoder's pre-defined architecture and parameters, it performs deep encoding of the multimodal features. During this process, the encoder uncovers more complex semantic relationships and patterns between features. Through multiple layers of encoding operations, the input features are transformed into a more suitable feature representation for generating summaries, preparing the model for subsequent summarization and enabling it to accurately generate compliant summary content based on these processed features.
[0039] Step 7: The features processed by the encoder are used for autoregressive prediction, and the output is a summary text that summarizes the key content of multimodal information.
[0040] The features processed by the encoder contain rich multimodal information and have been transformed into a form suitable for summarizing. At this point, the model generates a summary using an autoregressive approach. In this process, the model generates summary text that accurately summarizes the key content of the input multimodal information based on language patterns, semantic relationships, and a comprehensive understanding of multimodal information learned during training. The final output summary concisely and accurately reflects the main information contained in the original text and image, meeting the user's need for rapid understanding and retrieval of key content from multimodal information.
[0041] This invention relates to electronic devices, characterized in that they comprise: At least one processor and at least one memory communicatively connected to the processor, wherein the processor is capable of performing at least the following: Preprocessing multimodal data: cleaning text and images, extracting structured information to form samples;
[0042] Bidirectional masking modeling: randomly masking words in text and predicting and reconstructing them, and randomly masking image patches in images and restoring pixels to enhance the semantic understanding of text and images;
[0043] Feature fusion and visual training: Integrating text and image features, generating summaries using only images to enhance independent modeling of visual semantics;
[0044] Generate multimodal summaries: Generate summaries through autoregression using a decoder, and output the results of condensing key information.
Claims
1. A multimodal digest generation method based on bidirectional masking, characterized in that, include: Step 1: Obtain multimodal data containing text and images as input to the model; Step 2: Using corresponding techniques, extract feature vectors from text and images respectively, capturing key information such as text semantics and syntax, and image shape and color; Step 3: Randomly mask words in the text according to a certain proportion, and randomly mask image patches according to a certain probability in the image to enhance the model's understanding of text semantics and image structural semantics; Step 4: Use an attention mechanism to merge text and image features to form a unified multimodal feature representation; Step 5: Based on the fusion features, the model generates text descriptions (summaries) solely from the images, enhancing image semantic understanding and independent modeling capabilities; Step 6: Input the fused and visually trained features into the encoder for further deep encoding to uncover complex semantic relationships between features; Step 7: The features processed by the encoder are used for autoregressive prediction, and the output is a summary text that summarizes the key content of multimodal information.
2. The multimodal digest generation method based on bidirectional masking as described in claim 1, characterized in that, Step 1 includes: This is the starting point for model processing, where multimodal data, including text and images, will be input into the system.
3. The multimodal digest generation method based on bidirectional masking as described in claim 1, characterized in that, Step 2 includes: Text Feature Extraction: Text contains rich semantic information, but computers cannot directly understand raw text. Using the BART advanced natural language processing model based on the Transformer architecture, elements such as words and sentences in the text are transformed into numerical feature vectors. In this process, the model captures the semantic information of the text through multi-layer encoding, such as semantic relationships between words and the core meaning of sentences; it can also learn grammatical information, such as parts of speech and sentence structure. Image Feature Extraction: Images are collections of visual information composed of a large number of pixels. The Vision Transformer (ViT) computer vision model is used to process the input image. ViT divides the image into multiple image patches, treating them as a sequence input into the Transformer architecture, and uses a self-attention mechanism to capture the relationships between image patches and extract key features of the image.
4. The multimodal digest generation method based on bidirectional masking as described in claim 1, characterized in that, Step 3 includes: Text masking: To help the model better understand the contextual semantics of the text, 15% of the tokens in the text are randomly selected for masking. Multiple strategies are combined for masking. 80% of the masked tokens are replaced with special [MASK] markers, so the model needs to infer the original word at the [MASK] position based on the surrounding unmasked tokens during subsequent predictions. 10% of the tokens are replaced with other random words, which increases the difficulty and generalization of the model's predictions and prevents the model from over-relying on fixed patterns. The remaining 10% of the tokens remain unchanged, allowing the model to utilize the original text information during processing. Image masking: Images consist of numerous image patches. To improve the model's understanding of image structure and semantics, 60% of the image patches are randomly selected for masking, which hides the original information of these image patches. In subsequent steps, the model needs to use information from the surrounding visible image patches to reconstruct the masked image patches.
5. The multimodal digest generation method based on bidirectional masking as described in claim 1, characterized in that, Step 4 includes: After the preceding steps, text features and image features were obtained, but these are independent information from different modalities. In order for the model to comprehensively utilize information from these two modalities, they need to be fused. Using the attention mechanism, the model can automatically learn the degree of correlation between text features and image features, assign different weights to different features according to the strength of the correlation, and then perform weighted fusion. This allows the fused features to more effectively contain the key information of both text and image, forming a unified multimodal feature representation, and providing a more comprehensive information foundation for subsequent processing.
6. The multimodal digest generation method based on bidirectional masking as described in claim 1, characterized in that, Step 5 includes: After obtaining the fused multimodal features, additional visual task training is performed specifically for images. The purpose of this step is to enhance the model's understanding of image semantics and its ability to independently model images, and to avoid the model relying too much on text information when processing multimodal information. Specifically, the model is required to generate corresponding text descriptions based solely on image information, without relying on previous text input. By repeatedly performing this training, the model can establish a direct mapping relationship between image information and text summarization, improve its understanding and abstraction of image content, and thus better utilize the information contained in the image in multimodal summarization generation.
7. The multimodal digest generation method based on bidirectional masking as described in claim 1, characterized in that, Step 6 includes: The fused multimodal features, along with the features trained and adjusted for the visual task, are input into the BART decoder. The decoder plays a crucial role in the overall model architecture, further processing and transforming the input features. Based on the encoder's pre-defined architecture and parameters, it performs deep encoding on the multimodal features. During this process, the encoder uncovers more complex semantic relationships and patterns between features. Through multiple layers of encoding operations, it transforms the input features into a more suitable feature representation for generating summaries, preparing for subsequent summarization. This enables the model to accurately generate compliant summary content based on these processed features.
8. The multimodal digest generation method based on bidirectional masking as described in claim 1, characterized in that, Step 7 includes: The features processed by the decoder contain rich multimodal information and have been transformed into a form suitable for generating summaries. At this point, the model generates summaries through autoregression. In this process, the model generates summary text that can accurately summarize the key content of the input multimodal information based on the language patterns, semantic associations, and comprehensive understanding of multimodal information learned during training. The final output summary can concisely and accurately reflect the main information contained in the original text and image, meeting the user's need for quick understanding and acquisition of key content of multimodal information.
9. This invention relates to electronic devices, characterized in that, include: At least one processor and at least one memory communicatively connected to the processor, wherein the processor is capable of performing at least the following: Preprocessing multimodal data: cleaning text and images, extracting structured information to form samples; Bidirectional masking modeling: randomly masking words in text and predicting and reconstructing them, and randomly masking image patches in images and restoring pixels, thereby enhancing the semantic understanding of text and images; Feature fusion and visual training: Integrating text and image features, generating summaries using only images to enhance independent modeling of visual semantics; Generate multimodal summaries: Generate summaries through autoregression using a decoder, and output the results of condensing key information.