Vision-text collaborative abstract generation method and system based on multi-modal learning

Abstract

The invention belongs to the field of natural language processing crossing, and particularly relates to a visual-text collaborative abstract generation method and system based on multi-modal learning, and the system comprises a multi-modal data receiving module which is used for receiving multi-modal input data in parallel, and the multi-modal input data comprises text data and visual data; the visual semantic understanding module is used for extracting high-level semantic features of the image by adopting a visual semantic understanding model and generating text description; the semantic fusion module is used for carrying out semantic fusion on the original text and the generated image description by adopting a vision-text semantic fusion method based on direction consistency and self-adaptive semantic completion; and the abstract optimization module is used for carrying out semantic reconstruction on the fusion features by adopting a multilayer deep fusion hybrid neural network architecture, and generating an abstract text conforming to context semantics and accurate expression. The method has the advantages that visual and text information is accurately aligned, high-quality abstracts are generated, and the method is particularly suitable for news reports, conference records, video content analysis and other scenes needing cross-modal information fusion.

Description

Technical Field

[0001] The present application belongs to the interdisciplinary field of natural language processing, and specifically relates to a visual-text collaborative summary generation method and system based on multimodal learning. Background Art

[0002] With the rapid development of artificial intelligence technology, summary generation tasks have become a hot topic in current research. However, traditional summary generation methods often rely on a single image description generation technology or a text summary generation model, but both have their own limitations. Image description generation models usually find it difficult to accurately capture the subtle semantic information in the image, and the generated descriptions are often too brief or vague, lacking a deep understanding of the image content. This makes the image description unable to fully reflect the complex details of the image, which in turn affects the quality of the summary. Although text summary generation models perform well at the text level, they usually ignore the deep semantic association between images and text, resulting in the generated summary being inconsistent with the image content and lacking semantic accuracy and contextual coherence.

[0003] In order to solve the limitations brought by a single modality, multimodal learning came into being. Multimodal learning is committed to improving the understanding and generation capabilities of the model by integrating data of different modalities (such as images, text, videos, etc.). In visual-text tasks, images and text, as two different modal information, need to be effectively fused in order to extract richer semantic information from them, thereby improving the quality of natural language generation tasks. However, when dealing with complex situations, existing multimodal summary generation methods often cannot dynamically adjust the information fusion strategy between images and texts, resulting in the inability to fully explore the potential semantic relationship between images and texts in some scenarios, and unable to generate summaries that better match the input text. Therefore, how to efficiently fuse visual and text information and improve the accuracy and semantic consistency of summary generation is still a problem that needs to be solved. Summary of the invention

[0004] The present invention provides a visual-text collaborative summary generation method based on multimodal learning, which aims to effectively integrate image and text information and improve the accuracy and semantic consistency of summary generation. The present invention can generate a summary that conforms to the context semantics while ensuring that there is no semantic conflict between the image and the text by introducing a visual understanding module, a text and visual understanding combination module, and a summary generation module. The technical solution is as follows: A visual-text collaborative summary generation method based on multimodal learning includes the following steps: S1. Receive multimodal data: including text data and visual data as the basis for subsequent summary generation; S2. Visual semantic understanding: Input visual data into the visual semantic understanding model, extract visual features, identify key information in images or videos, and generate text descriptions of the visual data content; S3. Combining text with visual semantic understanding: effectively fusing the text data in step S1 with the image description text or video parsing text generated in step S2 to provide semantic information for the summary generation model; S4. Summary generation: Input the fused text in step S3 into the summary generation model, use the model to understand and process the input text, and finally generate a text summary that conforms to the contextual semantics.

[0005] Preferably, the visual semantic understanding model in step S2 includes video key frame recognition, calculates the importance weight of the pixel area through adaptive spatiotemporal weight fusion, identifies the core changes of the video content, removes redundant information, and obtains a set of representative and semantically rich key frame images: Adaptive spatiotemporal weight fusion: Based on the standard spatiotemporal attention, a dynamic weight factor is added to adaptively adjust the attention in the temporal and spatial dimensions; Temporal attention focuses on the relationship between different frames in a video sequence, reflecting the dynamic changes between frames; the temporal attention calculation formula is as follows: ; The attention in the spatial dimension focuses on the regional relationship within a single frame image, emphasizing the local and global structure of the object. The calculation formula is as follows: ; The calculation formula of adaptive spatiotemporal weight fusion is as follows: ; is the temporal attention factor, learning the global importance of each frame, is the spatial attention factor, , All of them are learnable parameters, so that the model can dynamically adjust the time and space weights through the sigmoid function Normalize to ensure and There will be no negative values ​​or values ​​that are too large or too small.

[0006] Preferably, the visual semantic understanding model in step S2 includes unimodal visual feature encoding, and a hybrid convolution strategy is adopted to combine standard convolution and depthwise separable convolution. The unimodal visual feature encoding adopts a dual-branch structure, and extracts global features through dilated convolution and local features through hybrid convolution through two parallel computing paths, and performs adaptive feature fusion through dual-branch fusion.

[0007] Preferably, dilated convolution: by introducing a dynamic dilation factor strategy, using dilation factors of different scales Multi-level information fusion is performed so that the network can capture long-distance pixel dependencies while keeping local information intact. The calculation formula is as follows: ; ; in, is the position in the output feature map The pixel value at ; Represents input features In Location The pixel value at is the index of the convolution kernel, indicating the position of the convolution kernel in the sliding window; R represents the receptive field, the set of neighborhood pixels involved in the convolution operation; Indicates that the convolution kernel is at position The weight of is the dynamic expansion factor, which controls the sampling interval of the convolution kernel. is the current feature map at position The local features of and is the parameter to be learned; when When , it is a standard convolution, when When , there are gaps between the sampling points of the convolution kernel; Depthwise separable convolutions include: Channel-wise convolution: ; in, Represents the first Channels, positions The pixel value at represents the index of the convolution kernel, is the convolution kernel at position The weight of Point-wise convolution: ; Where C is the number of input channels, is the output of channel-wise convolution, Is the convolution kernel, used for cross-channel weighted summation, by adjusting The number of controls the number of output channels; Mixed convolution formula: ; in, Indicates that the mixed convolution operation is at position The output feature map at is the balance coefficient, , used to control the weight between standard convolution and depth-wise separable convolution; Two-branch fusion formula: ; in, Represents the fusion formula of multi-scale dilated convolution and mixed convolution, It is the dynamic fusion weight that controls the proportion of global and local information.

[0008] Preferably, the visual semantic understanding model in step S2 includes image-text contrast loss and image-text matching loss; Image-text contrast loss: By activating the unimodal visual feature encoder, aligning the feature spaces of vision and text, positive sample image-text pairs are encouraged to have similar representations, while negative sample pairs have greater discrimination, thus enhancing the relevance between image and text. The contrast loss formula is as follows: ; in, Representing image features and text features The similarity between is the logarithmic loss function, N is the number of samples; Image-text matching loss: By activating the image-based text encoder, we learn to capture the image-text multimodal representation that captures the fine-grained alignment between vision and language. We use the image-text matching loss to predict whether the image-text pair given the multimodal features matches. The matching loss formula is as follows: ; ; in, is the label of the image-text pair, Indicates that the image and text match, Indicates that the image and text do not match, is the probability that the model predicts whether the image and text match, using the logarithmic loss function To optimize the model, is the image feature and text features The similarity calculation between is the sigmoid function, which is used to normalize the similarity value to , ensuring that it can be used as a probability.

[0009] Preferably, the visual semantic understanding model in step S2 includes a language modeling loss, and generates a text description of a given image by activating an image-based text encoder, and maximizes the possibility of generating text in an autoregressive manner. The calculation formula is as follows: ; in, Given the previous text and image information Next, generate the next word The probability of Q represents the length of the generated text sequence.

[0010] Preferably, the combination of text and visual semantic understanding in step S3 includes image-text consistency assessment, text-visual semantic coordination detection, and context supplementation based on dynamic semantic fusion; Image-text consistency assessment: Directional consistency is used to measure the similarity between the input text and the image description text. The higher the directional consistency, the higher the similarity between the text description image. The calculation formula is as follows: ; in, and are the vector representations of the input text and image description text respectively, and T is the transpose.

[0011] Text-visual semantic coordination detection: Based on the similarity calculation results, determine whether there is a semantic conflict between the input text and the image description text. If the similarity between the two is low or there is an obvious contradiction, mark them as a conflict to avoid incorrect fusion; Contextual supplementation based on dynamic semantic fusion includes keyword alignment, complementary weight calculation and dynamic information fusion; Keyword alignment: extract keywords from the input text and image description text, calculate their similarity, and evaluate the degree of semantic alignment between the two; Complementarity weight calculation: The adaptive alignment algorithm is used to calculate the complementary weight between text and visual content. This weight measures the relative importance of the input text and the image description text. The complementary weight is calculated through a gating mechanism and is expressed as ,in is the weight matrix, and are the encoded features of the input text and image description text, respectively. Represents the complementarity weight of text and image description: ; Dynamic Information Fusion: Based on Complementarity Weights , fuse the input text with the image description text by adjusting the weights of the two proportionally to generate an enhanced text representation , which includes the combination of image and text information, is calculated as follows: .

[0012] Preferably, in step S4, the model for summary generation includes a text encoding layer, a position encoding layer, a DeepFusion-Transformer layer, and an autoregressive decoding layer; the text fused in step S3 is input into the trained summary generation model to generate a text summary; Text encoding layer: converts each word in the text into a vector representation of fixed dimension and encodes the semantics of each word with its contextual relationship, thereby obtaining a high-quality representation of each word as the input of DeepFusion-Transformer; Position encoding layer: Use sine and cosine functions to generate position encoding so that the encoding of each position has different frequencies in different dimensions; DeepFusion-Transformer layer: It adopts a stacked multi-layer structure. Each layer gradually learns more advanced features. The shallow Transformer layer focuses on capturing local semantic information, while the deep Transformer layer focuses on learning global semantic representation. Autoregressive decoding layer: The decoder generates text step by step in an autoregressive manner. After each new word is generated, the decoder updates its state, takes the generated word as a new input, and continues to generate the next step in combination with the current context information until a complete summary text is generated.

[0013] Preferably, the formula for the autoregressive generation process is as follows: ; in, is given Step word and input features Next, generate The probability distribution of the step vocabulary; is the current decoder hidden state, and are the weights and biases used to calculate the vocabulary distribution; the hidden state of the decoder is updated at each time step, assuming that The hidden state of step is , then in Step , the new hidden state This can be calculated recursively: ; in, yes Step generated words, is the hidden state at the previous moment, It is the multimodal information of the input.

[0014] A visual-text collaborative summary generation system based on multimodal learning, comprising a multimodal input module, a visual semantic understanding module, a text and visual understanding combination module and a summary generation module; Multimodal input module: accepts input text data and visual data. The main function of this module is to provide raw input for subsequent visual understanding and text fusion; Visual semantic understanding module: The visual data is input into the visual semantic understanding model. The model extracts visual features, identifies key information in images or videos, and generates a detailed description of the visual data content. Text and visual understanding combination module: effectively combines the input text with the image description text or video analysis text generated by the visual understanding model to obtain a fusion representation of text and image content; Summary generation module: The text generated by the text and visual understanding module is input into the generation model. Through the model's understanding and processing of the text, a text summary that conforms to the contextual semantics is finally generated.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) Efficient fusion of multimodal information: The present invention effectively achieves deep fusion of visual information and text information by introducing a visual understanding module and a text-visual understanding combination module. Compared with traditional single-modality summary generation methods, the present invention can fully utilize the complementarity of images and text in the same summary, thereby improving the semantic richness and accuracy of the summary.

[0016] (2) Dynamic semantic alignment and information supplementation: The dynamic semantic alignment algorithm proposed in this paper can dynamically adjust the fusion strategy according to the semantic relationship between text and image, effectively avoiding the semantic conflict between image description and text content, and ensuring that the fused text summary is more coherent and consistent. Compared with the static fusion method of existing methods, this paper can flexibly adjust the information fusion strategy under different contextual situations, thereby improving the quality of the generated summary.

[0017] (3) Adaptability to complex scenarios: The summary generation method of the present invention can handle complex application scenarios, such as long news, meeting minutes, and assistive technology for the blind. In particular, in scenarios where both visual information and text information exist, it can generate accurate and comprehensive text summaries by combining precise image descriptions with context. When existing technologies cannot effectively handle these scenarios, the present invention provides a more advanced and efficient solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1An overall flow chart of a visual-text collaborative summary generation method based on multimodal learning disclosed in an embodiment of the present invention; Figure 2 Schematic diagram of the visual semantic understanding stage of an embodiment of the present invention; Figure 3 This is a schematic diagram of the text fusion stage of an embodiment of the present invention; Figure 4 Schematic diagram of the summary generation stage of an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the embodiments of the present invention and the drawings of the specification to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The present invention provides a visual-text collaborative summary generation method based on multimodal learning, which aims to effectively integrate image and text information and improve the accuracy and semantic consistency of summary generation. The present invention can generate a summary that conforms to the context semantics while ensuring that there is no semantic conflict between the image and the text by introducing a visual understanding module, a text and visual understanding combination module, and a summary generation module.

[0021] like Figure 1 As shown, the specific embodiments are as follows: A visual-text collaborative summary generation method based on multimodal learning includes the following steps: (1) Receiving text data and visual data: Receive multimodal input data including text data and visual data. The text data can be input in the form of articles, reports, or other text, while the visual data can be visual information of image files, image sequences, or videos. This step is intended to provide the multimodal input data to subsequent processing modules.

[0022] (2) Visual semantic understanding stage: Video key frame recognition. The importance weight of the pixel area is calculated through the adaptive spatiotemporal attention mechanism, the core changes of the video content are identified, and redundant information is removed to obtain a set of representative and semantically rich key frame images. The calculation process is as follows: Adaptive spatiotemporal weight fusion. On the basis of standard spatiotemporal attention, a dynamic weight factor is added to adaptively adjust the attention in the temporal and spatial dimensions. The temporal dimension focuses on the relationship between different frames in a video sequence, mainly reflecting the dynamic changes between frames, such as motion trajectories and scene switching; the spatial dimension focuses on the regional relationship within a single frame image, emphasizing the local and global structure of objects, such as object shape, texture, and boundaries. The calculation formula is as follows: Time attention: ; Spatial Attention: ; Adaptive Spatiotemporal Attention: ; in, represents the attention matrix after weighting the importance of different frames in the time dimension, Represents the attention matrix after weighting the importance of different regions in the spatial dimension; is the temporal attention factor, learning the global importance of each frame, is the spatial attention factor, , All of them are learnable parameters, so that the model can dynamically adjust the time and space weights through the sigmoid function Normalize to ensure and There will be no negative values ​​or values ​​that are too large or too small, which prevents unstable training.

[0023] Unimodal visual feature encoding. The present invention proposes a dilated convolution hybrid visual encoder, which enhances the modeling capability of global information through dilated convolution. At the same time, it combines the hybrid convolution structure and utilizes the combination of depthwise separable convolution and standard convolution to improve the model's compatibility with multi-scale targets. Dilated convolution expands the receptive field by introducing a dilation factor, so that the encoder can capture the dependencies between distant pixels in the image, and is suitable for processing targets with a large spatial distribution. The hybrid convolution structure can effectively extract local and global features of the image, thereby reducing computational complexity while improving the expressiveness and efficiency of the model.

[0024] ; ; in, is the position in the output feature map The pixel value at ; Represents input features In Location The pixel value at is the index of the convolution kernel, indicating the position of the convolution kernel in the sliding window; Indicates that the convolution kernel is at position The weight of is the dynamic expansion factor, which controls the sampling interval of the convolution kernel. is the current feature map at position The local features of and is the parameter to be learned; when When , it is a standard convolution, when When , there are gaps between the sampling points of the convolution kernel; Depthwise separable convolutions include: Channel-wise convolution: ; in, Represents the first Channels, positions The pixel value at represents the index of the convolution kernel, is the convolution kernel at position The weight of Point-wise convolution: ; in, is the output of channel-wise convolution, Is the convolution kernel, used for cross-channel weighted summation, by adjusting The number of controls the number of output channels; Mixed convolution formula: ; in, Indicates that the mixed convolution operation is at position The output feature map at is the balance coefficient, , used to control the weight between standard convolution and depth-wise separable convolution; Two-branch fusion formula: ; in, Represents the fusion formula of multi-scale dilated convolution and mixed convolution, It is the dynamic fusion weight that controls the proportion of global and local information.

[0025] Image-based text encoding: The features extracted by the image encoder in a single modality are converted into text features, and visual information is injected by introducing additional cross-attention between the self-attention layer and the feed-forward network layer to obtain a multimodal representation containing the image content; Image-based text decoding. The fused image-text information is further processed to generate a natural language description. By adopting the causal self-attention mechanism, the generated description text conforms to the contextual semantics and can generate an accurate description that matches the input image. Image-text contrast loss: By activating the unimodal visual feature encoder, the visual and text feature spaces are aligned, which encourages positive image-text pairs to have similar representations, while negative pairs have greater discrimination, thus enhancing the relevance between image and text.

[0026] The contrast loss formula is as follows: ; in, Representing image features and text features The numerator is the similarity between the positive samples. For a given image and the corresponding text , calculate the similarity between the two. The denominator is the sum of the similarities between all candidate texts (including negative samples) and the image. is the logarithmic loss function, which aims to maximize the probability of positive samples and minimize the probability of negative samples. is the sample size.

[0027] Image-text matching loss. We learn an image-text multimodal representation that captures fine-grained alignment between vision and language by activating an image-based text encoder, and use the image-text matching loss to predict whether an image-text pair matches given multimodal features.

[0028] The matching loss formula is as follows: ; ; in, is the label of the image-text pair, Indicates that the image and text match, Indicates that the image and text do not match, is the probability that the model predicts whether the image and text match, using the logarithmic loss function To optimize the model, is the image feature and text features The similarity calculation between is the sigmoid function, which is used to normalize the similarity value to , ensuring that it can be used as a probability.

[0029] Language modeling loss. By activating an image-based text encoder to generate a text description of a given image, an autoregressive approach is used to maximize the likelihood of generating text, thereby improving the quality of the text description and its match with the image semantics.

[0030] The formula for language modeling loss is as follows: ; in, Given the previous text and image information Next, generate the next word probability.

[0031] (3) Text fusion stage: The steps of combining text and visual understanding include image-text consistency assessment, text-visual semantic coordination detection, and context supplementation based on dynamic semantic fusion; the steps of accurately combining the input text with the visually understood text include: Image-text consistency assessment. Determine whether the input text and the text generated by visual understanding describe the same thing or concept. Directional consistency is used to measure the similarity between the input text and the image description text. The higher the directional consistency, the higher the similarity between the text description and the image. In order to avoid text redundancy problems, the similarity between the image description and the input text is calculated to determine whether to add new text. This method helps ensure that descriptions that are repeated with the original input content are not introduced during the text fusion process. The calculation formula is as follows: ; in, and are the vector representations of the input text and image description text respectively.

[0032] Text-visual semantic coordination detection. Based on the similarity calculation results, determine whether there is a semantic conflict between the input text and the image description text. If the similarity between the two is low or there is an obvious contradiction, it will be marked as a conflict to avoid incorrect fusion; Contextual supplementation based on dynamic semantic fusion. Under the premise of ensuring that there is no conflict between the input text and the image description text, an information supplementation method based on dynamic semantic fusion and keyword alignment is proposed. This method uses an adaptive semantic alignment algorithm to dynamically adjust the information fusion strategy by calculating the complementary weights of text and visual content, so that the generated text is optimized in terms of semantic integrity, contextual coherence and description accuracy. Its steps include: Keyword alignment: First, extract keywords from the input text and image description text, calculate their similarity, and evaluate the degree of semantic alignment between the two.

[0033] Complementarity weight calculation: The adaptive alignment algorithm is used to calculate the complementary weight between text and visual content. This weight measures the relative importance of the input text and the image description text. The complementary weight is calculated through a gating mechanism and is expressed as ,in is the weight matrix, and are the encoded features of the input text and image description text, respectively. Represents the complementarity weight of text and image description: .

[0034] Dynamic Information Fusion: Based on Complementarity Weights , fuse the input text with the image description text by adjusting the weights of the two proportionally to generate an enhanced text representation , which includes the combination of image and text information, is calculated as follows: .

[0035] (4) Abstract generation stage: Text encoding layer: Each word in the text is converted into a vector representation of fixed dimension. These vectors can effectively capture the semantic information of the text and encode the semantics of each word with its contextual relationship, thereby obtaining a high-quality representation of each word as the input of DeepFusion-Transformer.

[0036] Position encoding layer. Simple text encoding does not retain the positional relationship of words in a sentence, but only presents the semantic representation of words. It lacks understanding of the internal word order of a sentence, so the introduction of position encoding enables the model to understand the relative and absolute positions of words. Through position encoding, the model can fully consider the order of words in a sequence when performing self-attention calculations, thereby improving the accuracy and fluency of text understanding. Specifically, the position encoding is generated using sine and cosine functions, so that the encoding of each position has different frequencies in different dimensions, ensuring that the model can distinguish words in different positions.

[0037] DeepFusion-Transformer layer: Using a stacked multi-layer structure, each layer gradually learns more advanced features. The shallow Transformer layer focuses on capturing local semantic information, while the deep Transformer layer focuses on learning global semantic representation.

[0038] Autoregressive decoding layer. The decoder generates text step by step in an autoregressive manner. After each new word is generated, the decoder updates its state, takes the generated word as a new input, and continues to generate the next step in combination with the current context information until a complete summary text is generated.

[0039] The formula for the autoregressive generating process is as follows: ; in, is given Step word and input features Next, generate The probability distribution of the step vocabulary. is the current decoder hidden state, and are the weights and biases used to calculate the vocabulary distribution. The hidden state of the decoder is updated at each time step. The hidden state of step is , then in Step , the new hidden state This can be calculated recursively: ; in, yes Step generated words, is the hidden state at the previous moment, It is the multimodal information of the input.

[0040] A visual-text collaborative summary generation system based on multimodal learning, comprising: Multimodal input module: accepts input text data (such as news reports, meeting minutes, etc.) and visual data (such as images, image sequences or videos related to the text content). The main function of this module is to provide raw input for subsequent visual understanding and text fusion; Visual semantic understanding module: Visual data is input into the visual semantic understanding model. The model extracts visual features, identifies key information in images or videos, and generates a detailed description of the visual data content.

[0041] Text and visual understanding combination module: effectively combines the input text with the image description text or video analysis text generated by the visual understanding model to obtain a fused representation of the text and image content.

[0042] Summary generation module: The text generated by the text and visual understanding module is input into the generation model. Through the model's understanding and processing of the text, a text summary that conforms to the contextual semantics is finally generated.

[0043] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visual-text collaborative summary generation method based on multimodal learning, characterized in that: The following steps are involved: S1. Receive multimodal data: including text data and visual data as the basis for subsequent summary generation; S2. Visual semantic understanding: Input visual data into the visual semantic understanding model, extract visual features, identify key information in images or videos, and generate text descriptions of the visual data content; S3. Combining text with visual semantic understanding: effectively fusing the text data in step S1 with the image description text or video parsing text generated in step S2 to provide semantic information for the summary generation model; S4. Summary generation: Input the fused text in step S3 into the summary generation model, use the model to understand and process the input text, and finally generate a text summary that conforms to the contextual semantics.

2. The method for generating a visual-textual collaborative summary based on multimodal learning according to claim 1, characterized in that: The visual semantic understanding model in step S2 includes video key frame recognition, calculates the importance weight of the pixel area through adaptive spatiotemporal weight fusion, identifies the core changes of the video content, removes redundant information, and obtains a set of representative and semantically rich key frame images: Adaptive spatiotemporal weight fusion: Based on the standard spatiotemporal attention, a dynamic weight factor is added to adaptively adjust the attention in the temporal and spatial dimensions; Temporal attention focuses on the relationship between different frames in a video sequence, reflecting the dynamic changes between frames; the temporal attention calculation formula is as follows: ； The attention in the spatial dimension focuses on the regional relationship within a single frame image, emphasizing the local and global structure of the object. The calculation formula is as follows: ； The calculation formula of adaptive spatiotemporal weight fusion is as follows: ； is the temporal attention factor, learning the global importance of each frame, is the spatial attention factor, through the sigmoid function Normalize to ensure and There will be no negative values ​​or values ​​that are too large or too small.

3. The method for generating a visual-textual collaborative summary based on multimodal learning according to claim 1, characterized in that: The visual semantic understanding model in step S2 includes unimodal visual feature encoding, which combines standard convolution and depth-wise separable convolution using a hybrid convolution strategy. The unimodal visual feature encoding adopts a dual-branch structure, extracts global features through dilated convolution and local features through hybrid convolution through two parallel computing paths, and performs adaptive feature fusion through dual-branch fusion.

4. The method for generating a visual-textual collaborative summary based on multimodal learning according to claim 3 is characterized in that: Dilated convolution: By introducing a dynamic dilation factor strategy, dilation factors of different scales are used Multi-level information fusion is performed so that the network can capture long-distance pixel dependencies while keeping local information intact. The calculation formula is as follows: ； ； in, is the position in the output feature map The pixel value at ; Represents input features In Location The pixel value at is the index of the convolution kernel, indicating the position of the convolution kernel in the sliding window; Indicates that the convolution kernel is at position The weight of is the dynamic expansion factor, which controls the sampling interval of the convolution kernel. is the current feature map at position The local features of and is the parameter to be learned; when When , it is a standard convolution, when When , there are gaps between the sampling points of the convolution kernel; Depthwise separable convolutions include: Channel-wise convolution: ； in, Represents the first Channels, positions The pixel value at represents the index of the convolution kernel, is the convolution kernel at position The weight of Point-wise convolution: ； in, is the output of channel-wise convolution, Is the convolution kernel, used for cross-channel weighted summation, by adjusting The number of controls the number of output channels; Mixed convolution formula: ； in, Indicates that the mixed convolution operation is at position The output feature map at is the balance coefficient, , used to control the weight between standard convolution and depth-wise separable convolution; Two-branch fusion formula: ； in, Represents the fusion formula of multi-scale dilated convolution and mixed convolution, It is the dynamic fusion weight that controls the proportion of global and local information.

5. The method for generating a visual-textual collaborative summary based on multimodal learning according to claim 3, characterized in that: The visual semantic understanding model in step S2 includes image-text contrast loss and image-text matching loss; Image-text contrast loss: By activating the unimodal visual feature encoder, aligning the feature spaces of vision and text, positive sample image-text pairs are encouraged to have similar representations, while negative sample pairs have greater discrimination, thus enhancing the relevance between image and text. The contrast loss formula is as follows: ； in, Representing image features and text features The similarity between is the logarithmic loss function, N is the number of samples; Image-text matching loss: By activating the image-based text encoder, we learn to capture the image-text multimodal representation that captures the fine-grained alignment between vision and language. We use the image-text matching loss to predict whether the image-text pair given the multimodal features matches. The matching loss formula is as follows: ； ； in, is the label of the image-text pair, Indicates that the image and text match, Indicates that the image and text do not match, is the probability that the model predicts whether the image and text match, using the logarithmic loss function To optimize the model, is the image feature and text features The similarity calculation between is the sigmoid function, which is used to normalize the similarity value to , ensuring that it can be used as a probability.

6. The method for generating a visual-textual collaborative summary based on multimodal learning according to claim 2, characterized in that: The visual semantic understanding model in step S2 includes language modeling loss. By activating the image-based text encoder, a text description of a given image is generated, and the possibility of generating text is maximized in an autoregressive manner. The calculation formula is as follows: ； in, Given the previous text and image information Next, generate the next word probability.

7. The method for generating a visual-textual collaborative summary based on multimodal learning according to claim 1, characterized in that: The combination of text and visual semantic understanding in step S3 includes image-text consistency assessment, text-visual semantic coordination detection, and context supplementation based on dynamic semantic fusion; Image-text consistency assessment: Directional consistency is used to measure the similarity between the input text and the image description text. The higher the directional consistency, the higher the similarity between the text description image. The calculation formula is as follows: ； in, and are the vector representations of the input text and image description text respectively; Text-visual semantic coordination detection: Based on the similarity calculation results, determine whether there is a semantic conflict between the input text and the image description text. If the similarity between the two is low or there is an obvious contradiction, mark them as a conflict to avoid incorrect fusion; Contextual supplementation based on dynamic semantic fusion includes keyword alignment, complementary weight calculation and dynamic information fusion; Keyword alignment: extract keywords from the input text and image description text, calculate their similarity, and evaluate the degree of semantic alignment between the two; Complementarity weight calculation: The adaptive alignment algorithm is used to calculate the complementary weight between text and visual content. This weight measures the relative importance of the input text and the image description text. The complementary weight is calculated through a gating mechanism and is expressed as ,in is the weight matrix, and are the encoded features of the input text and image description text, respectively. Represents the complementarity weight of text and image description: ； Dynamic Information Fusion: Based on Complementarity Weights , fuse the input text with the image description text by adjusting the weights of the two proportionally to generate an enhanced text representation , which includes the combination of image and text information, is calculated as follows: 。 8. The method for generating a visual-textual collaborative summary based on multimodal learning according to claim 1, characterized in that: In step S4, the summary generation model includes a text encoding layer, a position encoding layer, a DeepFusion-Transformer layer, and an autoregressive decoding layer; the text fused in step S3 is input into the trained summary generation model to generate a text summary; Text encoding layer: converts each word in the text into a vector representation of fixed dimension and encodes the semantics of each word with its contextual relationship, thereby obtaining a high-quality representation of each word as the input of DeepFusion-Transformer; Position encoding layer: Use sine and cosine functions to generate position encoding so that the encoding of each position has different frequencies in different dimensions; DeepFusion-Transformer layer: It adopts a stacked multi-layer structure. Each layer gradually learns more advanced features. The shallow Transformer layer focuses on capturing local semantic information, while the deep Transformer layer focuses on learning global semantic representation. Autoregressive decoding layer: The decoder generates text step by step in an autoregressive manner. After each new word is generated, the decoder updates its state, takes the generated word as a new input, and continues to generate the next step in combination with the current context information until a complete summary text is generated.

9. The method for generating a visual-textual collaborative summary based on multimodal learning according to claim 8, characterized in that: The formula for the autoregressive generating process is as follows: ； in, is given Step word and input features Next, generate The probability distribution of the step vocabulary; is the current decoder hidden state, and are the weights and biases used to calculate the vocabulary distribution; the hidden state of the decoder is updated at each time step, assuming that The hidden state of step is , then in Step , the new hidden state This can be calculated recursively: ； in, yes Step generated words, is the hidden state at the previous moment, It is the multimodal information of the input.

10. A visual-text collaborative summary generation system based on multimodal learning, characterized in that: The method according to any one of claims 1 to 9 comprises a multimodal input module, a visual semantic understanding module, a text and visual understanding combination module and a summary generation module; Multimodal input module: accepts input text data and visual data. The main function of this module is to provide raw input for subsequent visual understanding and text fusion; Visual semantic understanding module: The visual data is input into the visual semantic understanding model. The model extracts visual features, identifies key information in images or videos, and generates a detailed description of the visual data content. Text and visual understanding combination module: effectively combines the input text with the image description text or video analysis text generated by the visual understanding model to obtain a fusion representation of text and image content; Summary generation module: The text generated by the text and visual understanding module is input into the generation model. Through the model's understanding and processing of the text, a text summary that conforms to the contextual semantics is finally generated.

Images