Multimodal summary generation method, apparatus, and electronic device

Abstract

Embodiments of the present application provide a multi-modal summary generation method, device and electronic equipment, relating to the technical field of artificial intelligence and data processing, the multi-modal summary generation method comprising: obtaining multi-modal data to be processed; the multi-modal data comprising first text data and first image data associated with each other; inputting the multi-modal data and summary length constraint information into a pre-trained summary generation model to extract a summary, obtaining summary data of the multi-modal data; the summary data comprising a text summary and an image summary; the sample summary data satisfying the following conditions: a total importance score of a sample sentence in the sample summary data is the highest, and a total length of the sample sentence matches the sample summary length constraint information. The present application not only meets the user's demand for summary length, but also ensures that the summary data can retain the core information and readability of the original multi-modal data, improving the overall quality of the multi-modal summary.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and data processing technology, and in particular to a multimodal summarization method, apparatus and electronic device. Background Technology

[0002] With the surge in multimedia data, controllable summarization has become a focal point in the field of AI content generation. Summarization is a key area in multimodal generation tasks, representing a complex information compression technique. In today's information-overloaded era, extracting key information from massive amounts of unstructured text data and forming concise and coherent summaries has become a focus of academic research. With the diversification of modal content, the fusion of various modal information has injected new vitality into summarization research. For example, news reports are no longer limited to single-text descriptions but utilize multimodal data, including images and videos, to depict richer narratives. This necessitates the extraction of summaries from multiple modalities, such as text summaries and image summaries. Text summarization aims to extract concise and coherent summaries from single or multiple documents to accurately capture the essence of the original text, while image summarization focuses on extracting key visual attributes from image collections or sequences and presenting them in a concise summary format, thereby helping users quickly understand the image content. This multimodal summarization extraction method not only improves the efficiency of information acquisition but also enriches the user's information retrieval experience, providing users with a more comprehensive and richer perspective.

[0003] However, traditional summary generation methods mostly employ a fixed summary length, a one-size-fits-all approach that often fails to meet diverse user needs in practice. This can result in summaries that are either too long or too short, failing to satisfy varied reading preferences. As user preferences and reading habits continue to evolve, the ability to customize summary length based on user feedback is becoming increasingly important. Therefore, how to flexibly adjust summary length according to actual user needs remains a pressing issue. Summary of the Invention

[0004] This application provides a multimodal summary generation method, apparatus, and electronic device to alleviate or solve one or more technical problems existing in the prior art.

[0005] In a first aspect, embodiments of this application provide a multimodal summary generation method, characterized in that it includes:

[0006] Acquire multimodal data to be processed; the multimodal data includes interrelated first text data and first image data;

[0007] The multimodal data and summary length constraint information are input into a pre-trained summary generation model for summary extraction to obtain summary data of the multimodal data; the summary data includes text summary and image summary;

[0008] The summary generation model is trained based on sample multimodal data, sample summary length constraint information, and sample summary data. The sample summary data satisfies the following conditions: the total importance score of the sample sentences in the sample summary data is the highest, and the total length of the sample sentences matches the sample summary length constraint information.

[0009] Secondly, embodiments of this application provide a multimodal summary generation apparatus, characterized in that it includes:

[0010] An acquisition module is used to acquire multimodal data to be processed; the multimodal data includes interrelated first text data and first image data;

[0011] An extraction module is used to input the multimodal data and summary length constraint information into a pre-trained summary generation model for summary extraction, thereby obtaining summary data of the multimodal data; the summary data includes text summaries and image summaries;

[0012] The summary generation model is trained based on sample multimodal data, sample summary length constraint information, and sample summary data. The sample summary data satisfies the following conditions: the total importance score of the sample sentences in the sample summary data is the highest, and the total length of the sample sentences matches the sample summary length constraint information.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0016] According to the technical solution of this application embodiment, multimodal data to be processed is acquired, including interrelated first text data and first image data; the multimodal data and summary length constraint information are input into a pre-trained summary generation model for summary extraction, resulting in summary data of the multimodal data, including text summaries and image summaries; wherein, the summary generation model is trained based on sample multimodal data, sample summary length constraint information, and sample summary data. Since the sample summary data satisfies the following conditions: the total importance score of the sample sentences in the sample summary data is the highest, and the total length of the sample sentences matches the sample summary length constraint information, the pre-trained summary generation model treats the summary generation problem as a knapsack problem when generating summaries, ensuring that the final generated summary data satisfies the conditions of the highest total importance score of the sentences in the summary data and the total sentence length matching the summary length constraint information, that is, maximizing the quality of the summary while satisfying the summary length constraint. Because the summary length constraint is flexible and controllable, the summary data generated by the summary generation model not only meets the user's requirements for summary length but also ensures that the summary data retains the core information and readability of the original multimodal data, improving the overall quality of the multimodal summary.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0018] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0019] Figure 1 A flowchart of the multimodal summary generation method provided in an embodiment of this application is shown;

[0020] Figure 2 A schematic diagram of the multimodal summary generation method provided in an embodiment of this application is shown.

[0021] Figure 3 A schematic diagram of the abstract generation model provided in an embodiment of this application is shown;

[0022] Figure 4 The diagram shows the effect of the multimodal summary generation method provided in the embodiment of this application;

[0023] Figure 5A block diagram of the multimodal summary generation apparatus provided in an embodiment of this application is shown;

[0024] Figure 6 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0025] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0026] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0027] The technical concept of this application is as follows: The problem of summary extraction under length constraints is treated as a combinatorial optimization problem, analogous to the knapsack problem. The goal is to maximize the total value of the items in the knapsack without exceeding its capacity limit. This is similar to extracting the most informative and representative sentence combination within a summary length constraint. By weighing the value of each sentence—for example, the predicted value based on the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score, sentence length, and the coherence and redundancy between sentences—the overall quality of the summary is maximized while satisfying the summary length constraint. Furthermore, combining this combinatorial optimization problem with a neural network model enables the automation of summary generation and length control. By training a neural network model that simultaneously considers sentence value, length, and contextual coherence, predicting sentence value, satisfying the summary length constraint, and contextual coherence are integrated into a unified optimization objective. This allows the model to automatically generate summaries that meet the specified length constraint while retaining the core information and readability of the original text, thereby improving the overall quality of the summary and achieving a balance between precise length control and high-quality summaries.

[0028] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1 A flowchart of the multimodal summary generation method provided in this application embodiment is shown, such as... Figure 1 As shown, the method may include steps S101 and S102.

[0030] Step S101: Obtain the multimodal data to be processed. The multimodal data includes interrelated first text data and first image data.

[0031] The first text data may include one or more texts, such as one or more news texts. The first image data may include one or more images.

[0032] Step S102: Input the multimodal data and summary length constraint information into the pre-trained summary generation model for summary extraction to obtain summary data of the multimodal data, which includes text summary and image summary.

[0033] The summary generation model is trained based on sample multimodal data, sample summary length constraint information, and sample summary data. The sample summary data meets the following conditions: the total importance score of the sample sentences in the sample summary data is the highest, and the total length of the sample sentences matches the sample summary length constraint information.

[0034] The summary length constraint information may include: text summary constraint information and / or image summary constraint information. Text summary constraint information may include at least one of the following: total text length, number of sentences, length of each sentence, etc., wherein the total text length may be characterized as the total length of all sentences included in the text summary. Image summary constraint information may include at least one of the following: number of images, total image size, size of a single image, etc.

[0035] According to the technical solution of this application embodiment, multimodal data to be processed is acquired, including interrelated first text data and first image data; the multimodal data and summary length constraint information are input into a pre-trained summary generation model for summary extraction, resulting in summary data of the multimodal data, including text summaries and image summaries; wherein, the summary generation model is trained based on sample multimodal data, sample summary length constraint information, and sample summary data. Since the sample summary data satisfies the following conditions: the total importance score of the sample sentences in the sample summary data is the highest, and the total length of the sample sentences matches the sample summary length constraint information, the pre-trained summary generation model treats the summary generation problem as a knapsack problem when generating summaries, ensuring that the final generated summary data satisfies the conditions of the highest total importance score of the sentences in the summary data and the total sentence length matching the summary length constraint information, that is, maximizing the quality of the summary while satisfying the summary length constraint. Because the summary length constraint is flexible and controllable, the summary data generated by the summary generation model not only meets the user's requirements for summary length but also ensures that the summary data retains the core information and readability of the original multimodal data, improving the overall quality of the multimodal summary.

[0036] In some embodiments, the summary generation model includes an attention-based encoder and decoder. Figure 2 A schematic diagram of the multimodal summary generation method provided in this application is shown. Combined with... Figure 2 When multimodal data and summary length constraint information are input into a pre-trained summary generation model for summary extraction, the following steps can be performed to obtain summary data of multimodal data:

[0037] The encoder converts the first text data and the first image data into vector representations to obtain the first text context vector corresponding to the first text data and the first image context vector corresponding to the first image data. The first text context vector and the first image context vector are fused based on the attention mechanism to obtain the fused multimodal context vector. The multimodal context vector includes the second text context vector and the second image context vector.

[0038] The decoder determines the vector length and importance score of each second text context vector. Based on the vector length and importance score, the target text context vector is selected from the second text context vectors, and a text summary is generated based on the target text context vectors. Additionally, the decoder determines the visual coverage score of each second image context vector. Based on the visual coverage score, the target image context vector is selected from the second image context vectors, and the image corresponding to the target image context vector is determined as the image summary.

[0039] When generating a text summary based on the target text context vector, the sentences corresponding to the target text context vector can be concatenated sequentially, and the concatenated text is the text summary. Alternatively, the concatenated text can be processed, and the processed text can serve as the text summary. At least one of the following processing methods can be applied to the concatenated text: removing redundant words, adding conjunctions between adjacent sentences, shortening lengthy sentences, etc.

[0040] In the summary generation model, the encoder can use a single-layer Bi-LSTM (Bidirectional Long Short-Term Memory) encoder, and the decoder can use a unidirectional LSTM decoder. To effectively fuse image and text data, a multimodal attention mechanism is introduced, which can dynamically adjust the weights between different modalities.

[0041] Suppose the multimodal data is represented as {T, I}, where T represents the first text data, denoted as T = {t1, t2, ..., t...}. n}, where n represents the number of sentences in the first text data, which can also be understood as the total length of the first text data. I represents the first image data, which may include multiple images related to the news text, denoted as I = {i1, i2, ..., i...} m}, where m represents the number of images.

[0042] The output data of a summary generation model can include a concise text S and a selected image G, where text S is the text summary and image G is the image summary. The summary S is represented as a set S = {S1, S2, ..., S...}. l} where l represents the length of the text summary, i.e., the total length of all sentences included in the text summary. l is significantly less than n to ensure the brevity of the text summary. The summary generation model can extract key information from text and image content and present the main content of the news text in a coherent and fluent manner, i.e., the summary data. The image G selected by the summary generation model is determined based on the visual coverage vector, aiming to represent the image that best reflects the key features of the image.

[0043] In this embodiment, when the encoder converts the first text data into a vector representation, an embedding layer is used to convert each word in the first text data into a corresponding word vector. This not only captures the semantic essence of the words but also accurately preserves the relevance of the words in the context. Then, the encoder encodes the first text context vector, thereby mapping the first text context vector to the encoder's hidden state h. i Above, each hidden state represents information about the text data in different dimensions.

[0044] The principle of text encoding can be expressed by the following formula:

[0045]

[0046] in, This represents the text attention score corresponding to the first text data, where t is the time step. h represents the text attention vector. i This indicates the hidden state of the encoder. x t This represents the input data of the decoder (i.e. the output data of the encoder) at time step t. It represents the current state or context information of the decoder in the process of generating the summary. It can indirectly affect the attention allocation of text encoding through the attention mechanism. That is, the decoder can decide the degree of attention to the text information according to the current state when generating the summary. This represents the time step before time step t. Indicates time step Text attention weights. txt W represents the first text context vector. i W x W c This represents the weight matrix.

[0047] When converting the first image data into a vector representation using an encoder, VGG19 can be selected as the pre-trained model for image feature extraction. During the preprocessing stage, deep features of the first image data are extracted to capture its comprehensive attributes. During encoding, the feature maps generated by each convolutional layer undergo pooling to reduce their size while preserving key details. This not only reduces computational complexity but also enhances the model's robustness. The compressed feature maps are then processed sequentially through subsequent convolutional layers, ultimately generating a fixed-size feature vector, i.e., the first image context vector. This first image context vector contains the core information of the first image data, providing accurate and crucial data input for subsequent summarization tasks.

[0048] The principle of image encoding can be expressed by the following formula:

[0049]

[0050] in, This represents the image attention score corresponding to the first image data, where t is the time step. G represents the image attention vector. i This represents the image feature information extracted from the first image data, which can be characterized as an image feature vector. t This represents the input data of the decoder (i.e. the output data of the encoder) at time step t. It represents the current state or context information of the decoder in the process of generating the summary. It can indirectly affect the attention allocation of image encoding through the attention mechanism. That is, the decoder can decide the degree of attention to the image information according to the current state when generating the summary. This represents the visual coverage vector corresponding to the first image data. c represents the image attention weights at time step t. img W represents the context vector of the first image. i W x g represents image feature information respectively i and encoder input data x t The corresponding weight matrix.

[0051] Optionally, during the encoder stage, when fusing the first text context vector and the first image context vector based on the attention mechanism to obtain the fused multimodal context vector, the following steps A1 to A3 can be performed:

[0052] Step A1: Based on the attention mechanism, calculate the first similarity between the first text context vector and the first image context vector.

[0053] Step A2: Based on the first similarity, determine the text attention weight corresponding to the first text context vector and the image attention weight corresponding to the first image context vector.

[0054] Step A3: Based on the text attention weight and the image attention weight, fuse the first text context vector and the first image context vector to obtain the multimodal context vector.

[0055] The working principle of vector fusion by the encoder can be expressed by the following formula:

[0056]

[0057] Among them, C t Represents the multimodal context vector at time step t. This represents the text attention weight corresponding to the first text context vector. This represents the image attention weight corresponding to the first image context vector. J represents the modality set, including the text modality txt and the image modality img.

[0058] In this embodiment, the first text data and the first image data are converted into vector representations by an encoder to dynamically adjust the weights between different modalities. By calculating the first similarity between the first text context vector and the first image context vector, the attention weights corresponding to the context vectors of different modalities can be determined, thereby obtaining the importance of each modality in the current processing task. Attention weights can reflect the importance of different modal information to a certain extent, and therefore can be used to decide which modal information to prioritize, enabling the summarization generation model to focus on the most relevant information for accurate processing.

[0059] In some embodiments, the decoder employs an autoregressive generation mechanism when generating summary data. During this process, the decoder updates its hidden state for each word it generates to reflect contextual changes during generation. Subsequently, an attention mechanism configured in the decoder is used to calculate the second text context vector, thereby cleverly focusing on the most relevant content in the encoded sequence.

[0060] To ensure the conciseness of the text summary, a knapsack algorithm is used to control its length. The knapsack algorithm is a combinatorial optimization problem that aims to select the most valuable subset of items from a set of items without exceeding a given capacity. In the scenario of this application, each sentence in the first text data can be considered an item, whose "value" is determined by the sentence's information content and importance, while the "capacity" corresponds to the maximum allowed length of the text summary, i.e., the text summary constraint information.

[0061] Suppose that for text summarization, the constraint is that the total length of the text does not exceed C. That is, we need to select sentences from n initial text data such that the total length of the selected sentences does not exceed C, while containing as much key information as possible. In this case, the knapsack algorithm can be applied to the text summarization generation scenario.

[0062] Based on this, in the decoder stage, when determining the vector size and importance score of each second text context vector, and selecting the target text context vector from the second text context vectors according to the vector length and importance score, the following steps B1 to B3 can be performed:

[0063] Step B1: For each second text context vector, calculate the second similarity between the second text context vector and other second text context vectors.

[0064] Step B2: Calculate the importance score of the second text context vector based on the second similarity.

[0065] Step B3: Select the target text context vector from the second text context vector that meets the text summarization criteria. The text summarization criteria include: the target text context vector has the highest total importance score, and the total length of the vector matches the summary length constraint.

[0066] Optionally, the summary length constraint information includes the total text length of the text summary. Then, in the decoder stage, step B3 above can be executed as steps C1 to C3:

[0067] Step C1: Select one second text context vector from the second text context vectors in descending order of importance score.

[0068] Step C2: After selecting a second text context vector each time, determine whether the sum of the lengths of the selected second text context vectors matches the total length of the text.

[0069] Step C3: If a match is found, the currently selected second text context vector is added to the candidate vector set, and the next second text context vector is selected from the candidate vectors; the candidate vector set includes the already selected second text context vectors. If no match is found, the second text context vector in the candidate vector set is determined as the target text context vector.

[0070] Optionally, in the decoder stage, when determining the visual coverage score of each second image context vector and filtering the target image context vector from the second image context vectors based on the visual coverage score, the following steps D1 to D2 can be performed:

[0071] Step D1: For each second image context vector, calculate the visual coverage score of the second image context vector; the visual coverage score is used to represent the degree of attention accumulation of the second image context vector.

[0072] Step D2: Select target image context vectors from the second image context vectors whose visual coverage scores meet the visual coverage conditions; the visual coverage conditions include at least one of the following: the highest visual coverage score, or the visual coverage score is higher than or equal to the first score threshold.

[0073] By filtering target image context vectors that meet visual coverage criteria, we can ensure that the selected image summaries can better reflect the relevance between the image and text summaries, highlight the salient features of the images, and thus improve the accuracy and relevance of the summary data.

[0074] The visual coverage score of the second image context vector can be calculated using the following formula:

[0075]

[0076] Among them, G img The visual coverage score represents the context vector of the second image. The visual coverage vector represents time step t.

[0077] In this embodiment, a visual overlay vector is introduced. Since the visual overlay vector can integrate the attention distribution of all previous decoding time steps, it can represent the cumulative sum of image attention to a certain extent. Therefore, it can highlight the most salient features in the image and efficiently capture key information in the image. Thus, through this mechanism, the decoder can not only consider the instantaneous attention distribution of the current time step, but also the cumulative attention distribution of all previous time steps, always paying attention to the information previously focused on, further ensuring that the summarization generation model can generate coherent and non-repetitive summary data.

[0078] Figure 3 The schematic diagram of the abstract generation model provided in the embodiments of this application is shown, such as... Figure 3 As shown, the input multimodal data includes text and images, which are then processed by an encoder and a decoder, respectively. In the encoder stage, the text attention layer encodes the text to obtain the text context vector c. txt The image attention layer encodes the image to obtain the image context vector c. img Subsequently, the text context vector and image context vector are fused using an attention mechanism to obtain the fused multimodal context vector C. t In the decoder stage, the input data includes the summary length constraint information "rt" and the encoder's hidden state "h". The encoder's hidden state "h" contains the multimodal context vector C. t Next, the decoder receives the value vector and the size vector, integrates them into a 2*n matrix, processes it through a linear layer to the hidden dimension d, then passes it through a Transformer module (a neural network structure based on self-attention), and finally returns it to the one-dimensional space Cf through another linear layer. The output is directly used as the score for the knapsack problem of selecting sentences under length constraints.

[0079] In this embodiment, a greedy algorithm is used to solve the aforementioned knapsack problem, namely, how to select the target text context vector with the highest total importance score and whose total length matches the summary length constraint from multiple second text context vectors. When filtering target text context vectors and target image context vectors, the summary generation model can convert the knapsack score to 0 or 1. Specifically, the calculated knapsack score is compared with a preset threshold. If the knapsack score is greater than the preset threshold, it is converted to 1. If the knapsack score is less than or equal to the preset threshold, it is converted to 0. Then, the sentences corresponding to the target text context vectors with a knapsack score of 1 are used to form a text summary, and the images corresponding to the target image context vectors with a knapsack score of 1 are used to form an image summary.

[0080] Depend on Figure 3As shown in the abstract generation model, in the encoder stage, the model performs deep semantic understanding on the input first text data to extract key information. In the decoder stage, a greedy algorithm is used to solve the knapsack problem, reasonably allocate words and accurately select sentences, and precisely control the text length of the generated abstract data according to the preset abstract length. Secondly, the abstract generation model cleverly embeds words from the text, ensuring a high degree of semantic consistency between the text abstract and the original multimodal data. Furthermore, by introducing visual overlay vectors, the abstract generation model can consider the semantic relationship between images and text when generating abstract data. This not only improves the semantic accuracy of the abstract data and reduces image output, but also enhances the information saliency of the selected images, making the generated abstract data more in line with human reading habits and comprehension. This length control method, combining optimization algorithms and greedy strategies, enables the abstract generation model to achieve an excellent balance between maintaining abstract quality and controlling abstract length.

[0081] In the summarization generation model, the encoder employs a Long Short-Term Memory (LSTM) network. Since LSTM encoders are particularly well-suited for capturing long-term dependencies in text data, text encoding using LSTM extracts rich textual information, providing a rich and accurate data foundation for the subsequent summarization process. In the decoder stage, an attention mechanism is introduced, enabling the summarization generation model to dynamically focus on the relevant parts of the text data at each step. This not only enhances the model's understanding of the input data but also significantly improves the accuracy and coherence of the summarization data. Furthermore, by calculating attention weights based on the similarity between the input sequence and the current decoding state and performing normalization, it is ensured that the generated summarization data contains key information from the multimodal data and does not exceed the summarization length constraint.

[0082] Figure 4 The following diagram illustrates the effect of the multimodal summarization generation method provided in the embodiments of this application: Figure 4 As shown, the first input text data (i.e., the source text) is quite long, and the first image data (i.e., the source images) includes three images. The summary length constraint is "output length = 120". Therefore, after processing by the summary generation model, it can output a text summary of no more than 120 characters: "Meat or frozen food in the refrigerator should be thawed at 5°C or below; cooked food should be refrozen after cooling. Avoid refrozen thawed food at room temperature to prevent microbial growth. Plan ahead for large items and do not refrozen hot food without cooling it. Image caption: If you have thawed frozen meat or chicken, you cannot refrozen it again," along with one image summary highly relevant to the text summary.

[0083] In some embodiments, when extracting text summaries from the first text data, a sentence centrality selection method can be used, where the centrality of a sentence reflects its importance. Optionally, a graph-based ranking method can be used, representing sentences as nodes in a graph, with edges representing the strength or similarity between sentences. The centrality of a sentence can be calculated by summing its similarity to other sentences. The higher the sum of similarities, the higher the centrality of the sentence, meaning the sentence is more central and plays a crucial role in connecting different parts of the text. Sentences with higher centrality are then selected as the text summary. This method can be expressed by the following formula:

[0084]

[0085] Wherein, C(x) i ) indicates the centrality of sentence i, e ij This indicates the similarity between sentence i and sentence j.

[0086] In some embodiments, the training of the summary generation model includes the following steps:

[0087] First, acquire multimodal sample data; multimodal sample data includes interrelated sample text data and sample image data.

[0088] Secondly, the multimodal sample data, sample summary length constraint information, and sample summary data are input into the base model to be trained so that the base model can extract the summary of the sample data to obtain the predicted summary data. The predicted summary data includes predicted text summary and predicted image summary. The sample sentences in the sample summary data have the highest total importance score, and the total length of the sample sentences matches the sample summary length constraint information.

[0089] Next, based on the predicted summary data and the sample summary data, the base model is iteratively trained to obtain the summary generation model.

[0090] Optionally, iteratively training the base model based on the predicted summary data and the sample summary data may include the following steps:

[0091] Calculate the third similarity between the predicted summary data and the sample summary data, and determine whether the base model satisfies the iteration termination condition based on the third similarity.

[0092] If the iteration termination condition is not met, the model parameters of the base model are adjusted, and the next round of iteration training is carried out based on the adjusted base model; if the iteration termination condition is met, the base model obtained in the current iteration is determined to be the summary generation model.

[0093] The model parameters can include the model learning rate, batch size, number of encoder layers, and number of decoder layers. By iteratively adjusting these model parameters, it can be ensured that the final trained summarization model can effectively capture key information while effectively controlling the summarization length of the data.

[0094] Optionally, to improve the generalization ability of the summarization generation model, the gains of each sample's multimodal data can be normalized to a sum of 1, reflecting the proportion of different sentences contributing to the total value of the summary. Simultaneously, normalization is performed by dividing the sentence length by the total length of the text summary, which helps the summarization generation model make effective choices under different length constraints. To enhance robustness, a feedback loop is introduced to evaluate the model performance of the summarization generation model, and automatic metrics such as ROUGE are used for tuning.

[0095] For example, let f be the evaluation function of the summary generation model, and V represent the optimal subset of the extracted summary data. If the quality of the summary data is represented by the total score of the selected sentences, then the evaluation function f(S) can be expressed as:

[0096]

[0097] Among them, v i Let S represent the importance score of sentence i, and let S represent the set of sentences included in the summary data.

[0098] Simultaneously, while ensuring the overall quality of the summary data, a constraint on the total text length must also be met. For the summary generation model, the learning objective that satisfies the total text length constraint can be expressed as the following formula:

[0099]

[0100] Among them, v i x represents the importance score of sentence i. i Let x be a binary variable with a value of 0 or 1. Optionally, if sentence i is selected as a sentence in the text summary, then x corresponding to sentence i... i The value is 1. If sentence i is not selected as a sentence in the text summary, then the value of x corresponding to sentence i is 1. i The value is 0.

[0101] The performance of the summarization model can be evaluated based on the value of the evaluation function f(S). If the value of the evaluation function f(S) is less than or equal to a preset evaluation threshold, it indicates that the performance of the summarization model is low, and the model can be optimized. The optimization method is similar to the model training method, except that the amount of sample data used for optimization can be less than the amount of sample data used for model training.

[0102] Experiments have shown that the summary generation model trained in this application, compared with existing summary generation methods, can not only generate semantically coherent summary data and precisely control the summary length, but also ensure a high degree of correlation between image summaries and text summaries in the summary data.

[0103] For example, the common LEDA-3 method extracts only the first three sentences as a summary, which misses key semantic content in multimodal data, resulting in summary data that cannot accurately cover the core information of the multimodal data. Furthermore, although the seq2seq method controls the summary length by introducing context vectors, it still struggles to ensure semantic consistency when handling summary length. These methods sacrifice ROUGE scores to strictly limit summary length, resulting in generated summary data lacking important semantic information.

[0104] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a multimodal summary generation device.

[0105] Figure 5 A block diagram of the multimodal summary generation apparatus provided in an embodiment of this application is shown, such as... Figure 5 As shown, the multimodal summarization generation device includes:

[0106] The acquisition module 51 is used to acquire multimodal data to be processed; the multimodal data includes interrelated first text data and first image data;

[0107] Extraction module 52 is used to input the multimodal data and summary length constraint information into a pre-trained summary generation model for summary extraction, to obtain summary data of the multimodal data; the summary data includes text summary and image summary;

[0108] The summary generation model is trained based on sample multimodal data, sample summary length constraint information, and sample summary data. The sample summary data satisfies the following conditions: the total importance score of the sample sentences in the sample summary data is the highest, and the total length of the sample sentences matches the sample summary length constraint information.

[0109] In some embodiments, the summary generation model includes an attention-based encoder and decoder;

[0110] When the extraction module 52 inputs the multimodal data and the summary length constraint information into a pre-trained summary generation model to extract the summary data of the multimodal data, it performs the following steps:

[0111] The encoder converts the first text data and the first image data into vector representations to obtain a first text context vector corresponding to the first text data and a first image context vector corresponding to the first image data. The first text context vector and the first image context vector are then fused based on an attention mechanism to obtain a fused multimodal context vector. The multimodal context vector includes a second text context vector and a second image context vector.

[0112] The decoder determines the vector length and importance score of each second text context vector, selects target text context vectors from the second text context vectors based on the vector length and importance score, and generates the text summary based on the target text context vectors; and determines the visual coverage score of each second image context vector, selects target image context vectors from the second image context vectors based on the visual coverage score, and determines the image corresponding to the target image context vector as the image summary.

[0113] In some embodiments, when the extraction module 42 performs the following steps to fuse the first text context vector and the first image context vector based on an attention mechanism to obtain a fused multimodal context vector:

[0114] Based on the attention mechanism, a first similarity is calculated between the first text context vector and the first image context vector;

[0115] Based on the first similarity, determine the text attention weight corresponding to the first text context vector and the image attention weight corresponding to the first image context vector;

[0116] The first text context vector and the first image context vector are fused based on the text attention weight and the image attention weight to obtain the multimodal context vector.

[0117] In some embodiments, when the extraction module 42 determines the vector size and importance score of each of the second text context vectors, and filters out the target text context vector from the second text context vectors based on the vector length and the importance score, it performs the following steps:

[0118] For each of the second text context vectors, calculate the second similarity between the second text context vector and the other second text context vectors;

[0119] Based on the second similarity, calculate the importance score of the second text context vector;

[0120] Select the target text context vector that satisfies the text summarization conditions from the second text context vector; the text summarization conditions include: the target text context vector has the highest total importance score, and the total length of the vector matches the summary length constraint information.

[0121] In some embodiments, the summary length constraint information includes the total text length of the text summary;

[0122] When the extraction module 42 selects the target text context vector that meets the text summarization conditions from the second text context vector, it performs the following steps:

[0123] According to the importance scores from high to low, select one of the second text context vectors in sequence;

[0124] After selecting a second text context vector each time, determine whether the sum of the lengths of the selected second text context vectors matches the total length of the text;

[0125] If a match is found, the currently selected second text context vector is added to the candidate vector set, and the next second text context vector is selected from the second text context vectors; the candidate vector set includes the already selected second text context vectors.

[0126] If there is no match, the second text context vector in the candidate vector set is determined to be the target text context vector.

[0127] In some embodiments, when the extraction module 42 determines the visual coverage score of each of the second image context vectors and filters out the target image context vector from the second image context vectors based on the visual coverage score, it performs the following steps:

[0128] For each of the second image context vectors, a visual coverage score for the second image context vector is calculated; the visual coverage score is used to represent the degree of attention accumulation of the second image context vector.

[0129] Select the target image context vectors from the second image context vectors whose visual coverage scores meet the visual coverage conditions; the visual coverage conditions include at least one of the following: the highest visual coverage score, or the visual coverage score is higher than or equal to a first score threshold.

[0130] In some embodiments, the apparatus further includes:

[0131] The second acquisition module is used to acquire sample multimodal data; the sample multimodal data includes interrelated sample text data and sample image data;

[0132] The prediction module is used to input the sample multimodal data, sample summary length constraint information, and sample summary data into the base model to be trained, so as to extract the summary from the sample data through the base model to obtain predicted summary data; the predicted summary data includes predicted text summary and predicted image summary; the sample sentences in the sample summary data have the highest total importance score, and the total length of the sample sentences matches the sample summary length constraint information;

[0133] The training module is used to iteratively train the base model based on the predicted summary data and the sample summary data to obtain the summary generation model.

[0134] In some embodiments, when the training module iteratively trains the base model based on the predicted summary data and the sample summary data, it performs the following steps:

[0135] Calculate the third similarity between the predicted summary data and the sample summary data, and determine whether the base model satisfies the iteration termination condition based on the third similarity;

[0136] If the iteration termination condition is not met, the model parameters of the base model are adjusted, and the next round of iteration training is carried out based on the adjusted base model.

[0137] If the iteration termination condition is met, then the base model obtained in the current iteration is determined to be the summary generation model.

[0138] According to the apparatus of this application embodiment, multimodal data to be processed is acquired, including first text data and first image data that are interrelated; the multimodal data and summary length constraint information are input into a pre-trained summary generation model for summary extraction to obtain summary data of the multimodal data, including text summary and image summary; wherein, the summary generation model is trained based on sample multimodal data, sample summary length constraint information, and sample summary data. Since the sample summary data satisfies the following conditions: the total importance score of the sample sentences in the sample summary data is the highest, and the total length of the sample sentences matches the sample summary length constraint information, the pre-trained summary generation model treats the summary generation problem as a knapsack problem when generating summaries, ensuring that the final generated summary data satisfies the conditions of maximizing the total importance score of the sentences in the summary data and matching the total sentence length with the summary length constraint information, that is, maximizing the quality of the summary while satisfying the summary length constraint. Because the summary length constraint is flexible and controllable, the summary data generated by the summary generation model not only meets the user's requirements for summary length but also ensures that the summary data retains the core information and readability of the original multimodal data, improving the overall quality of the multimodal summary.

[0139] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0140] Figure 6 This is a block diagram for implementing the electronic device provided in the embodiments of this application. Figure 6 As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the method described in the above embodiments. The number of memories 601 and processors 602 can be one or more. In a specific implementation, the electronic device may also include a communication interface 603 for communicating with external devices and exchanging data.

[0141] In practical implementation, if the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0142] Optionally, in a specific implementation, if the memory 601, processor 602 and communication interface 603 are integrated on a single chip, the memory 601, processor 602 and communication interface 603 can communicate with each other through an internal interface.

[0143] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0144] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0145] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0146] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0147] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0148] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0149] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0150] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0151] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0152] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0153] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0154] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0156] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating multimodal summaries, characterized in that, include: Acquire multimodal data to be processed; the multimodal data includes interrelated first text data and first image data; The multimodal data and summary length constraint information are input into a pre-trained summary generation model for summary extraction to obtain summary data of the multimodal data; the summary data includes text summary and image summary; The summary generation model is trained based on sample multimodal data, sample summary length constraint information, and sample summary data. The sample summary data satisfies the following conditions: the total importance score of the sample sentences in the sample summary data is the highest, and the total length of the sample sentences matches the sample summary length constraint information. The summary generation model includes an encoder and a decoder based on an attention mechanism; the process of inputting the multimodal data and summary length constraint information into the pre-trained summary generation model for summary extraction to obtain summary data of the multimodal data includes: The encoder converts the first text data and the first image data into vector representations to obtain a first text context vector corresponding to the first text data and a first image context vector corresponding to the first image data. The first text context vector and the first image context vector are then fused based on an attention mechanism to obtain a fused multimodal context vector. The multimodal context vector includes a second text context vector and a second image context vector. The decoder determines the vector length and importance score of each second text context vector, selects target text context vectors from the second text context vectors based on the vector length and importance score, and generates the text summary based on the target text context vectors; and determines the visual coverage score of each second image context vector, selects target image context vectors from the second image context vectors based on the visual coverage score, and determines the image corresponding to the target image context vector as the image summary.

2. The method according to claim 1, characterized in that, The process of fusing the first text context vector and the first image context vector based on an attention mechanism to obtain the fused multimodal context vector includes: Based on the attention mechanism, a first similarity is calculated between the first text context vector and the first image context vector; Based on the first similarity, determine the text attention weight corresponding to the first text context vector and the image attention weight corresponding to the first image context vector; The first text context vector and the first image context vector are fused based on the text attention weight and the image attention weight to obtain the multimodal context vector.

3. The method according to claim 1, characterized in that, The step of determining the vector size and importance score of each of the second text context vectors, and selecting the target text context vector from the second text context vectors based on the vector length and the importance score, includes: For each of the second text context vectors, calculate the second similarity between the second text context vector and the other second text context vectors; Based on the second similarity, calculate the importance score of the second text context vector; Select the target text context vector that satisfies the text summarization conditions from the second text context vector; the text summarization conditions include: the target text context vector has the highest total importance score, and the total length of the vector matches the summary length constraint information.

4. The method according to claim 3, characterized in that, The summary length constraint information includes the total text length of the text summary; The step of selecting the target text context vector that satisfies the text summarization conditions from the second text context vector includes: According to the importance scores from high to low, select one of the second text context vectors in sequence; After selecting a second text context vector each time, determine whether the sum of the lengths of the selected second text context vectors matches the total length of the text; If a match is found, the currently selected second text context vector is added to the candidate vector set, and the next second text context vector is selected from the second text context vectors; the candidate vector set includes the already selected second text context vectors. If there is no match, the second text context vector in the candidate vector set is determined to be the target text context vector.

5. The method according to claim 1, characterized in that, The step of determining the visual coverage score of each of the second image context vectors, and filtering the target image context vectors from the second image context vectors based on the visual coverage score, includes: For each of the second image context vectors, a visual coverage score for the second image context vector is calculated; the visual coverage score is used to represent the degree of attention accumulation of the second image context vector. Select the target image context vectors from the second image context vectors whose visual coverage scores meet the visual coverage conditions; the visual coverage conditions include at least one of the following: the highest visual coverage score, or the visual coverage score is higher than or equal to a first score threshold.

6. The method according to claim 1, characterized in that, The method further includes: Acquire sample multimodal data; the sample multimodal data includes interrelated sample text data and sample image data; The sample multimodal data, sample summary length constraint information, and sample summary data are input into the base model to be trained, so that the base model can extract summaries from the sample data to obtain predicted summary data; the predicted summary data includes predicted text summaries and predicted image summaries; the sample sentences in the sample summary data have the highest total importance score, and the total length of the sample sentences matches the sample summary length constraint information; The base model is iteratively trained based on the predicted summary data and the sample summary data to obtain the summary generation model.

7. The method according to claim 6, characterized in that, The step of iteratively training the base model based on the predicted summary data and the sample summary data includes: Calculate the third similarity between the predicted summary data and the sample summary data, and determine whether the base model satisfies the iteration termination condition based on the third similarity; If the iteration termination condition is not met, the model parameters of the base model are adjusted, and the next round of iteration training is carried out based on the adjusted base model. If the iteration termination condition is met, then the base model obtained in the current iteration is determined to be the summary generation model.

8. A multimodal summarization generation device, characterized in that, include: An acquisition module is used to acquire multimodal data to be processed; the multimodal data includes interrelated first text data and first image data; An extraction module is used to input the multimodal data and summary length constraint information into a pre-trained summary generation model for summary extraction, thereby obtaining summary data of the multimodal data; the summary data includes text summaries and image summaries; The summary generation model is trained based on sample multimodal data, sample summary length constraint information, and sample summary data. The sample summary data satisfies the following conditions: the total importance score of the sample sentences in the sample summary data is the highest, and the total length of the sample sentences matches the sample summary length constraint information. The summary generation model includes an encoder and decoder based on an attention mechanism; the extraction module is also used for: The encoder converts the first text data and the first image data into vector representations to obtain a first text context vector corresponding to the first text data and a first image context vector corresponding to the first image data. The first text context vector and the first image context vector are then fused based on an attention mechanism to obtain a fused multimodal context vector. The multimodal context vector includes a second text context vector and a second image context vector. The decoder determines the vector length and importance score of each second text context vector, selects target text context vectors from the second text context vectors based on the vector length and importance score, and generates the text summary based on the target text context vectors; and determines the visual coverage score of each second image context vector, selects target image context vectors from the second image context vectors based on the visual coverage score, and determines the image corresponding to the target image context vector as the image summary.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 7.

Images