Text summary generation model training method, text summary generation method and device

Abstract

The present application relates to a text summarization model training method, a text summarization method, an apparatus, a computer device, a storage medium, and a computer program product. The method comprises: inputting a training text and a corresponding training image set into an initial text summarization model to obtain a predicted text summary, and generating a target loss based on the difference between the predicted text summary and the labeled text summary; inputting masked training data corresponding to the training text and first training data into the initial text summarization model to obtain masked predicted data, and generating a reconstruction loss based on the difference between the masked labeled data and the masked predicted data; adjusting model parameters of the initial text summarization model based on the target loss and the reconstruction loss until convergence conditions are met, thereby obtaining a target text summarization model; and using the target text summarization model to generate a text summary of the text. This method can improve the prediction accuracy of the model and the quality of the generated text summary.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a text summary generation model training method, a text summary generation method, an apparatus, a computer device, a storage medium, and a computer program product. Background Art

[0002] With the development of computer technology, the amount of information on the internet has grown exponentially. People are exposed to a vast amount of information every day, including current affairs news, trending topics, social chats, lectures, and academic papers. To help users quickly access useful information from the internet, automatic text summarization technology has emerged. Automatic text summarization uses various techniques to extract, summarize, or refine key information from text, creating a summary that summarizes and presents the main content or gist of the original text.

[0003] Currently, machine learning models can be used to generate text summaries. Traditionally, summarization models are trained based on the text and summary information of articles with known summaries. However, training models based solely on text information makes it difficult to learn the most important information in the article for the summary. Text summarization models trained using current methods suffer from low prediction accuracy. Summary of the Invention

[0004] Based on this, it is necessary to provide a text summary generation model training method, text summary generation method, device, computer equipment, computer-readable storage medium and computer program product that can improve the model prediction accuracy in response to the above technical problems.

[0005] This application provides a method for training a text summarization model. The method includes:

[0006] Obtaining a training text and a training image set corresponding to the training text, and a label text summary; obtaining first training data, masked label data, and masked training data corresponding to the training text; the masked label data being masked data of the second training data after masking, and the masked training data being data other than the masked label data in the second training data; the first training data and the second training data being determined from the training image set and the label text summary;

[0007] Inputting the training text and the training image set into an initial text summary generation model to obtain a predicted text summary, and generating a target loss based on the difference between the predicted text summary and the label text summary;

[0008] Inputting the mask training data and the first training data into an initial text summarization model to obtain mask prediction data, and generating a reconstruction loss based on a difference between the mask label data and the mask prediction data;

[0009] Based on the target loss and the reconstruction loss, the model parameters of the initial text summary generation model are adjusted until a convergence condition is met, thereby obtaining a target text summary generation model; the target text summary generation model is used to generate a text summary of the text.

[0010] This application also provides a text summarization generation model training device. The device includes:

[0011] a data acquisition module configured to acquire a training text, a training image set corresponding to the training text, and a label text summary, and to acquire first training data, masked label data, and masked training data corresponding to the training text; the masked label data being masked data of the second training data after masking, and the masked training data being data other than the masked label data in the second training data; the first training data and the second training data being determined from the training image set and the label text summary;

[0012] a first data processing module, configured to input the training text and the training image set into an initial text summary generation model to obtain a predicted text summary, and generate a target loss based on a difference between the predicted text summary and the label text summary;

[0013] a second data processing module, configured to input the mask training data and the first training data into an initial text summarization model to obtain mask prediction data, and generate a reconstruction loss based on a difference between the mask label data and the mask prediction data;

[0014] A model parameter adjustment module is used to adjust the model parameters of the initial text summary generation model based on the target loss and the reconstruction loss until the convergence condition is met, thereby obtaining a target text summary generation model; the target text summary generation model is used to generate a text summary of the text.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in the above-mentioned text summary generation model training method when executing the computer program.

[0016] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps described in the above-mentioned text summary generation model training method are implemented.

[0017] A computer program product includes a computer program, which, when executed by a processor, implements the steps described in the above-mentioned text summary generation model training method.

[0018] The above-mentioned text summary generation model training method, device, computer equipment, storage medium and computer program product obtain training text and a training image set and a labeled text summary corresponding to the training text, and obtain first training data, masked label data and masked training data corresponding to the training text; the masked label data is the data masked after the second training data is masked, and the masked training data is other data in the second training data except the masked label data. The first training data and the second training data are determined from the training image set and the labeled text summary; the training text and the training image set are input into the initial text summary generation model to obtain a predicted text summary, and a target loss is generated based on the difference between the predicted text summary and the labeled text summary; the masked training data and the first training data are input into the initial text summary generation model to obtain masked prediction data, and a reconstruction loss is generated based on the difference between the masked label data and the masked prediction data; the model parameters of the initial text summary generation model are adjusted based on the target loss and the reconstruction loss until the convergence condition is met to obtain a target text summary generation model; the target text summary generation model is used to generate a text summary of the text. In this way, when training the model, the training loss includes target loss and reconstruction loss. The target loss corresponds to the summary generation task, which restores the original text summary through text information and image information. The target loss can guide the model to generate a text summary based on the text information and image information corresponding to the text. The reconstruction loss corresponds to the reconstruction task, which restores the masked data in the second training data through the unmasked data in the first training data and the second training data. The reconstruction loss helps to enhance the association between the first training data and the second training data, and guides the model to learn information related to the text summary from the image information corresponding to the text, thereby helping to improve the quality of the text summary generated by the model. Adjusting the model parameters based on the target loss and reconstruction loss can enable the model to output a higher quality text summary and effectively improve the prediction accuracy of the model. Generating a text summary through the target text summary generation model obtained by the final training can improve the quality of the generated text summary.

[0019] This application provides a method for generating a text summary. The method includes:

[0020] Obtain the target text and the target image set corresponding to the target text;

[0021] Inputting the target text and the target image set into a target text summary generation model to obtain a target text summary corresponding to the target text;

[0022] The training process of the target text summary generation model includes the following steps:

[0023] Obtain training text and a training image set and a label text summary corresponding to the training text, and obtain first training data, mask label data, and mask training data corresponding to the training text; the mask label data is the masked data of the second training data after masking, and the masked training data is other data in the second training data except the mask label data, and the first training data and the second training data are determined from the training image set and the label text summary; input the training text and the training image set into an initial text summary generation model to obtain a predicted text summary, and generate a target loss based on the difference between the predicted text summary and the label text summary; input the masked training data and the first training data into the initial text summary generation model to obtain masked prediction data, and generate a reconstruction loss based on the difference between the masked label data and the masked prediction data; adjust the model parameters of the initial text summary generation model based on the target loss and the reconstruction loss until the convergence condition is met, and obtain a target text summary generation model.

[0024] This application also provides a text summary generation device. The device includes:

[0025] A target text acquisition module is used to acquire the target text and the target image set corresponding to the target text;

[0026] A text summary generation module, configured to input the target text and the target image set into a target text summary generation model to obtain a target text summary corresponding to the target text;

[0027] The training process of the target text summary generation model includes the following steps:

[0028] Obtain training text and a training image set and a label text summary corresponding to the training text, and obtain first training data, mask label data, and mask training data corresponding to the training text; the mask label data is the masked data of the second training data after masking, and the masked training data is other data in the second training data except the mask label data, and the first training data and the second training data are determined from the training image set and the label text summary; input the training text and the training image set into an initial text summary generation model to obtain a predicted text summary, and generate a target loss based on the difference between the predicted text summary and the label text summary; input the masked training data and the first training data into the initial text summary generation model to obtain masked prediction data, and generate a reconstruction loss based on the difference between the masked label data and the masked prediction data; adjust the model parameters of the initial text summary generation model based on the target loss and the reconstruction loss until the convergence condition is met, and obtain a target text summary generation model.

[0029] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned text summary generation method when executing the computer program.

[0030] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned text summary generation method.

[0031] A computer program product includes a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned text summary generation method are implemented.

[0032] The above-mentioned text summarization method, apparatus, computer device, storage medium, and computer program product obtain a target text and a target image set corresponding to the target text, input the target text and the target image set into a target text summarization generation model, and obtain a target text summary corresponding to the target text. The target text summarization generation model is obtained by inputting the training text and the training image set into an initial text summarization generation model to obtain a predicted text summary, inputting masked training data and first training data into the initial text summarization generation model to obtain masked prediction data, generating a target loss based on the difference between the predicted text summary and the labeled text summary, generating a reconstruction loss based on the difference between the masked labeled data and the masked predicted data, and adjusting the model parameters of the initial text summarization generation model based on the target loss and the reconstruction loss. In this way, when training the model, the training loss includes target loss and reconstruction loss. The target loss corresponds to the summary generation task, which restores the original text summary through text information and image information. The target loss can guide the model to generate a text summary based on the text information and image information corresponding to the text. The reconstruction loss corresponds to the reconstruction task, which restores the masked data in the second training data through the unmasked data in the first training data and the second training data. The reconstruction loss helps to enhance the association between the first training data and the second training data, and guides the model to learn information related to the text summary from the image information corresponding to the text, thereby helping to improve the quality of the text summary generated by the model. Adjusting the model parameters based on the target loss and reconstruction loss can enable the model to output a higher-quality text summary and effectively improve the model's prediction accuracy. Inputting the target text and target image set into the target text summary generation model and generating a text summary through the trained target text summary generation model can effectively improve the quality of the generated text summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A diagram illustrating an application environment of a text summary generation model training method and a text summary generation method in one embodiment;

[0034] Figure 2 A flowchart of a text summary generation model training method according to an embodiment;

[0035] Figure 3 A schematic diagram of a process for generating a predictive text summary in one embodiment;

[0036] Figure 4 is a schematic diagram of initial image features in one embodiment;

[0037] Figure 5 A schematic diagram of the structure of a text summary generation model in one embodiment;

[0038] Figure 6 Schematic diagram of a flow chart of a method for generating a text summary in one embodiment;

[0039] Figure 7 is a schematic diagram of a document and a corresponding text summary in one embodiment;

[0040] Figure 8 Schematic diagram of the architecture of a multilingual multimodal summarization model in one embodiment;

[0041] Figure 9 A structural block diagram of a text summary generation model training device in one embodiment;

[0042] Figure 10 is a structural block diagram of a text summary generating device in one embodiment;

[0043] Figure 11 is a diagram of the internal structure of a computer device in one embodiment;

[0044] Figure 12 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0046] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0047] The solutions provided in the embodiments of this application involve artificial intelligence computer vision technology, natural language processing, machine learning and other technologies, which are specifically illustrated by the following embodiments:

[0048] The text summary generation model training method and text summary generation method provided in the embodiments of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other servers. The terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers or a cloud server.

[0049] Both the terminal and the server can be used independently to execute the text summary generation model training method and the text summary generation method provided in the embodiments of this application.

[0050] For example, the server obtains training text and a training image set corresponding to the training text, and a label text summary, inputs the training text and the training image set into the initial text summary generation model to obtain a predicted text summary, and generates a target loss based on the difference between the predicted text summary and the label text summary. The server obtains first training data, masked label data, and masked training data corresponding to the training text, inputs the masked training data and the first training data into the initial text summary generation model to obtain masked prediction data, and generates a reconstruction loss based on the difference between the masked label data and the masked prediction data. The server adjusts the model parameters of the initial text summary generation model based on the target loss and the reconstruction loss until the convergence condition is met, thereby obtaining a target text summary generation model. The masked label data is the masked data of the second training data after masking, the masked training data is the other data in the second training data except the masked label data, and the first training data and the second training data are determined from the training image set and the label text summary.

[0051] The server obtains the target text and the target image set corresponding to the target text, inputs the target text and the target image set into the target text summary generation model, and obtains the target text summary corresponding to the target text.

[0052] The terminal and the server can also be used in collaboration to execute the text summary generation model training method and the text summary generation method provided in the embodiments of this application.

[0053] For example, the server obtains training text, a training image set corresponding to the training text, and a labeled text summary from the terminal, and obtains first training data, masked label data, and masked training data corresponding to the training text from the terminal. The server inputs the training text and the training image set into the initial text summary generation model to obtain a predicted text summary, generates a target loss based on the difference between the predicted text summary and the labeled text summary, inputs the masked training data and the first training data into the initial text summary generation model to obtain masked prediction data, and generates a reconstruction loss based on the difference between the masked label data and the masked prediction data. The server adjusts the model parameters of the initial text summary generation model based on the target loss and the reconstruction loss until convergence conditions are met, thereby obtaining a target text summary generation model.

[0054] The terminal sends a text summary generation request to the server, which carries the text identifier corresponding to the target text. Based on the text identifier, the server obtains the target text and the corresponding target image set. It then inputs the target text and target image set into the target text summary generation model to generate a target text summary corresponding to the target text. The server then sends the target text summary to the terminal, which can then display the target text summary.

[0055] In one embodiment, Figure 2 As shown in FIG, a text summary generation model training method is provided, and the method is applied to a computer device as an example. The computer device can be a terminal or a server, and the method can be executed by the terminal or server alone, or through the interaction between the terminal and the server. Figure 2 ,The text summary generation model training method includes the following steps:

[0056] Step S202: Obtain a training text and a training image set and a label text summary corresponding to the training text, and obtain first training data, mask label data, and mask training data corresponding to the training text; the mask label data is the masked data after the second training data is masked, and the mask training data is the other data in the second training data except the mask label data. The first training data and the second training data are determined from the training image set and the label text summary.

[0057] The training text and the corresponding training image set constitute a training document containing both text and images. A training document refers to a document with a known text summary, which is used for model training. The training text refers to the text portion of the training document, that is, the text information in the training document. The training image set refers to the image portion of the training document, that is, the image information in the training document. The training image set may include at least one training image. It is understood that there is a corresponding relationship between the training text and the training image set in the same training document. The training document can be a variety of documents, for example, news, reports, letters, notifications, push notifications, etc.

[0058] The training text and training image collection serve as input data during model training. During model training, the model processes the training text and training image collection to predict a text summary. The labeled text summary refers to the ground truth text summary corresponding to the training text, that is, the ground truth text summary corresponding to the training document. The labeled text summary serves as a supervisory signal during model training, guiding the model to predict a text summary that is close to the labeled text summary and output an accurate text summary.

[0059] The first training data and the second training data are determined from a training image set and a labeled text summary. The first training data and the second training data are different data. For example, if the first training data is a training image set, the second training data is a labeled text summary; if the first training data is a labeled text summary, the second training data is a training image set. The first training data is used as input data for the model during training, and the second training data is used to determine the data that is input into the model in conjunction with the first training data during training, as well as to determine another supervisory signal for the model.

[0060] The masked label data and masked training data are obtained by masking the second training data. Masking refers to masking part of the data in the second training data. Masked label data refers to the masked portion of the second training data after masking, that is, masked label data refers to the masked data of the second training data after masking. Masked training data refers to the unmasked portion of the second training data after masking, that is, masked training data refers to the other data in the second training data except the masked label data. The masked training data is used as the input data of the model in conjunction with the first training data during model training. During model training, the model predicts the masked data in the second training data by performing data processing on the first training data and the masked training data. The masked label data is used as a supervisory signal of the model during model training to guide the model to predict data close to the masked label data.

[0061] Specifically, the computer device can obtain model training data locally or from another device, and perform model training based on the training data to train an accurate text summarization model. The model training data includes training text and a set of training images corresponding to the training text, a labeled text summary, first training data, masked label data, and masked training data.

[0062] In step S204, the training text and the training image set are input into the initial text summary generation model to obtain a predicted text summary, and a target loss is generated based on the difference between the predicted text summary and the label text summary.

[0063] The initial text summarization model refers to the text summarization model to be trained. The text summarization model must be trained before it can be used to generate a text summary for any document. During application, the text summarization model takes the text and images from the document as input and outputs the document's text summary.

[0064] A predicted text summary is the text summary predicted by the model after processing the input data. After processing the training text and training image sets, the model can output a predicted text summary.

[0065] Specifically, during model training, the computer device can input the training text and training image collection into the initial text summary generation model. After the model processes the data, the model outputs a predicted text summary corresponding to the training text. The training goal of the model is to make the predicted text summary and the true text summary increasingly close, so that when the model encounters a new document, it can generate an accurate text summary based on the text and image information of the new document. Therefore, the computer device can generate a target loss based on the difference between the predicted text summary and the labeled text summary. The target loss is used for backpropagation to adjust the model parameters so that the predicted text summary output by the model gradually approaches the labeled text summary.

[0066] In one embodiment, a training text and a training image set can be input into an initial text summary generation model to generate a predicted text summary. The model encodes the training text and the training image set, respectively, to obtain training text features and training image features. The training text features and training image features are then fused to obtain a training fused feature. The training fused feature is then decoded to obtain a predicted text summary. The decoding of the training fused feature can be performed in an orderly manner to obtain a sequence of words, thereby obtaining a predicted text summary. When decoding a word in the predicted text summary, the decoded word can be referenced.

[0067] In one embodiment, in order to improve the quality of model training, the training text, training image set and label text summary can be input into the initial text summary generation model to obtain a predicted text summary. The model encodes the training text and training image set respectively to obtain training text features and training image features, fuses the training text features and training image features to obtain fused image features, and decodes the fused image features based on the label text summary to obtain a predicted text summary. The fused image features are decoded based on the label text summary so that the model can refer to the historical part of the label text summary when decoding a certain word in the predicted text summary. For example, when decoding to generate the tth word, the previous t-1 words in the label text summary can be referred to. Decoding the fused image features based on the label text summary can correct the model's prediction during training and avoid further error amplification in the process of word sequence generation. Decoding the fused image features based on the label text summary can greatly accelerate the convergence speed of the model, making the model training process faster and smoother.

[0068] In one embodiment, to reduce the amount of model computation, feature extraction can be performed on the training image set in advance to obtain initial image features. The training text and initial image features are then input into the initial text summary generation model to obtain a predicted text summary. Alternatively, feature extraction can be performed on the training text in advance to obtain initial text features. The initial text features and initial image features are then input into the initial text summary generation model to obtain a predicted text summary.

[0069] Step S206 : Input the mask training data and the first training data into the initial text summarization generation model to obtain mask prediction data, and generate a reconstruction loss based on the difference between the mask label data and the mask prediction data.

[0070] The mask prediction data refers to the mask portion predicted by the model after data processing of the input data, and is the predicted masked portion of the second training data. After the model processes the mask training data and the first training data, it can output the mask prediction data.

[0071] Specifically, during model training, the computer device can input the mask training data and the first training data into the initial text summary generation model. After the model processes the data, the model outputs the mask prediction data. The training goal of the model is to make the mask prediction data and the mask label data closer and closer, so that the model can learn the association between the training image set and the label text summary. When faced with a new document, in addition to the text information of the new document, it can better refer to the image information of the new document to generate a more accurate text summary. Therefore, the computer device can generate a reconstruction loss based on the difference between the mask label data and the mask prediction data. The reconstruction loss is used for back propagation to adjust the model parameters so that the mask prediction data output by the model gradually approaches the mask label data, and can further make the predicted text summary output by the model gradually approach the label text summary.

[0072] In one embodiment, the masked training data and the first training data are input into the initial text summary generation model, the model encodes the masked training data and the first training data respectively to obtain masked training features and the first training features, performs inter-modal attention processing on the masked training features and the first training features to obtain fused features, and obtains mask prediction data based on the fused features.

[0073] Step S208 , adjusting the model parameters of the initial text summary generation model based on the target loss and the reconstruction loss until the convergence condition is met, thereby obtaining a target text summary generation model; the target text summary generation model is used to generate a text summary of the text.

[0074] The target text summarization model refers to a trained text summarization model. During application, the input data of the target text summarization model is the text and images in a document, and the output data is the text summary corresponding to the document. The convergence condition can be at least one of the following: the target loss and reconstruction loss are both less than their respective preset thresholds; the total loss based on the target loss and reconstruction loss is less than the target preset threshold; or the number of model iterations exceeds a preset number.

[0075] Specifically, the computer device performs backpropagation based on the target loss and the reconstruction loss to adjust the model parameters of the initial text summary generation model until a convergence condition is met, thereby obtaining a target text summary generation model. The computer device can perform backpropagation based on the target loss and the reconstruction loss to adjust the model parameters of the initial text summary generation model to obtain an intermediate text summary generation model, use the intermediate text summary generation model as the initial text summary generation model, and iterate the steps of obtaining the training text and the training image set corresponding to the training text, the labeled text summary, and obtaining the first training data corresponding to the training text, the masked label data, and the masked training data until the convergence condition is met, thereby obtaining the target text summary generation model.

[0076] In one embodiment, a computer device may obtain multiple training sets and randomly select one training set from each training set as the current training set. The computer device inputs the training data corresponding to the training text in the current training set into the initial text summarization model. After data processing by the initial text summarization model, a predicted text summary and mask prediction data corresponding to the training text are obtained. A target loss is calculated based on the predicted text summary and label text summary corresponding to the training text. A reconstruction loss is calculated based on the mask prediction data and mask label data corresponding to the training text. Model parameters of the initial text summarization model are adjusted based on the target loss and reconstruction loss to obtain an intermediate text summarization model. The computer device then uses the next training set as the new current training set and the intermediate text summarization model as the new initial text summarization model. It then returns to the step of inputting the training data corresponding to the training text in the current training set into the initial text summarization model to perform iterative training. This is repeated through multiple iterative trainings, continuously adjusting model parameters until convergence conditions are met, and obtaining a target text summarization model. For example, if, in a certain round of training, the total loss calculated based on the target loss and reconstruction loss is less than a target preset threshold, the model parameter adjustment is stopped, and the most recently adjusted text summarization model is used as the target text summarization model. If the number of model iterations after a round of training is greater than the preset number, the most recently adjusted text summary generation model will be used as the target text summary generation model.

[0077] In the above-mentioned text summary generation model training method, a training text and a training image set and a label text summary corresponding to the training text are obtained, and first training data, mask label data and mask training data corresponding to the training text are obtained; the mask label data is the masked data after the second training data is masked, and the masked training data is the other data in the second training data except the mask label data, and the first training data and the second training data are determined from the training image set and the label text summary; the training text and the training image set are input into the initial text summary generation model to obtain a predicted text summary, and a target loss is generated based on the difference between the predicted text summary and the label text summary; the masked training data and the first training data are input into the initial text summary generation model to obtain masked prediction data, and a reconstruction loss is generated based on the difference between the masked label data and the masked prediction data; the model parameters of the initial text summary generation model are adjusted based on the target loss and the reconstruction loss until the convergence condition is met to obtain a target text summary generation model; the target text summary generation model is used to generate a text summary of the text. In this way, when training the model, the training loss includes target loss and reconstruction loss. The target loss corresponds to the summary generation task, which restores the original text summary through text information and image information. The target loss can guide the model to generate a text summary based on the text information and image information corresponding to the text. The reconstruction loss corresponds to the reconstruction task, which restores the masked data in the second training data through the unmasked data in the first training data and the second training data. The reconstruction loss helps to enhance the association between the first training data and the second training data, and guides the model to learn information related to the text summary from the image information corresponding to the text, thereby helping to improve the quality of the text summary generated by the model. Adjusting the model parameters based on the target loss and reconstruction loss can enable the model to output a higher quality text summary and effectively improve the prediction accuracy of the model. Generating a text summary through the target text summary generation model obtained by the final training can improve the quality of the generated text summary.

[0078] In one embodiment, the reconstruction loss includes a summary reconstruction loss. Obtaining first training data, mask label data, and mask training data corresponding to the training text includes:

[0079] A training image set is obtained as first training data, and a label text summary is obtained as second training data; masking is performed on the label text summary to obtain summary mask label data and summary mask training data, the summary mask label data is obtained as mask label data, and the summary mask training data is obtained as mask training data.

[0080] Among them, the summary reconstruction loss corresponds to the summary reconstruction task. The summary reconstruction task refers to using the complete image to restore the occluded words in the text summary, so as to improve the direct connection between the text summary and the image, thereby improving the quality of the final generated text summary.

[0081] Summary mask label data refers to the masked or obscured data in the label text summary and is used as the model training label. Summary mask training data refers to the unmasked or unobscured data in the label text summary and is used as the model input data. The label text summary is masked, and the masked portion of the label text summary is used as the summary mask label data, while the remaining portion of the label text summary is used as the summary mask training data.

[0082] Specifically, when training a text summarization model, a computer device can utilize a summary reconstruction task to improve the model's training quality and the quality of the text summaries generated by the model. For the summary reconstruction task, the computer device can obtain a training image set as first training data, obtain a labeled text summary as second training data, obtain summary mask label data obtained by masking the labeled text summary as masked label data, and obtain summary mask training data obtained by masking the labeled text summary as masked training data. The masked training data and the first training data are used as input data for the model, and the masked label data is used as the model's desired output. The model is trained to use the complete image to recover obscured words in the text summary.

[0083] The mask training data and the first training data are input into the initial text summary generation model to obtain mask prediction data, and a reconstruction loss is generated based on the difference between the mask label data and the mask prediction data, including:

[0084] The training image set and summary mask training data are input into the initial text summary generation model to obtain summary mask prediction data; and a summary reconstruction loss is generated based on the difference between the summary mask label data and the summary mask prediction data.

[0085] The summary mask prediction data refers to the data predicted by the model for the masked and obscured data in the label text summary.

[0086] Specifically, for the summary reconstruction task, the computer device can input the training image set and the summary mask training data into the initial text summary generation model. After the model processes the data, the summary mask prediction data is obtained, and the summary reconstruction loss is generated based on the difference between the summary mask label data and the summary mask prediction data.

[0087] In the above embodiment, the reconstruction loss includes the summary reconstruction loss. The summary reconstruction loss corresponds to the summary reconstruction task, which refers to using the complete image to restore the occluded words in the text summary. The summary reconstruction loss can help the model improve the direct correlation between the text summary and the image, thereby improving the quality of the final generated text summary.

[0088] In one embodiment, the reconstruction loss includes an image reconstruction loss. Obtaining first training data, mask label data, and mask training data corresponding to the training text includes:

[0089] Obtain a label text summary as first training data, and obtain a training image set as second training data; perform mask processing on the training image set to obtain image mask label data and image mask training data, obtain the image mask label data as mask label data, and obtain the image mask training data as mask training data.

[0090] Among them, the image reconstruction loss corresponds to the image reconstruction task, which refers to using the complete text summary to restore the occluded image area in the image set, so as to enhance the direct correlation between the text summary and the image, thereby improving the quality of the final generated text summary.

[0091] Image mask label data refers to the masked or obscured data in the training image set and is used as the model's training labels. Image mask training data refers to the unmasked or unobscured data in the training image set and is used as the model's input data. The training image set is masked, and the masked portion of the training image set is used as the image mask label data, while the remaining portion of the training image set is used as the image mask training data.

[0092] Specifically, when training a text summary generation model, a computer device can utilize image reconstruction tasks to improve the model's training quality and the quality of the text summaries generated by the model. For the image reconstruction task, the computer device can obtain a labeled text summary as first training data, a training image set as second training data, image mask label data obtained by masking the training image set as mask label data, and image mask training data obtained by masking the training image set as mask training data. The mask training data and the first training data are used as input data for the model, and the mask label data is used as the model's expected output. The training model utilizes the complete text summary to restore occluded image regions in the image set.

[0093] The mask training data and the first training data are input into the initial text summary generation model to obtain mask prediction data, and a reconstruction loss is generated based on the difference between the mask label data and the mask prediction data, including:

[0094] The labeled text summary and image mask training data are input into the initial text summary generation model to obtain image mask prediction data; and the image reconstruction loss is generated based on the difference between the image mask labeled data and the image mask prediction data.

[0095] The image mask prediction data refers to the data predicted by the model for the masked and occluded data in the training image set.

[0096] Specifically, for the image reconstruction task, the computer device can input the label text summary and image mask training data into the initial text summary generation model. After the model processes the data, the image mask prediction data is obtained, and the image reconstruction loss is generated based on the difference between the image mask label data and the image mask prediction data.

[0097] It can be understood that the reconstruction loss may include at least one of a summary reconstruction loss or an image reconstruction loss.

[0098] In one embodiment, during masking, a preset proportion of data in the label text summary can be masked to obtain summary mask label data and summary mask training data. The preset proportion can be set based on actual needs, for example, 15%. Similarly, a preset proportion of data in the training image set can be masked to obtain image mask label data and image mask training data. The preset proportions corresponding to the label text summary and the training image set can be the same or different.

[0099] In the above embodiment, the reconstruction loss includes image reconstruction loss. The image reconstruction loss corresponds to the image reconstruction task, which refers to using the complete image to restore the occluded words in the text summary. The image reconstruction loss can help the model improve the direct correlation between the text summary and the image, thereby improving the quality of the final generated text summary.

[0100] In one embodiment, Figure 3 As shown, the training text and training image sets are input into the initial text summary generation model to obtain the predicted text summary, including:

[0101] Step S302: input the training text, the training image set and the label text summary into the initial text summary generation model.

[0102] Specifically, in order to improve the quality of model training, the computer device can input the training text, training image set and label text summary into the initial text summary generation model, and guide the model to better generate the predicted text summary based on the label text summary.

[0103] Step S304 , encoding the training text and training image sets respectively to obtain training text features and training image features.

[0104] Step S306: Fusing the training text features and the training image features to obtain training fusion features.

[0105] Among them, encoding processing refers to encoding data to quantify the data and convert the data into feature vectors that are convenient for the model to process the data.

[0106] Specifically, in the model, the training text is first encoded to obtain training text features, and the training image set is encoded to obtain training image features. Then, the training text features and training image features are fused to obtain training fusion features. The training fusion features integrate the effective text information and image information in the training documents and are a multimodal feature.

[0107] Step S308: Based on the label text summary, the training fusion features are decoded to obtain a predicted text summary.

[0108] The decoding process refers to decoding the feature vector so as to restore the feature vector to data that is easy to understand.

[0109] Specifically, in the model, the training fusion features can be decoded based on the label text summary. When generating a word sequence used as a predicted text summary, the relevant information of the label text summary is referenced, ensuring that the generated predicted text summary has a certain degree of reliability and accuracy, thereby preventing the model from being affected by completely incorrect prediction results during training. When decoding the training fusion features to generate a certain word in the predicted text summary, the historical portion of the label text summary can be referenced. This historical portion of the label text summary can be used to guide the generation of the next word, which can improve the accuracy of the generated word, and thus improve the accuracy of the generated predicted text summary.

[0110] In one embodiment, when decoding to generate a word sequence for use as a predicted text summary, each word in the word sequence may refer to its corresponding historical portion in the label text summary. For example, when decoding to generate the t-th word, the previous t-1 words in the label text summary may be referenced. When decoding to generate a word sequence for use as a predicted text summary, a portion of the words in the word sequence may refer to their corresponding historical portion in the label text summary, while another portion of the words may refer to words that have already been decoded. For example, when decoding to generate the t-th word, a probability p is used to select whether to refer to its corresponding historical portion in the label text summary or to refer to the currently decoded word.

[0111] In the above embodiment, training text, a training image set, and a labeled text summary are input into the initial text summary generation model. The training text and training image set are encoded to obtain training text features and training image features, respectively. The training text features and training image features are then fused to obtain a fused training feature. Based on the labeled text summary, the fused training feature is decoded to obtain a predicted text summary. In this way, guiding the model to generate a predicted text summary based on the labeled text summary can improve the training quality of the model and accelerate the model convergence speed.

[0112] In one embodiment, encoding processing is performed on the training text and training image sets respectively to obtain training text features and training image features, including:

[0113] Perform feature extraction on the training text to obtain initial text features, perform feature encoding on the initial text features to obtain training text features; perform object recognition on the training images in the training image set to obtain object image features corresponding to at least one target object; obtain initial image features corresponding to the training image set based on the object image features corresponding to the training images in the training image set; perform feature encoding on the initial image features to obtain training image features.

[0114] Feature extraction involves mapping high-dimensional raw data (e.g., images, sentences) to low-dimensional representation vectors, making the high-dimensional raw data separable. Feature encoding involves further encoding the initially extracted feature vectors to produce feature vectors with stronger semantic expression capabilities.

[0115] Object recognition refers to identifying objects in an image. It's understood that objects in an image can include animals, plants, and objects. The target object is the object identified in the training image. Object image features are the image features corresponding to the target object in the training image.

[0116] Specifically, for the training text, features can be first extracted from the training text to obtain initial text features, and then feature encoded on the initial text features to obtain training text features. For example, features can be extracted from the training text using an embedding layer to obtain initial text features, and feature encoded on the initial text features using an encoding layer to obtain training text features.

[0117] For a training image set, object recognition is first performed on the training images in the training image set to obtain object image features corresponding to at least one target object in the training image. The object image features corresponding to the training images in the training image set are then aggregated to obtain initial image features corresponding to the training image set. Finally, feature encoding is performed on the initial image features to obtain training image features. For example, object recognition is performed on the training images in the training image set using an object recognition layer, and initial image features are obtained based on the object recognition results. The initial image features are then feature encoded using an encoding layer to obtain training image features.

[0118] In one embodiment, the initial image features can be extracted using a pre-trained network. That is, when training the initial text summarization model, the initial image features corresponding to each training image set are fixed. Model training primarily involves adjusting the model parameters used to encode the initial image features.

[0119] In one embodiment, the initial text features can be calculated by referring to the following formula (1):

[0120]

[0121] in, represents the initial text features, X represents the token embedding (i.e., character text features) corresponding to the training text, and E pe Indicates the position embedding corresponding to the training text (i.e., character position feature). represents the length of the training text, and d represents the vector dimension.

[0122] In the above embodiment, feature extraction is performed on the training text to obtain initial text features, and feature encoding is performed on the initial text features to obtain training text features. Through feature extraction and feature encoding, training text features containing important semantic information can be obtained. Object recognition is performed on the training images in the training image set to obtain object image features corresponding to at least one target object. Based on the object image features corresponding to each training image in the training image set, initial image features corresponding to the training image set are obtained. Feature encoding is performed on the initial image features to obtain training image features. Through object recognition and feature encoding, training image features containing important semantic information can be obtained. Such training text features and training image features help improve the training quality of the model.

[0123] In one embodiment, obtaining initial image features corresponding to the training image set based on image features of each object corresponding to each training image in the training image set includes:

[0124] Feature extraction is performed on the image identification corresponding to each training image, the object identification corresponding to each target object, and the recognition position corresponding to each target object, respectively, to obtain the image identification features corresponding to each training image, the object identification features corresponding to each target object, and the recognition position features corresponding to each recognition position; based on the object identification features, recognition position features, object image features corresponding to the same target object, and the image identification features corresponding to the training image to which it belongs, the initial image sub-features corresponding to the target object are obtained; based on the initial image sub-features corresponding to each target object in each training image, the initial image features are obtained.

[0125] An image identifier is a unique identifier used to identify an image and may include a string of at least one of the following: letters, numbers, and symbols. An object identifier is a unique identifier used to identify an object and may include a string of at least one of the following: letters, numbers, and symbols. The identified position refers to the position of the object identified in the image.

[0126] Specifically, the training image set may include multiple training images, and each training image may include multiple target objects. In order to preserve the order information between images and the order information between objects, the object image features and the features fused with the order information can be combined to obtain the final initial image features. First, feature extraction is performed on the image identifiers corresponding to each training image to obtain the image identifier features corresponding to each training image, feature extraction is performed on the object identifiers corresponding to each target object to obtain the object identifier features corresponding to each target object, and feature extraction is performed on the recognition positions corresponding to each target object to obtain the recognition position features corresponding to each recognition position. Then, the object identifier features, recognition position features, object image features corresponding to the same target object, and the image identifier features corresponding to the training image to which they belong are combined to obtain the initial image sub-features corresponding to a single target object. Finally, based on the initial image sub-features corresponding to each target object in each training image in the training image set, the initial image features corresponding to the training image set are obtained.

[0127] In one embodiment, the initial image features may be calculated by referring to the following formulas (2) and (3):

[0128]

[0129]

[0130] Among them, O represents the initial image feature, o ij Represents the initial image sub-feature corresponding to the j-th target object in the i-th image. ij Represents the object image features corresponding to the j-th target object in the i-th image. Represents the identification position feature corresponding to the jth target object in the i-th image. represents the image identification feature corresponding to the i-th image, represents the object identification feature corresponding to the jth target object. n represents the number of training images in the training image set, and m represents the number of target objects in the training images.

[0131] refer to Figure 4 If the training image set includes multiple images, in order to preserve the order information between images, the extracted object embeddings (i.e. object image features) will be added with additional Image ID embeddings (image identification features), Object ID Embeddings (object identification features) and RoI box Embeddings (i.e. identification location features) to obtain the initial image features. Figure 5 The first row of data represents object embeddings, the second row of data represents Image ID embeddings, the third row of data represents Object ID Embeddings, and the fourth row of data represents RoI box Embeddings.

[0132] In the above embodiment, feature extraction is performed on the image identifier corresponding to each training image, the object identifier corresponding to each target object, and the recognition position corresponding to each target object, respectively, to obtain the image identifier features corresponding to each training image, the object identifier features corresponding to each target object, and the recognition position features corresponding to each recognition position. Based on the object identifier features, recognition position features, object image features corresponding to the same target object, and the image identifier features corresponding to the training image to which it belongs, the initial image sub-features corresponding to the target object are obtained. Based on the initial image sub-features corresponding to each target object in each training image, the initial image features are obtained. In this way, the object identifier features, recognition position features, object image features, and image identifier features can represent the sequential information of images and objects. The initial image features obtained based on the object identifier features, recognition position features, object image features, and image identifier features corresponding to the target object, and the image identifier features corresponding to the training image to which it belongs, contain the sequential information between images and between objects. When generating a text summary, the relevant information in the image can be referenced in an orderly manner, which helps to improve the accuracy of the generated text summary and improve the training quality of the model.

[0133] In one embodiment, the current initial feature is an initial text feature or an initial image feature. Feature encoding is performed on the current initial feature to obtain the corresponding current training feature, including:

[0134] Perform attention processing on the current initial feature to obtain the current attention feature; fuse the current initial feature and the current attention feature to obtain the initial fusion feature; perform feature transformation on the initial fusion feature to obtain the current transformation feature; fuse the current transformation feature and the initial fusion feature to obtain the target fusion feature; obtain the current training feature based on the target fusion feature.

[0135] Among them, attention processing is used to highlight the data containing important semantic information in the features, and feature transformation is used to map the features into a higher-dimensional space.

[0136] Specifically, the process of feature encoding the initial text features is similar to the process of feature encoding the initial image features. The feature encoding process is illustrated by taking the current initial features as an example. The current initial features are initial text features or initial image features. First, the current initial features are subjected to attention processing to obtain the current attention features. The current attention features highlight the key content of the current initial features. Then, the current initial features and the current attention features are fused to obtain the initial fusion features. Fusion of the current initial features and the current attention features can prevent the model from completely forgetting the current initial features during the training process. Then, the initial fusion features are subjected to feature transformation to obtain the current transformation features. The initial fusion features are mapped to a higher-dimensional space to improve the model's feature recognition ability. Furthermore, the current transformation features and the initial fusion features are fused to obtain the target fusion features. Fusion of the current transformation features and the initial fusion features can prevent the model from completely forgetting the initial fusion features during the training process. Finally, the current training features are obtained based on the target fusion features.

[0137] It can be understood that if the current initial feature is an initial text feature, the current training feature is a training text feature; if the current initial feature is an initial image feature, the current training feature is a training image feature.

[0138] In one embodiment, the initial text summarization model includes a text encoder and an image encoder. The text encoder is used to encode text, and the image encoder is used to encode images. The current encoder is a text encoder or an image encoder. The current encoder includes at least one encoding layer. Each encoding layer is used to feature encode input data. The output data of the current encoding layer is used as input data for the next encoding layer. The output data of the output encoding layer is the current training feature.

[0139] Taking the text encoder as an example, the text encoder includes at least one ordered connected encoding layer. The input data of the first encoding layer in the text encoder is the initial text feature, and the output data of the first encoding layer is the target fusion feature. The target fusion feature is input into the second encoding layer as the new initial text feature, and the second encoding layer outputs the new target fusion feature. The new target fusion feature is input into the third encoding layer as the new initial text feature, and the third encoding layer outputs the new target fusion feature. And so on. The output data of the current encoding layer is used as the input data of the next encoding layer, and the output data of the last encoding layer of the text encoder is used as the training text feature.

[0140] In one embodiment, the training text features can be calculated by referring to the following formulas (4) and (5):

[0141]

[0142]

[0143] in, It represents the output data of the l-1th encoding layer in the text encoder, that is, the input data of the lth encoding layer in the text encoder. MHA represents attention processing, that is, Multi-Head Attention operation. Represents the initial fused features corresponding to the lth encoding layer in the text encoder. FFN (Feed Forward Network) represents a feedforward neural network, which is used to provide nonlinear transformation and map data into a higher-dimensional space. Represents the output data of the lth encoding layer in the text encoder, that is, the target fusion feature corresponding to the lth encoding layer in the text encoder.

[0144] It can be understood that if the text encoder includes L encoding layers, the final training text features are

[0145] In one embodiment, the target fusion feature can be calculated by referring to the following formula (6) and formula (7):

[0146]

[0147]

[0148] in, It represents the output data of the l-1th encoding layer in the image encoder, that is, the input data of the lth encoding layer of the image encoder. MHA represents attention processing, that is, Multi-Head Attention operation. Represents the initial fused features corresponding to the lth encoding layer of the image encoder. FFN represents a feedforward neural network, which is used to provide nonlinear transformation and map data into a higher dimensional space. Represents the output data of the lth coding layer of the image encoder, that is, the target fusion feature corresponding to the lth coding layer of the image encoder. represents the number of target objects, d v The dimension of the vector representing the initial image features.

[0149] It can be understood that if the image encoder includes H layers of coding layers, the final training text features are

[0150] In one embodiment, attention processing can be implemented using a single-head attention mechanism or a multi-head attention mechanism.

[0151] In the above embodiment, attention processing is performed on the current initial feature to obtain the current attention feature; the current initial feature and the current attention feature are fused to obtain the initial fused feature; feature transformation is performed on the initial fused feature to obtain the current transformed feature; the current transformed feature and the initial fused feature are fused to obtain the target fused feature; and the current training feature is obtained based on the target fused feature. During feature encoding, accurate training text features and training image features can be obtained through attention processing and feature transformation, which in turn helps improve the accuracy of the generated text summary and the training quality of the model.

[0152] In one embodiment, the training text features and the training image features are fused to obtain the training fused features, including:

[0153] Inter-modal attention processing is performed on the training text features and the training image features to obtain fused attention features; image filtering features are obtained based on the training text features and the fused attention features; based on the image filtering features, feature filtering is performed on the fused attention features to obtain fused enhancement features; the fused enhancement features and the training text features are spliced ​​to obtain training fused features.

[0154] Intermodal attention processing refers to the process of applying attention to data from different modalities, allowing them to interact with each other and highlight relevant information between them. Modality refers to the source or form of information. For example, text features are generated based on text and are one type of modal data, while image features are generated based on images and are another type of modal data. Feature filtering refers to filtering out redundant and invalid data within features.

[0155] Specifically, when fusing the training text features and the training image features, inter-modal attention processing is first performed on the training text features and the training image features to obtain a fused attention feature. The fused attention feature highlights the related content in the training text features and the training image features. Then, based on the training text features and the fused attention features, an image filtering feature is obtained. The image filtering feature is used to filter out redundant and useless image features. Furthermore, based on the image filtering feature, the fused attention feature is feature filtered to obtain a fused enhancement feature. The fused enhancement feature contains the strongly related content in the text features and the image features. Finally, the fused enhancement feature and the training text features are spliced ​​together to obtain the training fused feature.

[0156] In one embodiment, inter-modal attention processing can be implemented based on commonly used attention mechanisms, such as key-value pair attention mechanisms, multi-head attention mechanisms, etc. The input information for inter-modal attention processing is data from different modalities. Of course, inter-modal attention processing can also be implemented based on custom formulas or algorithms.

[0157] In the above embodiment, inter-modal attention processing is performed on the training text features and the training image features to obtain a fused attention feature; based on the training text features and the fused attention features, an image filtering feature is obtained; based on the image filtering feature, the fused attention feature is feature filtered to obtain a fused enhancement feature; and the fused enhancement feature and the training text features are concatenated to obtain a training fused feature. When fusing the training text features and the training image features, inter-modal attention processing and feature filtering can produce a training fused feature that integrates important text information and important image information, thereby helping to improve the accuracy of the generated text summary and enhance the training quality of the model.

[0158] In one embodiment, image filtering features are obtained based on the training text features and the fused attention features, including:

[0159] The training text feature and the fused attention feature are spliced ​​to obtain a first spliced ​​feature; based on the first model parameter, the first spliced ​​feature is subjected to feature transformation to obtain a first transformed feature; and the first transformed feature is subjected to feature mapping to obtain an image filtering feature.

[0160] The first model parameter is a model parameter used to perform feature transformation on the first splicing feature. The first model parameter is a parameter to be learned in the model and is adjusted during model training. Feature mapping is used to map features to a preset range. For example, an activation function can be used for feature mapping.

[0161] Specifically, when calculating image filtering features, the training text features and the fused attention features can be first concatenated to obtain a first concatenated feature. The first concatenated feature is then transformed based on the first model parameters to obtain a first transformed feature. Finally, the first transformed feature is feature mapped to obtain an image filtering feature. Feature mapping can be used to uniformly map the data in the first transformed feature to a preset range. The image filtering feature obtained through feature mapping is equivalent to a weight coefficient. The subsequent model performs a weighted fusion of the image filtering feature and the fused attention feature to quickly filter out useless image features in the fused attention feature.

[0162] In the above embodiment, the training text features and the fused attention features are concatenated to obtain a first concatenated feature, a feature transformation is performed on the first concatenated feature based on the first model parameters to obtain a first transformed feature, and feature mapping is performed on the first transformed feature to obtain an image filtering feature. Through concatenation, feature transformation, and feature mapping, redundant features can be determined from the training text features and the fused attention features, resulting in an image filtering feature that is used to filter out useless image features from the fused attention features.

[0163] In one embodiment, the fusion enhancement feature and the training text feature are spliced ​​to obtain the training fusion feature, including: splicing the fusion enhancement feature and the training text feature to obtain a second spliced ​​feature; and performing feature transformation on the second spliced ​​feature based on the second model parameter to obtain the training fusion feature.

[0164] The second model parameter is a model parameter of the model used to perform feature transformation on the second splicing feature. The second model parameter is a parameter to be learned in the model and will be adjusted during the model training process.

[0165] Specifically, when performing splicing processing, in addition to splicing and fusion enhancement features and training text features to obtain training fusion features, you can also first splice and fusion enhancement features and training text features to obtain a second splicing feature, and then perform feature transformation on the second splicing feature based on the second model parameters to obtain a training fusion feature, so that the training fusion feature can be flexibly adjusted during the model training process, thereby speeding up the model training speed.

[0166] In one embodiment, the training fusion features can be calculated by referring to the following formulas (8) to (12):

[0167]

[0168]

[0169]

[0170]

[0171]

[0172] Among them, CMHA (Cross-modal Multi-Head Attention) represents cross-modal attention processing, that is, inter-modal attention processing, which enables text features and image features to interact. M represents the fused attention feature. CMHA is a query-key-value pair attention mechanism, where Q represents the query vector, K represents the key vector, and V represents the value vector. W q 、W k 、W v are parameters to be learned, which are used to linearly project the data into three different spaces to obtain the query vector, key vector, and value vector respectively. represents the training text features, Represents the training image features. d c Represents the vector dimensions of Q, K, and V.

[0173] G represents image filtering features, Concat represents concatenation, Sigmoid represents activation function, and W g 、b g Represents the first model parameter, which is the parameter to be learned. T+V Represents the training fusion feature, which can also be called multimodal feature. z 、b z Represents the second model parameter, which is the parameter to be learned. Represents dot product. G is a gate mechanism, and its dot product with M is used to filter out useless features in M.

[0174] In one embodiment, the label text summary includes a plurality of ordered training summary words. Based on the label text summary, the training fusion features are decoded to obtain a predicted text summary, including:

[0175] Determine the target position from each word position corresponding to the label text summary; obtain the training summary word before the target position from the label text summary as a reference summary word, encode the reference summary word to obtain a reference summary feature; obtain a predicted summary feature based on the training fusion feature and the reference summary feature; decode the predicted summary feature to obtain a predicted summary word corresponding to the target position; obtain the next word position as the target position, return to the step of obtaining the training summary word before the target position from the label text summary as a reference summary word, and execute until the end condition is met to obtain multiple predicted summary words; obtain a predicted text summary based on each predicted summary word.

[0176] The label text summary includes a plurality of ordered training summary words. For example, if the label text summary is "I love you", the plurality of ordered training summary words are I-love-you.

[0177] Specifically, when the training fusion features are decoded based on the label text summary, the various predicted summary words used to constitute the predicted text summary can be decoded in sequence. When decoding and generating a certain predicted summary word, it is necessary to refer to the historical part before the corresponding position in the label text summary. By referring to the historical part in the label text summary, a more correct predicted summary word can be decoded.

[0178] The computer device may select the first word position from each word position corresponding to the label text summary as the target position, obtain the training summary word before the target position from the label text summary as the reference summary word, and if the label text summary does not have a training summary word before the target position, obtain a preset character as the reference summary word, encode the reference summary word to obtain a reference summary feature, fuse the training fusion feature with the reference summary feature to obtain a predicted summary feature, and finally decode the predicted summary feature to obtain the predicted summary word corresponding to the target position, that is, obtain the predicted summary word corresponding to the first word position. The computer device may obtain the second word position from each word position corresponding to the label text summary as the target position, obtain the training summary word before the target position from the label text summary as the reference summary word, that is, obtain the training summary word at the first word position from the label text summary as the reference summary word, encode the reference summary word to obtain a reference summary feature, fuse the training fusion feature with the reference summary feature to obtain a predicted summary feature, and finally decode the predicted summary feature to obtain the predicted summary word corresponding to the target position, that is, obtain the predicted summary word corresponding to the second word position. The computer device obtains the third word position from each word position corresponding to the label text summary as the target position, obtains the training summary word before the target position from the label text summary as the reference summary word, that is, obtains the training summary words at the first word position and the second word position from the label text summary as the reference summary word, encodes the reference summary word to obtain a reference summary feature, fuses the training fusion feature and the reference summary feature to obtain a predicted summary feature, and finally decodes the predicted summary feature to obtain the predicted summary word corresponding to the target position, that is, obtains the predicted summary word corresponding to the third word position. This process is repeated until the termination condition is met, and multiple predicted summary words are obtained. Finally, the predicted summary words are arranged in order to obtain a predicted text summary.

[0179] It can be understood that the predicted summary feature incorporates the relevant information of the historical part of the label text summary. The predicted summary words obtained by decoding the predicted summary feature are better than the predicted summary words obtained by directly decoding the training fusion feature.

[0180] The termination condition may be that the total number of predicted summary words exceeds a preset number. If the total number of predicted summary words exceeds the preset number, decoding is stopped, and the decoded predicted summary words are arranged in order to obtain the predicted text summary. The termination condition may also be that a predicted summary word representing an end symbol is obtained. If a predicted summary word representing an end symbol is obtained during decoding, decoding is stopped, and the decoded predicted summary words are arranged in order to obtain the predicted text summary.

[0181] If there is no training summary word before the target position in the label text summary, a preset character is obtained as a reference summary word. The preset character can be set according to actual needs, for example, the start character is used as the preset character.

[0182] In one embodiment, decoding the predicted summary features to obtain the predicted summary words corresponding to the target position includes: performing feature transformation on the predicted summary features based on third model parameters to obtain third transformed features; and normalizing the third transformed features to obtain the predicted summary words corresponding to the target position.

[0183] The third model parameter is a model parameter used to perform feature transformation on the prediction summary feature. The third model parameter is a parameter to be learned in the model and is adjusted during the model training process.

[0184] Specifically, during decoding processing, the predicted summary features can be transformed based on the third model parameters to obtain third transformed features, and then the third transformed features can be normalized to obtain the predicted summary words corresponding to the target position, so that the predicted summary words can be flexibly adjusted during the model training process, thereby speeding up the model training speed.

[0185] In the above embodiment, predicted summary words are generated in order to form a predicted text summary. When generating the predicted summary words, the training summary words before the corresponding position in the label text summary are referred to. This can guide the model to output more accurate predicted summary words based on the forward correct answer, avoiding further amplification of errors in the process of word sequence generation, thereby greatly accelerating the convergence speed of the model and making the model training process faster and smoother.

[0186] In one embodiment, based on the training fusion features and the reference summary features, the predicted summary features are obtained, including:

[0187] Attention processing is performed on the reference summary features to obtain initial attention features, and initial summary features are obtained based on the initial attention features and the reference summary features; cross-attention processing is performed on the initial summary features and the training fusion features to obtain cross-attention features, and intermediate summary features are obtained based on the cross-attention features and the initial summary features; feature transformation is performed on the intermediate summary features to obtain target summary features, and predicted summary features are obtained based on the intermediate summary features and the target summary features.

[0188] Among them, cross-attention processing is used to interact different data. Cross-attention processing is an operation that combines feature crossing and attention processing.

[0189] Specifically, when obtaining predicted summary features based on the training fusion features and the reference summary features, attention processing is first performed on the reference summary features to obtain initial attention features. The initial attention features highlight the key content of the reference summary features. The initial attention features are then fused with the reference summary features to obtain initial summary features. This fusion of the initial attention features and the reference summary features prevents the model from completely forgetting the reference summary features during training. The initial summary features and the training fusion features are then subjected to cross-attention processing to obtain cross-attention features. This cross-attention processing allows the initial summary features and the training fusion features to fully interact, highlighting the related content between the initial summary features and the training fusion features. Furthermore, the cross-attention features are fused with the initial summary features to obtain intermediate summary features. This fusion of the cross-attention features and the initial summary features prevents the model from completely forgetting the initial summary features during training. Finally, feature transformation is performed on the intermediate summary features to obtain target summary features. The intermediate summary features are mapped to a higher-dimensional space to improve the model's feature recognition ability. Finally, the intermediate summary features and the target summary features are fused to obtain the predicted summary features.

[0190] In one embodiment, the cross-attention processing can be implemented based on a commonly used cross-attention mechanism or based on a custom formula or algorithm.

[0191] In one embodiment, attention processing is performed on the reference summary feature to obtain an initial attention feature, and the initial summary feature is obtained based on the initial attention feature and the reference summary feature; cross-attention processing is performed on the initial summary feature and the training fusion feature to obtain a cross-attention feature, and an intermediate summary feature is obtained based on the cross-attention feature and the initial summary feature; feature transformation is performed on the intermediate summary feature to obtain a target summary feature, and a predicted summary feature is obtained based on the intermediate summary feature and the target summary feature.

[0192] In one embodiment, the initial summary feature is subjected to attention processing to obtain a first attention feature, the training fusion feature is subjected to attention processing to obtain a second attention feature, and the first attention feature and the second attention feature are subjected to feature crossover to obtain a cross attention feature.

[0193] In the above embodiment, attention processing is performed on the reference summary feature to obtain an initial attention feature, and the initial summary feature is obtained based on the initial attention feature and the reference summary feature; cross-attention processing is performed on the initial summary feature and the training fusion feature to obtain a cross-attention feature, and an intermediate summary feature is obtained based on the cross-attention feature and the initial summary feature; feature transformation is performed on the intermediate summary feature to obtain a target summary feature, and a predicted summary feature is obtained based on the intermediate summary feature and the target summary feature. Through attention processing, cross-attention processing, and feature exchange, the reference summary feature and the training fusion feature can fully exchange data, ultimately obtaining a more accurate predicted summary feature.

[0194] In one embodiment, the predicted text summary can be calculated by referring to the following formulas (13) to (16):

[0195]

[0196]

[0197]

[0198]

[0199] in, Indicates y in the label text summary 1:t-1 (i.e., the word embedding vector representation of the 1st to t-1th words in the label text summary). MHA stands for attention processing. Denotes the initial summary features when generating the t-th predicted summary word. MHCA (Multi-Head Cross-Attention) represents cross-attention processing. represents the intermediate summary features when generating the t-th predicted summary word. FFN represents the feed-forward neural network. Represents the predicted summary features when generating the t-th predicted summary word.

[0200] It can be understood that the initial text summary generation model includes a decoder, which includes at least one decoding layer. Each encoding layer is used to perform a series of attention processing, cross-attention processing, and feature transformation on the input data. The output data of the current encoding layer is used as the input data of the next encoding layer, and the output data of the output encoding layer is the predicted summary feature. For example, the decoder includes three decoding layers, and the start symbol is encoded to obtain a reference summary feature corresponding to the start symbol, the reference summary feature corresponding to the start symbol is input into the first decoding layer, data processing according to formula (13), formula (14) and formula (15) is performed in the first encoding layer to obtain output data of the first decoding layer, the output data of the first decoding layer is input into the second decoding layer, data processing according to formula (13), formula (14) and formula (15) is performed in the second encoding layer to obtain output data of the second decoding layer, the output data of the second decoding layer is input into the third decoding layer, data processing according to formula (13), formula (14) and formula (15) is performed in the third encoding layer to obtain output data of the third decoding layer, the output data of the third decoding layer is used as the predicted summary feature corresponding to the first word position, the predicted summary feature is decoded with reference to formula (16) to obtain the first predicted summary word. Similarly, the first training summary word in the label text summary is encoded to obtain the reference summary feature corresponding to the first training summary word. This reference summary feature is input into the first decoding layer. After data processing at each decoding layer, the predicted summary feature corresponding to the second word position is finally obtained. This predicted summary feature is decoded to obtain the second predicted summary word. This process is repeated in this way until each predicted summary word is obtained, and each predicted summary word constitutes the predicted text summary.

[0201] If the decoder includes L decoding layers, the predicted summary feature corresponding to the t-th predicted summary word is W o 、b o Represents the third model parameter, which is the parameter to be learned. represents the t-th predicted summary word.

[0202] In one embodiment, reference Figure 5 The initial text summary generation model includes a text encoder, an image encoder, and a decoder. The training text or the initial text feature (X) is input into the text encoder for encoding to obtain the training text feature. Input the training image set or initial text features (O) into the image encoder for encoding to obtain the training image features The training text features and training image features are fused to obtain the training fusion features (Z T+V), the training fusion features and the label text summary (Y) are input into the decoder, the training fusion features are decoded based on the label text summary to obtain the predicted summary features, and the predicted summary features are feature decoded (for example, normalized by a softmax function) to obtain the predicted text summary (Y').

[0203] In one embodiment, the training set corresponding to the initial text summary generation model includes training data corresponding to multiple training texts, the training data includes training texts and corresponding training image sets, labeled text summaries, first training data, masked label data, masked training data and language labels, and the training set includes at least one language label.

[0204] The training text and training image collection are input into the initial text summary generation model to obtain the predicted text summary. The target loss is generated based on the difference between the predicted text summary and the label text summary, including:

[0205] The training text in the training set, the corresponding training image set, and the language label are input into the initial text summary generation model to obtain a predicted text summary that matches the language label corresponding to the training text; a target sub-loss is generated based on the difference between the predicted text summary and the label text summary corresponding to the same training text, and a target loss is obtained based on the target sub-loss corresponding to each training text.

[0206] The language tag instructs the model to output data that matches the language tag. For example, if the language tag is Chinese, the model ultimately outputs a predicted text summary in Chinese. If the language tag is English, the model ultimately outputs an English-language predicted text summary. The training set corresponding to the initial text summarization model includes training data corresponding to multiple training texts, each of which has its own corresponding language tag. The training set includes at least one language tag, meaning that the training set can include training texts in multiple languages.

[0207] Specifically, when generating a predicted text summary, the computer device may input the training text in the training set, the corresponding training image set, and the language label into the initial text summary generation model. After the model processes the data, the model ultimately outputs a predicted text summary that matches the language label corresponding to the training text. It will be understood that a predicted text summary corresponding to each training text in the training set can be obtained. When calculating the target loss, a target sub-loss is first generated based on the difference between the predicted text summary and the labeled text summary corresponding to the same training text. The target sub-losses corresponding to each training text are then aggregated to obtain the target loss.

[0208] In one embodiment, the computer device may input the training text and corresponding training image set, the labeled text summary, and the language label in the training set into the initial text summary generation model to obtain a predicted text summary that matches the language label corresponding to the training text.

[0209] The mask training data and the first training data are input into the initial text summary generation model to obtain mask prediction data, and a reconstruction loss is generated based on the difference between the mask label data and the mask prediction data, including:

[0210] The first training data, masked training data, and language labels corresponding to the training texts in the training set are input into the initial text summary generation model to obtain masked prediction data that matches the language labels corresponding to the training texts. A reconstruction sub-loss is generated based on the difference between the masked label data and the masked prediction data corresponding to the same training text, and a reconstruction loss is obtained based on the reconstruction sub-losses corresponding to each training text.

[0211] Specifically, similar to generating a predicted text summary, when generating masked prediction data, the computer device can input the first training data, masked training data, and language labels corresponding to the training text in the training set into the initial text summary generation model. After the model processes the data, the model ultimately outputs masked prediction data that matches the language label corresponding to the training text. It can be understood that masked prediction data corresponding to each training text in the training set can be obtained. When calculating the reconstruction loss, a reconstruction sub-loss is first generated based on the difference between the masked label data and the masked prediction data corresponding to the same training text. The reconstruction loss is then obtained by summing the reconstruction sub-losses corresponding to each training text.

[0212] Correspondingly, when the trained model is used in a specific application, the input data of the model can also include language labels, so that the model can produce a text summary that matches the language label.

[0213] In the above embodiment, the training set corresponding to the initial text summarization model may include relevant data corresponding to training documents in multiple languages. The target text summarization model trained based on this training set can be applied to generate text summaries for documents in various languages. During model training, the model's input data includes language labels. The language labels can instruct the model to learn relevant knowledge about text and images in the current language and output data that matches the language labels.

[0214] In one embodiment, the target loss can be calculated by referring to the following formula (17):

[0215]

[0216] in, represents the target loss, Denotes the tth predicted summary word in the predicted text summary. |y| denotes the total number of predicted text words. It can be understood that The smaller it is, the closer the predicted text summary is to the label text summary.

[0217] The reconstruction loss can be calculated by referring to the following formula (18) and formula (19):

[0218]

[0219]

[0220] in, Denotes the summary reconstruction loss. MLVM (Masked Language Vision Modeling) represents the summary reconstruction task, which uses complete image features to restore the occluded text summary, thereby improving the direct correlation between the text summary and the image, thereby improving the summary quality. K represents the number of occluded words in the text summary. Y k Represents the k masked words in the label text summary, that is, the summary mask label data. k ) represents Y k The characteristic distribution of can be considered as the prior distribution. represents the summary mask prediction data, express The characteristic distribution of can be considered as the posterior distribution.

[0221] Represents the image reconstruction loss. MVLM (Masked Vision Language Modeling) represents the image reconstruction task, which uses the complete text summary to restore the occluded image area, thereby improving the direct correlation between the text summary and the image, thereby improving the summary quality. m represents the number of occluded image areas in the training image set. r Represents the r image regions that are blocked in the training image set, i.e., the image mask label data. r ) means O r The characteristic distribution of can be considered as the prior distribution. represents image mask prediction data, express The characteristic distribution of can be considered as the posterior distribution. KL Refers to the KL divergence calculation.

[0222] The total loss can be calculated by referring to the following formula (20) and formula (21):

[0223]

[0224]

[0225] in, Represents the training function and loss function of the model on a single language, including the objective loss function of multimodal summarization and two auxiliary reconstruction loss functions. α and β are the balancing factors corresponding to the two reconstruction losses.

[0226] Represents the training function and loss function of the model on multiple languages. L represents the number of languages. Represents the training function and loss function of the model on the kth language

[0227] In one embodiment, α and β can be set to fixed values, for example, to a first preset value and a second preset value. The first preset value and the second preset value can be set according to actual needs, for example, both the first preset value and the second preset value can be set to 1. α and β can also be set to dynamic values ​​for dynamically adjusting model training. For example, α and β decrease as the number of model iterations increases.

[0228] In one embodiment, the initial text summary generation model includes a text encoder, an image encoder, and a decoder. Inputting the training text and the training image set into the initial text summary generation model to obtain a predicted text summary, and inputting the mask training data and the first training data into the initial text summary generation model to obtain mask prediction data, including:

[0229] The training text is input into the text encoder, the training image set is input into the image encoder, a fused encoding result is obtained based on the output results of the text encoder and the image encoder, the fused encoding result and the label text summary are input into the decoder to obtain a predicted text summary; the mask training data and the first training data are input into the image encoder to obtain mask prediction data.

[0230] Specifically, the initial text summarization model includes a text encoder, an image encoder, and a decoder. The training text is input into the text encoder for encoding, and the training image set is input into the image encoder for encoding. The outputs of the text encoder and image encoder are fused to produce a fused encoding result. The fused encoding result and the labeled text summary are input into the decoder for decoding to produce a predicted text summary. The summary generation task and the reconstruction task share the image encoder. The mask training data and the first training data are input into the image encoder for data processing to produce mask prediction data.

[0231] In one embodiment, in order to reduce the amount of model calculation, an external tool can be used to pre-extract features from the training image set to obtain initial image features, and the training text and initial image features are input into the initial text summary generation model, with the input data of the text encoder being the text and the input data of the image encoder being the initial image features. In one embodiment, an external tool can also be used to pre-extract features from the training text to obtain initial text features, and the initial text features and initial image features are input into the initial text summary generation model, with the input data of the text encoder being the initial text features and the input data of the image encoder being the initial image features.

[0232] In one embodiment, to reduce model training time, the text encoder and decoder in the initial text summarization model can be obtained from a pretrained text processing model. That is, the initial parameters of the text encoder and decoder are pretrained. For example, the text encoder and decoder can be obtained from a text processing model that generates text summaries based solely on text. The text encoder and decoder are then combined with an image encoder to produce the initial text summarization model.

[0233] In the above embodiment, the initial text summary generation model includes a text encoder, an image encoder and a decoder. The summary generation task and the reconstruction task share the same image encoder. The reconstruction task can help improve the processing capability of the summary generation task and improve the quality of the text summary generated by the model.

[0234] In one embodiment, Figure 6 As shown, a text summary generation method is provided, and the method is applied to a computer device as an example. The computer device can be a terminal or a server, and the method can be executed by the terminal or server alone, or through the interaction between the terminal and the server. Figure 6 , the text summary generation method includes the following steps:

[0235] Step S602: Acquire target text and a target image set corresponding to the target text.

[0236] Step S604: input the target text and the target image set into a target text summary generation model to obtain a target text summary corresponding to the target text.

[0237] The target text and the corresponding target image set constitute a target document, and the target document refers to a document whose corresponding text summary is to be determined.

[0238] Specifically, the computer device can train a target text summary generation model for generating a text summary, input the target text and the corresponding target image set into the target text summary generation model, and finally obtain the target text summary corresponding to the target text after data processing by the model.

[0239] It can be understood that the specific training process of the target text summary generation model can refer to the contents of the various embodiments of the aforementioned text summary generation method, and will not be repeated here.

[0240] In one embodiment, a target text and a target image set are input into a target text summary generation model. Within the model, the target text and the target image set are respectively encoded to obtain target text features and target image features. The target text features and the target image features are then fused to obtain a target fused feature. The target fused feature is then decoded to obtain a target text summary. The target fused feature can be decoded in an orderly manner to obtain a sequence of words, thereby obtaining a target text summary. When decoding a word in the target text summary, the decoded word can be referenced.

[0241] It is understood that the specific process of encoding the target text and target image set can refer to the specific process of encoding the training text and training image set in the aforementioned text summary generation method. The specific process of fusing the target text features and target image features can refer to the specific process of fusing the training text features and training image features in the aforementioned text summary generation method.

[0242] The above-mentioned text summarization method obtains a target text and a target image set corresponding to the target text, inputs the target text and the target image set into a target text summarization generation model, and obtains a target text summary corresponding to the target text. The target text summarization generation model is obtained by inputting the training text and the training image set into an initial text summarization generation model to obtain a predicted text summary, inputting masked training data and first training data into the initial text summarization generation model to obtain masked prediction data, generating a target loss based on the difference between the predicted text summary and the labeled text summary, generating a reconstruction loss based on the difference between the masked labeled data and the masked predicted data, and adjusting the model parameters of the initial text summarization generation model based on the target loss and the reconstruction loss. In this way, when training the model, the training loss includes target loss and reconstruction loss. The target loss corresponds to the summary generation task, which restores the original text summary through text information and image information. The target loss can guide the model to generate a text summary based on the text information and image information corresponding to the text. The reconstruction loss corresponds to the reconstruction task, which restores the masked data in the second training data through the unmasked data in the first training data and the second training data. The reconstruction loss helps to enhance the association between the first training data and the second training data, and guides the model to learn information related to the text summary from the image information corresponding to the text, thereby helping to improve the quality of the text summary generated by the model. Adjusting the model parameters based on the target loss and reconstruction loss can enable the model to output a higher-quality text summary and effectively improve the model's prediction accuracy. Inputting the target text and target image set into the target text summary generation model and generating a text summary through the trained target text summary generation model can effectively improve the quality of the generated text summary.

[0243] In a specific embodiment, the text summary generation model training method and text summary generation method of the present application can be applied to an online news summary generation system. Existing text summary technology cannot effectively utilize images in documents and cannot generate high-quality text summaries. For example, Figure 7 The underlined text in the main text of the news appears in the image, but it hardly contributes to the summary. Existing text summarization technology cannot filter out this useless information. Then, the text summary generation model training method of this application can effectively model the relationship between image information and summary information, thereby generating image features that can express the meaning of the summary, thereby producing more appropriate summary information.

[0244] The text summary generation model training method and text summary generation method of this application include the following steps:

[0245] 1. Model training

[0246] 1-1. Target loss

[0247] refer to Figure 8 The text summarization model consists of a text encoder, an image encoder (also called a visual encoder), and a decoder. The architecture of the text encoder and decoder can refer to the architecture of the pre-trained Transformer network or be customized.

[0248] The training text (X) is input into the text summary generation model to be trained (i.e., the initial text summary generation model). The training text obtains the initial text features through the Embedding layer. The initial text features are input into the text encoder for Multi-Head Attention operation to obtain the training text features. The training image set corresponding to the training text is input into the text summary generation model to be trained. The training image set is passed through the Embedding layer to obtain the initial image features (O). The initial image features are input into the image encoder for Multi-Head Attention operation to obtain the training text features. In addition, the initial image features (O) can be pre-extracted through external tools (for example, Faster R-CNN model), and the initial image features are input into the text summary generation model to be trained, and the initial image features are input into the image encoder for Multi-Head Attention operation to obtain the training text features. Then check and Perform feature fusion (also known as Text-Vision Fusion operation) to extract effective image information and obtain multimodal feature Z T+V . Then Z T+V The label text summary (Y) is input into the decoder, and the output of the decoder is normalized to generate a predicted text summary. For example, the output of the decoder is decoded by the Softmax function to generate a predicted text summary Y'. The target loss is generated based on the difference between the label text summary and the predicted text summary.

[0249] 1-2. Summary reconstruction loss

[0250] In order to improve the training quality of the model, a summary reconstruction task (also known as MLVM) is introduced during model training. MLVM: Use the complete image features to restore the occluded words in the summary, thereby improving the direct association between the words and the image, thereby improving the summary quality. The initial image features (O) and the unoccluded part of the label text summary (Y mask , i.e. summary mask training data) is input into the image encoder for data processing to obtain summary mask prediction data Based on the occluded part (Y k, i.e., the difference between the summary mask label data) and the summary mask prediction data generates the summary reconstruction loss

[0251] 1-3. Image reconstruction loss

[0252] In order to improve the training quality of the model, an image reconstruction task (also called MVLM) is introduced during model training. MVLM: The summary signal is directly applied to the image features that have been erased to enhance the correlation between the visual features and the summary, thereby improving the summary quality. mask ) and label text summary (Y) are input into the image encoder for data processing to obtain image mask prediction data Based on the occluded part of the initial image features ( r , i.e., the difference between the image mask label data) and the image mask prediction data generates the image reconstruction loss

[0253] The total loss is obtained based on the target loss, summary reconstruction loss and image reconstruction loss. Backpropagation is performed based on the total loss to update the model parameters until the convergence conditions are met, and the trained text summary generation model (i.e., the target text summary generation model) is obtained.

[0254] If the model's training set includes training data in multiple languages, you can first calculate the training loss for each language and then combine the training losses for each language to get the total loss. The training loss for a single language includes the target loss, summary reconstruction loss, and image reconstruction loss.

[0255] 2. Model Application

[0256] The text and images of the news to be summarized are fed into the target text summarization model. The model then processes the data and outputs a news summary corresponding to the news. The news summary can be displayed to allow users to quickly understand the main content of the news.

[0257] This embodiment proposes a multilingual and multimodal summary generation method that fully models both image and text information during model training. The method learns the most important information for the summary in the current language from the input text and multiple images, thereby improving the accuracy of the summary. During model training, the method learns effective information related to different languages ​​from image information, simultaneously enhancing the accuracy of summaries in multiple languages ​​and making them more practical.

[0258] It can be understood that the text summary generation model training method and text summary generation method of this application can be applied to online news summary generation systems, and can also be applied to other summary generation systems, such as paper summary generation systems, push summary generation systems, etc.

[0259] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0260] Based on the same inventive concept, the embodiments of the present application also provide a text summary generation model training device for implementing the above-mentioned text summary generation model training method, and a text summary generation device for implementing the above-mentioned text summary generation method. The implementation solution provided by the device is similar to the implementation solution described in the above-mentioned method, so the specific limitations in one or more text summary generation model training device embodiments provided below can refer to the above-mentioned limitations on the text summary generation model training method, and the specific limitations in one or more text summary generation device embodiments provided below can refer to the above-mentioned limitations on the text summary generation method, and will not be repeated here.

[0261] In one embodiment, Figure 9 As shown, a text summary generation model training device is provided, including: a data acquisition module 902, a first data processing module 904, a second data processing module 906 and a model parameter adjustment module 908, wherein:

[0262] Data acquisition module 902 is used to obtain training text and the training image set and label text summary corresponding to the training text, and obtain first training data, mask label data and masked training data corresponding to the training text; the mask label data is the masked data after the second training data is masked, and the masked training data is the other data in the second training data except the mask label data. The first training data and the second training data are determined from the training image set and the label text summary.

[0263] The first data processing module 904 is configured to input the training text and the training image set into the initial text summary generation model to obtain a predicted text summary, and generate a target loss based on the difference between the predicted text summary and the label text summary.

[0264] The second data processing module 906 is configured to input the mask training data and the first training data into the initial text summarization model to obtain mask prediction data, and generate a reconstruction loss based on the difference between the mask label data and the mask prediction data.

[0265] The model parameter adjustment module 908 is used to adjust the model parameters of the initial text summary generation model based on the target loss and the reconstruction loss until the convergence condition is met, thereby obtaining a target text summary generation model; the target text summary generation model is used to generate a text summary of the text.

[0266] The above-mentioned text summary generation model training device, when training the model, the training loss includes target loss and reconstruction loss. The target loss corresponds to the summary generation task, and the original text summary is restored through the text information and image information. The target loss can guide the model to generate a text summary based on the text information and image information corresponding to the text. The reconstruction loss corresponds to the reconstruction task, and the masked data in the second training data is restored through the unmasked data in the first training data and the second training data. The reconstruction loss helps to enhance the association between the first training data and the second training data, and guides the model to learn information related to the text summary from the image information corresponding to the text, thereby helping to improve the quality of the text summary generated by the model. Adjusting the model parameters based on the target loss and reconstruction loss can enable the model to output a higher quality text summary and effectively improve the prediction accuracy of the model. Generating a text summary through the target text summary generation model obtained by the final training can improve the quality of the generated text summary.

[0267] In one embodiment, the reconstruction loss includes a summary reconstruction loss. The data acquisition module is further configured to acquire a training image set as first training data and a label text summary as second training data; perform masking on the label text summary to obtain summary mask label data and summary mask training data; obtain the summary mask label data as mask label data; and obtain the summary mask training data as mask training data. The second data processing module is further configured to input the training image set and the summary mask training data into the initial text summarization model to obtain summary mask prediction data; and generate a summary reconstruction loss based on the difference between the summary mask label data and the summary mask prediction data.

[0268] In one embodiment, the reconstruction loss includes an image reconstruction loss. The data acquisition module is further configured to acquire a labeled text summary as first training data and a training image set as second training data; perform mask processing on the training image set to obtain image mask label data and image mask training data, obtain the image mask label data as mask label data, and obtain the image mask training data as mask training data. The second data processing module is further configured to input the labeled text summary and the image mask training data into an initial text summary generation model to obtain image mask prediction data; and generate an image reconstruction loss based on the difference between the image mask label data and the image mask prediction data.

[0269] In one embodiment, the first data processing module is further configured to:

[0270] The training text, training image set and label text summary are input into the initial text summary generation model; the training text and training image set are encoded respectively to obtain training text features and training image features; the training text features and training image features are fused to obtain training fusion features; based on the label text summary, the training fusion features are decoded to obtain a predicted text summary.

[0271] In one embodiment, the first data processing module is further configured to:

[0272] Perform feature extraction on the training text to obtain initial text features, perform feature encoding on the initial text features to obtain training text features; perform object recognition on the training images in the training image set to obtain object image features corresponding to at least one target object; obtain initial image features corresponding to the training image set based on the object image features corresponding to the training images in the training image set; perform feature encoding on the initial image features to obtain training image features.

[0273] In one embodiment, the first data processing module is further configured to:

[0274] Feature extraction is performed on the image identification corresponding to each training image, the object identification corresponding to each target object, and the recognition position corresponding to each target object, respectively, to obtain the image identification features corresponding to each training image, the object identification features corresponding to each target object, and the recognition position features corresponding to each recognition position; based on the object identification features, recognition position features, object image features corresponding to the same target object, and the image identification features corresponding to the training image to which it belongs, the initial image sub-features corresponding to the target object are obtained; based on the initial image sub-features corresponding to each target object in each training image, the initial image features are obtained.

[0275] In one embodiment, the first data processing module is further configured to:

[0276] Perform attention processing on the current initial feature to obtain the current attention feature; fuse the current initial feature and the current attention feature to obtain the initial fusion feature; perform feature transformation on the initial fusion feature to obtain the current transformation feature; fuse the current transformation feature and the initial fusion feature to obtain the target fusion feature; obtain the current training feature based on the target fusion feature.

[0277] In one embodiment, the first data processing module is further configured to:

[0278] Inter-modal attention processing is performed on the training text features and the training image features to obtain fused attention features; image filtering features are obtained based on the training text features and the fused attention features; based on the image filtering features, feature filtering is performed on the fused attention features to obtain fused enhancement features; the fused enhancement features and the training text features are spliced ​​to obtain training fused features.

[0279] In one embodiment, the first data processing module is further configured to:

[0280] The training text feature and the fused attention feature are spliced ​​to obtain a first spliced ​​feature; based on the first model parameter, the first spliced ​​feature is subjected to feature transformation to obtain a first transformed feature; and the first transformed feature is subjected to feature mapping to obtain an image filtering feature.

[0281] In one embodiment, the tag text summary includes a plurality of ordered training summary words. The first data processing module is further configured to:

[0282] Determine the target position from each word position corresponding to the label text summary; obtain the training summary word before the target position from the label text summary as a reference summary word, encode the reference summary word to obtain a reference summary feature; obtain a predicted summary feature based on the training fusion feature and the reference summary feature; decode the predicted summary feature to obtain a predicted summary word corresponding to the target position; obtain the next word position as the target position, return to the step of obtaining the training summary word before the target position from the label text summary as a reference summary word, and execute until the end condition is met to obtain multiple predicted summary words; obtain a predicted text summary based on each predicted summary word.

[0283] In one embodiment, the first data processing module is further configured to:

[0284] Attention processing is performed on the reference summary features to obtain initial attention features, and initial summary features are obtained based on the initial attention features and the reference summary features; cross-attention processing is performed on the initial summary features and the training fusion features to obtain cross-attention features, and intermediate summary features are obtained based on the cross-attention features and the initial summary features; feature transformation is performed on the intermediate summary features to obtain target summary features, and predicted summary features are obtained based on the intermediate summary features and the target summary features.

[0285] In one embodiment, the training set corresponding to the initial text summary generation model includes training data corresponding to multiple training texts, the training data includes training texts and corresponding training image sets, labeled text summaries, first training data, masked label data, masked training data and language labels, and the training set includes at least one language label.

[0286] The first data processing module is also used to input the training text and the corresponding training image set and language label in the training set into the initial text summary generation model to obtain a predicted text summary that matches the language label corresponding to the training text; generate a target sub-loss based on the difference between the predicted text summary and the label text summary corresponding to the same training text, and obtain a target loss based on the target sub-loss corresponding to each training text.

[0287] The second data processing module is also used to input the first training data, masked training data and language labels corresponding to the training text in the training set into the initial text summary generation model to obtain masked prediction data that matches the language label corresponding to the training text; generate a reconstruction sub-loss based on the difference between the masked label data and the masked prediction data corresponding to the same training text, and obtain a reconstruction loss based on the reconstruction sub-loss corresponding to each training text.

[0288] In one embodiment, the initial text summary generation model includes a text encoder, an image encoder, and a decoder. The first data processing module is further configured to input training text into the text encoder, input a set of training images into the image encoder, generate a fused encoding result based on the outputs of the text encoder and the image encoder, and input the fused encoding result and the label text summary into the decoder to generate a predicted text summary. The second data processing module is further configured to input mask training data and the first training data into the image encoder to generate mask prediction data.

[0289] In one embodiment, Figure 10 As shown, a text summary generation device is provided, including: a target text acquisition module 1002 and a text summary generation module 1004, wherein:

[0290] The target text acquisition module 1002 is used to acquire the target text and the target image set corresponding to the target text.

[0291] The text summary generation module 1004 is configured to input the target text and the target image set into the target text summary generation model to obtain a target text summary corresponding to the target text.

[0292] The training process of the target text summarization model includes the following steps:

[0293] Obtain training text and a training image set and a label text summary corresponding to the training text, and obtain first training data, mask label data, and mask training data corresponding to the training text; the mask label data is the masked data after the second training data is masked, and the masked training data is the other data in the second training data except the mask label data. The first training data and the second training data are determined from the training image set and the label text summary; input the training text and the training image set into the initial text summary generation model to obtain a predicted text summary, and generate a target loss based on the difference between the predicted text summary and the label text summary; input the mask training data and the first training data into the initial text summary generation model to obtain masked prediction data, and generate a reconstruction loss based on the difference between the masked label data and the masked prediction data; adjust the model parameters of the initial text summary generation model based on the target loss and the reconstruction loss until the convergence condition is met, and obtain the target text summary generation model.

[0294] In the above-mentioned text summary generation device, when training the model, the training loss includes target loss and reconstruction loss. The target loss corresponds to the summary generation task, and the original text summary is restored through the text information and image information. The target loss can guide the model to generate a text summary based on the text information and image information corresponding to the text. The reconstruction loss corresponds to the reconstruction task, and the masked data in the second training data is restored through the unmasked data in the first training data and the second training data. The reconstruction loss helps to enhance the association between the first training data and the second training data, and guides the model to learn information related to the text summary from the image information corresponding to the text, thereby helping to improve the quality of the text summary generated by the model. Adjusting the model parameters based on the target loss and reconstruction loss can enable the model to output a higher-quality text summary and effectively improve the prediction accuracy of the model. Inputting the target text and target image set into the target text summary generation model and generating a text summary through the trained target text summary generation model can effectively improve the quality of the generated text summary.

[0295] Each module in the aforementioned text summarization model training device and text summarization device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0296] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as training sets and target text summary generation models. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a text summary generation model training method is implemented.

[0297] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 12 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a text summary generation method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0298] Those skilled in the art will understand that Figure 11 、 12The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0299] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0300] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0301] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0302] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0303] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0304] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0305] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A text summarization generation model training method, characterized in that: The method comprises: Obtaining a training text and a training image set corresponding to the training text, and a label text summary; obtaining first training data, masked label data, and masked training data corresponding to the training text; the masked label data being masked data of the second training data after masking, and the masked training data being data other than the masked label data in the second training data; the first training data and the second training data being determined from the training image set and the label text summary; Inputting the training text and the training image set into an initial text summary generation model to obtain a predicted text summary, and generating a target loss based on the difference between the predicted text summary and the label text summary; Inputting the mask training data and the first training data into an initial text summarization model to obtain mask prediction data, and generating a reconstruction loss based on a difference between the mask label data and the mask prediction data; Based on the target loss and the reconstruction loss, the model parameters of the initial text summary generation model are adjusted until a convergence condition is met, thereby obtaining a target text summary generation model; the target text summary generation model is used to generate a text summary of the text.

2. The method according to claim 1, characterized in that The reconstruction loss includes a summary reconstruction loss, and the obtaining of first training data, mask label data, and mask training data corresponding to the training text includes: Acquire the training image set as first training data, and acquire the label text summary as second training data; Performing masking processing on the label text summary to obtain summary mask label data and summary mask training data, obtaining the summary mask label data as mask label data, and obtaining the summary mask training data as mask training data; Inputting the mask training data and the first training data into an initial text summarization model to obtain mask prediction data, and generating a reconstruction loss based on a difference between the mask label data and the mask prediction data, comprises: Inputting the training image set and the summary mask training data into an initial text summary generation model to obtain summary mask prediction data; A summary reconstruction loss is generated based on a difference between the summary mask label data and the summary mask prediction data.

3. The method according to claim 1 or 2, characterized in that The reconstruction loss includes image reconstruction loss, and the obtaining of first training data, mask label data, and mask training data corresponding to the training text includes: Acquire the label text summary as first training data, and acquire the training image set as second training data; Performing mask processing on the training image set to obtain image mask label data and image mask training data, obtaining the image mask label data as mask label data, and obtaining the image mask training data as mask training data; Inputting the mask training data and the first training data into an initial text summarization model to obtain mask prediction data, and generating a reconstruction loss based on a difference between the mask label data and the mask prediction data, comprises: Inputting the label text summary and the image mask training data into an initial text summary generation model to obtain image mask prediction data; An image reconstruction loss is generated based on a difference between the image mask label data and the image mask prediction data.

4. The method according to claim 1, wherein The step of inputting the training text and the training image set into an initial text summary generation model to obtain a predicted text summary comprises: Inputting the training text, the training image set and the label text summary into an initial text summary generation model; Encoding the training text and the training image set respectively to obtain training text features and training image features; fusing the training text features and the training image features to obtain training fusion features; Based on the label text summary, the training fusion feature is decoded to obtain a predicted text summary.

5. The method according to claim 4, characterized in that The encoding process is performed on the training text and the training image set respectively to obtain training text features and training image features, including: Performing feature extraction on the training text to obtain initial text features, and performing feature encoding on the initial text features to obtain the training text features; Performing object recognition on the training images in the training image set to obtain object image features corresponding to at least one target object; Based on the image features of each object corresponding to each training image in the training image set, obtaining the initial image features corresponding to the training image set; Feature encoding is performed on the initial image features to obtain the training image features.

6. The method according to claim 5, characterized in that The obtaining of the initial image features corresponding to the training image set based on the image features of each object corresponding to each training image in the training image set includes: Perform feature extraction on the image identifier corresponding to each training image, the object identifier corresponding to each target object, and the recognition position corresponding to each target object, respectively, to obtain the image identifier feature corresponding to each training image, the object identifier feature corresponding to each target object, and the recognition position feature corresponding to each recognition position; Based on the object identification features, recognition position features, object image features, and image identification features corresponding to the training image of the same target object, an initial image sub-feature corresponding to the target object is obtained; The initial image features are obtained based on the initial image sub-features corresponding to the respective target objects in the respective training images.

7. The method according to claim 5, characterized in that The current initial feature is the initial text feature or the initial image feature. Feature encoding is performed on the current initial feature to obtain the corresponding current training feature, including: Performing attention processing on the current initial feature to obtain a current attention feature; Fusing the current initial feature and the current attention feature to obtain an initial fused feature; Performing feature transformation on the initial fusion feature to obtain a current transformed feature; Fusing the current transformed feature and the initial fused feature to obtain a target fused feature; The current training feature is obtained based on the target fusion feature.

8. The method according to claim 4, characterized in that The fusing of the training text features and the training image features to obtain training fusion features includes: Performing inter-modal attention processing on the training text features and the training image features to obtain a fused attention feature; Obtaining image filtering features based on the training text features and the fused attention features; Based on the image filtering feature, the fused attention feature is subjected to feature filtering to obtain a fused enhancement feature; The fusion enhancement feature and the training text feature are spliced ​​to obtain the training fusion feature.

9. The method according to claim 8, characterized in that The obtaining of image filtering features based on the training text features and the fused attention features includes: Splicing the training text feature and the fused attention feature to obtain a first spliced ​​feature; Performing feature transformation on the first splicing feature based on the first model parameter to obtain a first transformed feature; Perform feature mapping on the first transformation feature to obtain the image filtering feature.

10. The method according to claim 4, characterized in that The label text summary includes a plurality of training summary words arranged in an orderly manner; The decoding process of the training fusion feature based on the label text summary to obtain a predicted text summary includes: Determining a target position from each word position corresponding to the tag text summary; From the label text summary, obtaining the training summary words before the target position as reference summary words, encoding the reference summary words to obtain reference summary features; Obtaining predicted summary features based on the training fusion features and the reference summary features; Decode the predicted summary features to obtain the predicted summary words corresponding to the target position; Obtaining the next word position as the target position, returning to the step of obtaining the training summary word before the target position as the reference summary word from the label text summary, and executing until the end condition is met, thereby obtaining multiple predicted summary words; The predicted text summary is obtained based on each predicted summary word.

11. The method according to claim 10, characterized in that The step of obtaining a predicted summary feature based on the training fusion feature and the reference summary feature includes: Performing attention processing on the reference summary feature to obtain an initial attention feature, and obtaining an initial summary feature based on the initial attention feature and the reference summary feature; Perform cross-attention processing on the initial summary features and the training fusion features to obtain cross-attention features, and obtain intermediate summary features based on the cross-attention features and the initial summary features; The intermediate summary features are transformed to obtain the target summary features, and the predicted summary features are obtained based on the intermediate summary features and the target summary features.

12. The method according to claim 1, characterized in that The training set corresponding to the initial text summary generation model includes training data corresponding to a plurality of training texts, the training data including training texts and corresponding training image sets, label text summaries, first training data, mask label data, mask training data, and language labels, and the training set includes at least one language label; The step of inputting the training text and the training image set into an initial text summary generation model to obtain a predicted text summary, and generating a target loss based on a difference between the predicted text summary and the label text summary, comprises: Inputting the training texts and corresponding training image sets and language labels in the training set into the initial text summary generation model to obtain a predicted text summary that matches the language labels corresponding to the training texts; Generate a target sub-loss based on the difference between the predicted text summary and the label text summary corresponding to the same training text, and obtain the target loss based on the target sub-loss corresponding to each training text; Inputting the mask training data and the first training data into an initial text summarization model to obtain mask prediction data, and generating a reconstruction loss based on a difference between the mask label data and the mask prediction data, comprises: Inputting the first training data, masked training data, and language labels corresponding to the training text in the training set into the initial text summarization model to obtain masked prediction data that matches the language labels corresponding to the training text; The reconstruction sub-loss is generated based on the difference between the mask label data and the mask prediction data corresponding to the same training text, and the reconstruction loss is obtained based on the reconstruction sub-loss corresponding to each training text.

13. The method according to claim 1, wherein The initial text summary generation model includes a text encoder, an image encoder and a decoder; The step of inputting the training text and the training image set into an initial text summary generation model to obtain a predicted text summary, and inputting the mask training data and the first training data into the initial text summary generation model to obtain mask prediction data comprises: Inputting the training text into the text encoder, inputting the training image set into the image encoder, obtaining a fused encoding result based on output results of the text encoder and the image encoder, and inputting the fused encoding result and the label text summary into a decoder to obtain the predicted text summary; The mask training data and the first training data are input into the image encoder to obtain the mask prediction data.

14. A text summary generation method, characterized in that: The method comprises: Obtain the target text and the target image set corresponding to the target text; Inputting the target text and the target image set into a target text summary generation model to obtain a target text summary corresponding to the target text; The training process of the target text summary generation model includes the following steps: Obtaining a training text and a training image set corresponding to the training text, and a label text summary; obtaining first training data, masked label data, and masked training data corresponding to the training text; the masked label data being masked data of the second training data after masking, and the masked training data being data other than the masked label data in the second training data; the first training data and the second training data being determined from the training image set and the label text summary; Inputting the training text and the training image set into an initial text summary generation model to obtain a predicted text summary, and generating a target loss based on the difference between the predicted text summary and the label text summary; Inputting the mask training data and the first training data into an initial text summarization model to obtain mask prediction data, and generating a reconstruction loss based on a difference between the mask label data and the mask prediction data; The model parameters of the initial text summary generation model are adjusted based on the target loss and the reconstruction loss until a convergence condition is met, thereby obtaining a target text summary generation model.

15. A text summary generation model training device, characterized in that: The device comprises: a data acquisition module configured to acquire a training text, a training image set corresponding to the training text, and a label text summary, and to acquire first training data, masked label data, and masked training data corresponding to the training text; the masked label data being masked data of the second training data after masking, and the masked training data being data other than the masked label data in the second training data; the first training data and the second training data being determined from the training image set and the label text summary; a first data processing module, configured to input the training text and the training image set into an initial text summary generation model to obtain a predicted text summary, and generate a target loss based on a difference between the predicted text summary and the label text summary; a second data processing module, configured to input the mask training data and the first training data into an initial text summarization model to obtain mask prediction data, and generate a reconstruction loss based on a difference between the mask label data and the mask prediction data; A model parameter adjustment module is used to adjust the model parameters of the initial text summary generation model based on the target loss and the reconstruction loss until the convergence condition is met, thereby obtaining a target text summary generation model; the target text summary generation model is used to generate a text summary of the text.

16. A text summary generation device, characterized in that: The device comprises: A target text acquisition module is used to acquire the target text and the target image set corresponding to the target text; A text summary generation module, configured to input the target text and the target image set into a target text summary generation model to obtain a target text summary corresponding to the target text; The training process of the target text summary generation model includes the following steps: Obtain training text and a training image set and a label text summary corresponding to the training text, and obtain first training data, mask label data, and mask training data corresponding to the training text; the mask label data is the masked data of the second training data after masking, and the masked training data is other data in the second training data except the mask label data, and the first training data and the second training data are determined from the training image set and the label text summary; input the training text and the training image set into an initial text summary generation model to obtain a predicted text summary, and generate a target loss based on the difference between the predicted text summary and the label text summary; input the masked training data and the first training data into the initial text summary generation model to obtain masked prediction data, and generate a reconstruction loss based on the difference between the masked label data and the masked prediction data; adjust the model parameters of the initial text summary generation model based on the target loss and the reconstruction loss until the convergence condition is met, and obtain a target text summary generation model.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 14 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Images