Cross-language abstract generation method and device based on multi-language model

Abstract

The application provides a cross-language abstract generation method based on a multi-language model, and relates to the technical field of cross-language abstract generation, and the method comprises the following steps: acquiring multi-language text data; pre-training a general language model based on the multi-language text data to obtain a multi-language model; acquiring multi-language abstract data, learning the multi-language model based on the multi-language abstract data to obtain a multi-language abstract generation model; acquiring to-be-processed text data, inputting the to-be-processed text data into the multi-language abstract generation model to generate abstracts corresponding to the to-be-processed text data in multiple different languages. The application realizes cross-language abstract generation, can generate abstracts in different languages, can summarize the gist of the text at the same time, and improves the precision of the abstracts.

Description

Technical Field

[0001] This application relates to the field of cross-language summarization technology, and in particular to a cross-language summarization method and apparatus based on a multilingual model. Background Technology

[0002] In recent years, large-scale language models pre-trained on internet text have significantly improved the state-of-the-art capabilities of various Natural Language Processing (NLP) tasks. For example, the BERT model, trained on English Wikipedia, achieved first place on various NLP leaderboards upon its release, and even achieved improvements of over 10% on some tasks. Since BERT's inception, numerous improvements have been published. Meta (Facebook)'s RoBERTa proposed improvements to BERT in terms of parameters and training objectives. In 2019, Google's T5 model first reached a scale of tens of billions of parameters, achieving state-of-the-art performance in multiple domains. In 2020, OpenAI's GPT-3 model increased the parameters of pre-trained language models to the hundreds of billions level and demonstrated remarkable few-shot learning capabilities, achieving performance comparable to BERT's massive training on large datasets with only 32 labeled data points on some tasks. Meanwhile, large language models pre-trained on internet text have each released their own multilingual versions. First, there's mBERT, a multilingual version of BERT, which uses the same model as BERT and is trained on Wikipedia in 104 languages. It performs well in downstream tasks such as sentence classification, entity tagging, and reading comprehension. Meta proposed XLM-R, a multilingual version of RoBERTa, which for the first time considered the imbalance between English and other language training corpora and used sampling. Google proposed mT5, a multilingual version of T5, which, while sampling high-frequency language (English), proposed a series of improvements to the model's language transfer learning capabilities. However, neither Google's MT5 nor Meta's XLM-R multilingual model achieved cross-language summarization functionality.

[0003] Existing cross-language summarization techniques are very complex, requiring models to learn to generate summaries without cross-language translation before translating. Furthermore, the generated summaries are often repetitions of the original text, failing to extract and summarize the main idea in a simpler way, resulting in poor accuracy. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in the related art.

[0005] Therefore, the first objective of this application is to propose a cross-language summary generation method based on a multilingual model, which solves the technical problem that existing methods cannot complete cross-language summary tasks, can only generate summaries first and then translate them, and the generated summaries have poor accuracy. By performing autoregressive fill-in pre-training on a general language model, a multilingual model is obtained, and cross-language summary generation is realized based on the multilingual model. It can generate summaries in different languages ​​based on text data, and can also extract and summarize the main idea of ​​the text, thereby improving the accuracy of the summary.

[0006] The second objective of this application is to propose a cross-lingual summarization generation device based on a multilingual model.

[0007] The third objective of this application is to propose a computer device.

[0008] The fourth objective of this application is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, the first aspect of this application proposes a cross-language summarization method based on a multilingual model, comprising: acquiring multilingual text data; pre-training a general language model based on the multilingual text data to obtain a multilingual model; acquiring multilingual summarization data, learning the multilingual model based on the multilingual summarization data to obtain a multilingual summarization generation model; acquiring text data to be processed, inputting the text data to be processed into the multilingual summarization generation model, and generating summaries in multiple different languages ​​corresponding to the text data to be processed.

[0010] Optionally, in one embodiment of this application, after acquiring the multilingual text data, the process includes:

[0011] For each language of the multilingual text data, multiple text segments are randomly sampled. Each sampled segment in the text sequence is replaced with a mask marker to obtain a corrupted text sequence, where each segment corresponds to a series of consecutive characters.

[0012] The corrupted text sequence was used as the first sample data;

[0013] The segment whose mask marker is replaced is used as the second sample data.

[0014] Optionally, in one embodiment of this application, a multilingual model is obtained by pre-training a general language model based on multilingual text data, including:

[0015] Based on the first and second sample data, the general language model is pre-trained to obtain a multilingual model.

[0016] Optionally, in one embodiment of this application, the method further includes:

[0017] By changing the length and number of sampled text fragments, pre-training targets adapted to different tasks are generated to pre-train the general language model, resulting in a multilingual model adapted to different tasks.

[0018] Optionally, in one embodiment of this application, pre-training targets adapted to different tasks are generated by changing the length and number of sampled text fragments, including:

[0019] By sampling the length of each segment from a Poisson distribution with a mean of 3, the total segment length is a first preset proportion of the original sequence length, generating a pre-training target adapted to classification tasks for natural language understanding;

[0020] By making each segment a complete sentence in the text sequence, and the total segment length being a second preset proportion of the original text sequence length, a pre-training objective for the adaptive text generation task is generated.

[0021] By sampling only one segment from a text sequence, and making the segment length a third preset proportion of the original text sequence length, a pre-training target adapted to the task of long text generation is generated.

[0022] Optionally, in one embodiment of this application, the multilingual summarization data includes original text data and summary data in different languages. Obtaining the multilingual summarization data and learning a multilingual model based on the multilingual summarization data to obtain a multilingual summarization generation model includes:

[0023] By using original text and summary data in different languages ​​as training data, and using prompt-based fine-tuning to train the multilingual model, a multilingual summary generation model is obtained.

[0024] To achieve the above objectives, a second aspect of this application proposes a cross-lingual summarization generation apparatus based on a multilingual model, comprising:

[0025] The acquisition module is used to acquire multilingual text data;

[0026] The pre-training module is used to pre-train a general language model based on multilingual text data to obtain a multilingual model;

[0027] The training module is used to acquire multilingual summarization data, learn the multilingual model based on the multilingual summarization data, and obtain a multilingual summarization generation model.

[0028] The generation module is used to acquire the text data to be processed, input the text data to be processed into the multilingual summarization generation model, and generate summaries in multiple different languages ​​corresponding to the text data to be processed.

[0029] Optionally, in one embodiment of this application, after acquiring the multilingual text data, the process includes:

[0030] For each language of the multilingual text data, multiple text segments are randomly sampled. Each sampled segment in the text sequence is replaced with a mask mark to obtain a corrupted text sequence, where each segment corresponds to a series of consecutive characters.

[0031] The corrupted text sequence was used as the first sample data;

[0032] The segment whose mask marker is replaced is used as the second sample data.

[0033] To achieve the above objectives, a third aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the cross-language summary generation method based on a multilingual model described in the above embodiments.

[0034] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a processor, enables the execution of a cross-language summary generation method based on a multilingual model.

[0035] The present application's method, apparatus, computer device, and non-transitory computer-readable storage medium for cross-language summarization based on a multilingual model solves the technical problem that existing methods cannot complete cross-language summarization tasks, can only generate summaries first and then translate them, and the generated summaries have poor accuracy. By performing autoregressive fill-in pre-training on a general language model, a multilingual model is obtained, and cross-language summarization is achieved based on the multilingual model. It can generate summaries in different languages ​​based on text data, and can also extract and summarize the main idea of ​​the text, thus improving the accuracy of the summarization.

[0036] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0037] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0038] Figure 1 This is a flowchart illustrating a cross-language summary generation method based on a multilingual model provided in Embodiment 1 of this application;

[0039] Figure 2 This is a flowchart illustrating the generation of a multilingual summary generation model using a multilingual model-based cross-language summary generation method according to an embodiment of this application.

[0040] Figure 3Here is a flowchart illustrating the pre-training process of the cross-language summarization generation method based on a multilingual model, as described in this application:

[0041] Figure 4 This is an example diagram illustrating the use of a generative model for prompting learning in the cross-language summary generation method based on a multilingual model according to embodiments of this application;

[0042] Figure 5 This is a schematic diagram of the structure of a cross-language summary generation device based on a multilingual model provided in Embodiment 2 of this application. Detailed Implementation

[0043] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0044] The following is an explanation of the terms used in this application:

[0045] Natural Language Processing: The field of communication between humans and computers using natural language, utilizing computer and artificial intelligence technologies.

[0046] Language model: A probabilistic model of natural language. By having the model read large-scale data, the model can predict words or phrases that were masked during training based on the context.

[0047] Pre-training: Enables the language model to learn from a large amount of data, allowing it to discover general rules in the language, learn language structure and grammar, and understand the semantics of each word.

[0048] Fine-tuning: After pre-training, the model is further trained to learn task-related language knowledge on a specific task. Taking summary generation as an example: by having the model specifically learn articles and their corresponding summaries, the model can determine which information is a summary and which is detail based on the semantic knowledge acquired during pre-training.

[0049] Transfer learning: Transfer learning is a research field within machine learning that focuses on storing solutions to existing problems and applying them to other different but related problems. Domain transfer is a branch of transfer learning; it involves a model learning in one domain and then transferring the learned general knowledge to another. For example, applying methods for summarizing news articles to summarizing academic papers.

[0050] The following description, with reference to the accompanying drawings, describes a cross-language summary generation method and apparatus based on a multilingual model according to embodiments of this application.

[0051] Figure 1This is a flowchart of a cross-language summary generation method based on a multilingual model provided in Embodiment 1 of this application.

[0052] like Figure 1 As shown, this cross-language summarization generation method based on a multilingual model includes the following steps:

[0053] Step 101: Obtain multilingual text data;

[0054] Step 102: Pre-train the general language model based on multilingual text data to obtain a multilingual model;

[0055] Step 103: Obtain multilingual summary data, and learn the multilingual model based on the multilingual summary data to obtain a multilingual summary generation model;

[0056] Step 104: Obtain the text data to be processed, input the text data to be processed into the multilingual summarization generation model, and generate summaries in multiple different languages ​​corresponding to the text data to be processed.

[0057] The cross-language summarization method based on a multilingual model in this application involves: acquiring multilingual text data; pre-training a general language model based on the multilingual text data to obtain a multilingual model; acquiring multilingual summarization data and learning from the multilingual model based on the multilingual summarization data to obtain a multilingual summarization generation model; acquiring text data to be processed and inputting the text data to be processed into the multilingual summarization generation model to generate summaries in multiple different languages ​​corresponding to the text data to be processed. This solves the technical problem that existing methods cannot complete cross-language summarization tasks, requiring the generation of summaries before translation and resulting in poor summarization accuracy. By performing autoregressive pre-training of the general language model to obtain a multilingual model, and then using this multilingual model to achieve cross-language summarization, the method can generate summaries in different languages ​​from text data, while also extracting and summarizing the main idea of ​​the text, thus improving the accuracy of the summarization.

[0058] This application proposes a Multilingual General Language Model (MGLM). The pre-trained MGLM can perform a series of downstream tasks, such as sentence classification, entity annotation, and reading comprehension, on languages ​​not encountered during fine-tuning. The MGLM demonstrates strong transfer learning capabilities. For example, directly training the MGLM to classify sentences in 101 languages ​​would be too costly. By training the MGLM to classify sentences in English, it can be enabled to perform sentence classification tasks in other languages. For other downstream tasks, the MGLM can learn task knowledge and the semantic knowledge required for the task simply by training on English. Furthermore, it can transfer semantic knowledge from English to other languages ​​obtained during pre-training and complete downstream tasks.

[0059] Furthermore, in this embodiment of the application, after acquiring the multilingual text data, the process includes:

[0060] For each language of the multilingual text data, multiple text segments are randomly sampled. Each sampled segment in the text sequence is replaced with a mask mark to obtain a corrupted text sequence, where each segment corresponds to a series of consecutive characters.

[0061] The corrupted text sequence was used as the first sample data;

[0062] The segment whose mask marker is replaced is used as the second sample data.

[0063] For example, given a text sequence, multiple text segments are randomly sampled, where each segment corresponds to a series of consecutive characters. Each segment is replaced by a mask marker, eventually forming a corrupted text sequence. The corrupted text sequence is used as the first sample data, and the segments replaced by the mask markers are used as the second sample data.

[0064] Furthermore, in this embodiment of the application, a multilingual model is obtained by pre-training a general language model based on multilingual text data, including:

[0065] Based on the first and second sample data, the general language model is pre-trained to obtain a multilingual model.

[0066] The pre-training method of the General Language Model (GLM) in this application combines BERT and GPT-3: it covers short sentences and lets the GLM generate these short sentences. In this way, GLM combines autoencoders and autoregressors.

[0067] GLM proposes a general pre-training framework based on autoregressive cloze tests. By simultaneously learning bidirectional and unidirectional attention mechanisms within a unified framework, the general language model learns both contextual representations and autoregressive generation during the pre-training phase. In the fine-tuning phase for downstream tasks, different types of downstream tasks can be unified through cloze tests, thus achieving a pre-trained model applicable to all natural language processing tasks.

[0068] GLM is pre-trained by optimizing an autoregressive fill-in-the-blank objective. The GLM model first predicts missing segments from a corrupted text sequence in an autoregressive manner, and then continuously updates the model parameters to reduce the difference between the predicted and actual results, thus pre-training the model parameters. When predicting missing segments, the GLM model can access both the corrupted text and previously predicted segments. To fully capture the interdependencies between different segments, the order of the segments is randomly arranged.

[0069] In this embodiment, first sample data and second sample data are input into a GLM (Generic Model). The first sample data is a corrupted text sequence, and the second sample data is a fragment replaced by masked markers. The GLM model learns to generate the second sample data using the first sample data and continuously updates its parameters to make the generated result more closely resemble the second sample data, ultimately resulting in a pre-trained GLM model. During encoding, characters in the first sample data can be associated with all characters in the first sample data, but cannot be associated with any characters in the second sample data. Characters in the second sample data can be associated with characters in the first sample data and characters preceding them in the second sample data, but cannot be associated with any subsequent characters in the second sample data.

[0070] During the pre-training phase, the GLM model uses a bidirectional encoder to perform bidirectional associative encoding on the text in the first sample data; and a unidirectional encoder to perform unidirectional associative encoding on the text in the second sample data on both the first sample data and the preceding parts of the text in the second sample data. The GLM model updates the weight parameters in the bidirectional and unidirectional encoders by learning the task of generating the second sample data from the first sample data.

[0071] To perform autoregressive generation, a special marker character is padded at the beginning of each segment of the second sample data for input to the general language model, and a special marker character is padded at the end of each segment of the second sample data for output to the general language model.

[0072] Furthermore, in the embodiments of this application, the method further includes:

[0073] By changing the length and number of sampled text fragments, pre-training targets adapted to different tasks are generated to pre-train the general language model, resulting in a multilingual model adapted to different tasks.

[0074] Furthermore, in this embodiment of the application, pre-training targets adapted to different tasks are generated by changing the length and number of sampled text fragments, including:

[0075] By sampling the length of each segment from a Poisson distribution with a mean of 3, the total segment length is a first preset proportion of the original sequence length, generating a pre-training target adapted to classification tasks for natural language understanding;

[0076] By making each segment a complete sentence in the text sequence, and the total segment length being a second preset proportion of the original text sequence length, a pre-training objective for the adaptive text generation task is generated.

[0077] By sampling only one segment from a text sequence, and making the segment length a third preset proportion of the original text sequence length, a pre-training target adapted to the task of long text generation is generated.

[0078] For example, the first preset ratio can be 15%, the second preset ratio can be 15%, and the third preset ratio can be 50%-100%.

[0079] In this embodiment, for the character level, the length of each segment is sampled from a Poisson distribution with a mean of 3, and the total segment length is 15% of the original sequence length. This generates a pre-training objective adapted to the classification task of natural language understanding. This pre-training objective is similar to the masked language model objective of BERT, which is beneficial for completing the classification task of natural language understanding. For the sentence level, each segment is made to be a complete sentence in the text sequence, and the total segment length is 15% of the original text sequence length. This generates a pre-training objective adapted to the conditional text generation task, which is beneficial for completing the conditional text generation task. For the document level, only one segment is sampled from a text sequence, and the segment length is 50%-100% of the original text sequence length. This generates a pre-training objective adapted to the long text generation task. This pre-training objective is similar to the autoregressive language model objective of GPT-2, which is beneficial for completing the long text generation task.

[0080] Furthermore, in this embodiment of the application, the multilingual summarization data includes original text data and summary data in different languages. Obtaining the multilingual summarization data and learning a multilingual model based on the multilingual summarization data to obtain a multilingual summarization generation model includes:

[0081] By using original text and summary data in different languages ​​as training data, and using prompt-based fine-tuning to train the multilingual model, a multilingual summary generation model is obtained.

[0082] This application uses prompt-based fine-tuning to train a pre-trained multilingual model. Specifically, it uses prompts to enable the multilingual model to perform corresponding tasks, such as enabling the multilingual model to freely generate Chinese or English summaries.

[0083] This application collects Chinese and English summaries from the NCLS dataset as training data for fine-tuning. For example, when training a multilingual model, for the Chinese summarization task, "Original text + Chinese summary:" is used as a prompt, while for the English summarization task, "Original text + TL;DR:" is used as a prompt, where TL;DR is an abbreviation for Too Long, Didn't Read.

[0084] The multilingual summarization generation model, after fine-tuning, can output summaries in different languages ​​by inputting the original text and adding different prompts.

[0085] Figure 2 This is a flowchart illustrating the generation of a multilingual summary generation model in the cross-language summary generation method based on a multilingual model, as described in this application embodiment.

[0086] like Figure 2 As shown, the process of generating a multilingual summary model includes: collecting multilingual text data; performing multilingual pre-training on a general language model based on the multilingual text data to obtain a large multilingual model; collecting English original text and Chinese summary data, fine-tuning the large multilingual model, and obtaining the multilingual summary generation model.

[0087] Figure 3 This is a flowchart illustrating the pre-training process of the cross-language summarization generation method based on a multilingual model, as described in this application.

[0088] like Figure 3 As shown, the multilingual GLM model first splits the multilingual text data into first sample data and second sample data; then the multilingual GLM model uses the first sample data and second sample data for pre-training; wherein, the multilingual GLM model performs bidirectional association encoding on the characters in the first sample data; and performs unidirectional association encoding on the characters in the second sample data to the first sample data and the parts preceding the character in the second sample data.

[0089] Figure 4 This is an example diagram illustrating the use of a generative model for prompting learning in the cross-language summary generation method based on a multilingual model according to embodiments of this application.

[0090] like Figure 4As shown, the diagram consists of two parts. The first part shows the multilingual GLM model performing a classification task using a cue learning framework. The multilingual GLM model generates text from input text and cue information, and maps the generated text to a category to achieve the classification task. The second part shows the multilingual GLM model performing a text generation task using a cue learning framework. The multilingual GLM model generates text from input text.

[0091] Figure 5 This is a schematic diagram of the structure of a cross-language summary generation device based on a multilingual model provided in Embodiment 2 of this application.

[0092] like Figure 5 As shown, the cross-language summarization generation device based on a multilingual model includes:

[0093] Module 10 is used to acquire multilingual text data;

[0094] Pre-training module 20 is used to pre-train a general language model based on multilingual text data to obtain a multilingual model;

[0095] Training module 30 is used to acquire multilingual summary data, learn the multilingual model based on the multilingual summary data, and obtain a multilingual summary generation model;

[0096] The generation module 40 is used to acquire the text data to be processed, input the text data to be processed into the multilingual summary generation model, and generate summaries in multiple different languages ​​corresponding to the text data to be processed.

[0097] The cross-language summarization generation device based on a multilingual model according to this application includes an acquisition module for acquiring multilingual text data; a pre-training module for pre-training a general language model based on the multilingual text data to obtain a multilingual model; a training module for acquiring multilingual summary data and learning the multilingual model based on the multilingual summary data to obtain a multilingual summarization generation model; and a generation module for acquiring text data to be processed, inputting the text data to be processed into the multilingual summarization generation model, and generating summaries in multiple different languages ​​corresponding to the text data to be processed. This solves the technical problem that existing methods cannot complete cross-language summarization tasks, requiring the generation of summaries before translation and resulting in poor summarization accuracy. By performing autoregressive fill-in-the-blank pre-training on the general language model to obtain a multilingual model, and realizing cross-language summarization based on the multilingual model, it can generate summaries in different languages ​​from text data, while also extracting and summarizing the main idea of ​​the text, thus improving the accuracy of the summarization.

[0098] Furthermore, in this embodiment of the application, after acquiring the multilingual text data, the process includes:

[0099] For each language of the multilingual text data, multiple text segments are randomly sampled. Each sampled segment in the text sequence is replaced with a mask mark to obtain a corrupted text sequence, where each segment corresponds to a series of consecutive characters.

[0100] The corrupted text sequence was used as the first sample data;

[0101] The segment whose mask marker is replaced is used as the second sample data.

[0102] To implement the above embodiments, this application also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the cross-language summary generation method based on a multilingual model as described in the above embodiments.

[0103] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the cross-language summary generation method based on a multilingual model as described above.

[0104] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0105] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0106] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0107] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0108] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0109] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0110] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0111] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A cross-linguistic summarization generation method based on a multilingual model, characterized in that, Includes the following steps: Acquire multilingual text data; The multilingual text data is used to pre-train a general language model to obtain a multilingual model, which includes: for each language text sequence in the multilingual text data, randomly sampling multiple text segments, replacing each sampled segment in the text sequence with a mask marker to obtain a corrupted text sequence, wherein each segment corresponds to a series of consecutive characters; The damaged text sequence is used as the first sample data; The segment whose mask marker is replaced is used as the second sample data; Based on the first sample data and the second sample data, the general language model is pre-trained to obtain a multilingual model; Specifically, by changing the length and number of sampled text fragments, pre-training targets adapted to different tasks are generated to pre-train the general language model, resulting in a multi-language model adapted to different tasks; including: By sampling the length of each segment from a Poisson distribution with a mean of 3, the total segment length is a first preset proportion of the original sequence length, generating a pre-training target adapted to classification tasks for natural language understanding; By making each segment a complete sentence in the text sequence, and the total segment length being a second preset proportion of the original text sequence length, a pre-training objective for the adaptive text generation task is generated. Obtain multilingual summary data, and learn the multilingual model based on the multilingual summary data to obtain a multilingual summary generation model; Obtain the text data to be processed, input the text data to be processed into the multilingual summarization generation model, and generate summaries in multiple different languages ​​corresponding to the text data to be processed.

2. The method as described in claim 1, characterized in that, The multilingual summarization data includes original text data and summary data in different languages. The process of obtaining the multilingual summarization data and learning the multilingual model based on the multilingual summarization data to obtain a multilingual summarization generation model includes: The original text data and summary data in the different languages ​​are used as training data, and the multilingual model is trained using prompt-based fine-tuning to obtain a multilingual summary generation model.

3. A cross-language summarization generation device based on a multilingual model, characterized in that, include: The acquisition module is used to acquire multilingual text data; The pre-training module is used to pre-train the general language model based on the multilingual text data to obtain the multilingual model; The training module is used to acquire multilingual summary data, and to learn the multilingual model based on the multilingual summary data to obtain a multilingual summary generation model; The generation module is used to acquire the text data to be processed, input the text data to be processed into the multilingual summarization generation model, and generate summaries in multiple different languages ​​corresponding to the text data to be processed. The pre-training module is also used to randomly sample multiple text segments for each language of the multilingual text data, and replace each sampled segment in the text sequence with a mask mark to obtain a corrupted text sequence, wherein each segment corresponds to a series of consecutive characters; The damaged text sequence is used as the first sample data; The segments that are replaced by masked tags are used as second sample data; based on the first and second sample data, the general language model is pre-trained to obtain a multilingual model; Specifically, by changing the length and number of sampled text fragments, pre-training targets adapted to different tasks are generated to pre-train the general language model, resulting in a multi-language model adapted to different tasks; including: By sampling the length of each segment from a Poisson distribution with a mean of 3, the total segment length is a first preset proportion of the original sequence length, generating a pre-training target adapted to classification tasks for natural language understanding; By making each segment a complete sentence in the text sequence, and the total segment length being a second preset proportion of the original text sequence length, a pre-training objective for the adaptive text generation task is generated.

4. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of claims 1-2.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-2.

Images