Language-aware multilingual pre-training and fine-tuning method based on language discrimination cues

Abstract

The application discloses a language perception multilingual pre-training and fine-tuning method based on language discrimination prompts, which comprises a multilingual perception pre-training stage, wherein a model is pre-trained by using input content containing different language prompts and different language contexts, so that the model learns a multilingual perception function of identifying or perceiving a language by taking a language prompt as a language discrimination signal or language perception meta-information during the pre-training process; and a multilingual perception fine-tuning stage, wherein the model is fine-tuned by using input content containing different language prompts and different language contexts based on a downstream task, the model perceives and identifies the language of current input content by using the language prompt, and completes a model fine-tuning process for the downstream task based on language perception. The application introduces multilingual explicit distinguishing language information into the pre-training and fine-tuning stages of a multilingual and training model, improves the language perception capability of the multilingual model for different languages, and improves the effect of the downstream task.

Description

Technical Field

[0001] This invention relates to deep learning technology, and in particular to a method for multilingual pre-training and fine-tuning of language perception based on language discrimination prompts. Background Technology

[0002] In recent years, large-scale pre-trained language models such as GPT-3 and T5 have achieved excellent performance in many natural language processing (NLP) tasks, and have also achieved significant results in the multilingual (language-specific) domain, becoming the most mainstream implementation scheme for solving downstream tasks (sentiment classification, text summarization, etc.). Multilingual models are typically pre-trained on large multilingual corpora and directly adapted to downstream tasks specific to a particular language. Existing transformer-based language models with millions or even billions of parameters directly represent different languages ​​in a unified semantic space and can be fine-tuned without explicit signals. Existing research also shows that in multilingual training, different languages ​​can benefit from each other in terms of semantics, linguistics, and syntax, resulting in significant portability.

[0003] The most similar existing implementation to this invention is based on the mT5 (Massively Multilingual Pre-trained Text-to-text Transformer) pre-trained language model. After inputting the input data for the corresponding downstream task and the task's prompts, the language model extracts the semantic features of the sentence and the features of the corresponding task. Through a Transformer-based multi-layer language model, the desired sentence for the corresponding task is finally output as the result. This method has achieved better performance than previous mainstream methods on various tasks.

[0004] However, existing implementations still have the following problems: multilingual models still only implicitly learn the language information of the training corpus, and the model does not receive explicit language cues during the pre-training and fine-tuning stages of downstream tasks. This results in existing language models not being aware of the language information of the input text, and therefore unable to use the model's capabilities for the corresponding language. There is still room for improvement in performance when handling downstream tasks in a single language.

[0005] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The main objective of this invention is to overcome the shortcomings of the aforementioned background technology and provide a method for multilingual pre-training and fine-tuning of language perception based on language discrimination prompts.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for multilingual pre-training and fine-tuning of language perception based on language discrimination prompts includes the following steps:

[0009] In the pre-training stage of multilingual perception, the model is pre-trained using input content containing different language prompts and different language contexts, so that the model learns the multilingual perception function of recognizing or perceiving languages ​​by using the language prompts as language discrimination signals or language perception meta-information during the pre-training process.

[0010] In the fine-tuning stage of multilingual awareness, the model is fine-tuned using input content containing different language cues and different language contexts based on downstream tasks. The model uses language cues to perceive and identify the language of the current input content, and completes the model fine-tuning process for downstream tasks based on language perception.

[0011] Furthermore:

[0012] In the pre-training phase of the multilingual awareness, the input content is formatted as key-value pairs, where a delimiter is used to distinguish between language prompts and context, and the language prompts and context are concatenated to obtain the input content for model pre-training.

[0013] In the fine-tuning stage of language perception, a corresponding prompt template is designed for each target task. The language prompts trained in the pre-training stage are added to the context using the corresponding language and formatted as key-value pairs to obtain the input content for model fine-tuning.

[0014] The separators include language indicators and text indicators. The language indicators and text indicators are tokens that are not originally in the vocabulary. During training, the model uses them as symbols to indicate the beginning of a language prompt and the beginning of a text prompt, respectively.

[0015] The model is an encoder-decoder model, trained using language cue embeddings in a text-to-text manner.

[0016] The model is an mT5 model, which is trained using the cross-entropy loss function to achieve language awareness.

[0017] Further pre-training is performed based on the pre-trained checkpoint t1 of the mT5 model.

[0018] In the pre-training phase of multilingual awareness, the multilingual corpus mC4 is used to add trainable and storable language cues before the multilingual input content, where random initialization of continuous embeddings is used as language cues to represent each language.

[0019] In the pre-training stage of the multilingual perception, the "span-corruption" target of masked language modeling is used as the pre-training target, and the language prompts are jointly trained with the model parameters.

[0020] The downstream tasks include one or more of the following: natural language reasoning tasks, text tasks, sentiment classification tasks, and question answering tasks.

[0021] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the language perception multilingual pre-training and fine-tuning method based on language discrimination prompts.

[0022] The present invention has the following beneficial effects:

[0023] This invention introduces explicit language information into the pre-training and fine-tuning stages of multilingual training models, improving the multilingual model's ability to perceive different languages ​​and enabling it to achieve further performance improvements in various downstream tasks.

[0024] This invention enhances the ability of multilingual pre-trained language models to perceive linguistic information in training data. Without reducing inference efficiency, the proposed solution outperforms existing methods in various downstream natural language processing tasks, playing a promoting role in these tasks.

[0025] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating the principle of a language perception multilingual pre-training and fine-tuning method based on language discrimination prompts according to an embodiment of the present invention. Detailed Implementation

[0027] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0028] This invention proposes a language-aware multilingual pre-training and fine-tuning method based on language discrimination prompts. It presents a language-aware multilingual pre-training and fine-tuning framework, providing language information to the model during the pre-training and fine-tuning stages to optimize the model's training and fine-tuning process. In the multilingual pre-training stage, language information is introduced as meta-information to enhance the multilingual capabilities of the pre-trained language. By introducing multilingual language information as meta-information, this invention improves the downstream task performance of the multilingual model by using a unified language-aware multilingual pre-training and fine-tuning framework across various downstream tasks.

[0029] See Figure 1This invention provides a method for multilingual pre-training and fine-tuning of language perception based on language discrimination prompts, which includes two stages: multilingual perception pre-training and multilingual perception fine-tuning.

[0030] In the pre-training stage of multilingual perception, the model is pre-trained using input content containing different language prompts and different language contexts, so that the model learns the multilingual perception function of recognizing or perceiving languages ​​by using the language prompts as language discrimination signals or language perception meta-information during the pre-training process.

[0031] In the fine-tuning stage of multilingual perception, the model is fine-tuned using input content containing different language cues and different language contexts based on downstream tasks. The model uses language cues to perceive and identify the language of the current input content, and completes the model fine-tuning process for downstream tasks based on language perception.

[0032] In some embodiments, during the pre-training phase of the multilingual awareness, the input content is formatted as key-value pairs, wherein a delimiter is used to distinguish between language prompts and context, and the language prompts and context are concatenated to obtain the input content for model pre-training.

[0033] In some embodiments, during the language awareness fine-tuning stage, a corresponding prompt template is designed for each target task, and the language prompts trained in the pre-training stage are added to the context using the corresponding language and formatted as key-value pairs to obtain the input content for model fine-tuning.

[0034] The separator may include language indicators and text indicators. The language indicators and text indicators are tokens that are not originally in the vocabulary. During training, the model uses them as symbols to indicate the beginning of a language prompt and the beginning of a text prompt, respectively.

[0035] In some embodiments, the model is an encoder-decoder model trained using language cue embeddings in a text-to-text manner.

[0036] In some embodiments, the model is an mT5 model, and a language-aware mT5 model is trained using a cross-entropy loss function. In a preferred embodiment, further pre-training is performed based on the pre-trained checkpoint t1 of the mT5 model.

[0037] In some embodiments, during the pre-training phase of multilingual awareness, a multilingual corpus mC4 is used to add trainable and storable language cues before multilingual input content, wherein randomized initialized sequential embeddings are used as language cues to represent each language.

[0038] In some embodiments, during the pre-training phase of the multilingual awareness, the masked language modeling "span-corruption" target is used as the pre-training target, and the language prompts are jointly trained with the model parameters.

[0039] In different embodiments, the downstream task includes one or more of the following: natural language reasoning task, text task, sentiment classification task, and question answering task.

[0040] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the language perception multilingual pre-training and fine-tuning method based on language discrimination prompts.

[0041] This invention introduces explicit language information into the pre-training and fine-tuning stages of multilingual training models, improving the multilingual model's ability to perceive different languages ​​and enabling it to achieve further performance improvements in various downstream tasks.

[0042] The following further describes specific embodiments of the present invention, its principles, and test results.

[0043] Figure 1 This invention demonstrates the principle of a language-aware multilingual pre-training and fine-tuning method based on language discrimination prompts. This invention uses multilingual prompts as explicit signals to "hint" to the model which language the data being processed belongs to during pre-training and fine-tuning. First, this invention pre-trains a prompt-enhanced mT5 model on a subset of the multilingual corpus mC4 (Multilingual Colossal, Cleaned version of Common Crawl's Web Crawl Corpus). This involves adding trainable and storable soft prompts before the multilingual input and training them together with the model. These pre-trained discriminative soft prompts for different languages ​​contain language-aware meta-information and are used for subsequent adaptations in later tasks. In other words, the model is explicitly prompted by the language information of each data instance.

[0044] like Figure 1As shown, the framework proposed in this embodiment of the invention consists of two complementary stages: language-aware pre-training and language-aware fine-tuning. In both stages, the model's input consists of two parts: language cues and context, connected with predefined markers. The language cues are language-aware information learned during model pre-training, serving as language discrimination signals during the fine-tuning stage, thereby helping the model better recognize the language of the current input and stimulating its knowledge and ability in that language. The context contains semantic information of the original input text, such as text to be summarized or categorized, questions, options, etc.

[0045] Specifically, the implementation process of this invention is divided into two parts: model pre-training and model fine-tuning. Downstream task testing consists of four tasks: natural language inference, text, sentiment classification, and question answering.

[0046] Language perception model pre-training

[0047] The language-aware pre-training uses a masked language modeling "span-corruption" target as the pre-training objective to maintain consistency with the mT5 pre-training stage as much as possible, and trains language cues and model parameters jointly. More specifically, the masked language modeling "span-corruption" target is operated as follows: For each pre-training data instance, this embodiment randomly samples and deletes approximately 15% of the tokens, replacing them with a special token, the mask token, as the target to be predicted. Compared to normal tokens, the mask token itself has no semantic representation; the model needs to predict the semantics of the context. The remaining 85% of tokens, without replacement, are concatenated with the mask token and then used as the model's decoding target. During the pre-training stage, the model needs to perceive the semantics of the context and predict the semantic information of the deleted tokens. Next, this embodiment uses a large number of learnable and pluggable randomly initialized continuous embeddings as soft cues to represent each language; therefore, different inputs of the same language can share the same language information during the pre-training stage. To better distinguish the two parts of the model input mentioned above, this embodiment formats the input as key-value pairs. Specifically, this invention uses special markers, namely language indicators and text indicators, as separators to distinguish between soft prompts and context, respectively. In the embodiments, the language indicators and text indicators are similar to mask tokens, serving as tokens not originally present in the vocabulary. During model training, these are used as symbols to represent the beginning of a language segment and the beginning of a text segment, respectively. Finally, the two parts are concatenated to obtain the input to be encoded.

[0048] The input is fed into the mT5 model, which is trained together with language soft cue embeddings. Once the input and target are ready, this embodiment trains the language-aware mT5 model using a cross-entropy loss function. Since experimenting on all languages ​​is impractical, considering the languages ​​supported by the mT5 language model and all benchmark datasets, this embodiment selects six representative languages—English, Chinese, Spanish, French, Arabic, and Hindi—for pre-training and downstream fine-tuning. These six languages ​​have different linguistic distances. For example, English is closer to French in linguistic distance than Arabic. Since pre-training from scratch requires significant time and computational resources, and validating the effectiveness of this method does not require training the model from zero, this embodiment chooses to further pre-train based on the previously disclosed mT5-based pre-training checkpoint t1, rather than pre-training from scratch, to reduce costs. This embodiment prepares a subset of the mC4 dataset for the six languages ​​(keeping the number of data instances constant across languages), with a total pre-training corpus size of 204GB.

[0049] Fine-tuning of the language perception model

[0050] Through the above language-aware pre-training, soft embeddings of the target language can be obtained and used for downstream fine-tuning. For encoder-decoder models like mT5, cue engineering is obviously a huge boost in order to better understand the correlation between input and task. Therefore, this embodiment of the invention manually designs corresponding cue templates for each target task to achieve better performance. As done in the pre-training phase of this embodiment, after formatting the input into key-value pairs, pre-trained language cues are added to the context, and this embodiment of the invention obtains the final input to be encoded. The input will be fed into the pre-trained language-aware mT5 model, and the model will be trained using language cue embeddings in a text-to-text manner. In the inference phase, the input will also be processed as above so that the model generates the required output.

[0051] Actual training examples

[0052] The following is an example using a real data sample from the pre-training phase. An initial Chinese text is as follows:

[0053] Initial text: The Mogao Caves in Dunhuang are a famous historical site in China with a long history.

[0054] First, we replace 15% of the tokens with mask tokens, resulting in the input and output:

[0055] Input text: Dunhuang <s1>China <s2>Famous historical sites.

[0056] Output text: <s1>Mogao Caves <s2>Long history <s3>

[0057] Then we concatenate the text indicator, language indicator, and learnable language prompt (in this case, Chinese prompt) to the beginning of the text to obtain the final training input and output:

[0058] Input text:<zh_lang> [zh_prompt] <text>Dunhuang <s1>China <s2>Famous historical sites.

[0059] Output text: <s1>Mogao Caves <s2>Long history <s3>

[0060] In the downstream task stage of fine-tuning the model, the same processing methods as in the pre-training stage are used to improve the model's generation performance as much as possible. Further examples will not be provided here.

[0061] Downstream task testing

[0062] To fully evaluate the method proposed in this embodiment of the invention, extensive experiments were conducted on a dataset covering four different tasks, including two classification tasks: natural language inference and sentiment classification, and two generation tasks: text summarization and question answering. The dataset is as follows.

[0063] I. XNLI is a multilingual natural language inference dataset. It is a subset of MNLI and has been translated into 14 languages. The task is to infer whether the semantic relationship between two sentences is entailment, contradiction, or neutral. This embodiment of the invention treats it as a three-label classification task.

[0064] II. The Multilingual Amazon Review Corpus (MARC) is a multilingual sentiment classification dataset containing Amazon product reviews in six languages. Each data instance includes information such as review title, review content, and star rating. To enhance the effectiveness of the suggestion engineering, this embodiment of the invention changes the original five-label classification task to a three-label classification task. Specifically, the labels originally scored "4" and "5" are merged into a "positive" label, the labels originally scored "3" are merged into a "neutral" label, and the labels originally scored "1" and "2" are merged into a "negative" label.

[0065] Third, XLSUM is a multilingual summary dataset that uses design heuristics to extract summaries from BBC news articles. The dataset covers 44 languages, ranging from high-resource to low-resource, and is accompanied by professional annotations.

[0066] IV. XQuAD is a multilingual question-answering dataset. Its English dataset is a subset of SQuAD v1.1, and the other language datasets are translated from the English dataset. Therefore, the datasets are completely parallel. More detailed information includes manually designed hints for each dataset used in the experiments. Some datasets do not contain all six pre-trained languages; this embodiment of the invention experimented with the remaining languages. To reduce computational costs and ensure the same amount of training data for different languages, this embodiment of the invention randomly sampled 3000 / 600 instances for each language from the original training / validation sets of each dataset. This embodiment of the invention evaluated the model on the entire test set.

[0067] For classification tasks, embodiments of the present invention balance the dataset across labels, so that the number of data instances on each label is exactly the same.

[0068] The method proposed in this invention outperforms all baselines on average across all tasks. On XNLI, MARC, and XLSUM, the method achieves better performance across all languages, and on XQuAD, it achieves best or comparable results. This demonstrates the facilitating effect of the language-aware multilingual pre-training and fine-tuning framework of this invention on these downstream tasks.

[0069] In summary, this invention enhances the ability of multilingual pre-trained language models to perceive linguistic information in training data. Without reducing inference efficiency, the proposed solution outperforms existing methods in various downstream natural language processing tasks, playing a promoting role in these tasks.

[0070] This invention overcomes the previous problem of difficulty in using corresponding languages ​​to solve tasks in different languages, thus improving the performance of multilingual models. For example, this invention can help solve multilingual question-answering problems and can be applied to commercial question-answering robots to answer personalized and complex questions from native-speaker customers, improving the accuracy of the answers.

[0071] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0072] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0073] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0074] The storage medium can be implemented by any type of volatile or non-volatile storage device, or a combination thereof. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0075] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0076] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0077] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0078] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0079] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0080] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0081] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0082] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0083] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention. < / text>

Claims

1. A language-aware multilingual pre-training and fine-tuning method based on language discrimination cues, characterized in that, The method comprises the following steps: A multilingual perception pre-training stage, in which the model is pre-trained using input content containing different language cues and different language contexts, so that the model learns a multilingual perception function in the pre-training process, which uses the language cues as language discrimination signals or language perception meta-information to identify or perceive the language; A multilingual perception fine-tuning stage, in which the model is fine-tuned using input content containing different language cues and different language contexts based on downstream tasks, and the model perceives and identifies the language of the current input content using the language cues, and completes the model fine-tuning process for downstream tasks based on language perception.

2. The method of claim 1, wherein, In the multilingual perception pre-training stage, the input content is formatted as a key-value pair, in which a separator is used to distinguish the language cue and the context, and the language cue and the context are spliced to obtain the input content for model pre-training.

3. The method of claim 1, wherein, In the language perception fine-tuning stage, a corresponding prompt template is designed for each target task, the language cue trained in the pre-training stage is added before the context in the corresponding language, and is formatted as a key-value pair to obtain the input content for model fine-tuning.

4. The method of claim 2, wherein, The separator includes a language indicator and a text indicator, which are tokens that do not exist in the original vocabulary, and are used as symbols representing the beginning of the language cue and the beginning of the text, respectively, in the training process of the model.

5. The method according to any one of claims 1 to 4, characterized in that, The model is an encoder-decoder model, which is trained in a text-to-text manner using language cue embedding.

6. The method of claim 5, wherein, The model is an mT5 model, and the language perception mT5 model is trained using a cross-entropy loss function and further pre-trained based on a pre-training checkpoint of the mT5 model.

7. The method according to any one of claims 1 to 4, wherein In the multilingual perception pre-training stage, a multilingual corpus mC4 is used, and a trainable and storable language cue is added before the multilingual input content, in which a randomly initialized continuous embedding is used as the language cue to represent each language.

8. The method according to any one of claims 1 to 4, wherein In the multilingual perception pre-training stage, a mask language modeling "span-corruption" objective is used as the pre-training objective, and the language cue is trained jointly with the model parameters.

9. The method according to any one of claims 1 to 4, wherein The downstream tasks include one or more of natural language inference tasks, text tasks, sentiment classification tasks, and question and answer tasks.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any one of claims 1 to 9. The computer program, when executed by the processor, implements the method of any one of claims 1 to 9.

Images