Machine translation model training method and device for low-resource language and medium

Abstract

The invention belongs to the technical field of machine translation, and provides a low-resource language machine translation model training method and device and a medium, and the method comprises the steps: S1, constructing a pseudo parallel corpus based on a bilingual dictionary; s2, data enhancement based on language mixing: (21) word-level language mixing data enhancement; (22) language hybrid data of a sentence level is increased; s3, outputting a low-resource machine translation model based on lightweight fine tuning of the language family: selecting a machine translation model of the same language family as a basic model, and performing model fine tuning; and S4, completing machine translation model training of the low-resource language based on the setting of the steps S1 to S3. On one hand, the problem of low-resource training corpus shortage is solved, and on the other hand, the problems of large language span and high training cost of low-resource machine-upset training are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine translation, and in particular, relates to a method, device and medium for training a machine translation model for a low-resource language. Background Art

[0002] There are more than 7,000 languages ​​in the world, and currently only a few hundred of them are covered by publicly available machine translation engines. Although most of the remaining low-resource languages ​​have a lot of translation needs, there are no available machine translation engines. There are several difficulties in training low-resource / zero-resource machine translation engines: First, there is a lack of training corpus. For some low-resource languages, such as Sinhalese, there is limited public monolingual data and almost no bilingual data. Second, many low-resource languages ​​are not a single language, but are mixed with a large number of other languages. For example, Balochi is the language of the Arab region, which includes Urdu, Persian, Pashto and other languages. Third, most of the existing low-resource machine translation models use a full-variable fine-tuning method based on transfer learning, which has a large language span and high fine-tuning costs. Summary of the invention

[0003] The purpose of the present invention is to provide a machine translation model training method for low-resource languages ​​to solve the technical problems existing in the prior art.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows: A method for training a machine translation model for a low-resource language comprises the following steps: Step S1: construct a pseudo-parallel corpus based on a bilingual dictionary; Step S2: Data enhancement based on language mixing: (21) word-level language mixing data enhancement; (22) sentence-level language mixing data enhancement; Step S3: Output a low-resource machine translation model based on lightweight fine-tuning of the language family: Select a machine translation model of the same language family as the base model and fine-tune the model according to the following method: Step S3.1: For all pre-trained parameters Perform the first fine-tuning on the new corpus data to obtain the updated parameters , by selecting the parameter set with the maximum absolute difference Update, these parameters are the core parameters for training new models:

[0005] In the formula, Indicates the parameter amount to be updated; Step S3.2: Reset all pre-trained parameters to state, and perform a second fine-tuning to update the parameter set , other parameters remain frozen, and after two rounds of fine-tuning, the training parameters of the first round of fine-tuning are recorded as ; Step S3.3: In training parameters Based on this, the second round of first fine-tuning is performed, and a new parameter set is selected through the parameter selection strategy. , and then reset all training parameters to state, and perform the second round of fine-tuning to update the parameter set , other parameters remain frozen, and the training parameters for the second round of fine-tuning are recorded as ;go through After rounds of fine-tuning, the final updated parameter set is obtained:

[0006] In the formula, Indicates the parameter amount to be updated, The number of rounds to fine-tune the model; Step S4: Based on the settings of steps S1 to S3, complete the machine translation model training for low-resource languages.

[0007] Preferably, the method further includes the step of using a parameter controller to control the position and quantity of model parameters in different languages ​​during the first model fine-tuning process:

[0008] in, Indicates the parameter amount to be updated, represents the parameter controller, which is a fully connected function. represents the training data input, Indicates language information.

[0009] Preferably, during the model fine-tuning process, regularization is used to ensure the fine-tuning results.

[0010] Preferably, the specific method of step S1 is as follows: Step S1.1: For low-resource monolingual data, for each sentence in each language, use a bilingual dictionary to generate new translations through sentence or phrase matching to construct a new parallel corpus; for bilingual data of low-resource language pairs, select some words in the target language / source language sentences and replace them with other target language / source language words, and replace the corresponding words in the target language / source language according to the vocabulary to construct a new parallel corpus; Step S1.2: Filter the parallel corpus using a corpus quality assessment method to obtain a pseudo-parallel corpus.

[0011] Preferably, the (21) word-level language mixed data enhancement includes: Step S21.1: For a language with mixed language characteristics, randomly select a group of words in the pseudo-parallel corpus, including words of the language and its parallel words; Step S21.2: For the selected words, randomly specify another language that is mixed with the language, and replace the words in the language with words in other languages ​​as training corpus for data enhancement.

[0012] Preferably, the (22) sentence-level language mixed data is increased by using other languages ​​mixed with the language and their pseudo-parallel corpora as training corpora for data enhancement.

[0013] In order to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the machine translation model training method for low-resource languages.

[0014] In order to achieve the above-mentioned objectives, the present invention also provides a machine translation model training device for low-resource languages, comprising: a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory, and is used to execute the computer program stored in the memory, so that the machine translation model training device for low-resource languages ​​executes the machine translation model training method for low-resource languages.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention constructs low-resource pseudo-parallel corpus through bilingual dictionaries and performs data enhancement by mixing languages, thereby solving the problem of lack of low-resource training corpus.

[0016] (2) The present invention selects the machine translation model of the same language as the basic model, and reduces the number of fine-tuning parameters through lightweight fine-tuning, thereby solving the problem of large language span and high training cost of low-resource machine translation training. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic diagram of the process principle of Example 1 of the present invention.

[0018] Figure 2 This is a principle block diagram of Example 3 of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to have a clearer understanding and knowledge of the present invention, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described below are only used to explain the present invention, and are convenient for understanding. The technical solutions provided by the present invention are not limited to the technical solutions provided by the following embodiments, and the technical solutions provided by the embodiments should not limit the protection scope of the present invention.

[0020] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0021] Example 1 like Figure 1 As shown, this embodiment provides a method for training a machine translation model for a low-resource language. The method is based on an innovative design of lightweight fine-tuning of the same language and mixed language data enhancement. The method mainly implements the training of the translation model from aspects such as using a dictionary to construct a pseudo-parallel corpus, performing data enhancement through a mixed language scheme, using a machine translation model of the same language as a basic model, and using lightweight fine-tuning of some parameters.

[0022] In this embodiment, the machine translation model training method for low-resource languages ​​specifically includes the following steps: 1. Step S1: Constructing a pseudo-parallel corpus based on a bilingual dictionary Step S1.1: Construct a new parallel corpus. The new parallel corpus includes two aspects: (1) For low-resource monolingual data, for each sentence in each language, a bilingual dictionary is used to generate new translations through sentence or phrase matching to construct a new parallel corpus. The low-resource languages ​​mentioned in this embodiment include multiple languages. For each language pair that needs to be translated, such as translating Sinhalese into Chinese, a bilingual dictionary containing two languages ​​is used. The bilingual dictionary can be an existing one or manually compiled. If an existing one is available, it is preferred to use the existing one. If no existing one is available, it needs to be manually compiled.

[0023] (2) For bilingual data of low-resource language pairs, select some words in the target language / source language sentences (wherein, the selection method can use a variety of selection methods at the same time, including random selection, selection of rare words (words with low frequency of occurrence), selection of high frequency, selection of nouns, etc.), and replace them with other target language / source language words. At the same time, replace the corresponding words in the target language / source language according to the vocabulary to construct a new parallel corpus; wherein, the vocabulary processing method is the same as that of the bilingual dictionary, and can use an existing one or can be manually constructed. If there is an existing one, it is preferred to use the existing one. If not, it is manually sorted and constructed. It should be noted that partial words refer to a part of the words in the target language / source language sentence, which can be two, three, four, five or even more. Those skilled in the art can select the number of partial words as needed.

[0024] Step S1.2: In order to ensure the fluency of the generated corpus, the new parallel corpus is filtered through corpus quality assessment methods (such as syntactic rules, language models, etc.), and high-quality parallel corpora are retained, thereby obtaining a pseudo-parallel corpus based on a bilingual dictionary.

[0025] Step S2: Data enhancement based on language mixing This step is mainly to solve the language mixing problem of low-resource languages. In this embodiment, it mainly includes two aspects: (21) Word-level language mixing data enhancement: Step S21.1: For a language with language mixing characteristics, randomly select a group of words in the pseudo-parallel corpus, including words of the language and its parallel words; Step S21.2: For the selected words, randomly specify another language that is mixed with the language, and replace the words of the language with words of the other language as the training corpus for data enhancement.

[0026] (22) Increasing the language-mixed data at the sentence level; using other languages ​​mixed with the language and their pseudo-parallel corpora as training corpora for data enhancement; taking Balochi to Chinese as an example, we added parallel corpora from Urdu to Chinese, Persian to Chinese, and Pashto to Chinese for data enhancement.

[0027] Step S3: Language-based Lightweight Fine-tuning Output low-resource machine translation model: In order to reduce the gap between languages ​​and narrow the language span, select the machine translation model of the same language family (it should be noted that in this field, there is a recognized standard for the same language family, which includes the language families of all languages ​​in the case of mixed languages) as the basic model (using the machine translation model of the same language family as the basic model can reduce the fine-tuning language span), and fine-tune the model according to the following method: Step S3.1: For all pre-trained parameters Perform the first fine-tuning on the new corpus data to obtain the updated parameters , by selecting the parameter set with the maximum absolute difference Update, these parameters are the core parameters for training new models:

[0028] In the formula, Indicates the parameter amount to be updated; The pre-trained model is trained on common language corpus. For example, when translating Balochi into Chinese, the pre-trained model can be trained on bilingual corpus from Arabic to Chinese, because there are more public corpora in Arabic, and bilingual data is easier to obtain. It is not a low-resource language. The new corpus in this step refers to all the corpora generated by the above process.

[0029] Step S3.2: Reset all pre-trained parameters to state, and perform a second fine-tuning to update the parameter set , other parameters remain frozen, and after two rounds of fine-tuning, the training parameters of the first round of fine-tuning are recorded as ; Step S3.3: In training parameters Based on this, the second round of first fine-tuning is performed, and a new parameter set is selected through the parameter selection strategy. , and then reset all training parameters to state, and perform the second round of fine-tuning to update the parameter set , other parameters remain frozen, and the training parameters for the second round of fine-tuning are recorded as ;go through After rounds of fine-tuning, the final updated parameter set is obtained:

[0030] In the formula, Indicates the parameter amount to be updated, The number of rounds to fine-tune the model; Step S4: Based on the settings of steps S1 to S3, complete the machine translation model training for low-resource languages.

[0031] In a further preferred solution, in order to ensure the fine-tuning results and prevent them from deviating too much from the pre-training results, regularization is used during the model fine-tuning process to ensure the fine-tuning results. Take regularization as an example. ,in is a hyperparameter.

[0032] In order to fit the situation of mixed languages, this embodiment also designs a parameter controller, which is applied in the first model fine-tuning process. The parameter controller is used to control the position and quantity of model parameters in different languages:

[0033] in, Indicates the parameter amount to be updated, represents the parameter controller, which is a fully connected function. represents the training data input, Indicates language information.

[0034] The above method can be used to train machine translation models for low-resource / zero-resource languages, which can effectively solve the problems of insufficient low-resource / zero-resource training data and the difficulty and high cost of model fine-tuning.

[0035] Example 2 This embodiment provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the machine translation model training method for low-resource languages ​​provided in Example 1. A person of ordinary skill in the art can understand that all or part of the steps of implementing the method provided in Example 1 can be completed by hardware related to the computer program, and the above-mentioned computer program can be stored in a computer-readable storage medium, and when the program is executed, the steps of the method provided in Example 1 are executed; and the above-mentioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0036] Example 3 like Figure 2 As shown, this embodiment provides a machine translation model training device for low-resource languages, including: a processor and a memory; the memory is used to store computer programs; the processor is connected to the memory, and is used to execute the computer program stored in the memory, so that the machine translation model training device for low-resource languages ​​executes the machine translation model training method for low-resource languages ​​provided in Example 1.

[0037] Specifically, the memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.

[0038] Preferably, the processor can be a general-purpose processor, including a central processing unit, a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0039] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for training a machine translation model for a low-resource language, characterized in that: The following steps are involved: Step S1: construct a pseudo-parallel corpus based on a bilingual dictionary; Step S2: Data enhancement based on language mixing: (21) word-level language mixing data enhancement; (22) sentence-level language mixing data enhancement; Step S3: Output a low-resource machine translation model based on lightweight fine-tuning of the language family: Select a machine translation model of the same language family as the base model and fine-tune the model according to the following method: Step S3.1: For all pre-trained parameters Perform the first fine-tuning on the new corpus data to obtain the updated parameters , by selecting the parameter set with the maximum absolute difference Update, these parameters are the core parameters for training new models: In the formula, Indicates the parameter amount to be updated; Step S3.2: Reset all pre-trained parameters to state, and perform a second fine-tuning to update the parameter set , other parameters remain frozen, and after two rounds of fine-tuning, the training parameters of the first round of fine-tuning are recorded as ; Step S3.3: In training parameters Based on this, the second round of first fine-tuning is performed, and a new parameter set is selected through the parameter selection strategy. , and then reset all training parameters to state, and perform the second round of fine-tuning to update the parameter set , other parameters remain frozen, and the training parameters for the second round of fine-tuning are recorded as ;go through After rounds of fine-tuning, the final updated parameter set is obtained: In the formula, Indicates the parameter amount to be updated, The number of rounds of fine-tuning for the model; Step S4: Based on the settings of steps S1 to S3, complete the machine translation model training for low-resource languages.

2. The method for training a machine translation model for a low-resource language according to claim 1, wherein: It also includes the steps of using parameter controllers to control the location and number of model parameters in different languages ​​during the first model fine-tuning process: in, Indicates the parameter amount to be updated, represents the parameter controller, which is a fully connected function. represents the training data input, Indicates language information.

3. The method for training a machine translation model for a low-resource language according to claim 2, wherein: During model fine-tuning, regularization is used to ensure the fine-tuning results.

4. The method for training a machine translation model for a low-resource language according to claim 3, wherein: The specific method of step S1 is as follows: Step S1.1: For low-resource monolingual data, for each sentence in each language, use a bilingual dictionary to generate new translations through sentence or phrase matching to construct a new parallel corpus; for bilingual data of low-resource language pairs, select some words in the target language / source language sentences and replace them with other target language / source language words, and replace the corresponding words in the target language / source language according to the vocabulary to construct a new parallel corpus; Step S1.2: Filter the parallel corpus using a corpus quality assessment method to obtain a pseudo-parallel corpus.

5. The method for training a machine translation model for a low-resource language according to claim 4, wherein: The (21) word-level language mixed data enhancement includes: Step S21.1: For a language with mixed language characteristics, randomly select a group of words in the pseudo-parallel corpus, including words of the language and its parallel words; Step S21.2: For the selected words, randomly specify another language that is mixed with the language, and replace the words in the language with words in other languages ​​as training corpus for data enhancement.

6. The method for training a machine translation model for a low-resource language according to claim 5, characterized in that: The (22) sentence-level language mixed data increase: other languages ​​mixed with the language and their pseudo-parallel corpora are used as training corpora for data enhancement.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the machine translation model training method for a low-resource language as described in any one of claims 1 to 6.

8. A machine translation model training device for low-resource languages, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute a computer program stored in the memory, so that the low-resource language machine translation model training device executes the low-resource language machine translation model training method as described in any one of claims 1 to 6.

Images