A method and system for transfer of machine translation style based on curricular pre-training

Abstract

The application discloses a kind of based on course pre-training's machine translation style's migration method and system, wherein based on course pre-training's machine translation style's migration method, including: parallel corpus and monolingual corpus are respectively processed to get training corpus;Using training corpus and the model task corresponding to training corpus, course pre-training is carried out to machine translation model;Using style vocabulary alignment algorithm constructs pseudo-parallel data, using pseudo-parallel data, the machine translation model after course training is pre-trained, obtains final pre-training model;The final pre-training model is fine-tuned to translation style, and translation style migration model is obtained.The technical scheme of the application can solve the problem that the prior art does not pay attention to the style characteristics of the sentence, and it is difficult to realize text style conversion.

Description

Technical Field

[0001] This invention relates to the field of machine translation technology, and in particular to a method and system for transferring machine translation styles based on curriculum-based pre-training. Background Technology

[0002] Machine translation refers to the process by which a computer translates sentences in a source language into semantically equivalent sentences in a target language. Machine translation is mainly divided into three methods: rule-based machine translation, statistical machine translation, and neural network-based machine translation. Rule-based machine translation relies heavily on grammatical and syntax rules established by language experts, thus having certain limitations. Statistical machine translation is a corpus-based method, and its performance is poor when data is limited. With the development of machine learning technology, neural network-based machine translation, or neural machine translation, has gradually emerged. Neural machine translation has achieved significant performance advantages in most translation fields and is gradually becoming the mainstream translation method.

[0003] Neural machine translation typically employs an encoder-decoder framework. Specifically, the encoder encodes the source language sequence and extracts information from the source language to create a distributed representation. The decoder then generates the target language sequence based on the extracted distributed representation. The structure of neural machine translation models has undergone multiple iterations, resulting in significant improvements in training speed and translation quality. General-domain neural machine translation possesses a large number of parallel sentence pairs and exhibits high translation quality. However, in practical applications, specific scenarios require translation results with specific styles, necessitating the implementation of style transfer in neural machine translation. Currently, there are two main approaches to achieving style transfer: one is a pivot-based method where the translation model and the text style transfer model decode in two consecutive steps to obtain the target style translation. However, multi-step decoding leads to error propagation and excessive decoding time. The other approach utilizes knowledge distillation or back-translation to construct pseudo-parallel corpora. However, the pseudo-parallel corpora constructed by this method contain noise, which affects model performance and reduces the accuracy of style transfer. Currently, neural machine translation has achieved relatively limited results in the direction of translation style transfer, mainly due to the lack of large-scale parallel sentence pairs with corresponding styles, which limits data-driven end-to-end translation models.

[0004] Machine translation model training requires large-scale parallel data, making it difficult to learn language and style transfer simultaneously when corpora are scarce. Although pre-trained models can provide some prior knowledge, current pre-training tasks only focus on understanding the semantic information of sentences, neglecting the stylistic features, which are crucial for achieving text style transfer. Summary of the Invention

[0005] This invention provides a method and system for machine translation style transfer based on curriculum-based pre-training. It can design new pre-training tasks according to the translation style transfer task, enabling the model to learn more semantic and style features, while reducing the training difficulty of the model. This solves the problem in the prior art that does not pay attention to the style features of sentences and is difficult to achieve text style transfer.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a method for transferring machine translation style based on curriculum-based pre-training is proposed, comprising:

[0007] The parallel corpus and the monolingual corpus were segmented separately to obtain the training corpus;

[0008] The machine translation model is pre-trained using training corpus and the corresponding model tasks.

[0009] Pseudo-parallel data is constructed using a style vocabulary alignment algorithm. The pseudo-parallel data is then used to pre-train the course-trained machine translation model to obtain the final pre-trained model.

[0010] Fine-tuning the translation style of the final pre-trained model yields a translation style transfer model.

[0011] Preferably, in the above-mentioned machine translation style transfer method, the steps of performing word segmentation processing on parallel corpora and monolingual corpora respectively include:

[0012] The parallel corpus and monolingual corpus were segmented using word segmentation tools respectively;

[0013] The Byte-Pair Encoding (BPE) tool was used to perform sub-word segmentation on the segmented parallel and monolingual corpora to obtain the segmented training corpora.

[0014] The training corpus is analyzed using word frequency to obtain word lists for the source and target sides, respectively.

[0015] Preferably, in the above-mentioned machine translation style transfer method, the step of pre-training the machine translation model using training corpus and the corresponding model task includes:

[0016] With the goal of reconstructing noisy text, the machine translation model was pre-trained using target style monolingual data and a masked language model task to obtain the first pre-trained model.

[0017] The first pre-trained model is pre-trained using text style transfer data and a style transfer language model task to obtain the second pre-trained model.

[0018] With the training objective of predicting the complete target sentence, the second pre-trained model was pre-trained using translation parallel data and sentence-level masking language model tasks to obtain the third pre-trained model;

[0019] The third pre-trained model was pre-trained using translation parallel data and a style lexical masking language model task to obtain the fourth pre-trained model.

[0020] Preferably, in the above-mentioned machine translation style transfer method, the steps of pre-training the machine translation model using target style monolingual data and a masked language model task, with the reconstruction of noisy text as the training objective, include:

[0021] Noise is added to the target style text containing target style monolingual data to obtain the target noisy text;

[0022] The target noisy text is input into the encoder of the machine translation model to obtain a semantic representation vector;

[0023] Based on the reconstruction loss function, the semantic representation vector is decoded using the decoder of the machine translation model to obtain the original target style text.

[0024] Preferably, in the above-mentioned machine translation style transfer method, the step of pre-training the second pre-trained model using translation parallel data and sentence-level masking language model tasks includes:

[0025] Noise is added to the parallel translation data to obtain the translated noisy text;

[0026] The noisy text is translated and fed into the encoder of the second pre-trained model;

[0027] The complete target sentence is predicted using the decoder of the second pre-trained model according to the preset loss function.

[0028] Preferably, in the above-mentioned machine translation style transfer method, the step of constructing pseudo-parallel data using a style vocabulary alignment algorithm includes:

[0029] A style classification model is trained using text style transfer data, and style words are selected using the style classification model.

[0030] The source-target aligned dictionary is used to filter style terms to obtain the source style term-target style term aligned dictionary;

[0031] Pseudo-parallel data is obtained by matching and replacing words in the target sentence of the translated parallel data using source-target style word alignment dictionary.

[0032] Preferably, in the above-mentioned machine translation style transfer method, the step of pre-training the course-trained machine translation model using pseudo-parallel data includes:

[0033] The fourth pre-trained model is trained using pseudo-parallel data and style lexical masking language modeling tasks, resulting in the final pre-trained model.

[0034] Preferably, in the above-mentioned machine translation style transfer method, the step of fine-tuning the translation style of the final pre-trained model includes:

[0035] Obtain parallel data on translation style transfer;

[0036] Parallel data on translation style transfer is input into the final pre-trained model, and the translation style of the final pre-trained model is fine-tuned to obtain the translation style transfer model.

[0037] According to a second aspect of the present invention, the present invention also provides a machine translation style transfer system based on curriculum-based pre-training, comprising:

[0038] The word segmentation module is used to segment parallel corpora and monolingual corpora separately to obtain training corpora.

[0039] The course pre-training module is used to perform course-based pre-training of the machine translation model using training corpus and the corresponding model tasks.

[0040] The pseudo-parallel data training module is used to construct pseudo-parallel data using a style vocabulary alignment algorithm. The pseudo-parallel data is then used to pre-train the course-trained machine translation model to obtain the final pre-trained model.

[0041] The style fine-tuning module is used to fine-tune the translation style of the final pre-trained model to obtain a translation style transfer model.

[0042] Preferably, in the above-mentioned machine translation style transfer system, the course pre-training module includes:

[0043] The target style pre-training module is used to pre-train the machine translation model with the training objective of reconstructing noisy text, using target style monolingual data and masked language model task to obtain the first pre-trained model.

[0044] The text style pre-training module is used to pre-train the first pre-trained model using text style transfer data and style transfer language model task to obtain the second pre-trained model.

[0045] The translation data pre-training module is used to pre-train the second pre-trained model with the training objective of predicting the complete target sentence, and to obtain the third pre-trained model by using translation parallel data and sentence-level masked language model tasks.

[0046] The translation style pre-training module is used to pre-train the third pre-trained model using translation parallel data and a style lexical mask language model task, resulting in the fourth pre-trained model.

[0047] In summary, the machine translation style transfer scheme based on curriculum-based pre-training provided by the present invention obtains training data for training neural networks by segmenting parallel and monolingual corpora separately. Then, using this training data and corresponding model tasks, the machine translation model is pre-trained using curriculum-based methods. Because this curriculum-based training process trains the target style, text style, and translation style of the machine translation model, it reduces the difficulty of style transfer. Furthermore, a pseudo-parallel data is constructed using a style-vocabulary alignment algorithm, and this pseudo-parallel data is used to pre-train the curriculum-trained machine translation model, resulting in a final pre-trained model that adjusts the translation style of the machine translation model. Finally, fine-tuning the translation style of this final pre-trained model yields the final translation style transfer model. In conclusion, by utilizing the aforementioned training data, including target style monolingual data, text style transfer data, and general translation parallel data, the target style, text style, and translation style of the model can be trained, thereby achieving the goal of translation style transfer in the machine translation model and enabling the model to learn more semantic and stylistic features. In summary, the above methods can solve the problem that existing technologies do not focus on sentence style features, making it difficult to achieve text style transfer. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of the structure of a curriculum-based pre-trained model for machine translation provided in an embodiment of the present invention;

[0050] Figure 2 This is a flowchart illustrating a machine translation style transfer method based on curriculum-based pre-training provided in an embodiment of the present invention.

[0051] Figure 3 yes Figure 2 The illustrated embodiment provides a flowchart of a word segmentation processing method;

[0052] Figure 4 yes Figure 2 The illustrated embodiment provides a flowchart of a course-based pre-training method;

[0053] Figure 5 yes Figure 4 The illustrated embodiment provides a flowchart of a machine translation model pre-training method;

[0054] Figure 6 yes Figure 4 The illustrated embodiment provides a flowchart of a pre-training method for a second pre-trained model.

[0055] Figure 7 yes Figure 2 The illustrated embodiment provides a flowchart of a method for constructing pseudo-parallel data;

[0056] Figure 8 yes Figure 2 The illustrated embodiment provides a flowchart of a method for fine-tuning translation style;

[0057] Figure 9 This is a schematic diagram of the structure of a machine translation style transfer system based on curriculum-based pre-training provided in an embodiment of the present invention;

[0058] Figure 10 yes Figure 9 The illustrated embodiment provides a structural diagram of a course pre-training module.

[0059] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0060] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0061] The main technical problem solved by the embodiments of the present invention is:

[0062] Currently, there are two main approaches to implementing translation style transfer: one is a pivot-based method, where the translation model and the text style transfer model decode in two consecutive steps to obtain the target style translation. However, multi-step decoding leads to error propagation and excessive decoding time. The other approach utilizes knowledge distillation or back translation to construct pseudo-parallel corpora. However, these pseudo-parallel corpora contain noise, which affects model performance and reduces the accuracy of style transfer. Current pre-training tasks focus only on understanding the semantic information of sentences, neglecting the stylistic features, which are crucial for achieving text style transfer.

[0063] To address the aforementioned issues, the following embodiments of the present invention provide a machine translation style transfer scheme based on curriculum-based pre-training. This scheme utilizes large-scale monolingual and parallel corpora—i.e., general translation data—to reduce the training difficulty of the translation model, enabling it to extract the semantic and stylistic features required for style transfer. Simultaneously, it accelerates the model's convergence speed, reduces the difficulty of style transfer in machine translation, and ultimately improves the style transfer performance of machine translation. Furthermore, obtaining monolingual data of the target style is not difficult and contains rich style information. Therefore, pre-training with the monolingual data of the target style can be fully utilized to obtain better initialization parameters, thereby alleviating the problem of insufficient training data.

[0064] To achieve the above objectives, please refer to Figure 2 , Figure 2 A flowchart illustrating a machine translation style transfer method based on curriculum-based pre-training, as provided in this embodiment of the invention, is shown below. Figure 2 As shown, this method for transferring machine translation styles based on curriculum-based pre-training includes:

[0065] S110: Parallel and monolingual corpora are segmented separately to obtain training corpora. Monolingual data of the target style is relatively easy to obtain and contains rich style information. Therefore, the target style monolingual data can be fully utilized for pre-training to obtain better initialization parameters, thereby alleviating the problem of insufficient training data. The training corpora obtained after segmentation preprocessing include target style monolingual data, text style transfer data, and translation parallel data. Utilizing target style monolingual data, text style transfer data, and general translation parallel data to improve the results of translation style transfer can reduce the cost of manually annotating translation data.

[0066] As a preferred embodiment, such as Figure 3 As shown, the specific steps for segmenting parallel and monolingual corpora include:

[0067] S111: Use word segmentation tools to segment parallel corpora and monolingual corpora respectively.

[0068] S112: Use the Byte-Pair Encoding (BPE) tool to perform sub-word segmentation on the segmented parallel corpus and monolingual corpus to obtain the segmented training corpus.

[0069] For example, Jieba segmentation tool is used for Chinese, and Mose segmentation tool is used for English. Then, BPE is used to perform sub-word segmentation on the processed parallel and monolingual corpora.

[0070] S113: Use word frequency analysis to parse the training corpus and obtain the word lists for the source and target sides respectively.

[0071] The source and target vocabularies are obtained based on word frequency. The source vocabulary uses the source language and the target language vocabulary, while the target vocabulary uses the target language vocabulary. Words outside the vocabulary are replaced with UNK to ensure that the sentence length remains unchanged.

[0072] Training data is obtained by segmenting parallel and monolingual corpora separately. Since monolingual data of the target style is not difficult to obtain and contains rich style information, it can be fully utilized for pre-training to obtain better initial parameters, thereby alleviating the problem of insufficient training data.

[0073] S120: Using the training corpus and the corresponding model task, the machine translation model is pre-trained using a curriculum. Because this curriculum-based pre-training process trains the machine translation model in the target style, text style, and translation style, it reduces the difficulty of style transfer in the machine translation model, achieving the goal of style transfer and enabling the model to learn more semantic and stylistic features.

[0074] As a preferred embodiment, such as Figure 4 As shown, the steps for pre-training a machine translation model using training corpora and the corresponding model tasks include:

[0075] S121: With the goal of reconstructing noisy text, the machine translation model is pre-trained using target style monolingual data and masked language model task to obtain the first pre-trained model.

[0076] Based on target style monolingual data and masked language model training tasks, the machine translation model can be pre-trained by reconstructing noisy text with the training objective of adding noise to the target style. This enables the first pre-trained model to have a strong modeling ability for target style text.

[0077] As a preferred embodiment, such as Figure 5 As shown, the steps for pre-training a machine translation model using target-style monolingual data and a masked language model task, with the training objective of reconstructing noisy text, include:

[0078] S1211: Add noise to the target style text containing target style monolingual data to obtain the target noisy text.

[0079] In this embodiment of the application, since the masked language model task is used for training, noise can be added to the target style text in four ways: character masking, word masking, word deletion, and word insertion, so as to obtain the target style text with noise, namely the above-mentioned target noisy text.

[0080] S1212: Input the target noisy text into the encoder of the machine translation model to obtain the semantic representation vector.

[0081] Machine translation models, also known as neural machine translation models, employ an encoder-decoder framework. The target noisy text is input into the encoder, which then encodes the noisy text into a semantic vector representation.

[0082] S1213: According to the reconstruction loss function, the decoder of the machine translation model decodes the semantic representation vector to obtain the original target-style text. The decoder can generate the complete original text based on the semantic vector, thereby enhancing the model's ability to model target-style text. The reconstruction loss function is as follows: in, Let x represent the target style single sentence after adding noise information, θ represent the original sentence, θ represent the parameters of the pre-trained model, and L(θ;X) represent the reconstruction loss function.

[0083] After obtaining the target noise text in step S121 above... Figure 4 The method shown also includes the following steps:

[0084] S122: Use text style transfer data and style transfer language model task to pre-train the first pre-trained model to obtain the second pre-trained model.

[0085] Pre-training is performed using text style transfer data, namely text style transfer parallel sentence pairs and style transfer masking tasks. First, noise information is added to the source style sentences of the text style transfer sentence pairs. The method of introducing noise is the same as in step S121 above. Then, the model predicts the complete target style sentence based on the source style sentence with noise, so that the model has a certain text style transfer capability.

[0086] S123: Using the prediction of the complete target sentence as the training objective, the second pre-trained model is pre-trained using translation parallel data and sentence-level masking language model tasks to obtain the third pre-trained model.

[0087] As a preferred embodiment, such as Figure 6 As shown, the steps for pre-training the second pre-trained model using parallel translation data and sentence-level masking language modeling tasks include:

[0088] S1231: Add noise to the translated parallel data to obtain the translated noisy text.

[0089] S1232: Input the translated noisy text into the encoder of the second pre-trained model.

[0090] S1233: The complete target sentence is predicted using the decoder of the second pre-trained model according to the preset loss function.

[0091] In the technical solution provided in this application embodiment, parallel sentence pairs of translation style transfer are concatenated, and then noise is introduced into the concatenated sentence and used as input data for the encoder. The decoder then predicts the complete target sentence with partial source and target sentence information, thereby reducing the training difficulty of the model. The calculation formula of the above-mentioned preset loss function is as follows:

[0092]

[0093] in, This indicates the source sentence after noise has been introduced. This represents the target sentence after noise has been introduced. Indicates the sentence and sentences The concatenated sentence is θ, where θ is the model parameters of the pre-trained model, M is the total number of sentences, and L is the loss function.

[0094] S124: The third pre-trained model is pre-trained using translation parallel data and a style lexical masking language model task to obtain the fourth pre-trained model.

[0095] Furthermore, the style words of the source sentence in the translation style transfer parallel sentence pair are masked, and the model is trained to predict the complete target style translation based on contextual information, thereby specifically enhancing the model's style transfer and translation capabilities for style words.

[0096] By pre-training the machine translation model using the above-mentioned steps, including training the model with target style monolingual data, text style transfer data, and translation parallel data respectively, the machine translation model's ability to recognize and transfer target, style text style, and translation style can be improved, thus reducing the difficulty of translation.

[0097] S130: A pseudo-parallel data is constructed using a style vocabulary alignment algorithm. This pseudo-parallel data is then used to pre-train the course-trained machine translation model to obtain the final pre-trained model. In this embodiment, words with significant influence are selected as style words. Then, an alignment dictionary is constructed using the style vocabulary alignment algorithm. This alignment dictionary is used to match and replace sentences in the target sentence. The sentences that replace the target sentence are selected as pseudo-parallel sentence pairs, thereby introducing the target style into the general translation corpus.

[0098] Specifically, the steps for pre-training the course-trained machine translation model using pseudo-parallel data include: training a fourth pre-trained model using pseudo-parallel data and a style lexical mask language model task to obtain the final pre-trained model.

[0099] As a preferred embodiment, such as Figure 7As shown, the steps for constructing pseudo-parallel data using the style vocabulary alignment algorithm include:

[0100] S131: A style classification model is trained using text style transfer data, and style words are selected using the style classification model. In this embodiment, a style classification model is trained based on text style transfer data, using the pre-trained model BERT as the model framework. The attention scores of each attention head in the highest layer of the pre-trained model BERT are used to measure the influence of each word on the style, and words with greater influence are selected as style words. The relevant calculation formula is as follows:

[0101]

[0102]

[0103] Where, α h (w) represents the degree of influence of word w on style, Q and k represent query and key respectively, k represents the number of multi-head attention heads, and score is the final score of word w.

[0104] S132: Use the source-target aligned dictionary to filter style terms and obtain the source style term-target style term aligned dictionary.

[0105] S133: Use source style words - target style words alignment dictionary matching and replacement of words in the target sentence of the translated parallel data to obtain pseudo-parallel data.

[0106] After obtaining the aforementioned style vocabulary, the fast-align tool is used to align the source and target sentences of the text style transfer corpus, resulting in an alignment dictionary. Then, based on the style vocabulary filtering method from the previous step, an alignment dictionary of source style words and target style words is obtained. Finally, the alignment dictionary is used to match the target sentences in the general corpus. Successful matches are replaced with words of the target style. If the replacement ratio exceeds a certain scale, these are selected as pseudo-parallel sentence pairs, thus introducing the target style into the translation corpus within the aforementioned training corpus.

[0107] After pre-training the machine translation model using pseudo-parallel data to obtain the final pre-trained model... Figure 2 The machine translation style transfer method shown also includes the following steps:

[0108] S140: Fine-tune the translation style of the final pre-trained model to obtain the translation style transfer model. The fine-tuning method provided in this embodiment uses parallel translation style transfer data to fine-tune the pre-trained model, thereby obtaining the final translation style transfer model.

[0109] As a preferred embodiment, such as Figure 8As shown, the steps for fine-tuning the translation style of the final pre-trained model include:

[0110] S141: Obtain parallel data on translation style transfer.

[0111] S142: Input parallel translation style transfer data into the final pre-trained model, fine-tune the translation style of the final pre-trained model, and obtain the translation style transfer model.

[0112] By acquiring parallel translation style transfer data and then inputting this parallel data into the final training model, the accuracy of the model's translation style transfer can be improved, thus obtaining the final translation style transfer model.

[0113] Combination Figure 1 As shown in the transfer model, the above embodiments of the present invention, based on the idea of ​​curriculum learning, design four pre-training tasks with progressively increasing difficulty, including masked language modeling, style transfer masking modeling, sentence-level masked language modeling, and style word masking language modeling. This allows the model to gradually learn translation knowledge and style transfer knowledge, thereby effectively reducing the training difficulty of the machine translation style transfer model. The above method can fully utilize readily available target style monolingual data and general machine translation data, thus greatly reducing manual annotation costs and improving translation efficiency. Furthermore, the style word alignment data enhancement method proposed in the above embodiments of the present invention, by explicitly introducing target style information into large-scale general translation parallel sentence pairs to construct pseudo-parallel sentence pairs, can alleviate the problem of insufficient training data in translation style transfer tasks. Finally, the present invention combines curriculum learning with pre-training, enabling the model to extract all the features required for translation style transfer. Simultaneously, the enhanced pseudo-parallel data can be used in the pre-training process, effectively improving the performance of the translation style transfer model.

[0114] In summary, the machine translation style transfer method based on curriculum-based pre-training provided in this invention obtains training data for training neural networks by segmenting parallel and monolingual corpora separately. Then, using this training data and corresponding model tasks, the machine translation model is pre-trained using curriculum-based methods. Because this curriculum-based training process trains the target style, text style, and translation style of the machine translation model, it reduces the difficulty of style transfer. Furthermore, a pseudo-parallel data is constructed using a style-vocabulary alignment algorithm, and this pseudo-parallel data is used to pre-train the curriculum-trained machine translation model, resulting in a final pre-trained model that adjusts the translation style of the machine translation model. Finally, fine-tuning the translation style of this final pre-trained model yields the final translation style transfer model. In summary, by utilizing the aforementioned training data, including target style monolingual data, text style transfer data, and general translation parallel data, the model's target style, text style, and translation style can be trained, thereby achieving the goal of translation style transfer in the machine translation model and enabling the model to learn more semantic and stylistic features. In conclusion, this method solves the problem in existing technologies that do not focus on sentence style features, making text style transfer difficult.

[0115] Furthermore, to verify the effectiveness of this method, this invention applies a course-based pre-trained machine translation style transfer method, as shown in Table 1, to the Modern-Classical dataset for a translation style transfer task, with the style transfer direction being from modern Chinese to classical Chinese. This paper first performs course-based pre-training on monolingual classical Chinese data and parallel translation corpora to reduce the training difficulty of the model. Then, it uses translation style transfer data to fine-tune the pre-trained model to obtain the final translation style transfer model. Experimental results show that the method proposed in this invention can significantly improve the performance of machine translation style transfer, achieving a 3.9-point improvement in the BLEU score, while also achieving good style transfer accuracy.

[0116] Table 1 Comparison of Experimental Results on Style Transfer in Different Neural Machine Translations

[0117]

[0118] In addition, based on the same concept of the above method embodiments, the present invention also proposes a machine translation style transfer system based on curriculum-based pre-training to implement the above method of the present invention. Since the principle and method of solving the problem in this system embodiment are similar, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, and will not be described in detail here.

[0119] See Figure 9 , Figure 9This is a schematic diagram illustrating the structure of a machine translation style transfer system based on curriculum-based pre-training, provided as an embodiment of the present invention. Figure 9 As shown, this machine translation style transfer system includes:

[0120] The word segmentation module 110 is used to perform word segmentation on parallel corpora and monolingual corpora to obtain training corpora.

[0121] The course pre-training module 120 is used to perform course-based pre-training of the machine translation model using training corpus and the corresponding model tasks.

[0122] The pseudo-parallel data training module 130 is used to construct pseudo-parallel data using a style vocabulary alignment algorithm, and to pre-train the course-trained machine translation model using the pseudo-parallel data to obtain the final pre-trained model.

[0123] The style fine-tuning module 140 is used to fine-tune the translation style of the final pre-trained model to obtain the translation style transfer model.

[0124] In summary, the machine translation style transfer system based on curriculum-based pre-training provided in this invention obtains training data for training the neural network by segmenting parallel and monolingual corpora separately. Then, using this training data and corresponding model tasks, the machine translation model is pre-trained using curriculum-based methods. Because this curriculum-based training process trains the target style, text style, and translation style of the machine translation model, it reduces the difficulty of style transfer. Furthermore, a pseudo-parallel data is constructed using a style-vocabulary alignment algorithm, and this pseudo-parallel data is used to pre-train the curriculum-trained machine translation model, resulting in a final pre-trained model that adjusts the translation style of the machine translation model. Finally, fine-tuning the translation style of this final pre-trained model yields the final translation style transfer model. In summary, by utilizing the aforementioned training data, including target style monolingual data, text style transfer data, and general translation parallel data, the model's target style, text style, and translation style can be trained, thereby achieving the goal of translation style transfer in the machine translation model and enabling the model to learn more semantic and stylistic features. In conclusion, this method solves the problem in existing technologies that do not focus on sentence style features, making text style transfer difficult.

[0125] As a preferred embodiment, such as Figure 10 As shown, in the above machine translation style transfer system, the course pre-training module 120 includes:

[0126] The target style pre-training module 121 is used to pre-train the machine translation model with target style monolingual data and masked language model task with the training objective of reconstructing noisy text, so as to obtain the first pre-trained model.

[0127] The text style pre-training module 122 is used to pre-train the first pre-trained model using text style transfer data and a style transfer language model task to obtain the second pre-trained model.

[0128] The translation data pre-training module 123 is used to pre-train the second pre-trained model with the training objective of predicting the complete target sentence, using translation parallel data and sentence-level masking language model tasks to obtain the third pre-trained model.

[0129] Translation style pre-training module 124 is used to pre-train the third pre-trained model using translation parallel data and style lexical masking language model task to obtain the fourth pre-trained model.

[0130] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0135] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0136] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for transferring machine translation style based on curriculum-based pre-training, characterized in that, include: The parallel corpus and the monolingual corpus were segmented separately to obtain the training corpus; Using the training corpus and the model task corresponding to the training corpus, the machine translation model is pre-trained in a curriculum manner. The specific steps include: taking the reconstruction of noisy text as the training objective, using target style monolingual data and masked language model task to pre-train the machine translation model to obtain a first pre-trained model. The first pre-trained model is pre-trained using text style transfer data and a style transfer language model task to obtain the second pre-trained model. With the training objective of predicting the complete target sentence, the second pre-trained model is pre-trained using translation parallel data and sentence-level masking language model tasks to obtain the third pre-trained model; The third pre-trained model is pre-trained using translation parallel data and a style vocabulary masking language model task to obtain the fourth pre-trained model. Pseudo-parallel data is constructed using a style vocabulary alignment algorithm, and the pseudo-parallel data is used to pre-train the machine translation model after curriculum training to obtain the final pre-trained model. The steps of pre-training the course-trained machine translation model using pseudo-parallel data include: The fourth pre-trained model is trained using the pseudo-parallel data and the style lexical mask language model task to obtain the final pre-trained model. The translation style is fine-tuned on the final pre-trained model to obtain the translation style transfer model.

2. The machine translation style transfer method according to claim 1, characterized in that, The steps of performing word segmentation on parallel corpora and monolingual corpora respectively include: The parallel corpus and monolingual corpus were segmented using a word segmentation tool respectively. The Byte-Pair Encoding (BPE) tool was used to perform sub-word segmentation on the segmented parallel corpus and monolingual corpus to obtain the segmented training corpus. The training corpus is analyzed using word frequency analysis to obtain word lists for the source and target sides, respectively.

3. The machine translation style transfer method according to claim 1, characterized in that, The step of pre-training the machine translation model using target-style monolingual data and a masked language model task, with the goal of reconstructing noisy text, includes: Noise is added to the target style text containing target style monolingual data to obtain the target noisy text; The target noisy text is input into the encoder of the machine translation model to obtain a semantic representation vector; According to the reconstruction loss function, the semantic representation vector is decoded using the decoder of the machine translation model to obtain the original target style text.

4. The machine translation style transfer method according to claim 1, characterized in that, The step of pre-training the second pre-trained model using translation parallel data and sentence-level masked language modeling tasks includes: Noise is added to the translated parallel data to obtain translated noisy text; The translated noisy text is input into the encoder of the second pre-trained model; The complete target sentence is predicted using the decoder of the second pre-trained model according to the preset loss function.

5. The machine translation style transfer method according to claim 1, characterized in that, The step of constructing pseudo-parallel data using the style vocabulary alignment algorithm includes: A style classification model is trained using the text style transfer data, and style vocabulary is selected using the style classification model. The style terms are filtered using a source-target aligned dictionary to obtain a source style term-target style term aligned dictionary; The pseudo-parallel data is obtained by matching and replacing words in the target sentence of the translated parallel data using the source style word-target style word alignment dictionary.

6. The machine translation style transfer method according to claim 1, characterized in that, The steps for fine-tuning the translation style of the final pre-trained model include: Obtain parallel data on translation style transfer; The translation style transfer parallel data is input into the final pre-trained model, and the translation style of the final pre-trained model is fine-tuned to obtain the translation style transfer model.

7. A machine translation style transfer system based on curriculum-based pre-training, characterized in that, include: The word segmentation module is used to segment parallel corpora and monolingual corpora separately to obtain training corpora. The course pre-training module is used to perform course-based pre-training of the machine translation model using the training corpus and the model task corresponding to the training corpus. The course pre-training module includes: The target style pre-training module is used to pre-train the machine translation model with target style monolingual data and masked language model task, with the goal of reconstructing noisy text, to obtain the first pre-trained model. The text style pre-training module is used to pre-train the first pre-trained model using text style transfer data and style transfer language model task to obtain the second pre-trained model. The translation data pre-training module is used to pre-train the second pre-trained model with the training objective of predicting the complete target sentence, using translation parallel data and sentence-level masked language model tasks to obtain the third pre-trained model. The translation style pre-training module is used to pre-train the third pre-trained model using translation parallel data and a style lexical mask language model task to obtain a fourth pre-trained model. The pseudo-parallel data training module is used to construct pseudo-parallel data using a style vocabulary alignment algorithm, and to pre-train the machine translation model after curriculum training using the pseudo-parallel data to obtain the final pre-trained model. The steps of pre-training the course-trained machine translation model using pseudo-parallel data include: The fourth pre-trained model is trained using the pseudo-parallel data and the style lexical mask language model task to obtain the final pre-trained model. The style fine-tuning module is used to fine-tune the translation style of the final pre-trained model to obtain a translation style transfer model.

Images