A method for constructing a machine translation model, a translation device and a computer-readable storage medium

Abstract

The present invention discloses a method for constructing a machine translation model, comprising: constructing a basic machine translation model for a general field; and fine-tuning the basic machine translation model to a specific field to construct a machine translation model for the specific field. The present invention also discloses a corresponding translation device and a computer-readable storage medium. Through the fine-tuning mechanism, the comprehensive improvement of the translation effect and the development time of the translation model is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to but are not limited to the field of machine translation technology, and in particular to a method for constructing a machine translation model, a translation device, and a computer-readable storage medium. Background Art

[0002] Machine translation, also known as automatic translation, is the process of using computers to convert one natural language (source language) into another natural language (target language). It is a branch of computational linguistics and one of the ultimate goals of artificial intelligence, and has important scientific research value. At the same time, machine translation has important practical value. With the rapid development of economic globalization and the Internet, machine translation technology plays an increasingly important role in promoting political, economic, and cultural exchanges.

[0003] Machine translation has important practical value. Compared with manual translation, machine translation has three obvious advantages: low cost, easy control and fast translation speed. The development of machine translation technology has been closely following the development of computer technology, information theory, linguistics and other disciplines. From early dictionary matching, to rule translation combining dictionary with linguistic expert knowledge, to corpus-based statistical machine translation.

[0004] Currently popular translation systems are mainly for general field translation, but they are not very effective in many scenarios in some proprietary fields, such as the medical field, the financial field, and the legal compliance field. Redeveloping a translation system for a proprietary field requires a lot of manpower and material resources. Therefore, a translation system suitable for a proprietary field that not only has good translation effects but also high development efficiency is needed. Summary of the invention

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] On the one hand, an embodiment of the present invention provides a method for constructing a machine translation model by fine-tuning to improve development efficiency and translation effect. The method includes:

[0007] Build basic machine translation models for general domains;

[0008] Based on the basic machine translation model, fine-tuning is performed towards a specific domain to construct a machine translation model for the specific domain.

[0009] On the other hand, an embodiment of the present invention provides a translation device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for constructing a machine translation model when executing the program.

[0010] On the other hand, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the above-mentioned method for constructing a machine translation model.

[0011] The embodiment of the present invention includes: based on a general domain training data set, a basic machine translation model suitable for a general domain is obtained; using the obtained basic machine translation model, based on a specific domain training data set, the model's hyperparameters: learning rate are adjusted, and fine-tuning is performed toward a specific domain to construct a result machine translation model for a specific domain. Therefore, according to the embodiment of the present invention, the idea of ​​fine-tuning is innovatively applied to the domain adaptation migration of machine translation, achieving a comprehensive improvement in translation effect and translation model development time. At the same time, the relevant parameters of the classic converter architecture are innovatively adjusted to achieve an improvement in the overall effect.

[0012] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation to the technical solution of the present invention.

[0014] Figure 1 is an overall flow chart of a method for constructing a machine translation model using fine-tuning according to an embodiment of the present invention;

[0015] Figure 2 is a flowchart of data preprocessing in a method for constructing a machine translation model by fine-tuning according to an embodiment of the present invention;

[0016] Figure 3 is a flowchart of a keyword protection process in a method for constructing a machine translation model by fine-tuning according to an embodiment of the present invention;

[0017] Figure 4 is a flowchart of vocabulary processing in a method for constructing a machine translation model by fine-tuning according to an embodiment of the present invention;

[0018] Figure 5 is a flowchart of training a basic machine translation model in a method for constructing a machine translation model by fine-tuning according to an embodiment of the present invention;

[0019] Figure 6It is a flowchart of a method for constructing a machine translation model by fine-tuning based on a basic model to a specific field so as to construct a machine translation model suitable for a specific field in one embodiment of the present invention;

[0020] Figure 7 Another embodiment of the present invention adopts the overall architecture of a machine translation model suitable for a specific field by fine-tuning the converter architecture, and shows the overall model design and training process in the system architecture;

[0021] Figure 8 yes Figure 7 A structural diagram of an implementation of a first data preprocessing unit of a machine translation model of an embodiment, wherein the preprocessing process of training data is shown with arrows;

[0022] Fig. 9 yes Figure 7 A flowchart of an implementation method of keyword protection in corpus processing of a machine translation model of an embodiment mainly shows the processing flow of training data for implementing keyword protection;

[0023] Fig.10 yes Figure 7 A schematic diagram of an implementation of a transformer architecture for a machine translation model of an embodiment, wherein arrows are used to illustrate the training steps through the architecture;

[0024] Fig.11 yes Figure 7 A schematic diagram of an implementation of a fine-tuning portion of a machine translation model of an embodiment, wherein the flow of arrows is used to illustrate the process of fine-tuning machine translation from a general domain to a specific domain;

[0025] Fig.12 is a schematic diagram of constructing a machine translation model architecture suitable for the telecommunications field by fine-tuning using a converter architecture according to another embodiment of the present invention;

[0026] Fig.13 yes Fig.12 A schematic diagram of the implementation process of the machine translation model of the embodiment;

[0027] Fig.14 is a schematic diagram of constructing a machine translation model architecture suitable for the legal field through fine-tuning using a converter architecture according to another embodiment of the present invention;

[0028] Fig.15 yes Fig.14 A schematic diagram of the implementation process of the machine translation model of the embodiment;

[0029] Fig.16 It is a structural block diagram of a machine translation model according to another embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0031] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0032] After years of development, the translation quality of neural machine translation (NMT) technology based on deep learning has been greatly improved compared with the previous statistical machine translation system. In recent years, a deep learning machine translation system based on the attention mechanism and the transformer architecture has been proposed, which has improved the accuracy and fluency of machine translation by one step. In June 2017, Google published an article titled "Attention is all you need" on Arxiv, which proposed a transformer model to solve the sequence to sequence problem, replacing the long short-term memory (LSTM) with the full attention structure, thereby abandoning the traditional encoder-decoder model that must be combined with the convolutional neural network (CNN) or the recurrent neural network (RNN). This model has made unprecedented breakthroughs in machine translation and brought neural network machine translation into a new era.

[0033] At present, several existing machine translation systems on the market have been recognized by users. Although they have not yet completely replaced human translation in terms of translation quality, it is undeniable that they have greatly improved the reference value of machine translation. However, the neural network models or neural networks of these machine translation systems are mainly trained and applied for general fields, and are not targeted at specific professional fields, such as communications, medical care, finance, law and other professional fields. Therefore, in many translation scenarios in some specific professional fields, the effect is not very good. For example, the deviation from the manual translation results in a specific field is greater than the deviation from the manual translation results in the general field.

[0034] Based on this, the present invention provides a method for constructing a machine translation model, a translation device, and a computer-readable storage medium. By collecting general corpus and training it using a converter architecture, a basic machine translation model under the converter architecture is obtained, and then based on the basic machine translation model, fine-tuning is performed toward a specific field to construct a machine translation model for a specific field. Through the embodiments of the present invention described below, it can be clearly understood that fine-tuning is performed toward a specific professional field on a pre-trained translation model based on a general field, so as to realize a fast and effective design of a specific field machine translation system.

[0035] The following is combined with Figure 1-6 , the embodiments of the present invention are further explained from the perspective of method flow.

[0036] Reference Figure 1 As shown, a method for constructing a machine translation device using fine-tuning according to an embodiment of the present invention includes but is not limited to: step S101, constructing a basic machine translation model for a general field; step S102, based on the basic machine translation model, fine-tuning to a specific field to construct a machine translation model for the specific field.

[0037] In fact, fine-tuning is a means of implementing transfer learning. Transfer learning is a machine learning method that uses the model developed for task A as the starting point and reuses it in the process of developing a model for task B. In fact, it is the influence of one kind of learned knowledge on the learning of another kind of knowledge. Transfer learning can be regarded as a learning system. The core issue is how to apply the "knowledge" learned previously to other fields. By fine-tuning the machine translation model for the general field to a specific field, rather than redeveloping it, a fast and effective machine translation system design for a specific field can be achieved.

[0038] The model according to the embodiment of the present invention includes a neural network model, which can also be simply referred to as a neural network, or even more simply referred to as a network in this article.

[0039] Reference Figure 2 As shown, the data preprocessing process in the method of constructing a machine translation device using fine-tuning according to an embodiment of the present invention includes but is not limited to: step S201, preparing a training data set, wherein the training data set includes corpus pairs; step S202, preprocessing the corpus pairs in the training data set.

[0040] In step S201, for obtaining better training effect, i.e. better model convergence, when building the basic machine translation model of general domain, first need to prepare a large amount of bidirectional corpus texts of general domain as training data set, for example English-Chinese bidirectional corpus, wherein contain a large amount of corpus pairs. Corpus quantity level can reach ten million level, for example about 40 million. Specifically, these bidirectional corpus can be stored in two texts respectively, for example, be stored in an English text respectively, and in another Chinese text. Each pair of corpus is respectively located in the same row in two texts, corresponding to a pair of English-Chinese parallel translation data. As described below, when building the machine translation model for specific domain, similarly need to prepare the bidirectional corpus text of specific domain as training data set.

[0041] After the data is acquired in step S201, in step S202, a series of processing may be performed on the model structure, that is, on the network structure, including but not limited to:

[0042] (1) Punctuation regression:

[0043] The punctuation regression here can be performed for the steps where specific punctuation marks in some training data sets are replaced by other string combinations: for example, in the Cwmt (China Workshop on Machine Translation) data set, the single quotation mark "'" is replaced by the string "'", the double quotation mark '"' is replaced by '";', etc. In this step, these strings are restored to the corresponding punctuation marks.

[0044] (2) Unification of punctuation and special symbols:

[0045] In order to solve the problem of inconsistent punctuation and special symbols in various corpora, normalization processing can be performed, including: converting full-width characters to half-width characters; unifying punctuation marks; for paired punctuation marks such as book title marks, brackets (large, medium and small), and double quotation marks, if there is a punctuation missing problem, delete the remaining half (note: single quotation marks are not included); modify single quotation marks that are mistakenly written as single double quotation marks, for example, it's is mistakenly written as it"s.

[0046] (3) Unification of simplified and traditional Chinese characters:

[0047] Convert all traditional Chinese characters in the corpus to simplified Chinese.

[0048] (4) Sentence start processing:

[0049] For some meaningless symbols such as '! @#&*:;,. / ' that appear at the beginning of sentences in some corpora, delete them.

[0050] (5) Delete the "dirty sentences" that are still confusing after processing:

[0051] After processing the data according to the above steps, if it is found that part of the corpus is still very chaotic (according to empirical statistics, it generally accounts for about 0.02% of the corpus), for example: the overall length of a sentence is 30-40 characters, and more than 10 of them are garbled symbols '@#&', and after removing these garbled characters, a complete sentence cannot be formed, then the entire corpus will be directly deleted from the corpus.

[0052] (6) Deduplication:

[0053] Since some corpora overlap each other, the corpora after the above processing will be deduplicated.

[0054] After the above data preprocessing, the cleaned bilingual forecast can be finally obtained. Similarly, when building a machine translation model for a specific field, the collected training data for the specific field also needs to be preprocessed in the same way.

[0055] In translation, there are some words that do not need to be translated and can be directly retained, which are referred to as keywords in this article. In order to solve the problem of keyword retention, the translation device of the present invention can further learn the special word protection function. Figure 3 As shown, the keyword protection processing flow in the method of constructing a machine translation device using fine-tuning in one embodiment of the present invention includes but is not limited to: step S301, matching the character strings in the corpus pair; step S302, identifying the keywords based on the same character strings in the corpus pair; step S303, replacing the keywords with special character strings.

[0056] In step S301, for the processed bilingual corpus, such as English-Chinese bilingual corpus, the words of each sentence in the English corpus are separated by spaces. Each sentence in the Chinese corpus has also completed natural word segmentation, in which the words are also separated by spaces. The work done by matching the strings in the corpus pair is, more specifically: for each pair of corpus, the English and the corresponding Chinese are matched (the basic string unit is an English word or a Chinese natural word segmentation). For example, Chinese: Insert a SIM card into the phone; English, Insert a SIM card into the phone.

[0057] In step S302, in each pair of matched bilingual corpora, for the string part that is completely identical, it can be determined that this string part is a keyword and does not need to be translated but is directly retained.

[0058] In step S303, the character strings that are determined to be identical after matching are replaced with a special character string, for example, '&SEP1;'. If there are multiple identical character string parts in a sentence, they can be replaced in order with '&SEP2;', '&SEP3;', and so on.

[0059] The machine translation system uses a vocabulary. Since the number of neurons in the output layer of the neural network must be consistent with the length of the vocabulary, the length of the vocabulary cannot be too large, otherwise the number of neurons in the output layer will also become too large, resulting in low training efficiency, slow model convergence speed, etc. Therefore, a method is needed to reduce the length of the vocabulary.

[0060] The present invention uses byte pair encoding (BPE) technology to build a vocabulary, the main purpose of which is data compression, which can be specifically described as a layer-by-layer iterative process in which a pair of characters with the highest frequency in a string is replaced by a character that does not appear in this string. The idea of ​​BPE is to use smaller sub-word combinations to represent words with a small number of occurrences and complex structures, thereby shortening the vocabulary and optimizing the OOV (out of vocabulary) problem. In neural network machine translation, an unregistered word may refer to a word that appears in a validation set or a test set but has never been seen in a training set.

[0061] Reference Figure 4 As shown, the vocabulary processing flow in the method of constructing a machine translation device using fine-tuning according to an embodiment of the present invention includes but is not limited to: step S401, segmenting the words in the preprocessed corpus pair based on the byte pair encoding (BPE) mechanism to establish a vocabulary; step S402, segmenting the words in the corpus pair of the training data set according to the vocabulary to establish a segmented training set.

[0062] In step S401, the vocabulary created is also called a sub-word vocabulary. According to an example, the form is as follows:

[0063] …

[0064] c@@ 352824

[0065] account 351725

[0066] groups 351454

[0067] framework 351102

[0068] capac@@ 20984

[0069] rever@@ 20983

[0070] …(where @@ stands for “ity”)

[0071] It can be seen that the sub-word list contains the "@" character, indicating that the word is a "sub-word", and the number behind it is the frequency of the word. The order of the word list is determined by the frequency from top to bottom. Compared with the length of the original natural word list, the length of the sub-word word list constructed in this way is greatly shortened.

[0072] In step S402, the bilingual corpus is segmented according to the sub-word list established in step S401 to establish a segmented training data set.

[0073] According to an example, the bilingual corpus after word segmentation is in the following form:

[0074] En:The definition of Rap@@e under the Myanmar Penal Code is asfollow@@:

[0075] Zh: The Myanmar Criminal Code defines rape as follows:

[0076] En: it was a roman@@tic interest in this attractive and unfortunate young man.

[0077] Zh: This is actually a romantic interest in an unfortunate handsome boy.

[0078] En:rather than meet Bu@@c@@ks officials after the draft in New York,the7-foot(2.@@13-me@@tre)Y@@i travel@@ed to D@@all@@as where he joined theChinese national team for an exhibition game on Sunday

[0079] Zh: 2.13 meters tall Yi Jianlian did not meet with Bucks officials after being selected by the Bucks in New York, but went to Dallas to meet with the Chinese national team to prepare for Sunday's warm-up game (here only represents a logo)

[0080] After using the sub-word list to segment the bilingual corpus, it is divided into the above forms. The "@@" symbol indicates that the current part is a "sub-word" rather than a natural word.

[0081] Similarly, when constructing a machine translation model for a specific domain, the bilingual corpus pairs in the specific domain also need to be processed as in steps S401 - S402 .

[0082] The machine translation system implemented by the present invention includes a neural network model, or a neural network. According to an embodiment of the present invention, considering the current development of neural networks and the pursuit of translation quality, its network architecture adopts an advanced transformer architecture. The advancement of this architecture lies at least in that the long short-term memory (LSTM) is replaced by a full attention structure, thereby abandoning the inherent mode that the traditional encoder-decoder model must be combined with a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0083] According to an embodiment of the present invention, the construction of a basic machine translation model for a general field further includes determining parameters of a converter architecture. The determined architecture parameters include but are not limited to:

[0084] Set the number of encoder and decoder layers. According to some examples, it can be set to 6-10 layers.

[0085] Set the number of neurons in the hidden layer. According to some examples, it can be set to 512-1024;

[0086] Set the number of neurons in the output layer to be consistent with the length of the vocabulary;

[0087] According to the size of the training dataset, the first learning rate of training is set according to a certain decay period as the learning rate of the basic machine translation model.

[0088] Regarding learning rate decay, when training a neural network, the learning rate is used to control the update speed of the parameters. When the learning rate is small, the update speed of the parameters will be greatly reduced, so that the convergence speed of the model training is slow. When the learning rate is large, oscillation may occur during the update process of parameter optimization, causing the parameters to hover around the optimal value.

[0089] At a small learning rate during preheating, the model can slowly stabilize. After the model is relatively stable, the pre-set learning rate can be selected for training, making the model converge faster and the model effect better. To this end, learning rate decay is introduced during the training process so that the learning rate gradually decays as the training progresses.

[0090] In some embodiments, constructing a basic machine translation model for a general domain further includes designing a loss function using the idea of ​​minimum risk training.

[0091] After determining the parameters of the basic machine translation model for the general domain, the training of the basic model will be carried out. Figure 5 FIG. 1 is a flow chart of training a basic machine translation model in a method for constructing a machine translation device by fine-tuning according to an embodiment of the present invention, including but not limited to:

[0092] Step S501, repeatedly training the training data set using the constructed basic machine translation model until the BLEU value of the basic machine translation model on the test set and the validation set reaches a stable value;

[0093] Step S502, after the basic machine translation model converges and stabilizes, a certain number of basic machine translation models are saved and weight fusion is performed on them;

[0094] Step S503, saving the basic machine translation model after weight fusion.

[0095] Among them, the training data set is divided into training set, test set and validation set according to a certain ratio.

[0096] Bilingual Evaluation Understudy (BLEU) means evaluating translation results instead of humans. Although this indicator was invented for translation, it can be used to evaluate texts generated by a set of natural language processing tasks.

[0097] After completing the training of the basic model, you can proceed to the critical next step of fine-tuning it to a specific domain. Figure 6 As shown, the process of fine-tuning a basic model to a specific field to construct a machine translation model suitable for a specific field in a method of constructing a machine translation device by fine-tuning according to an embodiment of the present invention includes but is not limited to:

[0098] Step S601, based on the constructed basic machine translation model, adjusting the learning rate according to the specific domain to obtain a second learning rate relative to the first learning rate;

[0099] Step S602, replace the verification set in the general domain training data set used for the basic machine translation model with the corpus of the specific domain training data set to be developed, and optimize the model through the BLEU value of the translation of the specific domain corpus until the model reaches convergence in the training of the specific domain.

[0100] Before performing step S601, the corpus in the specific field needs to be preprocessed as in steps S201-S202; keyword protection processing as in steps S301-S303; and word segmentation processing as in steps S401-S402.

[0101] In step S601, the first learning rate of the base model for the general domain is adjusted to a second learning rate according to the specific domain. According to some examples, the second learning rate can be adjusted to half of the end of the cycle of the cyclical learning rate of the first learning rate. According to some examples, the second learning rate can be adjusted to 0.5-1 times the first learning rate used when the base model converges completely, for fine-tuning to the specific domain. By adjusting the learning rate, the model fine-tunes the model parameters on the training data of the specific domain.

[0102] In step S602, it is necessary to determine the convergence effect of the model, and then retain the optimal model, perform weight fusion, and obtain the final result model.

[0103] Therefore, according to the embodiments of the present invention, the idea of ​​fine-tuning is innovatively applied to the domain adaptation migration of machine translation, achieving a comprehensive improvement in translation effect and translation model development time. At the same time, the relevant parameters of the classic converter architecture are innovatively adjusted to achieve an improvement in the overall effect.

[0104] The following is combined with Figure 7-11 , the embodiments of the present invention are further explained from the perspective of system architecture.

[0105] refer to Figure 7 As shown, the architecture of a machine translation system 700 suitable for a specific domain constructed by fine-tuning according to an embodiment of the present invention includes two parts: a pre-training part 701 and a fine-tuning part 703 .

[0106] The pre-training part 701 is used for pre-training of the basic model, mainly by learning general knowledge to build a machine translation model suitable for general fields. It can further include a first data pre-processing unit 7010 and a basic model pre-training unit 7020.

[0107] The first data preprocessing unit 7010 is used to preprocess the first data so that the first data can be used for training the basic model, wherein the first data includes a training data set containing bilingual corpus applicable to a general field. The first data preprocessing unit 7010 can further include a data collection module 7011 and a data cleaning module 7012.

[0108] The data collection module 7011 is used to prepare a large amount of bidirectional corpus text in a general field as a training data set, as described above.

[0109] The data cleaning module 7012 is used to perform a series of processing on the collected first data so that the cleaned first data can be used for training the basic model.

[0110] The basic model pre-training unit 7020 is used to pre-train the basic model, and may include a sub-vocabulary creation and word segmentation module, a network external parameter setting module, and a converter architecture setting module.

[0111] The fine-tuning part 703 is used to fine-tune the basic model to a specific field. Fine-tuning refers to fine-tuning the model. The fine-tuning part may include a second data preprocessing unit 7031 and a learning rate adjustment unit 7032.

[0112] The second data preprocessing unit 7031 is used to preprocess the second data so that the second data can be used for training a result model suitable for a specific field, wherein the second data includes a training data set containing bilingual corpus suitable for the specific field. The second data preprocessing unit 7031 can similarly include a data collection module and a data cleaning module (not shown in the figure).

[0113] The learning rate adjustment unit 7032 is used to adjust the learning rate used for basic model training so that the result model suitable for a specific field can achieve good convergence and reasonable training efficiency.

[0114] refer to Figure 8 , which is a detailed schematic diagram of an implementation of the first data preprocessing unit 7010, wherein the flow of arrows is used to illustrate the process of collecting and preprocessing the first data.

[0115] Box 801 corresponds to Figure 7 The data collection module 7011, in order to obtain better training effect, that is, better model convergence, when constructing a basic machine translation model in a general field, it is first necessary to prepare a large amount of bidirectional corpus texts in the general field as training data sets, such as English-Chinese bidirectional corpus, which contains a large number of corpus pairs. The corpus quantity level can reach tens of millions, such as about 40 million. Specifically, these bidirectional corpora can be stored in two texts respectively, for example, respectively stored in an English text, and another Chinese text. Each pair of corpora is located in the same row in the two texts, corresponding to a pair of English-Chinese parallel translation data. As described below, when constructing a machine translation model for a specific field, it is similarly necessary to prepare bidirectional corpus texts in the specific field as training data sets.

[0116] Box 802 corresponds to Figure 7 The data cleaning module 7012 may include submodules:

[0117] Punctuation regression submodule 8021 is a step for replacing specific punctuation marks in some training data sets with other string combinations: for example, in the Cwmt (China Workshop on Machine Translation) data set, single quotation mark "'" is replaced by "'" string, double quotation mark '"' is replaced by '";', etc. In this step, these strings are restored to the corresponding punctuation marks.

[0118] The character unification submodule 8022 can normalize the punctuation marks and special symbols that appear in different corpora, including: converting full-width characters to half-width characters; unifying punctuation marks; deleting the remaining half of paired punctuation marks such as book title marks, brackets (large, medium and small), and double quotation marks if there is a punctuation missing problem (note: single quotation marks are not included); and modifying single quotation marks that are mistakenly written as single double quotation marks, for example, mistakenly writing it's as it"s.

[0119] The simplified and traditional Chinese unification module 8023 converts the traditional Chinese characters in all corpora into simplified Chinese characters.

[0120] The sentence start processing submodule 8024 deletes meaningless symbols such as '! @#&*:;,. / ' that appear at the beginning of sentences in some corpora.

[0121] Cleaning submodule 8025, after processing the data according to the above steps, if it is found that part of the corpus is still very chaotic (according to empirical statistics, it generally accounts for about 0.02% of the corpus), for example: the overall length of a sentence is 30-40 characters, and more than 10 of them are garbled symbols '@#&', and after removing these garbled characters, a complete sentence cannot be formed, then the entire corpus will be directly deleted from the corpus.

[0122] The deduplication submodule 8026 deduplicates the corpus after the above processing because some corpora overlap each other.

[0123] Through data cleaning, that is, data preprocessing, we can finally obtain cleaned bilingual predictions. Similarly, when building a machine translation model for a specific field, it is also necessary to perform the same data cleaning on the collected training data in the specific field.

[0124] In translation, there are some words that do not need to be translated and can be directly retained, which are referred to as keywords in this article. In order to solve the problem of keyword retention, the translation system of the present invention can further learn the special word protection function. Figure 8 As shown, the corresponding Figure 3 A specific example process of implementing the keyword protection processing process.

[0125] In step 901, the Chinese corpus and the English corpus are segmented by spaces respectively to achieve word segmentation of the bilingual corpus.

[0126] In step 902, character matching is performed on the corpus that has completed word segmentation, or it is called left-right character matching, where "left" and "right" are only for the convenience of describing the relative and corresponding bilingual corpora, and do not limit the actual relative position, orientation or direction of the bilingual corpora. For example, assume that "left" is Chinese and "right" is English, and vice versa. The work done to match the strings in the corpus pair is: for each pair of corpora, the English and the corresponding Chinese are matched (the basic string unit is an English word or a Chinese natural segmentation word). For example, Chinese: Insert a SIM card into the phone; English, Insert a SIMcard into the phone.

[0127] In step 903, it is determined whether there is a string that satisfies the special character replacement rule, and the rule may be, for example, the same string part. Therefore, in each pair of matched bilingual corpora, if there is a string that satisfies the special character replacement rule, the process jumps to step 904, otherwise the process ends.

[0128] In step 904, the character strings that are determined to be identical after matching are replaced with a special character string, for example, '&SEP1;'. If there are multiple identical character string parts in a sentence, they can be replaced in sequence with '&SEP2;', '&SEP3;', and so on.

[0129] refer to Fig.10 As shown, Figure 7 A schematic diagram of an implementation of the converter architecture of the base model pre-training unit 7020 of the embodiment, wherein the training steps through the architecture are also shown with arrows. Fig.10 In the example, the basic model pre-training unit 1000 can be used with Figure 7 The basic model pre-training unit 1000 corresponds to the basic model pre-training unit 7020. The basic model pre-training unit 1000 may include a sub-word table creation module 1010, a sub-word table based word segmentation module 1020, a converter architecture setting module 1030, and a loss function setting module 1040.

[0130] The sub-word vocabulary creation module 1010 segments the words in the pre-processed corpus pairs based on the byte pair encoding (BPE) mechanism to establish a vocabulary, and the specific process is the same as the aforementioned step S401.

[0131] The sub-word list-based word segmentation module 1020 segments the words in the corpus pairs of the training data set according to the word list to establish a segmented training set. The specific process is as described in the aforementioned step S402.

[0132] The converter architecture setting module 1030 is used to determine the parameters of the converter architecture, which may include several sub-modules: encoder, decoder, learning rate, and number of hidden layer neuron nodes.

[0133] According to an example, the corresponding model parameter details are shown in Table 1 below:

[0134] Table 1 Parameters of converter architecture

[0135]

[0136]

[0137] As shown in the table, where:

[0138] The encoder layer refers to the number of layers in the encoder part of the converter structure, which is set to 6 here.

[0139] The decoder layer refers to the number of layers in the decoder part of the transformer structure, which is set to 6 here.

[0140] The encoder and decoder are both set to 6 layers. Considering the computational overhead, in some instances, the overall cost performance of setting it to 6 layers is higher than that of other layer selections.

[0141] The number of hidden layer neuron nodes is set to 512. The number set here can be determined by experience, and is usually hundreds to thousands. Considering the computational overhead, in some instances, setting the number of hidden layer neurons to 512 is more cost-effective.

[0142] The number of neurons in the output layer, K, is established according to the length of the sub-word vocabulary. In some examples, K is the length of the sub-word vocabulary.

[0143] The number of multi-head attention is set to 8 here.

[0144] The learning rate is set to a decaying learning rate with warm-up, in which the decay period is established according to the size of the training set. This is slightly different from the one introduced in the aforementioned "attention is all you need" and has been fine-tuned.

[0145] The batch size (batch_size) of training data entering the network is not determined by a fixed number of sentences, but by the total number of words entering the neural network. The training data for each batch is obtained by dynamically determining the sentence length distribution of each batch.

[0146] The maximum length of the decoder is set to 256, which is based on the longest single sentence that may appear under normal circumstances.

[0147] Regarding learning rate decay, when training a neural network, the learning rate is used to control the update speed of the parameters. When the learning rate is small, the update speed of the parameters will be greatly reduced, so that the convergence speed of the model training is slow. When the learning rate is large, oscillation may occur during the update process of parameter optimization, causing the parameters to hover around the optimal value.

[0148] At a small learning rate during preheating, the model can slowly stabilize. After the model is relatively stable, the pre-set learning rate can be selected for training, making the model converge faster and the model effect better. To this end, learning rate decay is introduced during the training process so that the learning rate gradually decays as the training progresses.

[0149] The loss function setting module 1040 is used to design the loss function using minimum risk training.

[0150] In some embodiments, constructing a basic machine translation model for a general domain further includes designing a loss function using the idea of ​​minimum risk training.

[0151] The basic idea of ​​minimum risk training is to use the loss function Δ(y,y (n) ) to describe the model predictions y and y (n) The difference between them is calculated, and an attempt is made to find a set of parameters that minimizes the expected value (i.e., risk) of the model loss on the training set:

[0152]

[0153] Where Y(x (n) ) represents x (n) The corresponding set of all possible outputs is also called the search space.

[0154] Table 2 shows an example of minimal risk training:

[0155] Table 2 Minimum risk training examples

[0156]

[0157] Assume that for input x (n) , search space Y(x (n) ) including y 1 ,y 2 ,y 3 For each candidate output, the loss with the standard answer can be calculated. In this example, the losses of the three candidates are -1.0, -0.3, and -0.5. In other words, the standard answer believes that y 1 Best, 3 Next, y 2The goal of minimum risk training is to find a set of model parameters that minimizes the expected value of loss. Four sets of probability distributions are given on the right side of Table 1. The first set of probability distributions assumes that y 2 >y 3 >y 1 , which is contrary to the standard answer, so a high risk value of -0.50 is obtained. The second set of probability distributions assumes that y 3 >y 1 >y 2 , which improves the correlation with the standard answer compared to the first set of probability distributions, thus obtaining a lower risk value of -0.61. The third set of probability distributions obtains a ranking y that is consistent with the standard answer 1 >y 3 >y 2 , thus further reducing the risk value to -0.71. The fourth group of probability distributions ensures the consistency of the order and increases the optimal output y 1 , thus reducing the risk value to -0.83. From this, we can see that minimum risk training believes that a good set of parameters should be as consistent as possible with the standard answer in the ranking of all candidate elements, and the loss function defines the calculation method of the ranking.

[0158] After completing the training of the basic model, we can start fine-tuning it to a specific field. Fig.11 As shown, Figure 7 An implementation of the fine-tuning part of the machine translation system of the embodiment, wherein the process of fine-tuning the machine translation from the general domain to the specific domain is shown by the arrow flow. This process can be divided into the specific domain corpus preprocessing as shown in box 1110, and the fine-tuning to the specific domain as shown in box 1120.

[0159] The function of domain-specific corpus preprocessing is equivalent to Figure 7 When the second data preprocessing unit 7031 is fine-tuned to a specific field, it is also necessary to first collect bilingual corpora in specific fields such as the telecommunications field and the legal field, or so-called professional fields as described below. Please note that although the present invention cites several specific fields as examples, the technical solution of the present invention is not limited to the exemplified fields and can also be applied to other professional fields. Then, the collected second data, that is, the bilingual corpus training data suitable for a specific field, is preprocessed. Fig.11 As shown in the boxes 1111-1116, its functions and implementation process are Figure 8 Similar to that shown in boxes 8021-8026.

[0160] In the domain-specific fine-tuning in box 1120 , two parts are included: learning rate adjustment 1121 and fine-tuning 1122 .

[0161] The learning rate adjustment 1121 adjusts the learning rate according to a specific domain based on the constructed basic machine translation model to obtain a second learning rate relative to the first learning rate.

[0162] Fine-tuning 1122, replacing the validation set in the general domain training dataset used for the basic machine translation model with the corpus of the training dataset for the specific domain to be developed, and optimizing the model through the BLEU value of the translation of the corpus in the specific domain until the model reaches convergence on the training in the specific domain.

[0163] According to some examples, the second learning rate can be adjusted to half of the end of the cycle of the cyclical learning rate of the first learning rate. According to some examples, the second learning rate can be adjusted to 0.5-1 times of the first learning rate used when the basic model converges completely, for fine-tuning to a specific field. By adjusting the learning rate, the model is fine-tuned on the training data of the specific field. When fine-tuning, it is necessary to judge the convergence effect of the model, and then retain the optimal model, perform weight fusion, and obtain the final result model. Thus, according to an embodiment of the present invention, the idea of ​​fine-tuning is innovatively applied to the field adaptation migration of machine translation, and the comprehensive improvement of translation effect and translation model development time is achieved. At the same time, the relevant parameters of the classical converter architecture are also innovatively adjusted to achieve an improvement in the overall effect.

[0164] The following is combined with Figure 12-15 , the embodiments of the present invention are further explained with respect to several application scenarios.

[0165] refer to Fig.12 As shown, it is an example of the architecture of a machine translation system 1200 suitable for the telecommunication field constructed through fine-tuning according to an embodiment of the present invention. Fig.12 System 1200 with Figure 7 is similar to system 700, except that Fig.12 In the telecommunication field corpus preprocessing unit, the second data processed is bilingual corpus specifically used in the telecommunication field.

[0166] refer to Fig.13 As shown, Fig.12 A more detailed system flow chart of the machine translation system architecture suitable for the telecommunications field in the embodiment includes more detailed functional subcomponents. The system flow chart describes how to use these functional subcomponents to implement a machine translation system suitable for the telecommunications field. From 1310 to 1390, each functional subcomponent participates in the process of implementing a machine translation system suitable for the telecommunications field in sequence. The whole process is similar to that from Figure 1-6 ,as well as Figure 7-11 The process described is similar.

[0167] refer to Fig.14 As shown, it is an example of the architecture of a machine translation system 1400 suitable for the legal field constructed through fine-tuning according to an embodiment of the present invention. Fig.14 System 1400 with Figure 7 is similar to system 700, except that Fig.14 In the telecommunication field corpus preprocessing unit, the second data processed is bilingual corpus specifically used in the legal field.

[0168] refer to Fig.15 As shown, Fig.14 A more detailed system flow chart of the machine translation system architecture suitable for the legal field in the embodiment includes more detailed functional sub-components. The system flow chart describes how to use these functional sub-components to implement the machine translation system suitable for the legal field. From 1510 to 1590, each functional sub-component participates in the process of implementing the machine translation system suitable for the legal field in sequence. The whole process is similar to that from Figure 1-6 ,as well as Figure 7-11 The process described is similar.

[0169] refer to Fig.16 FIG. 1 is a block diagram of a machine translation device according to another embodiment of the present invention. The machine translation device includes: a memory, a processor, and a computer program (not shown in the figure) stored in the memory and executable on the processor. The processor and the memory may be connected via a bus or other means. Fig.16 The example of connecting through bus is taken in the following.

[0170] The non-transient software program and instructions required to implement the machine translation device in the above embodiment are stored in the memory. When executed by the processor, the machine translation model implementation method in the above embodiment is executed, for example, the above described Figure 1-6 The method steps S101-S102, S201-S202, S301-S303, S401-S402, S501-S503, S601-S602.

[0171] In addition, an embodiment of the present invention further provides a computer-readable storage medium, the computer-readable storage medium storing computer-executable instructions, the computer-executable instructions being executed by a processor or controller, for example, Fig.16 The processor in the embodiment of the present invention may be executed by executing the method for implementing the machine translation model in the embodiment of the present invention, for example, executing the method described above. Figure 1-6 The method steps S101-S102, S201-S202, S301-S303, S401-S402, S501-S503, S601-S602.

[0172] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0174] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for constructing a machine translation model, comprising: Build a basic machine translation model for general domains, including: adopting a transformer architecture and determining architecture parameters; Determining the architecture parameters includes: According to the size of the training data set, set the first learning rate for training according to a certain decay period; Based on the basic machine translation model, fine-tuning is performed towards a specific domain to build a machine translation model for a specific domain, including: Based on the constructed basic machine translation model, the learning rate is adjusted according to the specific domain to obtain a second learning rate relative to the first learning rate; Replace the validation set in the general domain training dataset used for the basic machine translation model with the corpus of the specific domain training dataset to be developed, and optimize the model through the BLEU value of the translation of the specific domain corpus until the model reaches convergence on the training of the specific domain; The second learning rate is set to 0.5-1 times the first learning rate.

2. The method according to claim 1, characterized in that The constructing of a basic machine translation model for a general field and the constructing of a machine translation model for a specific field respectively include: Preparing a training data set, wherein the training data set includes corpus pairs; Preprocessing is performed on the corpus pairs in the training data set.

3. The method according to claim 2, characterized in that The preprocessing of the corpus pairs in the training data set includes protecting the keywords in the corpus pairs, including: Matching a string in the corpus pair; Identifying keywords based on identical character strings in the corpus pair; Use special strings to replace keywords.

4. The method according to claim 2, characterized in that: The constructing of a basic machine translation model for a general field and the constructing of a machine translation model for a specific field further include: Segment the words in the preprocessed corpus based on the byte pair encoding (BPE) mechanism to build a vocabulary; The words in the corpus pairs of the training data set are segmented according to the vocabulary to establish a segmented training set.

5. The method according to claim 1, characterized in that Determining the architecture parameters further includes: Set the number of encoder and decoder layers to 6-10; Set the number of hidden layer neurons to 512-1024; Set the number of neurons in the output layer to be consistent with the length of the vocabulary.

6. The method according to claim 1, characterized in that The construction of a basic machine translation model for a general domain also includes designing a loss function using minimum risk training.

7. The method according to claim 2, characterized in that Also includes: The training data set is divided into training set, test set and validation set according to a certain ratio; The training data set is repeatedly trained using the constructed basic machine translation model until the BLEU value of the basic machine translation model on the test set and the validation set reaches a stable value; After the basic machine translation model converges and stabilizes, a certain number of basic machine translation models are saved and weighted; Save the basic machine translation model after weight fusion.

8. The method according to claim 2, characterized in that: The training data set includes: General domain training dataset, used to train basic machine translation models in general domains; Domain-specific training datasets are used to train machine translation models in specific domains.

9. A translation device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for building a machine translation model as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method for constructing a machine translation model according to any one of claims 1 to 8.

Images