Tibetan-Chinese neural machine translation system construction method based on language model knowledge distillation

Abstract

The invention discloses a Tibetan-Chinese neural machine translation system construction method based on language model knowledge distillation, and is suitable for the field of artificial intelligence and machine translation. Comprising the steps that S1, Tibetan-Chinese bilingual corpora and Tibetan-Chinese monolingual corpora are collected and preprocessed; s2, selecting a neural machine translation model as a student model, and training the student model by using Tibetan-Chinese bilingual corpora; s3, selecting a monolingual language model as a teacher model, and training the teacher model by using Chinese and Tibetan monolingual corpora; s4, carrying out distillation training on the student model, and enabling the output probability of the student model to fit the output probability of the teacher model; and S5, for the student model, calculating the difference between the generated translation and the reference translation, and carrying out iterative training and parameter optimization. The monolingual language model is utilized to guide the neural machine translation model for training, the teacher model only participates in student model training and does not participate in translation decoding, the decoding speed can be increased, the resource and storage requirements can be reduced, and the method is favorable for being deployed in a platform with limited computing resources.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and machine translation, and in particular to a method for constructing a Tibetan-Chinese bidirectional neural machine translation system based on language model knowledge distillation. Background Art

[0002] Neural machine translation relies on large-scale, high-quality parallel corpora and powerful language modeling capabilities, and has become the mainstream machine translation method. However, massive bilingual corpora only exist in a few language pairs (such as Chinese and English), which makes its performance poor in low-resource neural machine translation where Tibetan-Chinese parallel corpora are scarce.

[0003] Compared with parallel corpora, monolingual data is usually huge on the Internet. Therefore, making full use of more easily accessible monolingual data has become one of the effective methods for low-resource neural machine translation. Among them, the language model that integrates large-scale monolingual data training has been proven to be very effective prior knowledge in machine translation. However, with the increase in the number of network parameters and model complexity, this model fusion method will seriously affect the decoding speed of machine translation. In addition, how to balance the proportion between the translation model and the language model has a huge impact on the translation performance. Therefore, improving the decoding speed and reducing resource and storage requirements are currently urgent problems to be solved. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the prior art, the present invention provides a method for constructing a Tibetan-Chinese bidirectional neural machine translation system based on language model knowledge distillation. By using more easily accessible large-scale Tibetan and Chinese monolingual data to train a monolingual language model (as a teacher model), the Tibetan-Chinese neural translation model (as a student model) is guided to perform distillation training, thereby solving the problems of poor translation effect due to the lack of Tibetan-Chinese parallel corpora and slow decoding speed caused by the traditional method of language model fusion.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] The present invention provides a method for constructing a Tibetan-Chinese bidirectional neural machine translation system based on language model knowledge distillation, the method being executed by a computer and comprising:

[0009] S1, collecting Tibetan-Chinese bilingual corpora and Tibetan-Chinese monolingual corpora respectively, and preprocessing the corpora;

[0010] S2, selecting a neural machine translation model as a student model, and using the preprocessed Tibetan-Chinese bilingual corpus to train the student model;

[0011] S3, selecting a monolingual language model as a teacher model, and using the pre-processed Chinese and Tibetan monolingual corpora to train the teacher model respectively;

[0012] S4, performing distillation training on the student model so that the output probability of the student model fits the output probability of the teacher model;

[0013] S5. For the student model, perform iterative training and model parameter optimization by calculating the difference between the generated translation and the reference translation.

[0014] Preferably, step S1 specifically includes: S101, for bilingual corpus, Tibetan-Chinese bilingual parallel sentence pairs, and preprocessing the Tibetan-Chinese bilingual parallel sentence pairs to obtain a first predetermined number of levels of Tibetan-Chinese bilingual text corpus set, the preprocessing includes: removing repeated sentence pairs, word segmentation, byte pair encoding processing and filtering noise data; S102, for monolingual corpus, using web crawling technology to collect Chinese text corpus in the news field from Chinese government news websites; and using web crawling technology to collect Tibetan text corpus in the news field from Tibetan-related government news websites, and preprocessing the Chinese and Tibetan monolingual text corpora respectively to obtain a second predetermined number of levels of Chinese text corpora and Tibetan text corpora, the preprocessing includes: sentence segmentation, removing repeated sentences, word segmentation, byte pair encoding processing and filtering noise data.

[0015] Preferably, the step S2 specifically includes: using the pre-trained model Transformer-Base model as a student model (Tibetan-Chinese and Chinese-Tibetan machine translation model); taking Chinese as the source language and Tibetan as the target language, the encoder of the Transformer-Base model successively converts the Tibetan in the Tibetan-Chinese bilingual text corpus into a fixed-length vector representation, and decodes the vector representation into Chinese through a decoder, and iteratively trains the "Tibetan-Chinese" direction neural machine translation model; conversely, taking Chinese as the source language and Tibetan as the target language, and iteratively training the "Chinese-Tibetan" direction neural machine translation model.

[0016] Preferably, the pre-trained model Transformer-Big is used as the teacher model.

[0017] Preferably, step S4 specifically includes: on the basis of the student model loss in step S3, introducing a distillation loss between the teacher model and the student model, wherein the distillation loss is used to minimize the divergence between the output probability of the target-end monolingual model and the output probability of the translation model from the source language to the target language, and configuring a weight coefficient λ for the distillation loss to control the influence of the distillation loss on the student model training.

[0018] Preferably, the step S4 further includes: introducing a temperature coefficient T in the distillation loss to obtain the probability distribution of the category labels of the teacher model, and smoothing the output probability by controlling the temperature coefficient T.

[0019] Preferably, the total loss function of the student model is the student loss of the student model plus the distillation loss between the teacher model and the student model, and the calculation formula is as follows:

[0020]

[0021] Where λ is the coefficient of the distillation loss; T is the temperature coefficient of the distillation loss softmax layer;

[0022] The student loss function L of the student model TM The calculation formula is as follows:

[0023]

[0024] Among them, p TM (y t |y <t ,x) means that for a bilingual sentence pair (x,y), x is the source language sentence and y is the <t is the translation sequence generated previously;

[0025] Distillation loss function L between teacher model and student model KL The calculation formula is as follows:

[0026] L KL =T 2 D KL (p LM (y t |y <t ;T)||p TM (y t |y <t ,x;T)

[0027] Among them, D KL (p LM (y t |y <t )||p TM (y t |y <t ,x) is an indicator marking the difference between the probability distributions of the student model and the teacher model;

[0028] D KL represents the divergence,

[0029] p LM (y t |y <t ) represents the output of the teacher model, pTM (y t |y <t ,x) represents the output of the student model, T is the temperature coefficient introduced in the Softmax layer of the distillation loss, and its calculation formula is as follows:

[0030]

[0031] Among them, i represents each category and j represents all categories;

[0032] T(T≥1) When T=1, it represents the traditional Softmax layer;

[0033] When T>1, the distribution gradually tends to be uniform and the generalization ability of the model is improved.

[0034] Preferably, the second predetermined number level is 2 million sentences level and / or 20 million sentences level.

[0035] Preferably, the obtained Tibetan-Chinese bilingual text corpus of the first predetermined number of levels is divided into a training set, a validation set and a test set.

[0036] (III) Beneficial effects

[0037] The present invention provides a method for constructing a Tibetan-Chinese bidirectional neural machine translation system based on language model knowledge distillation. Compared with the prior art, it has the following beneficial effects:

[0038] (1) In order to solve the problem of poor translation results due to the lack of Tibetan-Chinese parallel corpora, the present invention uses massive monolingual text data on the Internet to train a monolingual language model. By introducing the prior knowledge contained in the target-side monolingual language model, it can make up for the problem of insufficient generalization ability caused by the small number of Tibetan-Chinese machine translation training sets, and guide the translation model to learn the text sequence rules missing in the limited bilingual data, thereby improving the accuracy and fluency of the translation;

[0039] (2) Compared with the traditional "model fusion" method between the translation model and the language model, the present invention uses the target-side monolingual language model (teacher model) to guide the Tibetan-Chinese bidirectional neural machine translation model (student model) for distillation training. The target-side monolingual language model only participates in the machine translation model training and does not participate in the machine translation decoding stage. The number of parameters is greatly reduced, thereby achieving student model compression and acceleration, which can effectively improve the decoding speed and reduce resource and storage requirements, making it more conducive to deployment on platforms with limited computing resources;

[0040] (3) The monolingual language model is not a component of the machine translation neural network. For translation tasks of different language pairs and translation directions, it only needs to be replaced with the corresponding target language model. There is no need to train the monolingual language model from scratch, nor to change the translation model network structure, which greatly reduces the complexity of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A block diagram of a method for constructing a Tibetan-Chinese neural machine translation system based on language model knowledge distillation according to an embodiment of the present invention;

[0043] Figure 2 It is a block diagram of the composition of step S1 in the method for constructing a Tibetan-Chinese neural machine translation system based on language model knowledge distillation;

[0044] Figure 3 This is a structural diagram of the method for building a Tibetan-Chinese neural machine translation system based on language model knowledge distillation. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The embodiments of the present application provide a block diagram of a Tibetan-Chinese bidirectional neural machine translation method and system based on language model knowledge distillation, thereby solving the problems of poor translation effect caused by the shortage of Tibetan-Chinese parallel corpora, as well as slow decoding speed and high model complexity caused by the traditional method of fusing language models with translation models.

[0047] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0048] The present invention provides a method for constructing a Tibetan-Chinese bidirectional neural machine translation system based on language model knowledge distillation, which is executed by a computer. Figure 1 As shown, the following steps are included:

[0049] S1, collecting Tibetan-Chinese bilingual corpora and Tibetan-Chinese monolingual corpora respectively, and preprocessing the corpora;

[0050] S2, selecting a neural machine translation model as a student model, and using the preprocessed Tibetan-Chinese bilingual corpus to train the student model;

[0051] S3, selecting a monolingual language model as a teacher model, and using the pre-processed Chinese and Tibetan monolingual corpora to train the teacher model respectively;

[0052] S4, performing distillation training on the student model so that the output probability of the student model fits the output probability of the teacher model;

[0053] S5. For the student model, perform iterative training and model parameter optimization by calculating the difference between the generated translation and the reference translation.

[0054] The following is a detailed description of each step:

[0055] It should be noted that the embodiment of the present invention is a method for constructing a Tibetan-Chinese bidirectional neural machine translation system based on language model knowledge distillation, but it does not mean that the embodiment of the present invention can only be used for Tibetan-Chinese bidirectional neural machine translation tasks. Through the method of the embodiment of the present invention, low-resource neural machine translation models between other ethnic languages ​​and Chinese can also be trained. The low-resource neural machine translation model is distilled and trained through the target-end monolingual language model. The monolingual language model, as a priori knowledge, can make up for the problem of insufficient generalization ability of low-resource machine translation due to the small number of bilingual training sets, thereby further improving the translation effect.

[0056] S1, collect Tibetan-Chinese bilingual corpus and Tibetan and Chinese monolingual corpus respectively, and preprocess the corpus; specifically includes the following steps, see Figure 2 :

[0057] For bilingual corpus, S101 uses the Tibetan-Chinese bilingual parallel sentence pairs (government document field) released by the 17th National Conference on Machine Translation (CCMT2021), and preprocesses the Tibetan-Chinese bilingual parallel sentence pairs. These preprocessings include: removing duplicate sentence pairs, word segmentation, byte pair encoding processing, and filtering noise data. Among them, Jieba (Chinese word segmentation tool) is used to perform word segmentation on the Chinese data set in the Tibetan-Chinese bilingual parallel sentence pairs, and the TIP-LAS (Tibetan word segmentation and part-of-speech tagging system) word segmentation tool is used to perform word segmentation on the Tibetan in its sentence pairs. In the Byte-Pair En-coding (BPE) processing, the BPE fusion number is set to 16,000, and the noise data is filtered by deleting sentence pairs with less than 3 or greater than 120 in the parallel corpus. After the above preprocessing steps, a Tibetan-Chinese bilingual text corpus of limited size is obtained. In this embodiment, a Tibetan-Chinese parallel corpus (Chinese and Chinese-Tibetan bidirectional sentence pairs) of a predetermined size (first predetermined order) is obtained. These sentence pairs are divided into three groups, namely, a training set, a validation set and a test set.

[0058] Among them, the training set: 148,343 sentence pairs, the validation set: 1000 sentence pairs, and the test set: 1000 sentence pairs.

[0059] S102. For monolingual corpus, use web crawling technology to collect Chinese text corpus in the news field from Chinese government news websites, such as People's Daily Online, Xinhuanet, CCTV.com, China News Network, etc., and use web crawling technology to collect Tibetan text corpus in the news field from Tibetan-related government news websites, such as People's Daily Online Tibetan version, Xinhuanet Tibetan version, Voice of China Tibet Network, Qinghai Provincial People's Government Portal Tibetan version, etc.;

[0060] The collected Chinese and Tibetan monolingual text corpora are subjected to sentence segmentation, repeated sentence removal, word segmentation, byte pair encoding processing, and noise data filtering processing, wherein Jieba (Chinese word segmentation tool) is used to perform word segmentation processing on the collected Chinese data set, and TIP-LAS (Tibetan word segmentation and part-of-speech tagging system) word segmentation tool is used to perform word segmentation processing on the collected Tibetan. In the Byte-Pair En-coding (BPE) processing, the BPE fusion number is set to 16,000, and the sentence pairs with parallel corpora less than 3 or greater than 120 are deleted in the noise data filtering processing. After the above processing, a first predetermined number of Chinese text corpora and Tibetan text corpora are obtained; the second predetermined number of levels is 2 million sentence level and / or 20 million sentence level. That is, in this embodiment, 2 million sentence level Chinese and 2 million sentence level Tibetan monolingual text corpora and 20 million sentence level Chinese and 20 million sentence level Tibetan monolingual text corpora are obtained after collection and processing.

[0061] S2, selecting a neural machine translation model as a student model, and using the preprocessed Tibetan-Chinese bilingual corpus to train the student model;

[0062] In this embodiment, the neural machine translation model is based on the Transformer-Base neural network framework, adopts the open source fairseq system, and the operating system version is Ubuntu 20.04.3LTS. The main hyperparameter values ​​of the model training are shown in Table 1.

[0063] Main hyperparameter values ​​of the translation model

[0064]

[0065] Table 1

[0066] The neural machine translation model is trained using the limited-scale Tibetan-Chinese bilingual corpus obtained after the above preprocessing as the student model.

[0067] S2, selecting a monolingual language model as a teacher model, and using the preprocessed Chinese and Tibetan monolingual corpora to train the teacher model respectively;

[0068] In this embodiment, the teacher model is a monolingual language model, which is trained based on the Transformer-Big neural network framework, and the training parameter values ​​are shown in Table 2.

[0069] Monolingual language model training parameters

[0070]

[0071] Table 2

[0072] S4. The teacher model guides the student model training. Through knowledge distillation, the student model learns the knowledge contained in the teacher model and builds a Tibetan-Chinese bidirectional neural machine translation system based on language model knowledge distillation. Figure 1 and Figure 3 , the specific steps include:

[0073] The "teacher-student" model in knowledge distillation is adopted, with a complex and large monolingual language model as the teacher model, and a lightweight Tibetan-Chinese bidirectional neural machine translation as the student model. The teacher model has strong learning ability and can transfer its learned knowledge to the student model with relatively weak learning ability, thereby enhancing the generalization ability of the student model. The teacher model is used to assist the training of the student model. In theory, the result obtained will be better than that of the student model that simply fits the training data.

[0074] The goal of distillation training is to fit the output probability of the teacher model with the output probability of the student model. That is, on the basis of the loss of the original student model (before distillation training), the distillation loss between the teacher model and the student model is introduced. The distillation loss is used to minimize the KL (Kullback-Leibler) divergence between the output probability of the teacher model and the output probability of the student model. The KL divergence is an indicator used to measure the difference between the above two probability distributions.

[0075] Distillation training is performed on the "Tibetan→Chinese" neural machine translation model, with the target Chinese language model as the teacher model and the "Tibetan→Chinese" translation model as the student model. The goal is to use the output probability of the "Tibetan→Chinese" translation model (student model) to fit the output probability of the target Chinese language model (teacher model), that is, on the basis of the original "Tibetan→Chinese" translation model loss function (student loss), the distillation loss between the target Chinese language model and the "Tibetan→Chinese" translation model is introduced. The distillation loss can be used to minimize the KL (Kullback-Leibler) divergence between the output probability of the target Chinese language model (teacher model) and the output probability of the "Tibetan→Chinese" translation model.

[0076] Distillation training is performed on the "Chinese→Tibetan" neural machine translation model, with the target Tibetan language model as the teacher model and the "Chinese→Tibetan" translation model as the student model. The goal is to use the output probability of the "Chinese→Tibetan" translation model (student model) to fit the output probability of the target Tibetan language model (teacher model), that is, on the basis of the original "Chinese→Tibetan" translation model loss function (student loss), the distillation loss between the target Tibetan language model and the "Chinese→Tibetan" translation model is introduced. The distillation loss can be used to minimize the KL (Kullback-Leibler) divergence between the output probability of the target Tibetan language model (teacher model) and the output probability of the "Chinese→Tibetan" translation model.

[0077] In this embodiment, a weight coefficient λ is also configured for distillation training, and the coefficient λ is used to control the influence of distillation loss on model training.

[0078] In this embodiment, a temperature coefficient T is also configured for distillation training, and the temperature coefficient T is used to obtain the probability distribution of teacher model softening, which is used to calculate the distillation loss in the loss function.

[0079] The loss function of the original student model (not trained by distillation) is to make the predicted value as close to the true value as possible. The category probability output by the softmax layer is a hard label (except for the positive label which is 1, all other negative labels are 0). In knowledge distillation, the target monolingual language model is used, that is, the category probability Soft-label of the teacher model is used to assist in training the student model. That is, by assigning a probability to each category, the probability of the positive label is the highest, so that the student model can learn the similarity between samples and the generalization ability of the teacher model. The specific approach is to introduce the temperature coefficient T in the softmax layer of the distillation loss.

[0080] In summary, the calculation formula of the total loss function of the "Tibetan→Chinese" and "Chinese→Tibetan" bidirectional neural machine translation systems based on language model knowledge distillation provided in this embodiment is as follows:

[0081] The total loss function includes the loss function L of the student model mTM and the distillation loss function L between the teacher model and the student model KL It consists of two parts, and the calculation formula is as follows:

[0082]

[0083] in,

[0084] The student loss function L of the student model TM The calculation formula is as follows:

[0085]

[0086] Among them, p TM (y t |y <t ,x) means that for a bilingual sentence pair (x,y), x is the source language sentence and y is the <t is the translation sequence generated previously;

[0087] Distillation loss function L between teacher model and student model KL The calculation formula is as follows:

[0088] L KL =T 2 D KL (p LM (y t |y <t ;T)||p TM (y t |y <t ,x;T)

[0089] Among them, D KL (p LM (yt |y <t )||p TM (y t |y <t ,x) is an indicator that marks the difference between the output probabilities of the student model and the teacher model, D KL represents the divergence,

[0090] p LM (y t |y <t ) represents the output of the teacher model, p TM (y t |y <t ,x) represents the output of the student model, λ is the coefficient of distillation loss; T is the temperature coefficient introduced in the Softmax layer of distillation loss, and its calculation formula is as follows:

[0091]

[0092] Among them, i represents each category and j represents all categories;

[0093] Among them, T(T≥1) when T=1, represents the traditional Softmax

[0094] layer; when T>1, the distribution gradually tends to be uniform.

[0095] The generalization ability of the model is improved.

[0096] S5. A Tibetan-Chinese bidirectional neural machine translation system based on language model knowledge distillation measures the difference between the translation generated by the neural machine translation model and the reference translation, iteratively trains and guides model parameter optimization, improves model accuracy, and generates reliable translations.

[0097] The knowledge learned by the teacher model is used to guide the training of the student model, so that the student model can learn to imitate the teacher model, that is, learn the text sequence rules missing in the limited bilingual training data, and improve the accuracy and fluency of the translation. It should be noted that the complex teacher model does not participate in the machine translation decoding stage, and the number of parameters is greatly reduced, thereby achieving compression and acceleration of the student model. The experimental evaluation index uses the machine bilingual translation evaluation value (BLEU) to measure the difference between the translation generated by the neural machine translation model and the reference translation. Iterative training guides the optimization of model parameters and improves the model accuracy to generate reliable translations. In the training stage, the Adam gradient optimization algorithm is used to train to obtain the optimal model parameters. When there is no change in BLEU after 10 iterations, the training is automatically stopped. In the inference stage, the beam search algorithm is used to generate the target sentence for translation, and the beam size is set to 5.

[0098] In the present embodiment, the present invention adopts the most widely used machine translation evaluation index BLEU (Bilingual Evaluation Understudy) as the machine translation performance evaluation index of various methods. BLEU is an automatic evaluation method for machine translation based on n-gram matching, which is used to analyze the degree of co-occurrence of n-grams between candidate translations and reference translations. Its evaluation results also show a higher consistency compared with manual evaluation.

[0099] In terms of improving the translation model effect, in order to make the experiment of the present invention comparative and convincing, the present invention selects the Transformer-Base system and the representative language model fusion methods in recent years: the Shallow Fusion method proposed by Gulcehre et al. and the MSO method proposed by Miao et al. as the baseline system. The experimental evaluation indicator is the machine bilingual translation evaluation value (BLEU). In the training stage, the Adam gradient optimization algorithm is used to train to obtain the optimal model parameters. When there is no change in BLEU after 10 iterations, the training is automatically stopped. In the inference stage, the beam search algorithm is used to generate the target sentence for translation, and the beam size is set to 5.

[0100] Table 3 lists the language model knowledge distillation method (LM-Distillation) proposed in the present invention, including: LM-Distillation (200) based on 2 million monolingual language model distillation training, LM-Distillation (2000) based on 20 million monolingual language model distillation training, and the experimental results of the system on the validation set (dev) and test set (test) of CCMT2021 Tibetan-Chinese parallel corpus. It can be seen that:

[0101] (1) Although the Transformer-Base model performs well in Tibetan-Chinese low-resource neural machine translation, the LM-Distillation (200) method proposed in the present invention, which is based on the distillation training of 2 million monolingual language models, is still 2.82 BLUE values ​​higher (Tibetan→Chinese test set) and 2.74 BLUE values ​​higher (Chinese→Tibetan test set) than the Transformer model, and is higher than the MSO language model fusion method with the best performance in the baseline system, indicating that the prior knowledge contained in the monolingual language model at the distillation end can significantly improve the effect of Tibetan-Chinese bidirectional translation;

[0102] (2) LM-Distillation (2000) trained based on the distillation of the 20 million Tibetan target language model still improves 1.08 points on the "Chinese→Tibetan" test set compared to LM-Distillation (200) BLUE. However, based on the distillation of the 20 million Chinese target language model, the "Tibetan→Chinese" neural machine translation model is trained. Since the Chinese corpus has a relatively wide range of sources and a wide variety of topics, the prior knowledge provided by the larger but inconsistent Chinese language model will limit the improvement of the "Tibetan→Chinese" translation performance to a certain extent, indicating that the domain consistency of the monolingual corpus and the parallel corpus is also very important.

[0103] Model effect comparison

[0104]

[0105] Table 3

[0106] In terms of model decoding speed, unlike the traditional language model fusion method, the target-end monolingual language model of the present invention does not participate in the decoding stage, and the number of parameters is greatly reduced, thereby achieving translation model compression and acceleration. In order to verify the degree of improvement in decoding speed, the present invention takes the CCMT2021 "Tibetan→Chinese" translation task as an example, and compares the translation effect and decoding speed of the LM-Distillation (200) method based on 2 million Chinese language model distillation training with the baseline system fused with the language model. It can be seen from the experimental results in Table 4 that: under the same training conditions, based on the decoding speed of the shallow fusion method, the decoding speed of the method of the present invention is 1.06 times higher than that of the baseline system, and the Blue value is 1.34 points higher than the optimal MSO language model fusion method, indicating that the method of the present invention has advantages in both translation quality and decoding speed.

[0107] Comparative analysis of different models

[0108]

[0109]

[0110] Table 4

[0111] The present invention uses the prior knowledge learned by the target-side monolingual language model to guide the training of the Tibetan-Chinese bidirectional translation model, thereby improving the accuracy and fluency of the translation; and the target-side monolingual language model only participates in the machine translation model distillation training and does not participate in the machine translation decoding stage, which can effectively improve the decoding speed and reduce resource and storage requirements, and is more conducive to deployment on platforms with limited computing resources.

[0112] It should be noted that, through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0113] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0114] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a Tibetan-Chinese neural machine translation system based on language model knowledge distillation, characterized in that: The method is executed by a computer and comprises the following steps: S1, collecting Tibetan-Chinese bilingual corpora and Tibetan-Chinese monolingual corpora respectively, and preprocessing the corpora; S2, selecting a neural machine translation model as a student model, and using the preprocessed Tibetan-Chinese bilingual corpus to train the student model; S3, selecting a monolingual language model as a teacher model, and using the pre-processed Chinese and Tibetan monolingual corpora to train the teacher model respectively; S4, performing distillation training on the student model so that the output probability of the student model fits the output probability of the teacher model; S5. For the student model, perform iterative training and model parameter optimization by calculating the difference between the generated translation and the reference translation.

2. The method for constructing a Tibetan-Chinese neural machine translation system based on language model knowledge distillation as claimed in claim 1, characterized in that: Step S1 specifically includes: S101, for bilingual corpus, using Tibetan-Chinese bilingual parallel sentence pairs, and preprocessing the Tibetan-Chinese bilingual parallel sentence pairs to obtain a Tibetan-Chinese bilingual text corpus set of a first predetermined number of levels, wherein the preprocessing includes: removing repeated sentence pairs, word segmentation, byte pair encoding processing, and filtering noise data; S102. For monolingual corpus, using web crawling technology to collect Chinese text corpus in the news field from Chinese government news websites; and using web crawling technology to collect Tibetan text corpus in the news field from Tibetan-related government news websites, and preprocessing the Chinese and Tibetan monolingual text corpora respectively to obtain Chinese text corpora and Tibetan text corpora of a second predetermined number of levels, wherein the preprocessing includes: sentence segmentation, removal of repeated sentences, word segmentation, byte pair encoding processing, and filtering of noise data.

3. The method for constructing a Tibetan-Chinese neural machine translation system based on language model knowledge distillation as claimed in claim 1, characterized in that: The step S2 specifically includes: Use the pre-trained Transformer-Base model as the student model (Tibetan-Chinese and Chinese-Tibetan machine translation model); Taking Chinese as the source language and Tibetan as the target language, the encoder of the Transformer-Base model successively converts the Tibetan in the Tibetan-Chinese bilingual text corpus into a fixed-length vector representation, and decodes the vector representation into Chinese through the decoder, and iteratively trains the "Tibetan→Chinese" direction neural machine translation model; On the contrary, taking Chinese as the source language and Tibetan as the target language, the neural machine translation model in the "Chinese→Tibetan" direction is iteratively trained.

4. The method for constructing a Tibetan-Chinese neural machine translation system based on language model knowledge distillation as described in any one of claims 1 to 2, characterized in that: Use the pre-trained model Transformer-Big as the teacher model.

5. The method for constructing a Tibetan-Chinese neural machine translation system based on language model knowledge distillation as claimed in claim 4, characterized in that: The step S4 specifically includes: On the basis of the student model loss in step S3, a distillation loss is introduced between the teacher model and the student model. The distillation loss is used to minimize the divergence between the output probability of the target-side monolingual model and the output probability of the translation model from the source language to the target language. A weight coefficient λ is configured for the distillation loss to control the influence of the distillation loss on the student model training.

6. The method for constructing a Tibetan-Chinese neural machine translation system based on language model knowledge distillation as claimed in claim 5, characterized in that: The step S4 further comprises: A temperature coefficient T is introduced into the distillation loss to obtain the probability distribution of the category labels of the teacher model, and the output probability is smoothed by controlling the temperature coefficient T.

7. The method for constructing a Tibetan-Chinese neural machine translation system based on language model knowledge distillation as claimed in claim 6, characterized in that: The total loss function of the student model is the student loss of the student model plus the distillation loss between the teacher model and the student model. The calculation formula is as follows: Where λ is the coefficient of the distillation loss; T is the temperature coefficient of the distillation loss softmax layer; The student loss function L of the student model TM The calculation formula is as follows: Among them, p TM (y t |y <t ,x) means that for a bilingual sentence pair (x,y), x is the source language Sentences and y <t is the translation sequence generated previously; Distillation loss function L between teacher model and student model KL The calculation formula is as follows: L KL =T 2 D KL (p LM (y t |y <t ;T)||p TM (y t |y <t ,x;T) Among them, D KL (p LM (y t |y <t )||p TM (y t |y <t ,x) is an indicator that marks the difference between the probability distributions of the student model and the teacher model, D KL represents the divergence, p LM (y t |y <t ) represents the output of the teacher model, p TM (y t |y <t ,x) represents the output of the student model, T is the temperature coefficient introduced in the Softmax layer of the distillation loss, and its calculation formula is as follows: Among them, i represents each category and j represents all categories; T(T≥1) When T=1, it represents the traditional Softmax layer; When T>1, the distribution gradually tends to be uniform and the generalization ability of the model is improved.

8. The method for constructing a Tibetan-Chinese neural machine translation system based on language model knowledge distillation as claimed in claim 2, characterized in that: The second predetermined number level is 2 million sentences level and / or 20 million sentences level.

9. The method for constructing a Tibetan-Chinese neural machine translation system based on language model knowledge distillation as claimed in claim 2, characterized in that: The obtained Tibetan-Chinese bilingual text corpus of the first predetermined number of levels is divided into a training set, a verification set and a test set.

Images