Low-resource neural machine translation method based on bidirectional dependency self-attention mechanism

Abstract

The present invention relates to a low-resource neural machine translation method based on a bidirectional dependency self-attention mechanism, and belongs to the technical field of natural language processing. The present invention first obtains parallel corpus from the Internet, cleans the obtained data, then performs word segmentation and dependency parsing on the source language, and then uses the processed parallel data and dependency tag data as input, and inputs them into a Transformer model with a bidirectional dependency self-attention mechanism, and finally obtains the translation result. The low-resource neural machine translation method based on a bidirectional dependency self-attention mechanism proposed in the present invention has achieved good results in bidirectional translation between Chinese-Thai, Chinese-English, and English-German.

Description

Technical Field

[0001] The present invention relates to a low-resource neural machine translation method based on a bidirectional dependency self-attention mechanism, and belongs to the technical field of natural language processing. Background Art

[0002] In recent years, neural machine translation methods have made important progress, and the translation quality between major languages ​​has been close to the level of manual translation. Neural machine translation mainly relies on large-scale corpus. In the case of low resources, the translation quality of the neural machine translation model will be significantly reduced. In order to solve the problem of scarce resources, neural machine translation generally adopts the method of integrating syntactic information. Current research is mainly based on RNNs and LSTM model frameworks, but the Transformer model has become a baseline model, so the research of the present invention is based on the Transformer model. In terms of dependency syntax fusion, generally only the parent word information in the dependency syntax is integrated, and the direction from the child word to the parent word can determine the position information of the parent word in the sentence, obtain the parent word position vector, and enhance the influence of the parent word in the sentence on machine translation. The direction from the parent word to the child word is rarely integrated into the machine translation model, but the present invention believes that the direction from the parent word to the child word can provide the position information of the child word in the sentence, obtain the child word position information in the sentence by traversing, construct the child word weight matrix, and enhance the influence of the child word on machine translation. This clear bidirectional association relationship from the parent word to the child word and from the child word to the parent word may be more effective for machine translation. Therefore, the present invention proposes a method for fusing these two types of bidirectional information into the translation model, thereby improving the performance of machine translation through more comprehensive structural information fusion. Summary of the invention

[0003] The present invention provides a low-resource neural machine translation method based on a bidirectional dependency self-attention mechanism, so as to solve the problem of low translation performance of the low-resource neural machine translation method.

[0004] The technical solution of the present invention is: a low-resource neural machine translation method based on a bidirectional dependency self-attention mechanism, and the specific steps of the low-resource neural machine translation method based on a bidirectional dependency self-attention mechanism are as follows:

[0005] Step 1: Crawling parallel corpora of mutual translation from the Internet, manually screening and aligning them to obtain parallel sentence pairs.

[0006] Step 2: Perform word segmentation, dependency parsing and BPE on the processed parallel corpus. Word segmentation and BPE are performed on bilingual data, but dependency parsing is only performed on the source language side. After parsing, the dependency tags of the source language are obtained.

[0007] Step 3. Based on Step 2, the bidirectional dependency information is integrated into the Transformer model to obtain a Transformer model with a bidirectional dependency self-attention mechanism. The processed parallel corpus is then trained to obtain a trained Transformer model with a bidirectional dependency self-attention mechanism.

[0008] Step 4. Based on Step 3, use the trained Transformer model with bidirectional dependency self-attention mechanism to translate on the test set. After obtaining the translation results, use the BLUE value as an evaluation indicator to evaluate the translation quality.

[0009] As a further solution of the present invention, in Step 1, Chinese-Thai parallel corpora for mutual translation are crawled from the Internet, 1.06 million parallel sentence pairs are manually screened out, and 16,000 BPE dictionaries are constructed for all translation tasks.

[0010] As a further solution of the present invention, in Step 2, the processed parallel corpus is segmented and dependency parsed, and the position information of the sub-word to the parent word in the source language sentence is obtained after parsing, and the parent word position vector is constructed. The parent word position information is then used to obtain all the sub-word dependency information traversed from the parent word to the sub-word without using an external dependency parsing tool, thereby constructing a sub-word weight matrix to obtain bidirectional dependency information.

[0011] As a further solution of the present invention, Step 3 integrates the bidirectional dependency information from the child word to the parent word and from the parent word to the child word into the Transformer multi-head attention mechanism, modifies the multi-head attention mechanism, and constructs a bidirectional dependency self-attention mechanism by integrating the parent word position vector and the child word weight matrix. The bidirectional dependency information is used to provide the model with more comprehensive structural information, thereby improving the translation effect of low-resource neural machine translation.

[0012] The beneficial effects of the present invention are:

[0013] This paper proposes a low-resource neural machine translation method based on a bidirectional dependency self-attention mechanism. By integrating dependency knowledge into the multi-head attention mechanism of the Transformer encoder, the performance of low-resource neural machine translation is effectively improved. Using the bidirectional dependency self-attention mechanism, the translation model can simultaneously pay attention to the dependency information from the parent word to the child word and from the child word to the parent word. The method of integrating these two bidirectional information into the translation model improves the performance of machine translation through more comprehensive structural information fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Schematic diagram of the bidirectional dependency self-attention mechanism in the present invention;

[0015] Figure 2 Schematic diagram of the multi-head attention mechanism in the present invention. DETAILED DESCRIPTION

[0016] Example 1: Figure 1-Figure 2 As shown, the low-resource neural machine translation method based on bidirectional dependency knowledge of the present invention has been carried out in Chinese-Thai, Chinese-English, and English-German. The low-resource neural machine translation method based on bidirectional dependency self-attention mechanism has the following specific steps:

[0017] Step 1. The Chinese-Thai corpus is a parallel corpus of 1.06 million words crawled from the Internet, which is divided into training set, validation set and test set. The Chinese-English corpus is the CWMT corpus, which consists of 8.01 million sentence pairs, as the training set, newsdev2017 is used as the validation set, and newstest2017 is used as the test set. The English-German corpus is the IWSLT14 English-German corpus, which consists of 174,000 sentence pairs, as the training set and validation set, and newstest2015 is used as the test set, as shown in Table 1.

[0018] Table 1: Dataset

[0019]

[0020]

[0021] Step 2: Perform word segmentation, dependency parsing and BPE on the processed parallel corpus. Word segmentation and BPE are performed on bilingual data, but dependency parsing is only performed on the source language side. After parsing, the dependency tags of the source language are obtained.

[0022] The trained Transformer model with bidirectional dependency self-attention mechanism is used to translate on the test set. After obtaining the translation results, the BLUE value is used as an evaluation indicator to evaluate the translation quality.

[0023] Word segmentation: For Chinese, the present invention uses jieba word segmentation, and for Thai, ThaiNLP is used for word segmentation. For English and German, the present invention does not segment them into character levels.

[0024] Dependency syntactic parsing: Chinese uses LTP language cloud platform for word segmentation and dependency parsing, Thai uses spaCy-Thai for word segmentation and dependency parsing, and English and German use Stanford CoreNLP for dependency parsing. In order to ensure that the word segmentation results correspond to the results of dependency parsing, the present invention does not perform word segmentation on the source language sentence before performing dependency parsing, and directly uses the word segmentation results of the dependency parsing tool to ensure that there will be no errors in the input of the model.

[0025] BPE: The present invention uses a BPE vocabulary with a size of 16,000.

[0026] Step 3, integrate the bidirectional dependency information into the Transformer model, use the Transformer model based on the bidirectional dependency self-attention mechanism to train the processed parallel corpus, and obtain the trained Transformer model based on the bidirectional dependency self-attention mechanism.

[0027] Step 3 integrates the bidirectional dependency information from the child word to the parent word and from the parent word to the child word into the Transformer multi-head attention mechanism, modifies the multi-head attention mechanism, and constructs a bidirectional dependency self-attention mechanism by integrating the parent word position vector and the child word weight matrix. The bidirectional dependency information is used to provide the model with more comprehensive structural information, thereby improving the translation effect of low-resource neural machine translation.

[0028] In Step 3, the construction method of the Transformer model of the bidirectional dependency self-attention mechanism is as follows:

[0029] Based on the Transformer framework, the present invention uses the bidirectional dependency self-attention mechanism to improve the multi-head attention mechanism of the Transformer encoder. The input of the bidirectional dependency self-attention mechanism is the embedding matrix of the source language sentence. Source language parent word position vector P∈R L and the source language subword weight matrix C∈R L×L The output is the final representation M of the bidirectional dependent self-attention mechanism. H Step 3 mainly consists of bidirectional dependency information, bidirectional dependency self-attention mechanism, multi-head attention mechanism and bidirectional dependency ignoring. Figure 1 As shown, the figure shows the bidirectional dependency self-attention mechanism.

[0030] Bidirectional dependency information:

[0031] For the parent word information in the source language, the present invention uses an external parsing tool to obtain the parent word position sequence. For the child words, the present invention does not need to provide additional dependency parsing tools, and can construct the child word weight matrix using only the parent word position vector. For the root word, the present invention defines its parent word and child word as the root word itself.

[0032] The present invention first uses an external dependency parsing tool to obtain the parent word position tag sequence in dependency parsing, thereby obtaining the parent word position vector P∈R in the sentence sequence L ,according to Figure 1 The sentence dependency graph in shows the relationship between words in the sentence. The word pointed by the arrow is the child word, and the word pointed by the tail of the arrow is the parent word. From this, we can know the position of the parent word to which each child word belongs in the sentence, and thus we can get the parent word position vector P∈R L .like Figure 1 In the sentence, the parent word of "both" is "appear", and "appear" is in the third position in the sentence, so the first position in the parent word position vector is 3. By analogy, we can get the parent word position vector P∈R in the figure L .

[0033] Then according to the parent word position vector P∈R L Get the subword weight matrix C∈R in the source language sentence L×L Equation (1) gives the definition of the subword weight matrix C. Assume that x i is a possible parent word, then when x j is x i When the subword of ij 1, otherwise 0. For each sentence, each word in the sentence is made to correspond to itself. Since there may be multiple subwords for a parent word in each sentence, the present invention averages the weights of these subwords. For a root word, its subwords are taken as itself and the weights are recorded. In this way, each word is informed of its modifiers.

[0034]

[0035] where n i is x i The number of subwords. Figure 1 From the dependency graph in , we can see that each parent word has several subwords. For example, the word "appear" in the sentence has 4 subwords including itself. After the subword weights are averaged, the position of each subword in the third row of the subword weight matrix is ​​1 / 4, and the rows of the other words without subwords are all 0, which can be obtained. Figure 1 The subword weight matrix C∈R in L×L .

[0036] Bidirectional Dependency Self-Attention Mechanism:

[0037] exist Figure 1 In the example, for a source language sentence sequence of length L, the input of each head in the bidirectional dependency self-attention mechanism is the embedding matrix The parent word position vector P∈R of the source language sentence L and the subword weight matrix C∈R of the source language sentence L×L According to Vaswani et al.

[12] In each attention mechanism head H, calculations are performed for each tag, and three vectors are obtained, namely query, key and value, thus obtaining three matrices K h ∈R L×d , Q h ∈R L×d and V h ∈R L×d , where d = d model / H. Then, for each query, key, and value, we calculate the focus score that should be set at other positions in the input when encoding at a given position, and divide the score by It can alleviate the problem of gradient disappearance when the dot product is large:

[0038]

[0039] According to the dependent parent word position p of each token and position t t The distance between them is the mark score s at position t. t :

[0040]

[0041]

[0042] in is the parent word fusion matrix N h ∈R L×L The t-th row of represents the normalized score of proximity to the t-th parent word. is the parent word Gaussian matrix D p ∈R L×L The (t,j)th position of t Contains the distance relationship between each token j and each parent word position in the dependency knowledge. This distance calculation is defined as p t is centered and has variance σ 2 N(p t ,σ 2 ) is the value of the probability density of the normal distribution:

[0043]

[0044] According to the score matrix S h ∈RL×L And the parent word distance matrix D p ∈R L×L The parent word fusion matrix N can be obtained h ∈R L×L :

[0045] N h =S h D p (6)

[0046] Then, the parsed parent word information is used to construct the subword weight matrix C∈R L×L , this weight matrix is ​​constructed according to the number of subwords each word has in the input sequence. Equation (1) gives the definition of the subword weight matrix C.

[0047] Depend on Figure 1 It can be seen that the subword weight matrix C∈R L×L There are too many zero elements in the matrix. For these zero elements, the present invention uses Gaussian noise to process them, and the subword weight matrix C∈R L×L Add Gaussian noise to the elements in c. t is centered and has variance σ 2 , N(c t ,σ 2 ), (t,j) represents each element in the sub-word weight matrix. This method can be used to obtain the sub-word Gaussian weight matrix C g ∈R L×L :

[0048]

[0049] Get the subword Gaussian weight matrix C g ∈R L×L Finally, in order to improve the convergence speed of the model, the matrix is ​​normalized using the softmax function to obtain the subword dependency matrix C s ∈R L×L :

[0050] C s =softmax(C g ) (8)

[0051] According to the parent word fusion matrix N h ∈R L×L and the subword dependency matrix C s ∈R L×L , we can get the subword fusion matrix Z h ∈R L ×L :

[0052] Z h=N h C s (9)

[0053] Finally, the softmax function is applied to generate weight assignments for each token in the sentence, and the resulting subword fusion matrix Z is h ∈R L×L and value matrix V h ∈R L×d Multiply them together to obtain the final representation M of the bidirectional dependent self-attention mechanism head H H .

[0054] Since the present invention integrates dependency tags into the translation model, the cross entropy loss function is used when calculating the loss L, as shown in equation (10).

[0055] L=-[l i log(p i )+(1-l i )log(1-p i )] (10)

[0056] Among them l i Represents the tag of the i-th parent word, the positive class is 1 and the negative class is 0. i represents the probability that the i-th sample is positive.

[0057] The present invention utilizes both parent word information and child word information without configuring other additional training parameters. p ∈R L×L The distance depends only on the position vector of the parent word, while the subword weight matrix C s ∈R L×L It only depends on the number of subwords each word has in the sentence, and when constructing the subword weight matrix, the present invention does not need to use an external parser, which can save a lot of time for parsing the source language before training the model.

[0058] Multi-head attention mechanism:

[0059] The bidirectional dependency self-attention mechanism is an extension of the dot product attention mechanism in the multi-head attention mechanism. Figure 2 The bidirectional dependency self-attention mechanism is integrated into the multi-head attention mechanism. The present invention makes design choices in the integration method of the multi-head attention mechanism. The bidirectional dependency self-attention mechanism of the present invention is integrated only in the eight heads of the first layer of the multi-head attention mechanism, such as Figure 2 As shown in the figure, the overall structure of the encoding end has not changed. The input sentence x is word-embedded and position-encoded before being input into the multi-head attention mechanism. In the multi-head attention mechanism, the linearized query Q, key K, value V, and parent word position vector P∈R Land the subword weight matrix C∈R L×L As input, it is input into the bidirectional dependency self-attention mechanism to obtain the final representation M of each head H The rest of the processing is the same as the basic Transformer model, and the present invention does not change other positions on the encoding end. The bidirectional dependency self-attention mechanism is integrated into the 8 heads of the multi-head attention mechanism, and the best effect is achieved only in the first layer, which is verified in the subsequent experiments of the present invention, thereby proving the effectiveness of the present invention in integrating the bidirectional dependency self-attention mechanism.

[0060] Bidirectional dependency information is ignored:

[0061] Due to the lack of corpora parallel to standard parsing tools, the parent word dependency knowledge of the present invention relies on the results of external dependency parsing tools. To prevent the model from overfitting to noisy dependencies, the present invention introduces two regularization techniques for the bidirectional dependency self-attention mechanism, namely, parent word ignoring and child word ignoring. This method is similar to the dropout method of Srivastavaet et al., which ignores certain parent word and child word information during the model training phase. By adding the parent word distance matrix D p ∈R L×L and the subword dependency matrix C s ∈R L×L Each row in is randomly set to 1∈R L To ignore the position information of the parent word and the dependency information of the child word.

[0062] Step 4: Use the trained Transformer model with bidirectional dependency self-attention mechanism to translate on the test set, obtain the translation results and perform quality assessment.

[0063] The present invention conducts experiments on the Chinese-Thai dataset, the CWMT Chinese-English dataset, and the IWSLT14 English-German dataset. The experimental results are shown in Table 2.

[0064] Table 2 Experimental results of the model based on the bidirectional dependency self-attention mechanism

[0065]

[0066] Table 2 shows the experimental results of the present invention. In the bidirectional translation between Chinese and Thai, Bi-Dependency improves the translation results of Transformer by 0.71 and 0.86 BLUE values ​​respectively. In the Chinese-English translation task, Bi-Dependency also significantly improves the BLUE values ​​by 0.79 and 0.68. In English-German, the translation results of Bi-Dependency are improved compared with Transformer but not significantly. Compared with the LSTM model, the translation quality of Bi-Dependency has been greatly improved in the three groups of translation tasks. The present invention mainly targets low-resource languages ​​and improves the translation quality by integrating bidirectional dependency knowledge. According to Table 2, the present invention has a significant improvement in the Chinese-Thai translation task, indicating that the bidirectional dependency self-attention mechanism proposed in the present invention is effective.

[0067] In order to verify the role of integrating bidirectional dependency information of the source language, the present invention conducted an effectiveness experiment of integrating bidirectional dependency information on the Chinese-Thai dataset. The definition of "Transformer+CWord" indicates that only the subword information in the dependency knowledge is integrated, "Pascal" indicates that only the parent word information in the dependency knowledge is integrated, and the definition of "Bi-Dependency" indicates the model framework proposed by the present invention. The experimental results are shown in Table 3.

[0068] Table 3 Experimental results on the impact of bidirectional dependency information on translation results

[0069]

[0070] From the experimental results in Table 3, it can be seen that Bi-Dependency achieved the best results. According to the above results, it can be seen that the fusion of dependency knowledge on the source language side is significantly improved compared with the basic Transformer model, indicating that the fusion of dependency syntactic knowledge on the source language side is helpful for translation tasks. The Bi-Dependency translation model proposed in the present invention achieved the highest BLUE value in the Chinese-Thai bidirectional translation task and obtained the best translation effect, indicating that the fusion of bidirectional dependency knowledge on the source language side has a good help effect on translation under low-resource conditions.

[0071] The present invention also conducts bidirectional dependency self-attention mechanism layer experiments on the Han-Thai dataset at different layers of the encoding end to verify at which layer it is more effective to integrate bidirectional dependency knowledge. The experimental results are shown in Table 4.

[0072] Table 4 Experimental results on the impact of bidirectional dependency information fusion at different layers on the encoding end on translation results

[0073]

[0074]

[0075] Table 4 shows the experimental results of the bidirectional dependency self-attention mechanism at different layers of the encoding end. It can be seen from Table 4 that the Bi-Denpendency model achieves the best results by integrating bidirectional dependency syntactic knowledge in the first layer of the encoding end. Such results confirm that in the first layer, more attention is focused only on the words to be translated, rather than their context. It can be inferred that integrating syntactic relevance in the first layer can effectively learn better word representations, thereby further improving the translation accuracy of the Transformer model.

[0076] In order to verify the effect of adding Gaussian noise in the sub-word weight matrix, the present invention conducted a Gaussian weight matrix experiment on the Chinese-Thai dataset, and defined "Bi-Dependency-GWF" to indicate that no Gaussian weight function was added to the sub-word weight matrix. The experimental results are shown in Table 5.

[0077] Table 5 Experimental results on the impact of Gaussian weight matrix on translation results

[0078]

[0079] According to Table 5, Bi-Dependency achieves the best results. The translation results without adding Gaussian noise in the bidirectional dependency self-attention mechanism are significantly reduced, and the results are lower than those of Bi-Dependency and Transformer models. This proves that the addition of Gaussian noise is necessary and effective.

[0080] The present invention proposes a bidirectional dependency self-attention mechanism for low-resource translation tasks. Experiments have shown that, in low-resource situations, by integrating bidirectional dependency knowledge into the multi-head attention mechanism, low-resource translation tasks can be improved to a certain extent. Comparative experiments have also shown that the use of bidirectional dependency knowledge can provide more comprehensive structural information to the translation model, which is more effective for low-resource translation tasks.

[0081] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A low-resource machine translation method based on a bidirectional dependency self-attention mechanism, characterized in that: The specific steps of the method are as follows: Step 1: Crawling parallel corpora for mutual translation, and obtaining parallel sentence pairs through manual screening and alignment; Step 2: Perform word segmentation, dependency parsing and BPE on the processed parallel corpus. Word segmentation and BPE are performed on bilingual data, but dependency parsing is only performed on the source language side. After parsing, the dependency tags of the source language are obtained. Step 3: Based on Step 2, the bidirectional dependency information is integrated into the Transformer model to obtain a Transformer model with a bidirectional dependency self-attention mechanism. The processed parallel corpus is then trained to obtain a trained Transformer model with a bidirectional dependency self-attention mechanism. Step 4: Based on Step 3, use the trained Transformer model with bidirectional dependency self-attention mechanism to translate on the test set. After obtaining the translation results, use the BLUE value as an evaluation indicator to evaluate the translation quality. In the Step 2, the processed parallel corpus is segmented and dependency parsed, and the position information of the sub-word to the parent word in the source language sentence is obtained after parsing using an external dependency parsing tool, and a parent word position vector is constructed. The parent word position information is then used to obtain all sub-word dependency information traversed from the parent word to the sub-word without using an external dependency parsing tool, thereby constructing a sub-word weight matrix and obtaining bidirectional dependency information; Step 3 integrates the bidirectional dependency information from the subword to the parent word and from the parent word to the subword into the Transformer multi-head attention mechanism, modifies the multi-head attention mechanism, and constructs a bidirectional dependency self-attention mechanism by integrating the parent word position vector and the subword weight matrix, and uses the bidirectional dependency information to provide more comprehensive structural information for the model; In Step 3, the construction method of the Transformer model of the bidirectional dependency self-attention mechanism is as follows: Based on the Transformer framework, the multi-head attention mechanism of the Transformer encoder is improved by using the bidirectional dependency self-attention mechanism; the input of the bidirectional dependency self-attention mechanism is composed of the embedding matrix of the source language sentence Source language parent word position vector P∈R L and the source language subword weight matrix C∈R L×L The output is the final representation M of the bidirectional dependent self-attention mechanism. H ; Step 3 consists of bidirectional dependency information, bidirectional dependency self-attention mechanism, multi-head attention mechanism and bidirectional dependency ignoring; Bidirectional dependency information: For the parent word information in the source language, use external parsing tools to get the parent word position sequence. For the child words, only the parent word position vector can be used to construct the child word weight matrix. For the root word, its parent word and child word are defined as the root word itself. First, use the external dependency parsing tool to obtain the parent word position tag sequence in the dependency parsing, so as to obtain the parent word position vector P∈R in the sentence sequence L ; Then according to the parent word position vector P∈R L Get the subword weight matrix C∈R in the source language sentence L×L ; Equation (1) gives the definition of the subword weight matrix C, assuming x i is a possible parent word, then when x j is x i When the element C ij is 1, otherwise it is 0; for each sentence, make each word in the sentence correspond to itself; the parent word in each sentence may have multiple subwords, and these subwords are weighted averagely; for the root word, take its subwords as itself and record the weights; in this way, each word will be informed of its modifiers; Among them, n i is x i The number of subwords; Bidirectional Dependency Self-Attention Mechanism: For a source language sentence sequence of length L, the input of each head in the bidirectional dependency self-attention mechanism is the embedding matrix The parent word position vector P∈R of the source language sentence L and the subword weight matrix C∈R of the source language sentence L×L ; In each attention mechanism head H, calculations are performed for each tag to obtain three vectors, namely query, key and value, thus obtaining three matrices K h ∈R L×d , Q h ∈R L×d and V h ∈R L×d , where d = d model / H; then calculate for each query, key, and value the focus score to be set at other positions in the input when encoding at a given position, and divide the score by Alleviate the problem of gradient vanishing when the dot product is large: According to the dependent parent word position p of each token and position t t The distance between them is the mark score s at position t. t : in, is the parent word fusion matrix N h ∈R L×L The t-th row of represents the normalized score of proximity to the t-th parent word; is the parent word Gaussian matrix D p ∈R L×L The (t,j)th position of t Contains the distance relationship between each token j and each parent word position in the dependency knowledge. This distance calculation is defined as p t is centered and has variance σ 2 N(p t ,σ 2 ) is the value of the probability density of the normal distribution: According to the score matrix S h ∈R L×L And the parent word distance matrix D p ∈R L×L Get the parent word fusion matrix N h ∈R L×L : N h =S h D p (6) Then, the parsed parent word information is used to construct the subword weight matrix C∈R L×L , this weight matrix is ​​constructed according to the number of subwords each word has in the input sequence. Formula (1) gives the definition of the subword weight matrix C; Subword weight matrix C∈R L×L There are too many zero elements in the matrix, and these zero elements are processed by adding Gaussian noise, which is the subword weight matrix C∈R L×L Add Gaussian noise to the elements in c t is centered and has variance σ 2 , N(c t , σ 2 ), (i, j) represents each element in the sub-word weight matrix, and the sub-word Gaussian weight matrix C is obtained in this way. g ∈R L×L : Get the subword Gaussian weight matrix C g ∈R L×L Finally, the softmax function is used to normalize the matrix to obtain the subword dependency matrix C s ∈R L×L : C s =softmax(C g ) (8) According to the parent word fusion matrix N h ∈R L×L and the subword dependency matrix C s ∈R L×L , get the subword fusion matrix Z h ∈R L×L : Z h =N h C s (9) Finally, the softmax function is applied to generate weight assignments for each token in the sentence, and the resulting subword fusion matrix Z is h ∈R L×L and value matrix V h ∈R L×d Multiply them together to obtain the final representation M of the bidirectional dependent self-attention mechanism head H H ; When calculating the loss L, the cross entropy loss function is used, as shown in formula (10); L=-[l i log(p i )+(1-l i )log(1-p i )] (10) Among them l i Indicates the tag of the i-th parent word, the positive class is 1, the negative class is 0, p i represents the probability that the i-th sample is positive.

2. The low-resource neural machine translation method based on bidirectional dependency self-attention mechanism according to claim 1, characterized in that: In the Step 1, Chinese-Thai parallel corpora for mutual translation are crawled from the Internet, 1.06 million parallel sentence pairs are manually screened out, and 16,000 BPE dictionaries are constructed for all translation tasks.

Images