A neural machine translation method based on local phrase syntax enhancement mechanism

Abstract

The present invention relates to a neural machine translation method based on a local phrase syntax enhancement mechanism, and belongs to the technical field of natural language processing machine translation. First, a local phrase sequence of each element is constructed, and a relative phrase position vector matrix is ​​constructed through the local phrase sequence. Then, redundant heads are identified, and the importance of each head in the training process is dynamically determined according to the average weight of syntactic attention. Finally, an attention mechanism based on syntax enhancement of local phrases is used to replace the original redundant heads to activate their potential capabilities. The present invention can accurately identify important heads and redundant heads in a multi-head attention mechanism, and modify redundant heads through a syntax enhancement mechanism based on local phrases. The constructed self-attention network can be used for machine translation research, and has achieved good results.

Description

Technical Field

[0001] The present invention relates to a technology for utilizing local phrases to activate redundant heads in a multi-head self-attention mechanism in neural machine translation, and in particular to a neural machine translation method based on a local phrase syntax enhancement mechanism, belonging to the technical field of natural language processing machine translation. Background Art

[0002] The multi-head self-attention mechanism has attracted attention due to its significant parallel computing capabilities and flexible scalability. However, recent research has shown that not all self-attention heads provide positive feedback to neural networks, with some contributing little. Therefore, pruning these heads can improve the overall network performance.

[0003] In recent years, self-attention networks have been applied to various NLP (Natural Language Processing) tasks. Unlike recurrent neural networks and convolutional neural networks, self-attention networks capture both short-term and long-term relationships between words. By averaging weights, self-attention networks focus more on specific relationships. The Transformer, which performs well in machine translation, uses a multi-head attention network to combine attention mechanisms from different subspaces. Researchers have pointed out that the encoder-side attention heads in the Transformer contribute differently to the network and can be divided into important and redundant attention heads. Pruning redundant attention heads does not significantly affect performance. They also hypothesize that important heads have different degrees of influence in translation, including syntax, position, and functions based on rare words.

[0004] However, these methods consider the multi-head self-attention network as a whole, ignoring the uneven contribution of self-attention heads to the network. In addition, their understanding of the multi-head in the self-attention network is very limited, and there is no systematic analysis of the role of different heads. Summary of the Invention

[0005] This paper aims to address the technical problem of existing multi-head self-attention networks being unable to effectively utilize redundant self-attention heads. It creatively proposes a neural machine translation method based on a local phrase syntax enhancement mechanism. This method can reflect the relative phrase positions between words, enriching syntactic relationships.

[0006] The technical solution adopted in the present invention is as follows:

[0007] A neural machine translation method based on a local phrase syntax enhancement mechanism includes the following steps:

[0008] Step 1: Construct the local phrase sequence of each element.

[0009] Step 2: Construct a relative phrase position vector matrix based on the local phrase sequence.

[0010] Step 3: Identify redundant heads and dynamically determine the importance of each head during training based on the average weight of syntactic attention.

[0011] Step 4: Use the local phrase-based syntax-enhanced attention mechanism to replace the original redundant head to activate its potential capabilities.

[0012] Beneficial effects

[0013] Compared with the prior art, the present invention has the following beneficial effects and advantages:

[0014] 1. This paper constructs a component syntactic tree based on local phrase rules and converts it into a learnable relative local phrase position matrix, which is then added as a bias to the attention mechanism. Compared to the basic Transformer model, the BLEU value is improved by an average of 0.95 percentage points.

[0015] 2. The present invention can reflect the structural distribution of sentences and accurately express the hierarchical relationship between two words. It is superior to the existing dependency tree method in terms of overall structure.

[0016] 3. This paper simulates the specific functions of important heads through a syntactically enhanced attention mechanism, which is more effective than the existing improved self-attention method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flowchart of the method of the present invention.

[0018] Figure 2 Schematic diagram of the process of extracting relative phrase position offset in the present invention. DETAILED DESCRIPTION

[0019] The method of the present invention will be further described below with reference to the accompanying drawings.

[0020] like Figure 1 As shown, a neural machine translation method based on a local phrase syntax enhancement mechanism includes the following steps:

[0021] Step 1: Construct the local phrase sequence of each element.

[0022] The syntactic-enhanced attention mechanism based on local phrases is a syntactic extension of the standard self-attention mechanism. Unlike the standard mechanism, this method focuses on the relative position information between words.

[0023] The standard attention mechanism is as follows:

[0024]

[0025] Among them, Att() represents the attention mechanism, Q i represents the query query of word i, K j represents the key value of word j, x i Represents the word embedding vector of word i, x j Represents the word embedding vector of word j, W Q Represents the query weight matrix, W K represents the key value weight matrix, T represents the transpose operation of the matrix, d k Represents the vector dimension.

[0026] The present invention further adds a local phrase bias based on the standard attention mechanism, as shown in Formula 2:

[0027]

[0028] Among them, P i,j represents the local phrase bias matrix.

[0029] A local phrase contains syntactically related word pairs, rather than simply judging by sequence distance. Therefore, a local phrase contains hierarchical syntactic connections between words, and its rules are as follows:

[0030] Rule 1: Given an m-level component tree and word x i and its ancestor node sequence ast=(ast layer(xi)-1 ,…,ast0), where ast0 represents the ancestor node at level 0, ast layer(xi)-1 Represents word x i The ancestor node of the previous layer, layer(x i ) represents word x i The number of layers, then the local phrase local_phrase(x i ) including the lowest level multi-descendant ancestor node ast layer(xi)-m Words, where 0<=m<=layer(x i ).

[0031] Rule 2: If word x i Contained in word x j In the local phrase sequence of i The local phrase sequence also contains x j .

[0032] Step 2: Construct a relative phrase position vector matrix through the local phrase sequence.

[0033] like Figure 2As shown in Figure 1, to obtain the local phrase bias vector p, we first need to extract the relative phrase position matrix RP from the component tree.

[0034] Specifically, given a RP∈R u*u , where RP represents the relative phrase position matrix, u represents the matrix dimension, and each element represents the word x i and word x j The relative syntactic distance between them. For word pairs that are not in the same relative phrase sequence, their relative distance is set to positive infinity; for word pairs that are in the same relative phrase sequence, their relative phrase position distance is set to the difference in their relative phrase levels. This results in the matrix RP.

[0035] Since the precise relative phrase position information is not very useful when the distance is greater than a certain value, the maximum relative phrase distance is set to a fixed value k, and 2k+1 different labels are used for the relative phrase position vector, and the matrix RP is converted into the corresponding vector matrix M∈R u*u*dh ,in:

[0036]

[0037] Among them, M ij Represents the corresponding vector matrix, clip(x,k) means taking the maximum value within [-k,k], and taking the boundary value if it exceeds the range.

[0038] After that, we learn the relative phrase distance representation w = (w -k ,…,w k ), where w i ∈R dh , R dh Represents the vector space of dh dimension.

[0039] After obtaining the matrix M, a feedforward network is constructed to transform the relative local phrase position vector M ij is converted into a relative local phrase position hidden state. Further, through the linear projection U P ∈R dh * 1 Map the hidden state to a negative scalar P in the local phrase bias matrix P ij , that is, each element of the matrix P:

[0040] P=-|tanh(W P M+b P )U P | (4)

[0041] Among them, W P ∈R dh*dh and b P ∈R dhare all parameters of the model.

[0042] Step 3: Identify redundant heads and dynamically determine the importance of each head during training according to the average weight of syntactic attention.

[0043] Specifically, by dynamically identifying redundant heads that lack the ability to capture short-term and long-term syntactic relations, we combine them with a syntactic enhancement strategy to enhance their syntactic capabilities.

[0044] First, add the dependency mask Dep_mask to the attention weight matrix to obtain the corresponding syntactic attention weights reflecting the short-term and long-term syntactic relationships. Then, for each x in the sequence i Its syntax is related to x j , find the sum of its syntactic attention weights. Finally, calculate the average Syn attn as follows:

[0045]

[0046] Here, n represents the number of words.

[0047] The present invention further proposes criteria for determining important headers and redundant headers, which are as follows:

[0048] When the syntactic attention weight of an attention head is greater than the average of the maximum attention weights, it is considered an important head and has syntactic functionality. Existing methods simply use a fixed threshold to measure the importance of each attention head for all training instances, while this method dynamically identifies the role of the attention head for each sentence during training.

[0049] By comparing the syntactic attention weights Syn attn With dynamic learnable syntactic threshold Syn gate To determine the function of the attention head, where Syn gate is the function of confidence h conf The sigmoid activation function, confidence h conf Represents the mean of the sum of the maximum attention weights of an attention head for n words in the sequence:

[0050] Syn gate =sigmoid(h conf ) (6)

[0051]

[0052] If Syn attn Smaller than Syn gate , the current attention head is considered redundant:

[0053]

[0054] Among them, h label Indicates whether the attention head is redundant, 1 represents non-redundant and 0 represents redundant.

[0055] Step 4: Use the local phrase-based syntax-enhanced attention mechanism to replace the original redundant head to activate its potential capabilities.

[0056] Specifically, after distinguishing the important and redundant heads in the multi-head self-attention in step 2, the redundant heads are activated without affecting the important heads according to the syntactic bias mentioned in step 1.

[0057] Previous work has shown that redundant heads are primarily distributed in the lower layers of the encoder, and that directly using word embeddings as input to the lower layers of the encoder is more conducive to modeling local syntactic relationships. This paper evaluates the performance of this approach on both lower and higher-level encoders, maintaining the best performance when applied to the first-level encoder.

[0058] Example verification

[0059] The present invention is experimented on English-German (En-De) and English-Czech (En-Cs).

[0060] (1) Experimental data setting

[0061] In order to compare with the research results of predecessors and get closer to the actual translation scenario, the present invention conducted experiments on English-German and English-Czech. For English-German, the 4.5M-scale dataset of WMT14 and the 0.22M-scale dataset of WMT16 News Commentary were used as training sets, and newstest2013-2016 were used as development sets and test sets respectively; for English-Czech, the 0.18M-scale corpus of WMT16 News Commentary was used as training set, and newstest2015 and newstest2016 were used as development sets and test sets respectively. The present invention also used the Berkeley Neural Parser of the University of Berkeley to generate the component syntax tree on the English side. In addition, the present invention used the BPE method to segment words and used 4-gram NIST BLEU as the evaluation indicator.

[0062] (2) Baseline system experimental setup

[0063] Transformer: A standard self-attention model.

[0064] Rel_Pos: relative position encoding, taking into account the relative positions between sequence elements.

[0065] Locality: Local modeling enhances the ability of self-attention with learnable Gaussian bias to capture local context.

[0066] (3) Main experimental results

[0067] Table 1 Ablation experiment results

[0068] system En-De(WMT16) En-De(WMT14) En-Cs(WMT16) Transformer 25.28 27.31 15.46 +Rel_Pos 25.49 27.53 15.60 +Localness 25.53 27.61 15.65 +LPEA 25.90 27.96 15.97 +LPEA+RHE 26.28 28.27 16.27

[0069] Table 1 shows the results of ablation experiments using the Local Phrase-Based Syntactic Enhancement (LPEA) mechanism and the Redundant Head Activation (RHE) strategy. First, the "+LPEA" row represents the first encoding layer of all self-attention heads, including both important and redundant heads, replaced with LPEA. Second, the "+LPEA+RHE" row replaces only the redundant heads with LPEA, significantly improving the performance of the LPEA mechanism applied to language pairs and testing the effectiveness of identifying and modifying redundant heads without interfering with the functionality of important heads.

[0070] For the baseline system, although Rel_Pos and Localness both enhance the effect of the self-attention network in their own way, the LPEA and LPEA+RHE methods of the present invention surpass the standard Transformer and the above baseline systems and achieve superior results.

[0071] Furthermore, we evaluated the impact of varying the maximum relative local phrase position distance k on experimental performance. Table 2 shows that as k increases from 0 to 8, the average maximum phrase level in the training set is 11.13, similar to the values ​​for k = 5 and k = 6 (2k + 1 is 11 and 13, respectively). This result indicates that the best performance is achieved when the relative phrase position vector exactly covers the average maximum phrase level.

[0072] Table 2 The influence of the maximum relative local phrase position distance k on the results

[0073] k 0 2 4 6 8 BLEU 25.28 25.57 25.86 26.28 25.51 Difference - +0.29 +0.58 +1.0 +0.23

Claims

1. A neural machine translation method based on a local phrase syntax enhancement mechanism, characterized in that: The following steps are involved: Step 1: Construct local phrase sequences of each element: On the basis of the standard attention mechanism, a local phrase bias is further added: Among them, P i,j represents the local phrase bias matrix; Att() represents the attention mechanism; Q i represents the query query of word i; K j represents the key value of word j; x i Represents the word embedding vector of word i; x j Represents the word embedding vector of word j; W Q Represents the query weight matrix; W K Represents the key value weight matrix; T represents the transpose operation of the matrix, d k Represents vector dimension; Step 2: Construct a relative phrase position vector matrix through the local phrase sequence; The following steps are involved: To obtain the local phrase bias vector p, we first need to extract the relative phrase position matrix RP from the component tree; Given a RP∈R u*u , where RP represents the relative phrase position matrix, u represents the matrix dimension, and each element represents the word x i and word x j the relative syntactic distance between For word pairs that are not in the same relative phrase sequence, their relative distance is set to positive infinity; for word pairs that are in the same relative phrase sequence, their relative phrase position distance is set to the difference in their relative phrase levels, thereby obtaining the matrix RP; Set the maximum relative phrase distance to a fixed value k and use 2k+1 different labels for the relative phrase position vectors, transforming the matrix RP into the corresponding vector matrix M∈R u*u*dh ,in: M ij =w clip(j-i,k) clip(x,k))=max(-k,min(k,x)) Among them, M ij Represents the corresponding vector matrix, clip(x,k) means taking the maximum value within [-k,k], and taking the boundary value if it exceeds the range; After that, we learn the relative phrase distance representation w = (w -k ,…,w k ), where w i ∈R dh , R dh Represents the vector space of dh dimension; After obtaining the matrix M, a feedforward network is constructed to transform the relative local phrase position vector M ij Transformed into relative local phrase position hidden state; through linear projection U P ∈R dh*1 Map the hidden state to a negative scalar P in the local phrase bias matrix P ij , that is, each element of the matrix P: P=-|tanh(W P M+b P )U P | Among them, W P ∈R dh*dh and b P ∈R dh are all parameters of the model; Step 3: Identify redundant heads and dynamically determine the importance of each head during training based on the average weight of syntactic attention; Step 4: Use the local phrase-based syntax-enhanced attention mechanism to replace the original redundant head and activate its potential capabilities.

2. The neural machine translation method based on the local phrase syntax enhancement mechanism according to claim 1, characterized in that: The local phrases satisfy the following rules: Rule 1: Given an m-level component tree and word x i and its ancestor node sequence ast=(ast layer(xi)-1 ,…,ast0), where ast0 represents the ancestor node at level 0, ast layer(xi)-1 Represents word x i The ancestor node of the previous layer, layer(x i ) represents word x i The number of layers, then the local phrase local_phrase(x i ) including the lowest level multi-descendant ancestor node ast layer(xi)-m Words, where 0<=m<=layer(x i ); Rule 2: If word x i Contained in word x j In the local phrase sequence of i The local phrase sequence also contains x j .

3. The neural machine translation method based on the local phrase syntax enhancement mechanism according to claim 1, characterized in that: In step 3, first, the dependency mask Dep_mask is added to the attention weight matrix to obtain the corresponding syntactic attention weights reflecting the short-term and long-term syntactic relationships; then, for each x in the sequence i Its syntax is related to x j , find the sum of its syntactic attention weights; finally, calculate the average Syn of the syntactic attention weights attn as follows: Here, n represents the number of words.

4. The neural machine translation method based on the local phrase syntax enhancement mechanism according to claim 3, characterized in that: The criteria for determining important headers and redundant headers are as follows: When the syntactic attention weight of an attention head is greater than the average of the maximum attention weights, it is considered to be an important head and it has syntactic function; By comparing the syntactic attention weights Syn attn With dynamic learnable syntactic threshold Syn gate To determine the function of the attention head, where Syn gate is the function of confidence h conf The sigmoid activation function, confidence h conf Represents the mean of the sum of the maximum attention weights of an attention head for n words in the sequence: Syn gate =sigmoid(h conf ) If Syn attn Smaller than Syn gate , the current attention head is considered redundant: Among them, h label Indicates whether the attention head is redundant, 1 represents non-redundant and 0 represents redundant.

Images