A translation method based on Runge-Kutta high-order representation

Abstract

The present invention discloses a translation method based on Runge-Kutta high-order representation, the steps are: inputting the pre-processed source language into the encoder end of the translation model, establishing the implicit connection between the residual connection and the numerical solver of the ordinary differential equation; introducing the high-order Runge-Kutta method for the structural design of the network, using the 2nd-order Runge-Kutta and 4th-order Runge-Kutta methods to redesign the Transformer structure; adjusting the coefficient of the Runge-Kutta intermediate representation to 1; adaptively adjusting the weights of different intermediate terms with the update of the overall network parameters; obtaining the coefficient corresponding to each intermediate representation by automatic learning using the intermediate state of different time steps obtained by the high-order method, obtaining a more accurate layer representation, and realizing the translation method based on Runge-Kutta high-order representation. The method of the present invention has achieved significant performance improvement under different model capacities, and has achieved consistent performance improvement in scarce resource languages ​​and rich resource language tasks.

Description

Technical Field

[0001] The present invention relates to a machine language translation technology, in particular to a translation method based on Runge-Kutta high-order representation. Background Art

[0002] Machine translation refers to the process of mapping one language to another. This machine translation model, which is trained based on massive text data, has greatly alleviated the pressure of manual translation. With the iteration of machine translation technology, from the initial rule-based machine translation, to the statistical algorithm-based machine translation in the early 21st century, to the current neural network-based machine translation, the accuracy of machine translation has become higher and higher. At present, the Transformer model based on the self-attention mechanism has achieved translation accuracy comparable to that of humans, and its translation fluency has been significantly improved compared to the statistical-based machine translation model. This has also stimulated the development of the auxiliary translation industry that combines machine translation and manual translation.

[0003] At present, the Transformer model based on the self-attention mechanism has achieved the best performance in many machine translation tasks due to its highly parallel computing power and performance advantages. It is the most mainstream neural machine translation framework today. It still uses the encoder-decoder structure. The encoder and decoder are respectively stacked by multiple identical blocks, and independent parameters are used between different blocks to ensure the overall generalization ability of the network. The encoder contains an attention sublayer and a feedforward neural network sublayer. At the same time, for the stability of training, residual connections and layer regularization mechanisms are used between layers. Compared with the encoder, the decoder introduces an encoder-decoder attention network to capture the alignment relationship between the source language and the target language.

[0004] Neural network is not a brand new technology. As early as the 1950s, someone proposed a similar concept, but it was limited by the computing resources at that time and could not fully demonstrate its advantages. However, with the continuous improvement of computing power and the easy access to massive data in the big data era, applications based on neural networks have become the core technology of the artificial intelligence industry and have achieved remarkable results. However, the translation performance of machine translation models based on neural networks not only depends on the network architecture design, but also closely related to capacity. Capacity, in a broad sense, refers to the number of neurons in a neural network, and specifically the number of parameters in the model. Common ways to increase model capacity are to increase the width of the model, such as the Transformer-Big model; or the depth of the network stack, such as the Transformer-Deep model. In recent years, the deep Transformer model has become the mainstream baseline model in natural language processing tasks such as machine translation, language modeling, and text summarization. How to further improve the performance of the deep Transformer has become a widely discussed research hotspot. At present, in order to enable the deep Transformer model to be trained normally, the model structure needs to be adjusted, that is, the position of the layer regularization in the encoding layer and the decoding layer is adjusted from the output position of each sub-layer to the input position of the sub-layer, and additional layer regularization operations are added at the output position of the encoding end and the decoding end, which is called the Pre-Norm paradigm.

[0005] The earliest traceability of residual connection can be traced back to the ResNet network structure in the field of image processing. Through cross-layer connection, a convolutional neural network with more than 100 layers was successfully trained, and the excellent result of first place in the ImageNet task at that time was achieved. The introduction of this method has aroused the research interest of later generations, and made researchers realize that the use of multi-layer stacking for layer-by-layer feature extraction is a feasible technical means to improve classification accuracy. Later generations have also conducted extensive discussions on the interpretability of residual networks. From a mathematical point of view, the calculation method of residual connection is as follows:

[0006] y t+1 =y t +F(y t ,θ t )

[0007] where y t represents the input of the tth layer network, and F(·) represents the feature extraction function, which can be a convolutional network, a self-attention network, or a feedforward neural network. Formally speaking, the input of the t+1th layer does not only depend on the feature extraction vector F(y t ,θ t ), and also used y t information.

[0008] Some researchers pointed out that by deducing the above formula, the following formula can be obtained:

[0009]

[0010] here The derivative of the network at time t can be approximated by F(y(t),θ(t)), thus establishing the connection between the ordinary differential equation and the residual connection. The above formula refers to the simplest first-order Euler solution method in the ordinary differential equation, and its solution refers to the output of the middle layer of the network. In theory, there is a certain deviation from the true "solution", which is called truncation error.

[0011] Since the current mainstream Transformer and other neural networks are stacked in multiple layers, different layers can be regarded as independent ordinary differential solution methods, which also leads to error accumulation in network calculations, that is, global truncation error. The first-order Euler solution method used in the existing standard Transformer structure has a large stage error, so how to effectively further reduce the accumulation of errors is an urgent problem to be solved.

[0012] Although the mainstream multi-layer stacked neural network model can be stacked deeper and achieve better performance after the introduction of residual connections, the error accumulation problem has not been solved. The biased "solutions" predicted by different ordinary differential equation solvers increase with the increase of network depth, and the global truncation error further increases, which limits the network to a certain model depth. At the same time, the network parameters are not learned sufficiently with the same number of parameters. Summary of the invention

[0013] In order to solve the problem that the multi-layer neural network mentioned above causes error accumulation between different layers when using residual connections, resulting in a larger global truncation error in the end, the present invention adopts a high-order ordinary differential equation solving method to reduce the truncation error generated by different layers.

[0014] The technical solution adopted by the present invention is as follows:

[0015] The present invention provides a translation method based on Runge-Kutta high-order representation, comprising the following steps:

[0016] 1) Input the preprocessed source language into the encoder side of the translation model and encode it into a context vector through the word embedding layer and the stacked encoder layer; establish the implicit connection between the residual connection and the numerical solver of ordinary differential equations, and regard each individual residual block as a first-order solution method in ordinary differential equations;

[0017] 2) Introducing the high-order Runge-Kutta method for network structure design, by treating the operation of each sub-layer or the operation of the entire block as an independent solver, using the 2nd-order Runge-Kutta and 4th-order Runge-Kutta methods to redesign the Transformer structure;

[0018] 3) Further improve the structure based on step 2), and adjust the coefficient of the Runge-Kutta intermediate representation to 1;

[0019] 4) Based on step 3), the coefficients of the intermediate representation of Runge-Kutta are used as learnable network parameters, and their initial values ​​are set to 1. Then, the weights of different intermediate items are adaptively adjusted along with the update of the overall network parameters;

[0020] 5) Based on step 4), the intermediate states of different time steps obtained by high-order methods are used to automatically learn the coefficients corresponding to each intermediate representation, so as to obtain a more accurate layer representation and realize the translation method based on Runge-Kutta high-order representation.

[0021] The implicit connection between the residual connection and the numerical solver of the ordinary differential equation is established in step 1) through the following formula:

[0022] y t+1 =y t +G(LN(y t ),θ t )

[0023] Where LN(·) represents the regularization method, G(·) represents the self-attention network or feedforward neural network, and the above formula represents the calculation process of each sub-layer in the Pre-Norm Transformer. t Represents the input of the current layer, y t+1 Represents the output of the current layer;

[0024] The first-order ordinary differential equation is expressed as:

[0025]

[0026] Here f(y(t),t) defines a vector that is independent of time. By redefining G(LN(y t ),θ t ) function is expressed as follows:

[0027] y(t+Δt)=y(t)+Δt·F(y(t),θ(t))

[0028] Here, Δt represents the change in time t, and in the standard Transformer network, Δt = 1;

[0029] By adjusting the step size of Δt, we get:

[0030]

[0031]

[0032] The above two equations establish the connection between the Transformer model and ordinary differential equations.

[0033] In step 2), the model is constructed using the second-order Runge-Kutta method:

[0034]

[0035] F1=F(y t ,θ t )

[0036] F2=F(y t +F1,θ t )

[0037] Here F1 and F2 represent the first intermediate prediction result and the second intermediate prediction result of the model. The prediction calculation process of different intermediate results reuses the same network parameter θ t ;

[0038] The model constructed using the 4th-order Runge-Kutta method is:

[0039]

[0040] F1=F(y t ,θ t )

[0041]

[0042]

[0043] F4=F(y t +F3,θ t )

[0044] Among them, F3 and F4 represent the third and fourth intermediate prediction results of the model.

[0045] In step 3), the coefficient of the second-order method is set to 1, and the formula is:

[0046] y t+1 =y t +F(y t ,θ t )+F(y t +F(y t ,θ t ),θ t ).

[0047] In step 4), based on step 3), the intermediate coefficients are used as learnable network parameters γ1 and γ2, and their initialization values ​​are set to 1, and then they are automatically learned along with the update of the overall network parameters;

[0048] y t+1 =y t +γ1F(y t ,θ t )+γ2F(y t +F(y t ,θ t ),θ t ).

[0049] In step 5), based on step 4), the intermediate state obtained by the high-order method is used to automatically learn the weights of the representations at different time steps, and the contribution of different representations to the final output is dynamically determined;

[0050] y t+1 =y t +g·F(y t ,θ t )+(1-g)·F(y t +F(y t ,θ t ),θ t )

[0051] g=sigmoid(W1·F(y t ,θ t )+W2·F(y t +F(y t ,θ t ),θ t ))

[0052] Where g represents the coefficient obtained through learning, sigmoid represents the activation function, and W1 and W2 are learnable network parameters.

[0053] The present invention has the following beneficial effects and advantages:

[0054] 1. The present invention introduces a high-order Runge-Kutta ordinary differential equation solver to improve the accuracy of the representation of each layer and reduce the truncation error generated by the solver. Higher model accuracy can be achieved by using only fewer block stacks, reducing the global truncation error of the model.

[0055] 2. This paper establishes a corresponding relationship between residual connections and deep Transformer models, explains the mechanism behind the powerful capabilities of the Transformer model from the perspective of numerical analysis, and provides a new perspective for subsequent researchers to guide the design of network structures with interpretable numerical methods, rather than blindly optimizing the network through empirical adjustments.

[0056] 3. The translation method based on Runge-Kutta high-order representation proposed in this invention is not limited by the depth and width of the network, and has achieved significant performance improvement under different capacities. At the same time, consistent performance improvement has been achieved in both scarce resource languages ​​and rich resource language tasks.

[0057] 4. The method of the present invention only adds a small amount of additional storage overhead and computational cost, and its inference speed method meets industrial needs.

[0058] 5. The method of the present invention has achieved consistent performance improvement on common datasets and achieved the best translation performance on the WMT14 English-French task. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a structural diagram of the first-order Euler method in the prior art;

[0060] Figure 2 It is a schematic diagram of the model structure of the first-order baseline method involved in the method of the present invention;

[0061] Figure 3 It is a structural schematic diagram of the second-order Runge-Kutta model involved in the method of the present invention;

[0062] Figure 4 It is a structural schematic diagram of the 4th-order Runge-Kutta model involved in the method of the present invention;

[0063] Figure 5 The schematic diagram is a structural diagram of a second-order Runge-Kutta model with improved coefficients involved in the method of the present invention. DETAILED DESCRIPTION

[0064] The present invention will be further described below in conjunction with the accompanying drawings.

[0065] The present invention utilizes a high-order ordinary differential solution method to reduce the truncation error of each prediction step, thereby reducing the global error of the model and improving the representation ability of the model.

[0066] The present invention introduces a high-order ordinary differential equation solution method (such as Figure 1 As shown), replace the first-order Euler method in the standard Transformer model, that is, the residual connection (as shown Figure 2 shown).

[0067] The technical solution adopted by the present invention comprises the following steps:

[0068] 1) Input the preprocessed source language into the encoder side of the translation model and encode it into a context vector through the word embedding layer and the stacked encoder layer; establish the implicit connection between the residual connection and the numerical solver of ordinary differential equations, and regard each individual residual block as a first-order solution method in ordinary differential equations;

[0069] 2) Introduce a high-order Runge-Kutta method to design the network structure. By treating the operation of each sub-layer or the operation of the entire block as an independent solver, the second-order Runge-Kutta method (such as Figure 3 ) and 4th-order Runge–Kutta (as Figure 4 As shown) method to redesign the Transformer structure;

[0070] 3) Further improve the structure based on step 2) and adjust the coefficient of the Runge-Kutta intermediate representation to 1; implementing this step is equivalent to direct connection, which makes it easier to learn parameters;

[0071] 4) Based on step 3), the coefficients of the Runge-Kutta intermediate representation are used as learnable network parameters, and their initial values ​​are set to 1. Then, the weights of different intermediate terms are adaptively adjusted along with the update of the overall network parameters (such as Figure 5 shown);

[0072] 5) Based on step 4), the intermediate states of different time steps obtained by high-order methods are used to automatically learn the coefficients corresponding to each intermediate representation, so as to obtain a more accurate layer representation and realize the translation method based on Runge-Kutta high-order representation.

[0073] In the present invention, step 1) establishes the implicit connection between the residual connection and the ordinary differential numerical solver; step 2) introduces the high-order Runge-Kutta method to design the network structure. By considering the operation of each sub-layer or the operation of the entire block as an independent Euler solver, the Transformer structure is redesigned using more accurate 2nd and 4th order Runge-Kutta solvers;

[0074] Step 3) Further improve the structure based on step 2). When the network becomes deeper, the intermediate coefficient introduced by the high-order solution method will cause the gradient to vanish when the model updates the parameters in reverse. Adjusting its coefficient to 1 is equivalent to direct connection in practice, which makes it easier to learn parameters.

[0075] Step 4) Based on step 3), the intermediate term coefficient is used as a learnable network parameter, and its initial value is set to 1. Then, the weights of different intermediate terms are adaptively adjusted along with the update of the overall network parameters.

[0076] In step 1), the connection between the Transformer based on the Pre-Norm paradigm and the ordinary differential equation has been established. The implicit connection between the residual connection and the numerical solver of the ordinary differential equation is established through the following formula:

[0077] y t+1 =y t +G(LN(y t ),θ t )

[0078] Where LN(·) represents the regularization method, G(·) represents the self-attention network or feedforward neural network, and the above formula represents the calculation process of each sub-layer in the Pre-Norm Transformer. t Represents the input of the current layer, y t+1 Represents the output of the current layer;

[0079] The first-order ordinary differential equation is expressed as:

[0080]

[0081] Here f(y(t),t) defines a vector that is independent of time. By redefining G(LN(y t ),θ t ) function is expressed as follows:

[0082] y(t+Δt)=y(t)+Δt·F(y(t),θ(t))

[0083] Here, Δt represents the change in time t, and in the standard Transformer network, Δt = 1;

[0084] By adjusting the step size of Δt, we get:

[0085]

[0086]

[0087] The above two equations establish the connection between the Transformer model and ordinary differential equations.

[0088] In step 2), using the high-order Runge-Kutta method in numerical analysis, the residual connection between sublayers or layers in Transformer can be replaced by the following calculation method:

[0089] 201) The network is constructed using the second-order Runge-Kutta method as follows:

[0090]

[0091] F1=F(y t ,θ t )

[0092] F2=F(y t +F1,θ t )

[0093] Here F1 and F2 represent the first intermediate prediction result and the second intermediate prediction result of the model. From the formula, we can see that the prediction calculation process of different intermediate results reuses the same network parameter θ t , which improves the efficiency of parameter utilization to a certain extent. The second-order Runge-Kutta method is also called the improved Euler method.

[0094] 202) Similarly, the 4th-order Runge-Kutta method can be used to construct the network as follows:

[0095]

[0096] F1=F(y t ,θ t )

[0097]

[0098]

[0099] F4=F(y t +F3,θ t )

[0100] Among them, F3 and F4 represent the third and fourth intermediate prediction results of the model. Compared with the second-order method (such as Figure 3 As shown in the figure, the 4th-order method uses 4 intermediate predictions with forward and backward dependencies to obtain the final y t+1 Vector (such as Figure 4 shown).

[0101] In step 2), the Runge-Kutta method is explicitly expressed in the numerical analysis method as follows:

[0102]

[0103] F1=hf(y t ,t)

[0104]

[0105] Here h represents the size of the stride, F iis the intermediate prediction result, α, β and γ are the numerical coefficients of the intermediate representation, which are calculated by Tate expansion. The network calculation of each layer here not only depends on the feature extraction of the internal parameter matrix of the layer and the representation of the previous layer, but also performs multiple serial intermediate predictions internally, and the final output is the weighted sum of multiple intermediate representations.

[0106] In step 3), the method of solving the gradient by back propagation explains that the intermediate coefficients defined numerically are not easy to train deep networks. Here we take the second-order method as an example (such as Figure 3 shown):

[0107]

[0108] Let ε denote the training loss, L denote the overall depth of the network, and y L represents the final output of the model. Then for the yth t The gradient of can be calculated using the chain rule:

[0109]

[0110] here

[0111]

[0112] From the above formula, it can be observed that and Therefore, when the depth continues to increase, the gradient vanishing problem will occur. To solve this problem, the coefficient of the intermediate term can be modified to 1, and the modified RK2-block can be expressed as:

[0113] y t+1 =y t +F(y t ,θ t )+F(y t +F(y t ,θ t ),θ t )

[0114] Similarly, its t The gradient of can be obtained by the chain rule:

[0115]

[0116] Comparing the right sides of the two formulas, we can see that the improved RK2-block is easier to conduct gradients and is suitable for training deep networks.

[0117] In step 4), by further using the learnable coefficients as the representation of the intermediate terms, the network is initialized with 1 to adaptively determine the weights of different intermediate coefficients:

[0118] y t+1 =y t +γ1F(y t ,θ t )+γ2F(y t +F(y t ,θ t ),θ t )

[0119] Where γ1 and γ2 represent the numerical coefficients of the intermediate representation. This approach achieves better performance.

[0120] In step 5), the intermediate states obtained by the high-order method are used to automatically learn the weights of the representations at different time steps, and dynamically determine the contribution of different representations to the final output.

[0121] y t+1 =y t +g·F(y t ,θ t )+(1-g)·F(y t +F(y t ,θ t ),θ t )

[0122] g=sigmoid(W1·F(y t ,θ t )+W2·F(y t +F(y t ,θ t ),θ t ))

[0123] Where g represents the coefficient obtained through learning, sigmoid represents the activation function, and W1 and W2 are learnable network parameters.

[0124] This example uses the standard machine translation datasets WMT English-German and WMT English-French for experiments. On the WMT English-German task, the proposed method achieved a BLEU performance of nearly 29.03, which is 1.43 BLEU improvement compared to the baseline (27.60) with the same number of parameters. The experimental results are as follows:

[0125]

[0126] Table 1 Comparison of experimental results

[0127] The significant advantage of the present invention is that after using a high-order ordinary differential equation solving method, the output representation obtained by each block is more accurate than the original multi-block cumulative representation in the Transformer. In a sense, a more accurate representation is achieved by stacking fewer high-order ordinary differential equation blocks. It reduces error accumulation to a certain extent and significantly alleviates the problem of global truncation error. The present invention selects a high-order Runge-Kutta method as an ordinary differential equation solver. This method is a well-known method for solving high-order ordinary differential equations in numerical analysis, and has a smaller truncation error than the first-order Euler method. The Runge-Kutta method shown here is selected.

[0128] The method of the present invention uses a high-order solution method of ordinary differential equations to have a smaller truncation error characteristic, thereby reducing the error loss of each layer and significantly reducing the global error. The network structure has the advantages of high utilization of model parameters and strong performance, and can achieve significant performance gains under the same capacity.

Claims

1. A translation method based on Runge-Kutta high-order representation, characterized in that The following steps are involved: 1) Input the preprocessed source language into the encoder side of the translation model and encode it into a context vector through the word embedding layer and the stacked encoder layer; establish the implicit connection between the residual connection and the numerical solver of ordinary differential equations, and regard each individual residual block as a first-order solution method in ordinary differential equations; 2) Introducing the high-order Runge-Kutta method for network structure design, by treating the operation of each sub-layer or the operation of the entire block as an independent solver, using the 2nd-order Runge-Kutta and 4th-order Runge-Kutta methods to redesign the Transformer structure; 3) Further improve the structure based on step 2), and adjust the coefficient of the Runge-Kutta intermediate representation to 1; 4) Based on step 3), the coefficients of the intermediate representation of Runge-Kutta are used as learnable network parameters, and their initial values ​​are set to 1. Then, the weights of different intermediate items are adaptively adjusted along with the update of the overall network parameters; 5) Based on step 4), the intermediate states of different time steps obtained by high-order methods are used to automatically learn the coefficients corresponding to each intermediate representation, so as to obtain a more accurate layer representation and realize the translation method based on Runge-Kutta high-order representation.

2. The translation method based on Runge-Kutta high-order representation according to claim 1, characterized in that: The implicit connection between the residual connection and the numerical solver of the ordinary differential equation is established in step 1) through the following formula: y t+1 =y t +G(LN(y t ),θ t ) Where LN(·) represents the regularization method, G(·) represents the self-attention network or feedforward neural network, and the above formula represents the calculation process of each sub-layer in the Pre-Norm Transformer. t Represents the input of the current layer, y t+1 Represents the output of the current layer; The first-order ordinary differential equation is expressed as: Here f(y(t), t) defines a vector that is independent of time. By redefining G(LN(y t ), θ t ) function is expressed as follows: y(t+Δt)=y(t)+Δt·F(y(t),θ(t)) Here, Δt represents the change in time t, and in the standard Transformer network, Δt = 1; By adjusting the step size of Δt, we get: The above two equations establish the connection between the Transformer model and ordinary differential equations.

3. The translation method based on Runge-Kutta high-order representation according to claim 1, characterized in that: In step 2), the model is constructed using the second-order Runge-Kutta method: F1=F(y t ,θ t ) F2=F(y t +F1,θ t ) Here F1 and F2 represent the first intermediate prediction result and the second intermediate prediction result of the model. The prediction calculation process of different intermediate results reuses the same network parameter θ t ; The model constructed using the 4th-order Runge-Kutta method is: F1=F(y t ,θ t ) F4=F(y t +F3,θ t ) Among them, F3 and F4 represent the third and fourth intermediate prediction results of the model.

4. The translation method based on Runge-Kutta high-order representation according to claim 1, characterized in that: In step 3), the coefficient of the second-order method is set to 1, and the formula is: y t+1 =y t +F(y t ,θ t )+F(y t +F(y t ,θ t ),θ t )。 5. The translation method based on Runge-Kutta high-order representation according to claim 1, characterized in that: In step 4), based on step 3), the intermediate coefficients are used as learnable network parameters γ1 and γ2, and their initialization values ​​are set to 1, and then they are automatically learned along with the update of the overall network parameters; y t+1 =y t +γ1F(y t ,the t )+γ2F(y t +F(y t ,the t ),θ t )。 6. The translation method based on Runge-Kutta high-order representation according to claim 1, characterized in that: In step 5), based on step 4), the intermediate state obtained by the high-order method is used to automatically learn the weights of the representations at different time steps, and the contribution of different representations to the final output is dynamically determined; y t+1 =y t +g·F(y t ,θ t )+(1-g)·F(y t +F(y t ,θ t ),θ t ) g=sigmoid(W1·F(y t ,θ t )+W2·F(y t +F(y t ,θ t ),θ t )) Where g represents the coefficient obtained through learning, sigmoid represents the activation function, and W1 and W2 are learnable network parameters.

Images