Translation method and apparatus

Abstract

The application provides a translation method and device, wherein the translation method comprises: obtaining a sentence to be translated; inputting the sentence to be translated into a translation model to obtain a translated sentence corresponding to the sentence to be translated, wherein the translation model comprises an encoding layer and a weight layer, the encoding layer comprises at least one first stack layer, the first stack layer comprises a multi-head attention mechanism sublayer and a feedforward network layer, and the weight layer is used to generate a weight vector according to at least one encoding vector output by the encoding layer and a weight coefficient corresponding to each encoding vector, and the weight vector is used to generate the translated sentence. Since the translation model comprises the multi-head attention mechanism sublayer, richer information can be obtained by introducing the attention mechanism, so that the weight corresponding to each first stack layer is adjusted, the architecture of the translation model is optimized, and the translation effect of the sentence to be translated is further improved in the use process of the translation model.

Description

Translation methods and devices

[0001] This application is a divisional application of application number 201811513331.X, filed on December 11, 2018, entitled "A translation method and apparatus, a training method and apparatus for a translation model". Technical Field

[0002] This application relates to the field of computer technology, and in particular to a translation method and apparatus, a computing device, and a computer-readable storage medium. Background Technology

[0003] With the improvement of computing power, neural networks are being used more and more widely, such as in building translation models to convert source language to target language. Generally, the architecture of a translation model includes an encoder layer and a decoder layer. The encoder layer encodes the source sentence to be translated into a vector, and the decoder layer decodes the vector of the source sentence to generate the corresponding target sentence.

[0004] Traditional translation models mostly use Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs) as the basis for the encoder-decoder layer. Google's attention-based translation model does not use any CNN or RNN structure. This model can work in high parallelism, thus improving translation performance while also being particularly fast to train.

[0005] Figure 1 The architecture of an existing Google Translate model is shown. The model consists of two parts: an encoding layer and a decoding layer. The encoding layer consists of six identical encoding stacks, each of which includes two sub-layers: a multi-head attention mechanism sub-layer and a feedforward network layer.

[0006] The decoding layer consists of six identical decoding stack layers stacked together, each stack layer comprising three sub-layers: an implicit multi-head attention mechanism sub-layer, a multi-head attention mechanism sub-layer, and a feedforward network layer.

[0007] During use, on the encoding layer side, the original vector corresponding to the source statement serves as the input vector for the first encoding stack layer, and the output vector of each encoding stack layer serves as the input vector for the next encoding stack layer. The output vector of the last encoding stack layer serves as the input vector for the decoding stack layer, which is then fed into each decoding stack layer.

[0008] On the decoding layer side, the output vector of the previous decoding is used as the input vector for the current decoding, and is fed into the first decoding stack layer. The output vectors of each encoding stack layer and the output vector of the last encoding stack layer are used as the input vectors for the next encoding stack layer. The output vector of the last decoding stack layer is transformed through a linear layer and a classification function layer (softmax) to obtain the final target statement.

[0009] In the Google Translate model, the output vector of the last encoding stack layer is used as the final output vector of the encoding layer. Each encoding stack layer's output vector affects the final output vector, but the influence weight of each encoding stack layer is fixed, which can have an adverse effect on the final translation effect. Summary of the Invention

[0010] In view of this, embodiments of this application provide a translation method and apparatus, a computing device and a computer-readable storage medium to address the technical deficiencies existing in the prior art.

[0011] This application discloses a translation method, the method comprising:

[0012] Get the sentence to be translated;

[0013] The sentence to be translated is input into a translation model to obtain the translated sentence corresponding to the sentence to be translated. The translation model includes an encoding layer and a weight layer. The encoding layer includes at least one first stack layer, which includes a multi-head attention mechanism sublayer and a feedforward network layer. The weight layer is used to generate a weight vector based on at least one encoding vector output by the encoding layer and the weight coefficients corresponding to each encoding vector. The weight vector is used to generate the translated sentence.

[0014] Optionally, the translation model is trained in the following manner:

[0015] The first sentence in the target corpus and the translated second sentence are used as training samples and training features, respectively.

[0016] The training samples are input into the encoding layer of the translation model to obtain at least one encoding vector;

[0017] At least one encoded vector is input into the weight layer of the translation model to obtain a weight vector;

[0018] The training features and the weight vector are input into the decoding layer of the translation model to obtain the decoding vector, and the error of the decoding vector is obtained based on the decoding vector.

[0019] The weight layer is adjusted based on the error of the decoded vector, and the translation model is trained continuously until the training stopping condition is met.

[0020] Optionally, the step of inputting the training samples into the encoding layer of the translation model to obtain at least one encoding vector includes:

[0021] The training samples are input into the encoding layer of the translation model to generate a first vector;

[0022] The first vector is input into the first stack layer to obtain the encoded vector output by the first stack layer.

[0023] Optionally, the coding layer includes n first stack layers, and the n first stack layers are connected sequentially, where n≥2 and n is a positive integer;

[0024] The training samples are input into the encoding layer of the translation model to obtain at least one encoding vector, including:

[0025] 102. Input the training samples into the encoding layer of the translation model to generate a first vector;

[0026] 104. Input the first vector into the first stack layer to obtain the encoded vector of the output of the first stack layer;

[0027] 106. Input the encoding vector of the (i-1)th first stack layer into the ith first stack layer to obtain the encoding vector output by the ith first stack layer, where 2≤i≤n;

[0028] 108. Increment i by 1. Check if the incremented i is greater than n. If yes, end the process. If no, continue to step 106.

[0029] Optionally, the weight coefficients include a first weight coefficient; the step of inputting at least one encoded vector into the weight layer of the translation model to obtain a weight vector includes:

[0030] At least one encoding vector is input to the weight layer of the translation model, wherein the weight layer includes a first weight coefficient corresponding to each encoding vector;

[0031] A weight vector is calculated based on the encoded vector and the first weight coefficient.

[0032] Optionally, calculating a weight vector based on the encoded vector and the first weight coefficient includes:

[0033] The first weight coefficient is normalized to obtain the first mapping weight coefficient;

[0034] The weight vector is obtained by multiplying the encoded vector output by each first stack layer and the corresponding first mapping weight coefficient, and summing the results.

[0035] Optionally, the decoding layer includes at least one second stack layer;

[0036] The training features and the weight vector are input into the decoding layer of the translation model to obtain the decoding vector, including:

[0037] The trained features are input into the decoding layer of the translation model to generate a second vector;

[0038] The second vector and the weight vector are input into the second stack layer to obtain the decoding vector.

[0039] Optionally, the decoding layer includes m second stack layers, and the m second stack layers are connected sequentially, where m ≥ 2 and m is a positive integer;

[0040] The training features and the weight vector are input into the decoding layer of the translation model to obtain the decoding vector, including:

[0041] 202. Input the training features into the decoding layer of the translation model to generate a second vector;

[0042] 204. Input the second vector and the weight vector into the first second stack layer to obtain the output vector of the first second stack layer;

[0043] 206. Input the output vector of the (j-1)th second stack layer and the weight vector into the jth second stack layer to obtain the output vector of the jth second stack layer, and use the output vector of the mth second stack layer as the decoding vector, where 2≤j≤m;

[0044] 208. Increment j by 1, and check if j after incrementing by 1 is greater than m. If yes, end the process; otherwise, continue to step 206.

[0045] Optionally, the weighting coefficient includes a second weighting coefficient; the decoding layer includes m second stack layers, and the m second stack layers are connected sequentially, where m≥2 and m is a positive integer;

[0046] At least one encoding vector is input into the weight layer of the translation model to obtain a weight vector, including:

[0047] At least one encoding vector is input to the weight layer of the translation model, wherein the weight layer includes a second weight coefficient vector corresponding to each of the second stack layers, and each second weight coefficient vector includes a second weight coefficient corresponding to each of the encoding vectors;

[0048] The weight vector corresponding to each second stack layer is calculated based on the encoding vector and the second weight coefficient vector.

[0049] Optionally, calculating the weight vector corresponding to each of the second stack layers based on the encoding vector and the second weight coefficient vector includes:

[0050] The second weight coefficient vector is normalized to obtain the second mapped weight coefficient vector;

[0051] The weight vector is obtained by multiplying the encoded vector output from each first stack layer with the corresponding second mapping weight coefficient vector and summing the results.

[0052] Optionally, the training features and the weight vector are input into the decoding layer of the translation model to obtain a decoding vector, including:

[0053] 302. Input the training features into the decoding layer of the translation model to generate a second vector;

[0054] 304. Input the second vector and the weight vector corresponding to the first second stack layer into the first second stack layer to obtain the output vector of the first second stack layer;

[0055] 306. Input the output vector of the (j-1)th second stack layer and the weight vector corresponding to the jth second stack layer into the jth second stack layer to obtain the output vector of the jth second stack layer, and use the output vector of the mth second stack layer as the decoding vector, where 2≤j≤m;

[0056] 308. Increment j by 1, and check if j after incrementing by 1 is greater than m. If yes, end the process; otherwise, continue to step 306.

[0057] Optionally, the error of the decoding vector is obtained by the following method:

[0058] The decoded vector is compared with a preset vector verification set to obtain the error of the decoded vector.

[0059] Optionally, the training stopping condition includes: the rate of change of the error is less than a stability threshold.

[0060] This application also discloses a translation apparatus, the apparatus comprising:

[0061] The acquisition module is configured to acquire the sentences to be translated;

[0062] The translation module is configured to input the statement to be translated into a translation model to obtain the translated statement corresponding to the statement to be translated. The translation model includes an encoding layer and a weight layer. The encoding layer includes at least one first stack layer, which includes a multi-head attention mechanism sublayer and a feedforward network layer. The weight layer is used to generate a weight vector based on at least one encoding vector output by the encoding layer and the weight coefficients corresponding to each encoding vector. The weight vector is used to generate the translated statement.

[0063] This application also discloses a computing device, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein when the processor executes the instructions, it implements the translation method described above.

[0064] This application also discloses a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the translation method described above.

[0065] This application provides a translation method and apparatus that acquires a sentence to be translated; inputs the sentence to be translated into a translation model to obtain a translated sentence corresponding to the sentence to be translated. The translation model includes an encoding layer and a weight layer. The encoding layer includes at least one first stack layer, which includes a multi-head attention mechanism sublayer and a feedforward network layer. The weight layer generates a weight vector based on at least one encoding vector output by the encoding layer and the weight coefficients corresponding to each encoding vector. The weight vector is used to generate the translated sentence. Because the translation model includes a multi-head attention mechanism sublayer, by introducing an attention mechanism, the translation model can obtain richer information, thereby adjusting the weights corresponding to each first stack layer, optimizing the architecture of the translation model, and further improving the translation effect of the sentence to be translated during the use of the translation model. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the architecture of a translation model in the existing technology;

[0067] Figure 2 This is a schematic diagram of the structure of a computing device according to an embodiment of this application;

[0068] Figure 3a and Figure 3b These are architectural diagrams of the translation model in embodiments of this application;

[0069] Figure 4 This is a flowchart illustrating the training method of the translation model according to an embodiment of this application;

[0070] Figure 5This is a flowchart illustrating the training method of the translation model according to an embodiment of this application;

[0071] Figure 6 This is a flowchart illustrating the training method of the translation model according to an embodiment of this application;

[0072] Figure 7 This is a flowchart illustrating the translation method according to an embodiment of this application;

[0073] Figure 8 This is a schematic flowchart of the translation apparatus according to an embodiment of this application;

[0074] Figure 9 This is a flowchart illustrating the training apparatus for the translation model according to an embodiment of this application. Detailed Implementation

[0075] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0076] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0077] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0078] First, the terminology used in one or more embodiments of the present invention will be explained.

[0079] Translation model: Its main idea is to encode the sentence to be translated into an encoding vector through an encoder, then decode the encoding vector using a decoder to obtain a decoded vector, and finally translate the decoded vector into the corresponding translation sentence.

[0080] Google Translate: A neural network architecture proposed by Google for machine translation.

[0081] Encoder: Converts the text to be translated into an encoded vector;

[0082] Decoding: Converting encoded vectors into the translated language text;

[0083] BLEU score: a metric for evaluating machine translation results.

[0084] WMT (Workshop on Machine Translation) is a top international machine translation competition. WMT2013 is one of the official WMT machine translation datasets, used to test algorithm results.

[0085] In the embodiments of this application, a translation method and apparatus, a translation model training method and apparatus, a computing device, and a computer-readable storage medium are provided, which will be described in detail in the following embodiments.

[0086] Figure 2 This is a structural block diagram of a computing device 200 according to an embodiment of this specification. The components of the computing device 200 include, but are not limited to, a memory 210 and a processor 220. The processor 220 is connected to the memory 210 via a bus 230, and a database 250 is used to store data.

[0087] The computing device 200 also includes an access device 240 that enables the computing device 200 to communicate via one or more networks 260. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 240 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0088] In one embodiment of this specification, the computing device 200 described above and Figure 2 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 2The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0089] The computing device 200 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 200 can also be a mobile or stationary server.

[0090] The processor 220 can execute the translation method or the training method of the translation model in this embodiment.

[0091] First, the architecture of the translation model involved in this embodiment will be explained. Figure 3a and Figure 3b The two translation models shown in this embodiment are as follows.

[0092] Depend on Figure 3a and Figure 3b As can be seen, the translation model in this embodiment includes two parts: an encoding layer 31 and a decoding layer 32. The encoding layer 31 includes at least one first stack layer 311. Figure 3a and Figure 3b The encoding layer 31 comprises multiple first stack layers 311, which are stacked together. Each first stack layer 311 includes two sub-layers: a multi-head attention mechanism sub-layer and a feedforward network layer. The encoding layer 31 also includes a first embedding layer to receive the statement to be translated and convert it into a first vector input to the first first stack layer 311. The encoded vector output by the first first stack layer 311 is used as the input vector of the second stack layer 311, thereby obtaining the encoded vector output by the second first stack layer 311, and so on, until the encoded vectors output by all the first stack layers 311 are obtained.

[0093] In this embodiment, the encoded vector output from each first stack layer 311 is input to the weight layer, and the resulting weight vector is input to each second stack layer. Figure 3a In this process, the encoding vector of each first stack layer 311 corresponds to a weight coefficient in the weight vector, generating a weight vector that is input to each second stack layer; Figure 3b In this process, the encoding vector of each first stack layer 311 corresponds to a different weight coefficient in the weight vector, and the weight vectors input to each second stack layer are different from each other.

[0094] Decoding layer 32 includes at least one second stack layer 321. Figure 3a and Figure 3b The decoding layer 32 includes multiple second stack layers 321, which are connected in series. Each second stack layer 321 includes three sub-layers: an implicit multi-head attention mechanism sub-layer, a multi-head attention mechanism sub-layer, and a feedforward network layer. The decoding layer 32 also includes a second embedding layer, which, during the training phase, receives the translated statement and converts it into a second vector, which is then input to the first second stack layer 321. The first second stack layer 321 generates a corresponding output vector based on the second vector and the weight vector input to it; based on the output vector of the first second stack layer 321 and the weight vector input to the second second stack layer 321, it obtains the output vector of the second second stack layer 321, and so on, until the output vector of the final second stack layer 321 is obtained as the decoding vector.

[0095] See Figure 4 The training method for the translation model in this embodiment includes:

[0096] 402. Use the first sentence and the translated second sentence in the target corpus as training samples and training features, respectively.

[0097] In this embodiment, the target corpus includes multiple first statements and multiple second statements, with a one-to-one correspondence between the first and second statements. For example, the first statement is "I love China" and the second statement is "I love China"; the first statement is "I eat apples" and the second statement is "I eat apples".

[0098] 404. Input the training samples into the encoding layer of the translation model to obtain at least one encoding vector.

[0099] Specifically, the coding layer may include one first stack layer, or two or more first stack layers.

[0100] In the case where the encoding layer includes a first stack layer, step 404 specifically includes the following steps S1 to S2:

[0101] S1. Input the sentence to be translated into the encoding layer of the translation model to generate the first vector.

[0102] The first vector can be generated by the first embedding layer of the encoding layer. For example, if the sentence to be translated is "I love China", the first vector generated is X = (x0, x1, x2, x3), where x0 = "I", x1 = "love", x2 = "China", and x3 = "country".

[0103] S2. Input the first vector into the first stack layer to obtain the encoded vector output by the first stack layer.

[0104] In the case where the encoding layer includes n first stack layers, the n first stack layers are connected sequentially, where n≥2 and n is a positive integer. Step 404 specifically includes the following steps S3 to S5:

[0105] S3. Input the training samples into the encoding layer of the translation model to generate the first vector.

[0106] S4. Input the first vector into the first stack layer to obtain the encoded vector of the output of the first stack layer.

[0107] S5. Input the encoding vector of the (i-1)th first stack layer to the ith first stack layer to obtain the encoding vector output by the ith first stack layer, where 2≤i≤n.

[0108] S6. Increment i by 1, and determine whether the incremented i is greater than n. If yes, end the process; otherwise, continue with step S5.

[0109] As can be seen, for the case where the coding layer includes at least two first stack layers, except for the first first stack layer, the input vectors of the remaining first stack layers are all the coding vectors output by the previous first stack layer. This also ensures the correlation between the decoding vectors output by multiple first stack layers.

[0110] 406. Input at least one encoding vector into the weight layer of the translation model to obtain a weight vector.

[0111] In a specific implementation plan, see Figure 3a and Figure 5 Step 406 includes:

[0112] 502. Input at least one encoding vector into the weight layer of the translation model, wherein the weight layer includes a first weight coefficient corresponding to each encoding vector.

[0113] Specifically, during training, the first input weight coefficient can be the initial weight coefficient, which is continuously adjusted in subsequent training processes to obtain a suitable weight coefficient.

[0114] 504. A weight vector is calculated based on the encoded vector and the first weight coefficient.

[0115] Specifically, step 504 includes:

[0116] 1) Normalize the first weight coefficient to obtain the first mapping weight coefficient.

[0117] Specifically, the first mapping weight coefficient is implemented using the following formula (1):

[0118]

[0119] Where, α i The first weight coefficient; where i∈[0,…,n-1], n is the number of the first stack layers; t i This represents the first mapping weight coefficient.

[0120] 2) Multiply the output vector of each first stack layer with the corresponding first mapping weight coefficient, and then sum them to obtain a weight vector.

[0121] Specifically, the weight vector is calculated using the following formula (2):

[0122]

[0123] Where O represents the weight vector; t i Z represents the first mapping weight coefficient; i This represents the output vector of the first stack layer.

[0124] Steps 502-504 above ensure that the same weight vector is input to each second stack layer. By adjusting the first weight coefficient of the weight layer during the training of the translation model using input training samples and training labels, and obtaining the adjusted first weight coefficient, the translation model can achieve better translation results during use.

[0125] In a specific implementation plan, see Figure 3b and Figure 6 Step 406 includes:

[0126] 602. Input at least one encoding vector into the weight layer of the translation model, wherein the weight layer includes a second weight coefficient vector corresponding to each of the second stack layers, and each second weight coefficient vector includes a second weight coefficient corresponding to each of the encoding vectors.

[0127] 604. Calculate the weight vector corresponding to each of the second stack layers based on the encoding vector and the second weight coefficient vector.

[0128] Specifically, step 604 includes:

[0129] 1) Normalize the second weight coefficient vector to obtain the second mapped weight coefficient vector.

[0130] Specifically, the second mapping weight coefficient vector is implemented using the following formula (3):

[0131]

[0132] Where, α ij Let be the second weight coefficient vector; where i∈[0,…,n-1], n is the number of the first stack layer, and j∈[0,…,m-1];

[0133] t ij This represents the second mapping weight coefficient vector.

[0134] 2) Multiply the output vector of each first stack layer with the corresponding second mapping weight coefficient vector, and then sum them to obtain m weight vectors.

[0135] Where m is the number of the second stack layers.

[0136] Specifically, the m weight vectors are implemented using the following formula (4):

[0137]

[0138] Among them, O j Represents the weight vector; t ij Z represents the second mapping weight coefficient vector; i This represents the output vector of the first stack layer.

[0139] Steps 602-604 above allow for the input of different weight vectors to each second stack layer. By adjusting the second weight coefficient vector of the weight layer during the training of the translation model using input training samples and training labels, and obtaining the adjusted second weight coefficient vector, the translation model can achieve better translation results during use.

[0140] 408. Input the training features and the weight vector into the decoding layer of the translation model to obtain the decoding vector, and obtain the error of the decoding vector based on the decoding vector.

[0141] In the case where the decoding layer includes a second stack layer, the training features and the weight vector are input into the decoding layer of the translation model to obtain the decoding vector, including:

[0142] 1) Input the training features into the decoding layer of the translation model to generate a second vector;

[0143] 2) Input the second vector and the weight vector into the second stack layer to obtain the decoding vector.

[0144] Wherein, for a decoding layer comprising m sequentially connected second stack layers, and the weight vectors input to the second stack layers are identical, the training features and the weight vectors are input to the decoding layer of the translation model to obtain a decoding vector, including:

[0145] 1) Input the training features into the decoding layer of the translation model to generate a second vector;

[0146] 2) Input the second vector and the weight vector into the first second stack layer to obtain the output vector of the first second stack layer;

[0147] 3) Input the output vector of the (j-1)th second stack layer and the weight vector into the jth second stack layer to obtain the output vector of the jth second stack layer, and use the output vector of the mth second stack layer as the decoding vector, where 2≤j≤m.

[0148] 4) Increment j by 1, and check if the incremented j is greater than m. If yes, end the process; otherwise, continue with step 3.

[0149] Wherein, for the decoding layer comprising m sequentially connected second stack layers, and the m weight vectors input to the second stack layers are different from each other, the training features and the weight vectors are input to the decoding layer of the translation model to obtain the decoding vector, including:

[0150] 1) Input the training features into the decoding layer of the translation model to generate a second vector;

[0151] 2) Input the second vector and the weight vector corresponding to the first second stack layer into the first second stack layer to obtain the output vector of the first second stack layer;

[0152] 3) Input the output vector of the (j-1)th second stack layer and the weight vector corresponding to the jth second stack layer into the jth second stack layer to obtain the output vector of the jth second stack layer, and use the output vector of the mth second stack layer as the decoding vector, where 2≤j≤m.

[0153] 4) Increment j by 1, and check if the incremented j is greater than m. If yes, end the process; otherwise, continue with step 3.

[0154] Optionally, the error of the decoding vector is obtained by comparing the decoding vector with a preset vector verification set.

[0155] It should be noted that in this embodiment, the obtained decoded vector is not directly compared with the vector corresponding to the original translated sentence to calculate the error. Instead, a vector validation set is introduced. If the decoded vector is directly compared with the vector corresponding to the original translated sentence to calculate the error, it will lead to overfitting, causing the translation model to perform worse in the translation of other sentences, and the translation effect will be counterproductive.

[0156] 410. Adjust the weight layer according to the error of the decoding vector, and continue training the translation model until the training stopping condition is met.

[0157] The training stopping condition includes: the rate of change of the error is less than the stability threshold.

[0158] The stabilization threshold can be set according to actual needs, for example, 1%. In this way, when the error tends to stabilize, the model can be considered to have completed training.

[0159] This application provides a method for training a translation model. By adjusting the weight coefficients of the weight layer during the training process of the translation model using input training samples and training labels, a better weight layer can be obtained, thereby achieving better translation results during the use of the translation model.

[0160] Secondly, the weight vectors input to each second stack layer can be the same or different from each other, which further increases the adjustability of the translation model in this embodiment, thereby achieving better translation results.

[0161] This application also discloses a translation method, such as... Figure 7 As shown, it includes:

[0162] 702. Obtain the sentence to be translated.

[0163] 704. Input the sentence to be translated into the encoding layer of the translation model to obtain at least one encoding vector.

[0164] The encoding vector includes at least one word to be translated.

[0165] 706. Input at least one encoding vector into the weight layer of the translation model to obtain a weight vector.

[0166] 708. Input the reference translation result and weight vector into the decoding layer of the translation model to obtain the decoding vector.

[0167] The decoding vector includes at least one translated word.

[0168] Specifically, during the translation process, the corresponding translation words for the words to be translated in the encoding vector are obtained sequentially. For each word to be translated, the corresponding reference translation result needs to be input into the decoding layer.

[0169] For the first word to be translated, the reference translation result is the set initial value;

[0170] For the remaining words to be translated, excluding the first word, the reference translation result is the translation of the word preceding the current word.

[0171] For example, for the phrase "I love China", if the word to be translated is "love", then the reference translation result is the word "I" corresponding to "I".

[0172] 710. Obtain the translation statement corresponding to the statement to be translated based on the decoding vector.

[0173] The specific steps for obtaining at least one encoding vector of the encoding layer in step 704, the specific steps for obtaining the weight vector in step 706, and the specific steps for obtaining the decoding vector in step 708 have been described in detail in the foregoing embodiments and will not be repeated here.

[0174] In addition, inputting historical translations into the translation model during the translation process can improve the correlation between historical and current translations, which can help improve translation results.

[0175] Tables 1, 2, and 3 below compare experimental data using three different translation models with added weight layers. It should be noted that the weight layers added in Tables 1-3 include the first weight coefficient corresponding to each encoded vector, such as... Figure 3a As shown.

[0176] Table 1 shows the experimental results of the existing translation model (transformer) before and after adding weight layers. The data in Table 1 represent the BLEU scores of the translation model (transformer) before and after adding weight layers at different numbers of layers.

[0177] Table 1

[0178] Transformer with 4, 6, and 8 layers: Unweighted layer: 24.48, 24.78, 25.00; Weighted layer: 25.00, 25.07, 25.38. surface

[0179] As shown in Table 1, in the 4-layer, 6-layer, and 8-layer cases, the BLEU score of the translation model (transformer) with added weight layers is higher than that of the translation model (transformer) without added weight layers. Therefore, the translation performance of the model with added weight layers is superior to that of the model without weight layers.

[0180] Table 2 shows the experimental results of the existing weighted translation model before and after adding weight layers. The data in Table 2 represent the BLEU scores of the weighted translation model before and after adding weight layers at different numbers of layers.

[0181] Table 2

[0182] Weighted Transformer with 4, 6, and 8 layers: Unweighted layer: 24.70, 24.76, 24.92; Weighted layer: 24.71, 24.90, 25.10 surface

[0183] As shown in Table 2, in the 4-layer, 6-layer, and 8-layer cases, the BLEU score of the weighted translation model with added weight layers is higher than that of the weighted translation model without added weight layers. Therefore, the weighted translation model with added weight layers performs better than the weighted translation model without added weight layers.

[0184] Table 3 compares the experimental results of the existing universal transformer translation model before and after adding weight layers. The data in Table 3 represent the BLEU scores of the universal transformer translation model before and after adding weight layers at different numbers of layers.

[0185] Table 3

[0186]

[0187]

[0188] As shown in Table 3, in the 6-layer and 8-layer cases, the BLEU score of the universal translation model with added weight layers is higher than that of the universal translation model without weight layers. Therefore, the translation performance of the universal translation model with added weight layers is better than that of the universal translation model without weight layers.

[0189] The above is a detailed description of the training method for the translation model and the translation method according to embodiments of this application. Embodiments of this application also disclose a training device for a translation model and a translation device, which will be described below through embodiments.

[0190] This application also discloses a translation device, see [link to relevant documentation]. Figure 8 The device includes:

[0191] Module 802 is configured to retrieve the text to be translated.

[0192] Translation module 804 is configured to input the statement to be translated into a translation model to obtain the translated statement corresponding to the statement to be translated.

[0193] The translation model is trained using the following device:

[0194] The sample feature generation module 806 is configured to use the first sentence and the translated second sentence in the target corpus as training samples and training features, respectively.

[0195] The encoding vector generation module 808 is configured to input the training samples into the encoding layer of the translation model to obtain at least one encoding vector;

[0196] The weight vector generation module 810 is configured to input at least one encoded vector into the weight layer of the translation model to obtain a weight vector;

[0197] The decoding vector generation module 812 is configured to input the training features and the weight vector into the decoding layer of the translation model to obtain the decoding vector, and to obtain the error of the decoding vector based on the decoding vector;

[0198] The adjustment module 814 is configured to adjust the weight layer according to the error of the decoding vector and continue training the translation model until the training stopping condition is met.

[0199] Optionally, the coding layer includes a first stack layer, and the coding vector generation module 808 is further configured to:

[0200] The training samples are input into the encoding layer of the translation model to generate a first vector;

[0201] The first vector is input into the first stack layer to obtain the encoded vector output by the first stack layer.

[0202] Optionally, the encoding layer includes n first stack layers, and the n first stack layers are connected sequentially, where n ≥ 2 and n is a positive integer; the encoding vector generation module 808 is further configured to:

[0203] 1) Input the training samples into the encoding layer of the translation model to generate a first vector;

[0204] 2) Input the first vector into the first stack layer to obtain the encoded vector of the output of the first stack layer;

[0205] 3) Input the encoding vector of the (i-1)th first stack layer into the ith first stack layer to obtain the encoding vector output by the ith first stack layer, where 2≤i≤n;

[0206] 4) Increment i by 1, and check if the incremented i is greater than n. If yes, end the process; otherwise, continue with step 3.

[0207] Optionally, the weight vector generation module 810 is also configured to:

[0208] At least one encoding vector is input to the weight layer of the translation model, wherein the weight layer includes a first weight coefficient corresponding to each encoding vector;

[0209] A weight vector is calculated based on the encoded vector and the first weight coefficient.

[0210] Optionally, the decoding layer includes a second stack layer; the decoding vector generation module 812 is also configured to:

[0211] The trained features are input into the decoding layer of the translation model to generate a second vector;

[0212] The second vector and the weight vector are input into the second stack layer to obtain the decoding vector.

[0213] Optionally, the decoding layer includes m second stack layers, and m first stack layers are connected sequentially, where m ≥ 2 and m is a positive integer; the decoding vector generation module 812 is further configured to:

[0214] 1) Input the training features into the decoding layer of the translation model to generate a second vector;

[0215] 2) Input the second vector and the weight vector into the first second stack layer to obtain the output vector of the first second stack layer;

[0216] 3) Input the output vector of the (j-1)th second stack layer and the weight vector into the jth second stack layer to obtain the output vector of the jth second stack layer, and use the output vector of the mth second stack layer as the decoding vector, where 2≤j≤m;

[0217] 4) Increment j by 1, and check if the incremented j is greater than m. If yes, end the process; otherwise, continue with step 3.

[0218] Optionally, the decoding layer includes m second stack layers, and the m second stack layers are connected sequentially, where m ≥ 2 and m is a positive integer; the weight vector generation module 810 is further configured to:

[0219] At least one encoding vector is input to the weight layer of the translation model, wherein the weight layer includes a second weight coefficient vector corresponding to each of the second stack layers, and each second weight coefficient vector includes a second weight coefficient corresponding to each of the encoding vectors;

[0220] The weight vector corresponding to each second stack layer is calculated based on the encoding vector and the second weight coefficient vector.

[0221] Optionally, the decoding vector generation module 812 is also configured to:

[0222] 1) Input the training features into the decoding layer of the translation model to generate a second vector;

[0223] 2) Input the second vector and the weight vector corresponding to the first second stack layer into the first second stack layer to obtain the output vector of the first second stack layer;

[0224] 3) Input the output vector of the (j-1)th second stack layer and the weight vector corresponding to the jth second stack layer into the jth second stack layer to obtain the output vector of the jth second stack layer, and use the output vector of the mth second stack layer as the decoding vector, where 2≤j≤m.

[0225] 4) Increment j by 1, and check if the incremented j is greater than m. If yes, end the process; otherwise, continue with step 3.

[0226] Optionally, the decoding vector generation module 812 compares the decoding vector with a preset vector verification set to obtain the error of the decoding vector.

[0227] Optionally, the training stopping condition includes: the rate of change of the error is less than a stability threshold.

[0228] The translation device provided in this embodiment allows the weight vectors input to each second stack layer to be the same or different from each other, further increasing the adjustability of the translation model in this embodiment and thus achieving better translation results.

[0229] The above is a schematic representation of a translation device according to this embodiment. It should be noted that the technical solution of this translation device and the technical solution of the above-described translation method belong to the same concept. Details not described in detail in the technical solution of the translation device can be found in the description of the technical solution of the above-described translation method.

[0230] This application also discloses a training device for a translation model, see [link to relevant documentation]. Figure 9 ,include:

[0231] The sample feature generation module 902 is configured to use the first sentence and the translated second sentence in the target corpus as training samples and training features, respectively.

[0232] The encoding vector generation module 904 is configured to input the training samples into the encoding layer of the translation model to obtain at least one encoding vector;

[0233] The weight vector generation module 906 is configured to input at least one encoded vector into the weight layer of the translation model to obtain a weight vector;

[0234] The decoding vector generation module 908 is configured to input the training features and the weight vector into the decoding layer of the translation model to obtain the decoding vector, and to obtain the error of the decoding vector based on the decoding vector;

[0235] The adjustment module 910 is configured to adjust the weight layer according to the error of the decoding vector and continue training the translation model until the training stopping condition is met.

[0236] The translation model training device provided in this application adjusts the weight coefficients of the weight layer during the training process of the translation model by inputting training samples and training labels, and obtains the adjusted weight layer, thereby achieving better translation results during the use of the translation model.

[0237] The above is a schematic scheme of a translation model training device according to this embodiment. It should be noted that the technical solution of this translation model training device and the technical solution of the translation model training method described above belong to the same concept. For details not described in detail in the technical solution of the training device, please refer to the description of the technical solution of the translation model training method described above.

[0238] An embodiment of this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the training method or translation method of the translation model as described above.

[0239] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the above-described training method or usage method of the translation model. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above-described training method or usage method of the translation model.

[0240] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0241] A chip that stores computer instructions, which, when executed by a processor, implement the steps of the translation method or the training method of the translation model as described above.

[0242] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0243] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0244] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A translation method, characterized in that, The method includes: obtaining a sentence to be translated; inputting the sentence to be translated into a translation model to obtain a translated sentence corresponding to the sentence to be translated, wherein the translation model includes an encoding layer and a weight layer, the encoding layer includes at least one first stack layer, the first stack layer includes a multi-head attention mechanism sublayer and a feedforward network layer, the weight layer is used to generate a weight vector based on at least one encoding vector output by the encoding layer and the weight coefficients corresponding to each encoding vector, and the weight vector is used to generate the translated sentence.

2. The translation method as described in claim 1, characterized in that, The translation model is trained in the following way: the first sentence in the target corpus and the translated second sentence are used as training samples and training features, respectively. The training samples are input into the encoding layer of the translation model to obtain at least one encoding vector; At least one encoded vector is input into the weight layer of the translation model to obtain a weight vector; the training features and the weight vector are input into the decoding layer of the translation model to obtain a decoding vector, and the error of the decoding vector is obtained based on the decoding vector; the weight layer is adjusted based on the error of the decoding vector, and the translation model is trained until the training stopping condition is met.

3. The translation method as described in claim 2, characterized in that, The step of inputting the training samples into the encoding layer of the translation model to obtain at least one encoding vector includes: inputting the training samples into the encoding layer of the translation model to generate a first vector; and inputting the first vector into a first stack layer to obtain an encoding vector output by the first stack layer.

4. The translation method as described in claim 2, characterized in that, The encoding layer includes n first stack layers, which are connected sequentially, where n ≥ 2 and n is a positive integer. The training samples are input into the encoding layer of the translation model to obtain at least one encoding vector, including:

102. Inputting the training samples into the encoding layer of the translation model to generate a first vector; 104. Inputting the first vector into the first first stack layer to obtain the encoding vector output by the first first stack layer; 106. Inputting the encoding vector of the (i-1)th first stack layer into the ith first stack layer to obtain the encoding vector output by the ith first stack layer, where 2 ≤ i ≤ n; 108. Incrementing i by 1 and determining whether the incremented i is greater than n. If yes, the process ends; otherwise, continue with step 106.

5. The translation method as described in claim 2, characterized in that, The weight coefficients include a first weight coefficient; the step of inputting at least one encoded vector into the weight layer of the translation model to obtain a weight vector includes: inputting at least one encoded vector into the weight layer of the translation model, wherein the weight layer includes a first weight coefficient corresponding to each encoded vector; and calculating a weight vector based on the encoded vector and the first weight coefficient.

6. The translation method as described in claim 5, characterized in that, The step of calculating a weight vector based on the encoded vector and the first weight coefficient includes: normalizing the first weight coefficient to obtain a first mapping weight coefficient; and multiplying the encoded vector output by each first stack layer with the corresponding first mapping weight coefficient and summing the results to obtain the weight vector.

7. The translation method as described in claim 2, characterized in that, The decoding layer includes at least one second stack layer; inputting the training features and the weight vector into the decoding layer of the translation model to obtain a decoding vector includes: inputting the training features into the decoding layer of the translation model to generate a second vector; inputting the second vector and the weight vector into the second stack layer to obtain the decoding vector.

8. The translation method as described in claim 2, characterized in that, The decoding layer includes m second stack layers, which are sequentially connected, where m ≥ 2 and m is a positive integer. The training features and the weight vector are input into the decoding layer of the translation model to obtain a decoding vector, including:

202. Inputting the training features into the decoding layer of the translation model to generate a second vector; 204. Inputting the second vector and the weight vector into the first second stack layer to obtain the output vector of the first second stack layer; 206. Inputting the output vector of the (j-1)th second stack layer and the weight vector into the jth second stack layer to obtain the output vector of the jth second stack layer, and using the output vector of the mth second stack layer as the decoding vector, where 2 ≤ j ≤ m; 208. Incrementing j by 1, determining whether the incremented j is greater than m. If yes, the process ends; otherwise, continuing with step 206.

9. The translation method as described in claim 2, characterized in that, The weight coefficients include second weight coefficients; the decoding layer includes m second stack layers, and the m second stack layers are connected sequentially, where m ≥ 2 and m is a positive integer; inputting at least one encoded vector into the weight layer of the translation model to obtain a weight vector includes: inputting at least one encoded vector into the weight layer of the translation model, wherein the weight layer includes a second weight coefficient vector corresponding to each second stack layer, and each second weight coefficient vector includes a second weight coefficient corresponding to each encoded vector; calculating the weight vector corresponding to each second stack layer based on the encoded vector and the second weight coefficient vector.

10. The translation method as described in claim 9, characterized in that, The step of calculating the weight vector corresponding to each second stack layer based on the encoding vector and the second weight coefficient vector includes: normalizing the second weight coefficient vector to obtain a second mapped weight coefficient vector; and multiplying the encoding vector output by each first stack layer with the corresponding second mapped weight coefficient vector and summing them to obtain the weight vector.

11. The translation method as described in claim 9, characterized in that, The training features and the weight vector are input into the decoding layer of the translation model to obtain the decoding vector, including:

302. Inputting the training features into the decoding layer of the translation model to generate a second vector; 304. Inputting the second vector and the weight vector corresponding to the first second stack layer into the first second stack layer to obtain the output vector of the first second stack layer; 306. Inputting the output vector of the (j-1)th second stack layer and the weight vector corresponding to the jth second stack layer into the jth second stack layer to obtain the output vector of the jth second stack layer, and using the output vector of the mth second stack layer as the decoding vector, where 2≤j≤m; 308. Incrementing j by 1, and determining whether the incremented j is greater than m. If yes, the process ends; otherwise, continue with step 306.

12. The translation method as described in claim 2, characterized in that, The error of the decoding vector is obtained by comparing the decoding vector with a preset vector verification set.

13. The translation method as described in claim 2 or 12, characterized in that, The training stopping condition includes: the rate of change of the error is less than the stability threshold.

14. A translation device, characterized in that, The apparatus includes: an acquisition module configured to acquire a statement to be translated; and a translation module configured to input the statement to be translated into a translation model to obtain a translated statement corresponding to the statement to be translated. The translation model includes an encoding layer and a weight layer. The encoding layer includes at least one first stack layer, which includes a multi-head attention mechanism sublayer and a feedforward network layer. The weight layer generates a weight vector based on at least one encoding vector output by the encoding layer and weight coefficients corresponding to each encoding vector. The weight vector is used to generate the translated statement.

15. A computing device, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the instruction, it implements the steps of the translation method according to any one of claims 1-13 when the instruction is executed by the processor.

16. A computer-readable storage medium storing computer instructions, characterized in that, When executed by the processor, this instruction implements the steps of the translation method according to any one of claims 1-13.

Images