Method, apparatus and storage medium for training machine translation model

Abstract

The application discloses a training method and device of a machine translation model and a storage medium. The method comprises the following steps: obtaining an attention score of an i-th word to a t-th word; performing weighted summation on the attention score and a word vector of the i-th word to obtain a hidden layer vector; calculating a distance between the hidden layer vector and each sub-attribute value of a discrete hidden variable corresponding to the hidden layer vector, and taking the closest sub-attribute value as an original attribute value of the discrete hidden variable; performing weighted summation on the attention score and the original attribute value to obtain an attribute vector of the discrete hidden variable of the t-th word; calculating a distance between the attribute vector and each sub-attribute value of the discrete hidden variable of the i-th word, and taking the closest sub-attribute value as a new attribute value of the discrete hidden variable of the i-th word; and performing constraint on the original attribute value and the new attribute value by using a loss function to obtain a loss L C , the new attribute value is integrated into the hidden layer vector to obtain a fusion vector corresponding to the i-th word, the machine translation loss L nmt is calculated by using the fusion vector, the final loss function is obtained by adding the loss L C and the loss L nmt , and the machine translation model is trained by using the final loss function.

Description

Technical Field

[0001] This invention relates to the field of machine translation technology, specifically to a method, apparatus, and storage medium for training machine translation models. Background Technology

[0002] Machine translation refers to the technology of automatically translating text from one natural language into text in another natural language using computer programs. With the development of computer technology and deep learning, Neural Machine Translation (NMT) models based on deep neural networks have become dominant in machine translation research. Currently, the most mainstream model architecture in NMT is the Transformer, whose core is the attention mechanism. When the model generates a translation word, the attention mechanism causes the word at the current generation position to pay attention to all words in the input sequence and calculates a corresponding attention score (normalized to between 0 and 1) for each word. The higher this score, the more important the word is to the generation at the current position. This mechanism of dynamically calculating the contribution of each word has enabled the model to achieve very good results.

[0003] Most existing translation technologies focus on sentence-level translation, that is, translating a single sentence from a source language to a target language. However, in real-world translation scenarios, we need to translate passages composed of multiple sentences. This necessitates considering the contextual relationships between sentences to ensure translation consistency, such as tense consistency and gender consistency, thereby generating more natural and fluent translations. To capture contextual information, existing work typically uses various variants of attention mechanisms, enabling the currently unknown word to focus on words across sentences. However, these variants do not explicitly model the consistency we care about, thus remaining insufficient in terms of generating consistent translations.

[0004] To address the consistency issue, existing technical solutions employ attention regularization. This method requires manually labeled data to indicate which word pairs need to maintain consistency. For example, the label "he" for male needs to be consistent with the name "Bob." With such labeled data, during training, the attention regularization method attempts to guide the word "he" to focus on the word "Bob," thereby awarding it a higher attention score. Specifically, it uses a KL divergence as a loss function, causing the model's attention distribution to concentrate on "Bob," increasing its attention score and reducing attention to other words.

[0005] However, attention regularization only tells the model which words are more important, but it does not explicitly model the connotation of this consistency, that is, the implied consistency of tense or gender. Therefore, it is still insufficient in terms of translation effect. Summary of the Invention

[0006] To overcome the shortcomings of the attention regularization method in the aforementioned background technology, which only tells the model which words are more important but does not explicitly model the connotation of consistency, that is, the implied tense consistency or gender consistency, and therefore still has shortcomings in translation effect, this invention provides a training method for machine translation models.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a training method for a machine translation model, wherein a discrete latent variable is modeled to represent the discrete attributes of words at each position in the source and target ends, wherein the discrete latent variable assigns a sub-attribute value to each sub-attribute in the discrete attributes, specifically including the following:

[0009] S1: Convert the i-th and t-th words of the source and target ends into word vectors e respectively. i and word vector e t The input is fed into a neural network translation model to obtain the attention score a of the i-th word to the t-th word. it , where t < i;

[0010] S2: Regarding the attention score a it and word vector e i We perform a weighted summation to obtain the hidden vector x corresponding to the word at position i. i ;

[0011] S3: Calculate the hidden layer vector x respectively. i The distance to each sub-attribute value of the discrete latent variable of the i-th word is used as the original attribute value l of the discrete latent variable of the i-th word. i ;

[0012] S4: Based on the attention score a it and the original attribute value l i We perform a weighted summation of the discrete latent variables of the t-th word to obtain the attribute vector attr corresponding to the i-th word. i ; Calculate the attribute vector attr respectively i The distance to each sub-attribute value of the discrete latent variable of the i-th word is used, and the sub-attribute value with the closest distance is taken as the new attribute value l of the discrete latent variable of the i-th word. 2 i ;

[0013] S5: Apply the loss function to the original attribute value l i and the new attribute value l 2 i By applying constraints, the loss L is obtained. C The new attribute value l 2i Integrating into the hidden layer vector x i In the process, the fusion vector h corresponding to the i-th word is obtained. i Using the fusion vector h i Loss of computer translation L nmt ;

[0014] S6: Through the aforementioned loss L C With loss L nmt The sums are used to obtain the final loss function L for machine translation, and the machine translation model is trained using the final loss function L.

[0015] The beneficial effects of this invention are that by explicitly modeling the discrete attributes (such as gender and tense) of words in the translated text using discrete latent variables, the machine translation model becomes more sensitive to these discrete attributes; by learning and incorporating the attribute information implied by discrete latent variables through an end-to-end training method, the model can improve its accuracy in translation generation consistency by taking into account the contextual relationships between translated texts while ensuring translation accuracy.

[0016] In some possible implementations, when the discrete latent variable represents a discrete attribute of gender, it assigns a positive value to males and a negative value to females in the discrete attribute of gender.

[0017] In some possible implementations, the attribute vector attr described in S4 i The calculation formula is as follows:

[0018]

[0019] In some possible implementations, the loss function described in S5 uses KL divergence, and the loss L... C The calculation formula is as follows:

[0020] L C =kl_div(p1, p2);

[0021] p1 = softmax([d0, d1]);

[0022] p2 = softmax([d 2 0, d 2 1]);

[0023] Where kl_div represents the loss of KL divergence, p1 and p2 represent probability distributions; d0 represents the hidden vector x of the i-th word. i The distance to the positive value of the discrete latent variable of the i-th word, where d1 represents the latent vector x of the i-th word. i The distance d from the negative value of the discrete latent variable of the i-th word2 0 represents the attribute vector attr i The distance d from the positive value of the discrete latent variable of the i-th word 2 1 represents the attribute vector attr i The distance to the negative value of the discrete latent variable of the i-th word, where softmax is the normalized exponential function.

[0024] In some possible implementations, the new attribute value l is controlled by a gating mechanism in S5. 2 i Integrating into the hidden layer vector x i In the above, the fusion vector h i The calculation formula is as follows:

[0025] h i =g*x i +(1-g)*l 2 i ;

[0026] g = σ(concat(x) i , l 2 i )@W);

[0027] Where concat represents vector concatenation; @ represents matrix multiplication; W represents the model parameter matrix, W∈R 2d ×d σ represents the sigmoid function; g represents the gating unit.

[0028] A second aspect of the present invention provides a training apparatus for a machine translation model, wherein a discrete latent variable is modeled to represent the discrete attributes of words at each position in the source and target ends, wherein the discrete latent variable assigns a sub-attribute value to each sub-attribute among the discrete attributes; the machine translation model training apparatus includes:

[0029] Attention score acquisition module: Converts the i-th word and t-th word of the source and target ends into word vectors e respectively. i and word vector e t The input is fed into a neural network translation model to obtain the attention score a of the i-th word to the t-th word. it , where t < i;

[0030] Hidden layer vector calculation module: for the attention score a it and word vector e i We perform a weighted summation to obtain the hidden vector x corresponding to the word at position i. i ;

[0031] The module for calculating the original attribute values ​​of discrete latent variables: calculates the hidden layer vector x respectively.i The distance to each sub-attribute value of the discrete latent variable of the i-th word is used as the original attribute value l of the discrete latent variable of the i-th word. i ;

[0032] The module for calculating the new attribute value of discrete latent variables: based on the attention score a it and the original attribute value l i We perform a weighted summation of the discrete latent variables of the t-th word to obtain the attribute vector attr corresponding to the i-th word. i ; Calculate the attribute vector attr respectively i The distance to each sub-attribute value of the discrete latent variable of the i-th word is used, and the sub-attribute value with the closest distance is taken as the new attribute value l of the discrete latent variable of the i-th word. 2 i ;

[0033] Constraint loss calculation module: Utilizes the loss function to calculate the original attribute value l i and the new attribute value l 2 i By applying constraints, the loss L is obtained. C ;

[0034] The loss calculation module for machine translation: calculates the new attribute value l 2 i Integrating into the hidden layer vector x i In the process, the fusion vector h is obtained. i Using the fusion vector h i Loss of computer translation L nmt ;

[0035] Machine translation model training module: through the loss L C With loss L nmt The sums are used to obtain the final loss function L for machine translation, and the machine translation model is trained using the final loss function L.

[0036] In some possible implementations, when the discrete latent variable represents a discrete attribute of gender, it assigns a positive value to males and a negative value to females in the discrete attribute of gender.

[0037] In some possible implementations, the loss function described in S5 uses KL divergence, and the loss L... C The calculation formula is as follows:

[0038] L C =kl_div(p1, p2);

[0039] p1 = softmax([d0, d1]);

[0040] p2 = softmax([d 2 0, d 2 1]);

[0041] Where kl_div represents the loss of KL divergence, p1 and p2 represent probability distributions; d0 represents the hidden vector x of the i-th word. i The distance to the positive value of the discrete latent variable of the i-th word, where d1 represents the latent vector x of the i-th word. i The distance d from the negative value of the discrete latent variable of the i-th word 2 0 represents the attribute vector attr i The distance d from the positive value of the discrete latent variable of the i-th word 2 1 represents the attribute vector attr i The distance to the negative value of the discrete latent variable of the i-th word, where softmax is the normalized exponential function.

[0042] In some possible implementations, the new attribute value l is controlled by a gating mechanism in S5. 2 i Integrating into the hidden layer vector x i In the above, the fusion vector h i The calculation formula is as follows:

[0043] h i =g*x i +(1-g)*l 2 i ;

[0044] g = σ(concat(x) i , l 2 i )@W);

[0045] Where concat represents vector concatenation; @ represents matrix multiplication; W represents the model parameter matrix, W∈R 2d ×d σ represents the sigmoid function; g represents the gating unit.

[0046] A third aspect of the present invention provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the steps of the above-described machine translation model training method. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the overall steps of the training method for the machine translation model according to an embodiment of the present invention.

[0048] Figure 2 This is a flowchart illustrating the computational process of training a machine translation model according to an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram illustrating the translation process using and not using latent variables in embodiments of the present invention.

[0050] Figure 4 This is a schematic diagram of the structure of the training device for the machine translation model according to an embodiment of the present invention. Detailed Implementation

[0051] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0052] This application can be used in many scenarios involving discrete attributes such as gender or tense. Let's take gender as an example. Example: A translation from English to French, consisting of two sentences:

[0053] Source: "All those lessons...They'll be useful one day."

[0054] The source can be translated into two French translations:

[0055] The first French translation is: "Tous ces cours...Ils seront utiles un jour."

[0056] The second French translation is: "Toutes ces" ...Elles seront utiles un jour."

[0057] Among them, the first French translation includes Tous, cours, and Ils, while the second French translation includes Toutes, "Tous," "cours," and "Ils" are translations of "All," "lessons," and "They" from their source English counterparts. The difference lies in the masculine connotations: in the first French translation, "Tous," "cours," and "Ils" are all masculine words; in the second French translation, "Toutes," "cours," and "Ils" are masculine words. All three words, including *Elles*, are feminine. Since all three refer to lessons, in French they need to express the same masculine or feminine gender. However, in this context, whether lessons are translated as the masculine word *cours* or the feminine word... All of these are correct. Therefore, it is necessary to ensure that the masculine and feminine attributes of these three words are consistent.

[0058] Similarly, there are tense attributes. For example, a translation where the source is Chinese and the destination is English.

[0059] Source: He opened the door and went into the room.

[0060] The source can be translated into two English translations:

[0061] The first English translation is: He opens the door and walks into the room.

[0062] The second English translation is: He opened the door and walked into the room.

[0063] Both translations are correct; the only difference is the tense. Although this sentence can be translated using either the present or past tense, the tense must be consistent within the same sentence.

[0064] In the examples of masculine and feminine genders and tenses mentioned above, consistency needs to be maintained between words. This consistency does not mean that two words must be exactly the same on the surface, but rather that their implicit discrete attributes must be consistent (both must be masculine or both must be past tense). This attribute is not obvious to machine translation models.

[0065] This application explicitly models this discrete property by using discrete latent variables. See the appendix for a translation diagram illustrating the use and non-latent variables. Figure 3 As shown, without using latent variables, the model's input is the source document X, and the output is the target document Y; with latent variables, the model's input is the source document X, latent variables L are generated based on X, and the target document Y is output based on both X and L.

[0066] Because attributes like gender and tense are discrete—for example, gender can be either feminine or masculine—discrete latent variables can be used for modeling. Taking gender as an example, discrete latent variables are used to represent its discrete attributes. Since gender only has two attributes, the discrete latent variable includes two values: one representing feminine and the other representing masculine. Similarly, if discrete latent variables are used to represent the discrete attributes of tense, then the discrete latent variable includes four values: present, past, future, and perfect. The number of values ​​depends on the discrete attribute represented by the discrete latent variable. If it represents the gender attribute, then there are only two values: feminine and masculine. If it represents the tense attribute, since there are more than two tenses, there will be more values, each representing a tense.

[0067] In French, every word has a discrete gender attribute, meaning each word corresponds to a discrete latent variable. Each word can only be either feminine or masculine; therefore, each word's corresponding discrete latent variable will correspond to either a feminine or masculine value.

[0068] Based on this, this application provides a training method for a machine translation model, wherein the discrete latent variables are used to represent the discrete attributes of words at each position in the source and target ends, and each discrete latent variable includes a negative value and a positive value of the discrete attribute.

[0069] A specific example is as follows: When the discrete latent variable represents the discrete attribute of gender, it assigns a positive value to males and a negative value to females in the discrete attribute of gender, that is, the discrete latent variable L = {l0, l1}, where l0, l1 ∈ R. d {l0, l1} are the negative and positive values ​​of the discrete latent variable representing gender, respectively, both of which are d-dimensional (usually 512-dimensional) vectors.

[0070] See appendix Figure 1 and attached Figure 2 As shown, the training method for the machine translation model includes the following steps:

[0071] S1: Convert the i-th and t-th words of the source and target ends into word vectors e respectively. i and word vector e t The input is fed into a neural network translation model to obtain the attention score a of the i-th word to the t-th word. it Where t < i. In this embodiment, i = 4, t = 1, 2, 3.

[0072] Word vector e i and word vector e t These are all d-dimensional vectors. For each word i, an attention score is calculated with each word t. The attention score a of the i-th word to the t-th word is called a. it Attention score a it The calculation formula is as follows:

[0073] a it =norm(e i *e t ).

[0074] Where norm represents normalization, and this word vector e i and word vector e t The value of the inner product is transformed to be between 0 and 1. The purpose is to obtain the contribution of the word at position t to the word at position i.

[0075] When the model generates a translation word, the attention mechanism causes the word at the current generation position to pay attention to all words in the input sequence and calculates a corresponding attention score (normalized to between 0 and 1) for each word. The higher this score, the more important the word is to the generation at the current position. This mechanism of dynamically calculating the contribution of each word has enabled the model to achieve very good results.

[0076] S2: Regarding the attention score a it and word vector e i We perform a weighted summation to obtain the hidden vector x corresponding to the word at position i. i .

[0077] (Since neural networks have many layers, the word representation in the first layer is usually called a word vector, and those in other layers are called hidden vectors, both referring to the d-dimensional vector corresponding to a word in the neural network.) Hidden vectors are calculated for all words. A hidden vector is the weighted sum of the word vectors of other words, with a weight of 1.2, resulting in an attention score. The hidden vector corresponding to the i-th word is x. i The formula is as follows:

[0078] x i =∑ t a it *e t

[0079] Because t < i, the i-th word will only calculate attention to the words preceding it, and not to the words following it (this is because machine translation generates words one by one, and the later words are not generated yet, so the training phase also requires the model not to pay attention to the later words). Following the aforementioned translation example where the source is English and the target is French, the third word, "cours," will only pay attention to the preceding "Tous" and "ces," and not to all the words following it.

[0080] S3: Calculate the hidden layer vector x respectively. i The distances to the negative and positive values ​​of the discrete latent variables of the i-th word are used to determine the original attribute value l of the discrete latent variable of the i-th word. i .

[0081] The distance can be calculated using either the inner product or Euclidean distance. The calculation process using the inner product method is as follows: d0 = x i *l0, d0 represent the hidden vector x of the i-th word. i The distance to the positive value l0 of the discrete latent variable of the i-th word; d1 = x i *l1, d1 represent the hidden vector x of the i-th word. i The distance to the negative value l1 of the discrete latent variable of the i-th word; * denotes the inner product of vectors.

[0082] If d0 < d1, take l0 as the hidden vector x of the i-th word. i The original attribute value l of the corresponding discrete latent variable i .

[0083] If d1 < d0, take l1 as the hidden vector x of the i-th word. i The original attribute value l of the corresponding discrete latent variable i .

[0084] Thus, for each word's hidden vector x i Each corresponds to a primitive attribute value l of a discrete latent variable. i , l i The value can be either l0 or l1. Following the translation example where the source is English and the destination is French, Tous might correspond to the positive value l0, and cours might correspond to the negative value l1.

[0085] The purpose of S3 is to obtain the original attribute values ​​of the discrete latent variables corresponding to each word. i Through the original attribute value l i The discrete attributes of each word were explicitly modeled.

[0086] S4: Based on the attention score a it and the original attribute value l i The discrete latent variables of the t-th word are weighted and summed to obtain the attribute vector attr. i ; Calculate the attribute vector attr respectively i The distances to the negative and positive values ​​of the discrete latent variables of the i-th word are used to determine the new attribute value l of the discrete latent variable of the i-th word. 2 i .

[0087] The original attribute values ​​l of the discrete latent variables corresponding to all words were obtained through S3. i However, this is calculated based on the hidden layer vector. To ensure consistency in temporality or masculine / feminine properties, new attributes need to be obtained from the aforementioned discrete latent variables. The attribute vector attr... i The calculation formula is as follows:

[0088]

[0089] Distance can be calculated using either the inner product or Euclidean distance. The calculation process using the inner product method is as follows: d 2 0 = attr i *l0,d 2 0 represents the attribute vector attr i The distance to the positive value l0 of the discrete latent variable of the i-th word; d 21 = attr i *l1, d1 represent the attribute vector attr i The distance to the negative value l1 of the discrete latent variable of the i-th word; * denotes the inner product of vectors.

[0090] If d 2 0 < d 2 1. Take l0 as the hidden vector x of the i-th word. i The corresponding new attribute value l of the discrete latent variable 2 i .

[0091] If d 2 1 < d 2 0, take l1 as the hidden vector x of the i-th word. i The corresponding new attribute value l of the discrete latent variable 2 i .

[0092] The purpose of S4 is to obtain a new attribute of the discrete latent variable corresponding to the i-th word, and the new attribute value l. 2 i The consistency of the discrete attributes of the i-th word was modeled.

[0093] S5: Apply the loss function to the original attribute value l i and the new attribute value l 2 i By applying constraints, the loss L is obtained. C The new attribute value l 2 i Integrating into the hidden layer vector x i In the process, the fusion vector h is obtained. i Using the fusion vector h i Loss of computer translation L nmt .

[0094] To make the original attribute value l i and new attribute value l 2 i To maintain consistency, KL divergence is used as the loss function when the original attribute value l i and new attribute value l 2 i. When there is inconsistency, the loss increases, especially when the original attribute value l i and new attribute value l 2 i. When consistency is achieved, the loss decreases, allowing machine translation models to be more sensitive to positive and negative attributes. The loss function using KL divergence is used to optimize the original attribute values ​​l. i and new attribute value l 2 i.Maintaining consistency makes machine translation models more sensitive to discrete attributes, thereby enhancing the consistency of discrete attributes during the translation process and making the translation more accurate.

[0095] Loss L C The calculation formula is as follows:

[0096] L C =kl_div(p1, p2);

[0097] p1 = softmax([d0, d1]), where [d0, d1] means concatenating the two values ​​d0 and d1 into a single vector;

[0098] p2 = softmax([d 2 0, d 2 1]); [d 2 0, d 2 1] indicates that d 2 0 and d 2 1. Two values ​​are concatenated into a vector;

[0099] Where kl_div represents the loss of KL divergence, p1 and p2 are both discrete probability distributions, representing the probability of taking a negative or positive value; d0 represents the hidden vector x of the i-th word. i The distance to the positive value of the discrete latent variable of the i-th word, where d1 represents the latent vector x of the i-th word. i The distance d from the negative value of the discrete latent variable of the i-th word 2 0 represents the attribute vector attr i The distance d from the positive value of the discrete latent variable of the i-th word 2 1 represents the attribute vector attr i The distance to the negative value of the discrete latent variable of the i-th word, where softmax is the normalized exponential function.

[0100] The new attribute value l 2 i Integrating into the hidden layer vector x i In this process, the negative or positive information corresponding to the discrete latent variable L is incorporated into the hidden layer vector x. i In this process, we can choose direct addition, weighted summation, or a gating mechanism. Since the gating mechanism provides better fusion results, we use a gating mechanism to merge the new attribute value l. 2 i Integrating into the hidden layer vector x i In the above, the fusion vector h i The calculation formula is as follows:

[0101] h i =g*x i +(1-g)*l2 i ;

[0102] g = σ(concat(x) i , l 2 i )@W);

[0103] Here, `concat` represents vector concatenation; `@` represents matrix multiplication; and `W` represents the model parameter matrix, which is continuously optimized during machine translation model training and used to concat(x) the concatenated structure. i , l 2 i Map back to d dimensions, W∈R 2d ×d σ represents the sigmoid function; g represents the gating unit, which automatically controls the input x based on the parameters of the machine translation model. i With l 2 i Weighting during information fusion.

[0104] The above objective is to fuse the hidden vector of the current position word with the information of the discrete latent variables corresponding to the current position word, and the fused information (fused vector h) i The loss function L involved in computer translation nmt This allows machine translation models to explicitly distinguish these discrete attributes, improving the consistency of model translation.

[0105] S6: Through the aforementioned loss L C With loss L nmt The summation yields the final loss function L for machine translation, which is then used to train the machine translation model. The final loss function L is the target to be optimized; a lower loss function value indicates a better machine translation model performance. Compared to the original machine translation loss function, the final loss function L models the consistency of the discrete latent variables corresponding to each word in the source and target languages, thereby improving the consistency of the translation results during the optimization of the final loss function.

[0106] The preferred training method is gradient descent. Gradient descent updates the parameters of the machine translation model according to the direction of the gradient of the loss function, gradually optimizing the final loss function, and ultimately training a translation model with stronger translation consistency.

[0107] This application makes machine translation models more sensitive to discrete attributes (such as gender and tense) of words in translated text by explicitly modeling them using discrete latent variables. By learning and incorporating the attribute information implied by discrete latent variables through an end-to-end training method, the model can improve its accuracy in translation generation consistency by taking into account the contextual relationships between translated texts while ensuring translation accuracy.

[0108] This application also provides a training device for a machine translation model, which models a discrete latent variable to represent the discrete attributes of words at each position in the source and target ends, wherein the discrete latent variable assigns a sub-attribute value to each sub-attribute in the discrete attributes.

[0109] In some implementations, when the discrete latent variable represents a discrete attribute of gender, it assigns a positive value to males and a negative value to females in the discrete attribute of gender.

[0110] See appendix Figure 4 As shown, the machine translation model training device includes:

[0111] Attention score acquisition module: Converts the i-th and t-th words of the source and target ends into word vectors e respectively. i and word vector e t The input is fed into a neural network translation model to obtain the attention score a of the i-th word to the t-th word. it , where t < i;

[0112] Hidden layer vector calculation module: for the attention score a it and word vector e i We perform a weighted summation to obtain the hidden vector x corresponding to the word at position i. i .

[0113] The module for calculating the original attribute values ​​of discrete latent variables: calculates the hidden layer vector x respectively. i The distance between the negative and positive values ​​of the discrete latent variable of the i-th word is used to determine the nearest sub-attribute value, which is then taken as the original attribute value of the discrete latent variable of the i-th word. i .

[0114] In some implementations, the distance can be expressed as an inner product or Euclidean distance. The calculation process using the inner product method is as follows: d0 = x i *l0, d0 represent the hidden vector x of the i-th word. i The distance to the positive value l0 of the discrete latent variable of the i-th word; d1 = x i *l1, d1 represent the hidden vector x of the i-th word. iThe distance to the negative value l1 of the discrete latent variable of the i-th word; * denotes the inner product of vectors.

[0115] If d0 < d1, take l0 as the hidden vector x of the i-th word. i The original attribute value l of the corresponding discrete latent variable i .

[0116] If d1 < d0, take l1 as the hidden vector x of the i-th word. i The original attribute value l of the corresponding discrete latent variable i .

[0117] The module for calculating the new attribute value of discrete latent variables: based on the attention score a it and the original attribute value l i We perform a weighted summation of the discrete latent variables of the t-th word to obtain the attribute vector attr corresponding to the i-th word. i ; Calculate the attribute vector attr respectively i The distance between the negative and positive values ​​of the discrete latent variable of the i-th word is used to determine the new attribute value l of the discrete latent variable of the i-th word. 2 i .

[0118] The attribute vector attr i The calculation formula is as follows:

[0119]

[0120] In some implementations, the distance can be calculated using either an inner product or Euclidean distance. The calculation process using the inner product method is as follows: d 2 0 = attr i *l0,d 2 0 represents the attribute vector attr i The distance to the positive value l0 of the discrete latent variable of the i-th word; d 2 1 = attr i *l1, d1 represent the attribute vector attr i The distance to the negative value l1 of the discrete latent variable of the i-th word; * denotes the inner product of vectors.

[0121] If d 2 0 < d 2 1. Take l0 as the hidden vector x of the i-th word. i The corresponding new attribute value l of the discrete latent variable 2 i .

[0122] If d 2 1 < d 20, take l1 as the hidden vector x of the i-th word. i The corresponding new attribute value l of the discrete latent variable 2 i .

[0123] Constraint loss calculation module: Utilizes the loss function to calculate the original attribute value l i and the new attribute value l 2 i By applying constraints, the loss L is obtained. C ;

[0124] In some approaches, the loss function employs KL divergence, with a loss L... C The calculation formula is as follows:

[0125] L C =kl_div(p1, p2);

[0126] p1 = softmax([d0, d1]);

[0127] p2 = softmax([d 2 0, d 2 1]);

[0128] Where kl_div represents the loss of KL divergence, p1 and p2 represent probability distributions; d0 represents the hidden vector x of the i-th word. i The distance to the positive value of the discrete latent variable of the i-th word, where d1 represents the latent vector x of the i-th word. i The distance d from the negative value of the discrete latent variable of the i-th word 2 0 represents the attribute vector attr i The distance d from the positive value of the discrete latent variable of the i-th word 2 1 represents the attribute vector attr i The distance to the negative value of the discrete latent variable of the i-th word, where softmax is the normalized exponential function.

[0129] The loss calculation module for machine translation: calculates the new attribute value l 2 i Integrating into the hidden layer vector x i In the process, the fusion vector h is obtained. i Using the fusion vector h i Loss of computer translation L nmt .

[0130] In some implementations, the new attribute value l is controlled by a gating mechanism. 2 i Integrating into the hidden layer vector x i In the above, the fusion vector h iThe calculation formula is as follows:

[0131] h i =g*x i +(1-g)*l 2 i ;

[0132] g = σ(concat(x) i , l 2 i )@W);

[0133] Where concat represents vector concatenation; @ represents matrix multiplication; W represents the model parameter matrix, W∈R 2d ×d σ represents the sigmoid function; g represents the gating unit.

[0134] Machine translation model training module: through the loss L C With loss L nmt The sums are used to obtain the final loss function L for machine translation, and the machine translation model is trained using the final loss function L.

[0135] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the steps of the above-described machine translation model training method.

[0136] The storage medium stores program instructions capable of implementing all the above methods. These program instructions can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium can also be: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or other media capable of storing program code, or a terminal device such as a computer, server, mobile phone, or tablet.

[0137] A processor can also be called a CPU (Central Processing Unit). A processor may be an integrated circuit chip with signal processing capabilities. A processor can also be:

[0138] A DSP (Digital Signal Processor) is a processor composed of large-scale or very large-scale integrated circuit chips used to perform specific signal processing tasks. It gradually developed to meet the needs of high-speed, real-time signal processing tasks. With the development of integrated circuit technology and digital signal processing algorithms, the implementation methods of digital signal processors are constantly changing, and their processing capabilities are continuously improving and expanding.

[0139] ASIC (Application Specific Integrated Circuit) refers to an integrated circuit designed and manufactured to meet the specific requirements of a user and the needs of a specific electronic system.

[0140] FPGA (Field Programmable Gate Array) is a further development based on programmable devices such as PAL (Programmable Array Logic) and GAL (Generic Array Logic). It emerged as a semi-custom circuit in the field of Application-Specific Integrated Circuits (ASICs), solving the shortcomings of custom circuits while overcoming the limitation of the limited gate count of original programmable devices.

[0141] A general-purpose processor, which may be a microprocessor or any conventional processor.

[0142] A processor can also include a main processor and a coprocessor. The main processor is used to process data in the wake-up state and is also called the CPU (Central Processing Unit). The coprocessor is a low-power processor used to process data in the standby state.

[0143] In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content that the display screen needs to show.

[0144] In some embodiments, the processor may also include an AI (Artificial Intelligence) processor for handling computational operations related to machine learning.

[0145] The above embodiments are only for illustrating the technical concept and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it. They should not be used to limit the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A training method for a machine translation model, characterized by: For each position of a word in both the source and target ends, a discrete latent variable is modeled to represent its discrete attributes. This discrete latent variable assigns a sub-attribute value to each sub-attribute within the discrete attributes, specifically including the following: S1: Convert the i-th and t-th words of the source and target ends into word vectors e respectively. i and word vector e t The input is fed into a neural network translation model to obtain the attention score a between the i-th word and the t-th word. it , where t < i; S2: Regarding the attention score a it and word vector e i We perform a weighted summation to obtain the hidden vector x corresponding to the word at position i. i ; S3: Calculate the hidden layer vector x respectively. i The distance to each sub-attribute value of the discrete latent variable of the i-th word is used as the original attribute value l of the discrete latent variable of the i-th word. i ; S4: Based on the attention score a it and the original attribute value l i We perform a weighted summation of the discrete latent variables of the t-th word to obtain the attribute vector attr corresponding to the i-th word. i ; Calculate the attribute vector attr respectively i The distance to each sub-attribute value of the discrete latent variable of the i-th word is used as the new attribute value l of the discrete latent variable of the i-th word. 2 i ; S5: Apply the loss function to the original attribute value l i and the new attribute value l 2 i By applying constraints, the loss L is obtained. C The new attribute value l 2 i Integrating into the hidden layer vector x i In the process, the fusion vector h corresponding to the i-th word is obtained. i Using the fusion vector h i Loss of computer translation L nmt ; S6: Through the aforementioned loss L C With loss L nmt The final loss function L of machine translation is obtained by summing the results, and the machine translation model is trained using the final loss function L of machine translation. When the discrete latent variable represents the discrete attribute of gender, it assigns a positive value to male and a negative value to female in the discrete attribute of gender. The loss function described in S5 uses KL divergence, and the loss L... C The calculation formula is as follows: L C =kl_div(p1,p2); p1 = softmax([d0, d1]); p2=softmax([d 2 0, d 2 1]); Where kl_div represents the loss of KL divergence, p1 and p2 represent probability distributions; d0 represents the hidden vector x of the i-th word. i The distance to the positive value of the discrete latent variable of the i-th word, where d1 represents the latent vector x of the i-th word. i The distance d from the negative value of the discrete latent variable of the i-th word 2 0 represents the attribute vector attr i The distance d from the positive value of the discrete latent variable of the i-th word 2 1 represents the attribute vector attr i The distance to the negative value of the discrete latent variable of the i-th word, where softmax is the normalized exponential function.

2. The training method for the machine translation model according to claim 1, characterized in that: The attribute vector attr described in S4 i The calculation formula is as follows: 。 3. The training method for the machine translation model according to claim 1, characterized in that: In S5, the new attribute value l is controlled through a gating mechanism. 2 i Integrating into the hidden layer vector x i In the above, the fusion vector h i The calculation formula is as follows: h i =g×x i +(1-g)×l 2 i ; g=σ(concat(x i ,l 2 i )@W); Where concat represents vector concatenation; @ represents matrix multiplication; W represents the model parameter matrix, W∈R 2d×d σ represents the sigmoid function; g represents the gating unit.

4. A training device for a machine translation model, characterized in that: A discrete latent variable is modeled to represent the discrete attributes of words at each position in both the source and target ends. The discrete latent variable assigns a sub-attribute value to each sub-attribute in the discrete attributes. The training device for the machine translation model includes: Attention score acquisition module: Converts the i-th and t-th words of the source and target ends into word vectors e respectively. i and word vector e t The input is fed into a neural network translation model to obtain the attention score a between the i-th word and the t-th word. it , where t < i; Hidden layer vector calculation module: for the attention score a it and word vector e i We perform a weighted summation to obtain the hidden vector x corresponding to the word at position i. i ; The module for calculating the original attribute values ​​of discrete latent variables: calculates the hidden layer vector x respectively. i The distance to each sub-attribute value of the discrete latent variable of the i-th word is used as the original attribute value l of the discrete latent variable of the i-th word. i ; The module for calculating the new attribute value of discrete latent variables: based on the attention score a it and the original attribute value l i We perform a weighted summation of the discrete latent variables of the t-th word to obtain the attribute vector attr corresponding to the i-th word. i ; Calculate the attribute vector attr respectively i The distance to each sub-attribute value of the discrete latent variable of the i-th word is used as the new attribute value l of the discrete latent variable of the i-th word. 2 i ; Constraint loss calculation module: Utilizes the loss function to calculate the original attribute value l i and the new attribute value l 2 i By applying constraints, the loss L is obtained. C ; The loss calculation module for machine translation: calculates the new attribute value l 2 i Integrating into the hidden layer vector x i In the process, the fusion vector h is obtained. i Using the fusion vector h i Loss of computer translation L nmt ; Machine translation model training module: through the loss L C With loss L nmt The final loss function L of machine translation is obtained by summing the results, and the machine translation model is trained using the final loss function L. When the discrete latent variable represents the discrete attribute of gender, it assigns a positive value to male and a negative value to female in the discrete attribute of gender. The loss function uses KL divergence, and the loss L... C The calculation formula is as follows: L C =kl_div(p1,p2); p1 = softmax([d0, d1]); p2=softmax([d 2 0, d 2 1]); Where kl_div represents the loss of KL divergence, p1 and p2 represent probability distributions; d0 represents the hidden vector x of the i-th word. i The distance to the positive value of the discrete latent variable of the i-th word, where d1 represents the latent vector x of the i-th word. i The distance d from the negative value of the discrete latent variable of the i-th word 2 0 represents the attribute vector attr i The distance d from the positive value of the discrete latent variable of the i-th word 2 1 represents the attribute vector attr i The distance to the negative value of the discrete latent variable of the i-th word, where softmax is the normalized exponential function.

5. The training apparatus for the machine translation model according to claim 4, characterized in that: The machine translation loss calculation module uses a gating mechanism to process the new attribute value l 2 i Integrating into the hidden layer vector x i In the above, the fusion vector h i The calculation formula is as follows: h i =g×x i +(1-g)×l 2 i ; g=σ(concat(x i ,l 2 i )@W); Where concat represents vector concatenation; @ represents matrix multiplication; W represents the model parameter matrix, W∈R 2d×d σ represents the sigmoid function; g represents the gating unit.

6. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the steps of the training method for the machine translation model as described in any one of claims 1 to 3.

Images