Divergence modeling in neural machine translation
By treating the neural machine translation model as a multi-label classification task, using sigmoid activation function and SCONES loss function, the problem of intrinsic uncertainty processing is solved, the translation quality is improved and the cluster search curse is alleviated, and the translation performance is improved and reasoning accelerates.
Patent Information
- Application Number
- CN202280095134.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-28
- Publication Date
- 2024-11-19
AI Technical Summary
When dealing with inherent uncertainty, existing neural machine translation models cannot effectively distinguish different types of uncertainty, resulting in decreased translation quality and deterioration in performance, especially at larger cluster sizes.
Treat machine translation as a multi-label classification task, using sigmoid activation function and a single-label comparison target (SCONES) loss function of non-exclusive sequences, allowing the model to be trained on a single reference training data, and calculate the probability of each translation through a binary classifier. , and assign a probability between 0 and 1 to the word element at each time step.
Significantly improves intrinsic uncertainty processing, alleviates the drawbacks of traditional NMT models, improves translation quality, especially in medium resource language pairs and small cluster sizes, achieving 3.9 times inference acceleration, while maintaining or improving the BLEU score. .
Smart Images

Figure CN118984994A_ABST
Abstract
Description
Background Art
[0001] Machine translation (MT) is the automatic translation of written sentences in one language into another. Neural models of MT, namely neural machine translation (NMT), can try to resolve different kinds of uncertainty. This can include model uncertainty where the model itself is not sure which of two or more translations is correct, and intrinsic uncertainty that an input sentence has more than one correct translation. Existing NMT models can use a softmax layer to define a probability distribution over all target language sentences. While this approach can account for uncertainty by spreading the probability mass accordingly, it cannot distinguish between different kinds of uncertainty.
[0002] This shortcoming can lead to many pathologies in neural machine translation models. One such pathology is that when the cluster size is increased at inference time (when the model searches in a larger translation space), the model prefers short translations over reference translations. This pathology can be addressed by dividing the model score by a function of the translation length or using rewards for longer candidate translations. However, such fixes are post-hoc and do not address the root cause of these pathologies. Therefore, when using larger cluster sizes, such NMT methods can produce lower quality results or otherwise suffer from performance degradation. Therefore, applications and services that adopt such methods may not provide suitable or appropriate results. Summary of the invention
[0003] The aspect of the present technology regards machine translation as a multi-label classification task. The system is not for source sentence x learning a single distribution P (y|x) for all target sentences y, but for each sentence pair (x, y) learning indicating whether y is a valid translation of x. In this framework, intrinsic uncertainty can be represented by setting the probability of two (or more) correct translations y1 and y2 to 1 at the same time. A separate binary classifier can be used to calculate the probability of each translation, and therefore the sum of the probabilities of all translations is not required to be one. Compared with the typical machine translation model (e.g., Transformer model) using a softmax output layer, the divergence modeling method of this paper utilizes sigmoid activation (or another bounded activation function), which assigns the probability between 0 and 1 to each word in the vocabulary (all word units of this group) at each time step. In addition, the loss function referred to as "single label contrast target of non-exclusive sequences" (SCONES) herein allows training models on single reference training data. This allows non-exclusive outputs to be modeled. The method can provide significant improvements over existing techniques in terms of inherent uncertainty and mitigate the drawbacks encountered by those other techniques. While the method is particularly beneficial for text translation situations, it is also applicable to other types of classification problems for models for which the training corpus typically contains a single reference, but in practice multiple labels may be acceptable. One such type is text classification, such as in the case of classifying newspaper articles, website content, or medical diagnoses.
[0004] According to one aspect of the present technology, a system configured for a machine translation model. The system includes a memory configured to store a set of text examples in a source language and a set of rewritten texts in one or more languages different from the source language; and one or more processing elements, the one or more processing elements being operatively coupled to the memory. The one or more processing elements implement the machine translation model as a neural network having an encoder module and a decoder module. The encoder module includes an encoder neural network configured to receive a given text example (x = <x1, ..., x |x| >) and generates an encoded representation of a given text example. The decoder module includes a decoder neural network configured to receive the encoded representation and a set of translation prefixes (y <i = <y1, ..., y i-1 >) and outputs the unbounded function f(x, y <i ), the unbounded function is related to the given text example x and the set of translation prefixes y <iA set of word-grams is associated with each pair of translation prefixes in , wherein each word-gram is assigned a probability between 0 and 1 in the vocabulary of a given text example at each time step. The log probability module is configured to act on the unbounded function to generate a corresponding bounded conditional probability for each word-gram, wherein the conditional probability is not normalized over the vocabulary at each time step. The loss function module has a positive loss component and a scaled negative loss component, wherein the loss function module is configured to identify whether each target text in a set of target texts is a valid translation of a given text example.
[0005] The set of text examples may include a set of input sentences, and the system is configured to learn a binary classifier for each sentence pair (x, y) indicating whether y is a valid translation of x. Here, the inherent uncertainty may be represented by setting the conditional probabilities of at least two correct translations y1 and y2 to a maximum probability simultaneously. Alternatively or additionally, the log-odds module is configured to generate a probability for each translation using a separate binary classifier.
[0006] According to any of the above, the probability of a complete translation of a given text example can be decomposed into a product of word-level probabilities. Alternatively or additionally, the log-odds module is configured to apply a sigmoid activation to an unbounded function at each time step. Alternatively or additionally, the positive loss component applies a logarithmic function to a bounded conditional probability of each reference word, and the scaled negative loss component applies a logarithmic function to a bounded conditional probability of each non-reference word. Alternatively or additionally, during inference, the system is configured to search for a translation with the highest probability of being the translation of the text segment to be translated.
[0007] According to any of the above, the system is configured to express the inherent uncertainty associated with the machine translation model.Alternatively or additionally, the loss function module is configured to adjust the scaling of the negative loss component to maximize translation performance.
[0008] According to any of the above items, the encoder module and the decoder module include a self-attention neural network encoder-decoder architecture. Alternatively, the encoder module and the decoder module may include a sequence-to-sequence model architecture.
[0009] According to another aspect, a machine translation method employs a neural network, and the method includes: storing a set of text examples in a memory; receiving a given text example (x = <x1, ...,x |x| >); the encoder module generates an encoded representation of a given text example; the decoder module receives the encoded representation and a set of translation prefixes (y <i = <y1, ..., y i-1 >); the decoder module outputs the unbounded function f(x, y<i ), the unbounded function is related to the given text example x and the set of translation prefixes y <i A set of word-grams is associated with each pair of translation prefixes in , wherein each word-gram is assigned a probability between 0 and 1 in the vocabulary of a given text example at each time step; a corresponding bounded conditional probability is generated for each word-gram based on an unbounded function by a log-odds module, wherein the conditional probability is not normalized over the vocabulary at each time step; and a loss function module having a positive loss component and a scaled negative loss component is used to identify whether each target text in a set of target texts is a valid translation of the given text example.
[0010] In one example, the set of text examples includes a set of input sentences, and the method includes: for each sentence pair (x, y), learning a binary classifier indicating whether y is a valid translation of x. Here, the inherent uncertainty can be represented by setting the conditional probabilities of at least two correct translations y1 and y2 to a maximum probability simultaneously. Alternatively or additionally, generating a corresponding bounded conditional probability for each word-gram includes using a separate binary classifier to generate a probability for each translation.
[0011] According to any of the above, the probability of a complete translation of a given text example can be decomposed into a product of word-level probabilities. Alternatively or additionally, generating a corresponding bounded conditional probability for each word-gram can include applying a sigmoid activation to an unbounded function at each time step. Alternatively or additionally, the positive loss component can apply a logarithmic function to the bounded conditional probability of each word-gram, and the scaled negative loss component can apply a logarithmic function to the bounded conditional probability of each word-gram.
[0012] According to any of the above, the method may further include: searching during inference for a translation having the highest probability of being the translation of the text segment to be translated. Alternatively or additionally, the method may further include: adjusting the scaling of the negative loss component to maximize translation performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 An example of using a trained NMT model in a text-based application according to aspects of the present technology is illustrated.
[0014] Figure 2 A Transformer-type architecture that may be employed in accordance with aspects of the present technique is illustrated.
[0015] Figure 3 A multi-directional NMT Transformer architecture for non-exclusive target sequences in accordance with aspects of the present technique is illustrated.
[0016] FIG. 4A to FIG. 4Bis a table showing Transformer hyperparameters and MT training network sizes according to aspects of the present technique.
[0017] FIG. 5A to FIG. 5B A comparison of a SCONES-based NMT system according to aspects of the present technology with a softmax baseline is illustrated.
[0018] Figure 6 A table of language pairs and alpha values based on BLEU scores according to aspects of the present technology is illustrated.
[0019] 7A to 7F is a graph showing BLEU scores as a function of decoding speed for different translation directions in accordance with aspects of the present technique.
[0020] Figure 8 BLEU scores are plotted as a function of bundle size in accordance with aspects of the present technique.
[0021] FIG. 9A to FIG. 9B Test results involving length ratio, bundle size, and alpha value according to aspects of the present technique are plotted.
[0022] FIG. 10A to FIG. 10B Illustrated is a table comparing the results of a beam search and an exact search in accordance with aspects of the present technique.
[0023] Fig. 10C Illustrated are the mean and standard deviation of the log-probability of the global highest probability translation search in accordance with aspects of the present technique.
[0024] Fig.11 The percentages of beam search errors for different testing criteria in accordance with aspects of the present technique are illustrated.
[0025] FIG. 12A to FIG. 12B A table illustrating sample information when testing according to IBM-3 according to aspects of the present technique.
[0026] FIG. 13A to FIG. 13B Presented are graphs of BLEU scores at different sampling temperatures according to aspects of the present technology.
[0027] Fig.14 An example implementation of SCONES in JAX according to aspects of the present technique is provided.
[0028] FIG. 15A to FIG. 15B A system for use with aspects of the present technology is illustrated.
[0029] Fig.16 Methods according to aspects of the present technology are illustrated. DETAILED DESCRIPTION Overview
[0030] Figure 1 A high-level example 100 of utilizing a trained NMT model in a text-based application is illustrated. In this example, one or more source files (e.g., curated text files in one or more selected languages) in box 102 are applied to a trained model at box 104. The output of the model is a translated text 106 in a language different from the original language. The translated text can be used in a variety of applications, including machine translation, grammatical error correction, dialogue modeling, or automatic foreign language subtitles (e.g., for video content), etc. The translated text can be evaluated by comparison with a baseline translation (e.g., a manually created golden translation).
[0031] Some traditional NMT models can be based on the Transformer architecture. As described in detail below, this architecture uses a softmax function to generate an output probability set (probability distribution). However, the softmax layer in neural machine translation is designed to model the distribution over mutually exclusive tokens. However, machine translation is inherently nondeterministic: the same source sentence can have multiple semantically equivalent translations. Therefore, one aspect of the present technology is to replace the softmax activation with a multi-label classification layer that can more effectively model the disagreement.
[0032] The SCONES loss function enables modeling of non-exclusive outputs. The multi-label output layer can still be trained on a single reference training data using the SCONES loss function. SCONES produces consistent BLEU score gains across various translation directions, especially for medium-resource language pairs and small bundle sizes. By using a smaller bundle size, the method can speed up reasoning by a factor of approximately 3.9 times and still match or improve the softmax BLEU score, as shown in testing. In addition, SCONES can be used to train an NMT model that assigns the highest probability to a proper translation, thereby alleviating the "beam search curse". As detailed below, additional experiments on synthetic language pairs with different degrees of uncertainty indicate that the improvement from SCONES relative to other methods can be attributed to better handling of disagreements. General Transformer Methods
[0033] The SCONES method can adopt a self-attention architecture, for example, the Transformer neural network encoder-decoder architecture. Figure 2An exemplary general purpose Transformer-type architecture is shown in FIG. 1 , which is based on the arrangement shown in U.S. Patent No. 10,452,978, entitled “Attention-based sequence transduction neural networks,” the entire disclosure of which is incorporated herein by reference. Although a Transformer-type architecture may be employed, the methods described herein may also be used with different architectures. For example, a sequence-to-sequence model, such as a model using a long short-term memory (LSTM) architecture.
[0034] Figure 2 The system 200 may be implemented as a computer program by a processor of one or more computers in one or more locations. The system 200 receives an input sequence 202 and processes the input sequence 202 to transduce the input sequence 202 into an output sequence 204. The input sequence 202 has a corresponding network input at each of a plurality of input positions in an input order, and the output sequence 204 has a corresponding network output at each of a plurality of output positions in an output order.
[0035] The system 200 can perform any of a variety of tasks that require processing sequential inputs to generate sequential outputs. The system 200 includes an attention-based sequence transduction neural network 206, which in turn includes an encoder neural network 208 and a decoder neural network 210. The encoder neural network 208 is configured to receive an input sequence 202 and generate a corresponding encoded representation of each network input in the input sequence. The encoded representation is a vector or other ordered set of numerical values. Then, the decoder neural network 210 is configured to use the encoded representation of the network input to generate an output sequence 204. Generally speaking, both the encoder 208 and the decoder 210 are attention-based. In some cases, neither the encoder nor the decoder includes any convolutional layers or any recurrent layers. The encoder neural network 208 includes an embedding layer (input embedding) 212 and a sequence of one or more encoder sub-networks 214. The encoder neural network 208 may include N encoder sub-networks 214.
[0036] The embedding layer 212 is configured to map each network input in the input sequence to a digital representation of the network input in the embedding space, for example, to a vector in the embedding space. Then, the embedding layer 212 provides the digital representation of the network input to the first subnetwork in the sequence of the encoder subnetwork 214. The embedding layer 212 can be configured to map each network input to an embedding representation of the network input, and then combine (e.g., sum or average) the embedding representation of the network input with the position embedding of the input position of the network input in the input order to generate a combined embedding representation of the network input. In some cases, the position embedding is learned. As used herein, "learned" means that the operation or value has been adjusted during the training of the sequence transduction neural network 206. In other cases, the position embedding can be fixed and different for each position.
[0037] The combined embedded representation is then used as a digital representation of the network input. Each encoder sub-network in encoder sub-networks 214 is configured to receive a corresponding encoder sub-network input for each of a plurality of input positions and to generate a corresponding sub-network output for each of a plurality of input positions. The encoder sub-network output generated by the last encoder sub-network in the sequence is then used as the encoded representation of the network input. For the first encoder sub-network in the sequence, the encoder sub-network input is the digital representation generated by the embedding layer 212, and for each encoder sub-network in the sequence except the first encoder sub-network, the encoder sub-network input is the encoder sub-network output of the preceding encoder sub-network in the sequence.
[0038] Each encoder sub-network 214 includes an encoder self-attention sub-layer 216. The encoder self-attention sub-layer 216 is configured to receive a sub-network input for each of a plurality of input positions, and for each particular input position in the input order, apply an attention mechanism to the encoder sub-network input at the input position using one or more queries derived from the encoder sub-network input at the particular input position to generate a corresponding output for the particular input position. In some cases, the attention mechanism is a multi-headed attention mechanism as shown. In some implementations, each encoder sub-network in the encoder sub-network 214 may also include a residual connection layer that combines the output of the encoder self-attention sub-layer with the input of the encoder self-attention sub-layer to generate an encoder self-attention residual output, and a layer normalization layer that applies layer normalization to the encoder self-attention residual output. Figure 2 In , these two layers are collectively referred to as the "Add & Norm" operation.
[0039] Some or all of the encoder subnetworks may also include a position-by-position feed-forward layer 218 configured to operate separately on each position in the input sequence. Specifically, for each input position, the feed-forward layer 218 is configured to receive an input at the input position and apply a series of transformations to the input at the input position to generate an output for the input position. The input received by the position feed-forward layer 218 may be the output of a layer normalization layer when a residual and layer normalization layer are included, or may be the output of the encoder self-attention sublayer 216 when a residual and layer normalization layer are not included. The transformation applied by the layer 218 is typically the same for each input position (but different feed-forward layers in different subnetworks may apply different transformations).
[0040] Where the encoder subnetwork 214 includes a position-by-position feed-forward layer 218 as shown, the encoder subnetwork may also include a residual connection layer that combines the output of the position-by-position feed-forward layer with the input of the position-by-position feed-forward layer to generate an encoder position-by-position residual output and a layer normalization layer that applies layer normalization to the encoder position-by-position residual output. As noted above, these two layers are also collectively referred to as "add and normalize" operations. The output of the layer normalization layer may then be used as the output of the encoder subnetwork 214.
[0041] Once the encoder neural network 208 generates the encoded representation, the decoder neural network 210 is configured to generate an output sequence in an autoregressive manner. That is, the decoder neural network 210 generates the output sequence by generating, at each of a plurality of generation time steps, a network output for a corresponding output position conditioned on (i) the encoded representation and (ii) the network output at the output position that precedes the output position in the output order. Specifically, for a given output position, the decoder neural network generates an output that defines a probability distribution over possible network outputs at the given output position. The decoder neural network can then select a network output for the output position by sampling from the probability distribution or by selecting the network output with the highest probability.
[0042] Because the decoder neural network 210 is autoregressive, at each generation time step, the decoder network 210 operates on the network output that has been generated before the generation time step (i.e., the network output at the output position that precedes the corresponding output position in the output order). In some implementations, to ensure that this is the case during both inference and training, at each generation time step, the decoder neural network 210 shifts the already generated network output to the right by one output order position (i.e., introduces a position offset into the already generated network output sequence) and (as described in more detail below) masks certain operations so that a position can only focus on positions up to and including that position (rather than subsequent positions). Although the remainder of the following description describes that when a given output is generated at a given output position, the various components of the decoder 210 operate on the data at the output position that precedes the given output position (rather than operating on the data at any other output position), it should be understood that this type of conditioning can be effectively implemented by using shifts.
[0043] The decoder neural network 210 includes an embedding layer (output embedding) 220, a sequence of decoder subnetworks 222, a linear layer 224, and a softmax layer 226. Specifically, the decoder neural network may include N decoder subnetworks 222. However, although Figure 2 The example of shows that the encoder 208 and the decoder 210 include the same number of sub-networks, but in some cases, the encoder 208 and the decoder 210 include different numbers of sub-networks. The embedding layer 220 is configured to map the network output to a digital representation of the network output in the embedding space for each network output at an output position that precedes the current output position in the output order at each generation time step. The embedding layer 220 then provides the digital representation of the network output to the first sub-network 222 in the decoder sub-network sequence.
[0044] In some implementations, the embedding layer 220 is configured to map each network output to an embedding representation of the network output, and to combine the embedding representation of the network output with the position embedding of the output position of the network output in the output order to generate a combined embedding representation of the network output. The combined embedding representation is then used as a digital representation of the network output. The embedding layer 220 generates the combined embedding representation in the same manner as described above with reference to the embedding layer 212.
[0045] Each decoder sub-network 222 is configured to receive a corresponding decoder sub-network input for each of a plurality of output positions preceding the corresponding output position at each generation time step, and to generate a corresponding decoder sub-network output for each of the plurality of output positions preceding the corresponding output position (or equivalently, each network output at positions up to and including the current output position when the output sequence has been shifted to the right). Specifically, each decoder sub-network 222 includes two different attention sub-layers: a decoder self-attention sub-layer 228 and an encoder-decoder attention sub-layer 230. Each decoder self-attention sub-layer 228 is configured to receive an input for each output position preceding the corresponding output position at each generation time step, and for each particular output position, apply an attention mechanism to the input at the output position preceding the corresponding position using one or more queries derived from the input at the particular output position to generate an updated representation of the particular output position. That is, the decoder self-attention sub-layer 228 applies an attention mechanism that is masked so that it does not pay attention to or otherwise process any data that is not at a position preceding the current output position in the output sequence.
[0046] On the other hand, each encoder-decoder attention sublayer 230 is configured to receive input for each output position preceding the corresponding output position at each generation time step, and for each output position, apply an attention mechanism to the encoded representation at the input position using one or more queries derived from the input at the output position to generate an updated representation of the output position. Thus, the encoder-decoder attention sublayer 230 applies attention to the encoded representation, while the decoder self-attention sublayer 228 applies attention to the input at the output position.
[0047] exist Figure 2 In the example of , the decoder self-attention sublayer 228 is shown as being located before the encoder-decoder attention sublayer in the processing order within the decoder subnetwork 222. However, in other examples, the decoder self-attention sublayer 228 can be located after the encoder-decoder attention sublayer 230 in the processing order within the decoder subnetwork 222, or different subnetworks can have different processing orders. In some implementations, each decoder subnetwork 222 includes a residual connection layer that combines the output of the attention sublayer with the input of the attention sublayer to generate a residual output and a layer normalization layer that applies layer normalization to the residual output after the decoder self-attention sublayer 228, after the encoder-decoder attention sublayer 230, or after each of the two sublayers. These two layers are inserted after the two sublayers, both of which are referred to as "add and normalize" operations.
[0048] Part or all of the decoder sub-network 222 also includes a position-by-position feed-forward layer 232 configured to operate in a manner similar to the position-by-position feed-forward layer 218 from the encoder 208. Specifically, the layer 232 is configured to: at each generation time step: for each output position preceding the corresponding output position: receive an input at the output position, and apply a series of transformations to the input at the output position to generate an output at the output position. The input received by the position-by-position feed-forward layer 232 may be the output of the layer normalization layer (after the last attention sub-layer of the sub-network layer 222) when a residual layer and a layer normalization layer are included, or may be the output of the last attention sub-layer of the sub-network layer 222 when a residual layer and a layer normalization layer are not included. In the case where the decoder sub-network 222 includes the position-by-position feed-forward layer 232, the decoder sub-network may also include a residual connection layer that combines the output of the position-by-position feed-forward layer with the input of the position-by-position feed-forward layer to generate a decoder position-by-position residual output, and a layer normalization layer that applies layer normalization to the decoder position-by-position residual output. These two layers are also collectively referred to as an “add and normalize” operation. The output of the normalization layer can then be used as the output of the decoder subnetwork 222.
[0049] At each generation time step, the linear layer 224 applies the learned linear transformation to the output of the last decoder subnetwork 222 to project the output of the last decoder subnetwork 222 into an appropriate space for processing by the softmax layer 226. The softmax layer 226 then applies a softmax function to the output of the linear layer 224 to generate a probability distribution (output probability) 234 of possible network outputs at the generation time step. The decoder 210 can then use the probability distribution to select a network output from the possible network outputs. Overall SCONES Architecture
[0050] For text translation using a set of source text examples, other neural machine translation (NMT) models can learn the probability P(y|x) of a target sentence y given a source sentence x. This framework implies that for a given source sentence, there is a single best translation: if there are multiple valid translations y1 and y2, they will need to share probability mass (e.g., P(y1|x)=0.5 and P(y2|x)=0.5), but this distribution can also represent model uncertainty, i.e., whether y1 or y2 is the correct translation. Therefore, learning a single distribution over all target language sentences does not allow the model to naturally express intrinsic uncertainty, and the essence of the translation task is to allow for multiple semantically equivalent translations of a given source sentence. A single distribution over all sequences represents uncertainty by assigning probabilities, but they cannot distinguish between different kinds of uncertainty (e.g., model uncertainty vs. intrinsic uncertainty).
[0051] In contrast, aspects of the present technology treat machine translation as a multi-label classification task. In this approach, instead of learning a single distribution P(y|x) over all target sentences y for source sentences x, the system learns a binary classifier for each sentence pair (x, y) that indicates whether y is a valid translation of x. In this framework, intrinsic uncertainty can be represented by setting the probability of two (or more) correct translations y1 and y2 to the maximum value (e.g., to "1" on a scale of 0 to 1) at the same time.
[0052] The probability of each translation is calculated using a separate binary classifier, and therefore the probabilities of all translations are not required to sum to 1. In fact, the probability of a complete translation is decomposed into the product of word-level probabilities. In one aspect, the softmax output layer in the Transformer model is replaced with a sigmoid activation (e.g., Figure 2 226 in ), the sigmoid activation assigns a probability between 0 and 1 to each word in the vocabulary at each time step. Alternatively, instead of the sigmoid activation, another bounded activation function (such as tanh) can also be used. The SCONES loss function allows the system to train the model on a single reference training data. Unlike noise contrast estimation (NCE) whose main goal is to efficiently train models on large vocabularies, the use of the SCONES method enables modeling of non-exclusive outputs.
[0053] Testing has shown that training NMT models with SCONES has multiple benefits compared to standard cross entropy with regular softmax. This can include consistent BLEU score gains between 1% and 9% across six different translation directions. SCONES with greedy search generally outperforms softmax with beam search, yielding up to 3.9x inference speedup compared to softmax without any drop in BLEU score.
[0054] In addition, SCONES can also alleviate the drawbacks of traditional NMT models. Softmax-based models have been shown to assign the highest probability to empty or improper translation (pattern). This behavior itself manifests as the "beam search curse", in which increasing the size of the cluster may cause the translation quality to decline, as described in "Six challenges for neural machine translation" (its entire disclosure is incorporated herein by reference) published on pages 28 to 39 of "First Workshop on Neural Machine Translation" published by Association for Computational Linguistics (Association for Computational Linguistics) by Koehn and Knowles in 2017. Tests show that SCONES can be used for training the following model: a) the highest probability is assigned to proper translation; and b) do not suffer from the beam search curse.
[0055] SCONES can also be used to train models on synthetic translation pairs generated by sampling from IBM Model 3 ("IBM-3") (see "The mathematics of statistical machine translation: Parameter estimation" published by Brown et al. in 1993 in Computational Linguistics, the entire disclosure of which is incorporated herein by reference). By changing the sampling temperature (entropy of the distribution), the degree of divergence in the language pair can be controlled. Tests show that SCONES can effectively improve the appropriateness of the highest probability translation of highly divergent translation pairs, thereby confirming that SCONES can be highly effective in handling inherent uncertainty. Using SCONES to train NMT models
[0056] For model training, the (sub-word) vocabulary is represented as , the special end-of-sentence symbol is represented by , the source sentence is represented as , the translation is represented as , and the translation prefix is represented as . Center Point" ” is used for string concatenation. Unlike other NMT methods that can model a single distribution P(y|x) over all target language sentences, SCONES learns a separate binary classifier for each sentence pair (x,y). The Boolean function is defined as t( ) to indicate whether y is a valid translation of x, such that t(x,y) := true if y is a translation of x; false otherwise.
[0057] This method does not directly apply to t( ) modeling. To guide decoding, learn the variables , the variable will be t( ) generalized to translate prefixes: Right now, is a binary label for a pair (x,y) associated with a source sentence x and a translation prefix y: y is a prefix of a valid translation of x if and only if y is a prefix of x. = 1. Its probability is decomposed into a product of conditions to facilitate left-to-right bundle decoding as follows (as a base case, for the empty translation prefix, :
[0058] By using the sigmoid activation function Applied to the log-odds (where the log-odds are the raw prediction vectors generated by the model before normalization) to assign conditional probabilities: here Represents a single word. represents the log-odds of time step i, represents the log probability corresponding to word w.
[0059] The difference to the architecture of the standard NMT model is the output activations. Specifically, instead of a softmax function that produces a single distribution over the entire vocabulary, multiple sigmoid activations are used in each log-odds component to define a separate Bernoulli distribution for each item in the vocabulary. This reference Figure 3 , which illustrates a multi-directional NMT Transformer architecture for non-exclusive target sequences. In this figure, a source sentence x is applied to a Transformer encoder 302, and the output from the Transformer encoder 304 is compared with the translation prefix are applied together to the Transformer decoder 304. The output of the Transformer decoder 304 is applied to the log probability 306. The Transformer decoder output is bounded by the logistic activation function σ, where σ(f()) is a bounded activation function. For each word w (e.g., w1, w2, w3, w4, etc.), the corresponding conditional probability P308 is obtained.
[0060] However, using this multi-label classification view requires a different training loss function because, unlike the log-odds from softmax, the log-odds of the conditional probabilities in the equation above do not provide a normalized distribution over the vocabulary. An additional challenge is that existing MT training datasets typically do not provide more than one reference translation. The SCONES loss function is designed to use a scaling factor To balance the two token-level objectives: ,in
[0061] here, Designed to increase the gold label i The log probability of ), because it is the translation prefix For example, the positive loss component applies a logarithmic function to the bounded conditional probability of each reference token. Figure 3 Box 310 in exemplifies the probability that a bounded activation function is applied to the SCONES loss function 308. Aspects of the loss function include: Designed to reduce the gold label i The probability of all labels w other than :
[0062] For example, the negative loss component applies a logarithmic function to the bounded conditional probability of each non-reference token.
[0063] During inference, to generate a translation, the system searches for a translation y* that ends with y and is most likely the translation of x: , which is equal to:
[0064] This decision rule can be approximated with a vanilla beam search, which is the standard beam search algorithm for sequence-to-sequence models. This beam search takes the best N tokens and selects the best N partial sequences (at each time step), considering the probabilities of the combination of all preceding words in addition to the word at the current position. Examples are described in Sequence to Sequence Learning with Neural Networks, Sutskever et al., published on December 14, 2014, the entire disclosure of which is incorporated herein by reference.
[0065] For evaluation purposes, the softmax baseline and the SCONES trained model use the same inference code. Here, the only difference will be that the log odds from the SCONES model are transformed by sigmoid instead of Softmax activation, in other words, the entire vocabulary (the set of all tokens) does not need to be summed. Relation to noise contrastive estimation
[0066] The SCONES loss function is related to noise contrast estimation (NCE) because both methods reformulate next word prediction as a multi-label classification problem, and both losses have a "positive" component for the gold label and a "negative" component for the other labels. However, unlike NCE, the negative loss component in SCONES ) does not need to sample from a noise distribution because it utilizes all tokens in the vocabulary except the golden token. This is possible because the SCONES method is able to operate on a limited (e.g. 32K) sub-word vocabulary, while NCE is typically used to train language models with larger word-level vocabularies. In addition, NCE has a "self-normalization" property that can reduce computation by avoiding expensive partition functions for the distribution over the entire vocabulary. To this end, NCE uses the multi-label classification task as a proxy problem. In contrast, in SCONES, the multi-label classification perspective is used to express the inherent uncertainty in MT, rather than just a proxy for the entire softmax. Therefore, the main motivation of SCONES is not to self-normalize the entire vocabulary. Details about NCE can be found in “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models” by Gutmann and Hyvärinen, Proceedings of the thirteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, pp. 297-304, the entire disclosure of which is incorporated herein by reference. Experimental setup
[0067] To understand some of the benefits and advantages of the SCONES approach, it is helpful to compare an NMT model trained with SCONES to a well-trained standard softmax-based model. To do this, the setup should be kept simple, reproducible, and computationally economical. In one case, a Transformer model is based on Figure 4A The Transformer model was trained on the WMT19 training set provided by TensorFlow Datasets for six translation directions, namely German-English (de-en), Finnish-English (en-fi), Lithuanian-English 220 (lt-en), and the reverse direction. These language pairs were chosen to experiment with different training set sizes, such as Figure 4B As indicated in Table 2.
[0068] The training set was filtered using language ID and a simple length-based heuristic and split into subwords using the joint 32KSentencePiece model as described in “SentencePiece: A simple and languageindependent subword tokenizer and detokenizer for neural text processing” by Kudo and Richardson, Proceedings of the 2018 Conference on Empirical Methods in Natural LanguageProcessing: System Demonstrations, pages 66–71, Brussels, Belgium, the entire disclosure of which is incorporated herein by reference.
[0069] In this experimental setup, all models were trained using the LAMB optimizer in JAX until convergence on the development set (training steps ranged from 100K to 700K). Details on the LAMB optimizer can be found in You et al., “Large batch optimization for deep learning: Training BERT in 76 minutes,” International Conference on Learning Representations, 2020, the entire disclosure of which is incorporated herein by reference. Details on JAX can be found in Bradbury et al., “JAX: Composable transformations of Python+NumPy programs,” 2018, the entire disclosure of which is incorporated herein by reference.
[0070] The softmax baseline is trained by minimizing the cross entropy without label smoothing. The multi-directional NMT model is trained by minimizing the above SCONES loss function, also without label smoothing. The model is evaluated on the WMT19 test set using SacreBLEU, and α is adjusted using the WMT18 test set as the development set. As an additional optimization, the greedy search implementation is configured to operate directly on the log-odds without applying output activation. Details about SacreBLEU can be found in Matt Post, "A call for clarity in reporting BLEU scores" in "Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191", 2018, which is incorporated by reference in its entirety. Experimental Results
[0071] FIG. 5A to FIG. 5B Tables 500 and 520 in Figure 5 compare the SCONES-based NMT system with a softmax baseline, where α is based on Figure 6 conditioned on the BLEU scores on the development set shown in the table. As shown, SCONES produces consistent improvements across the board. For four of the six language pairs (all except en-de and fi-en), SCONES with greedy search is even able to outperform the softmax model with beam search. The low-resource language pairs (fi↔en, lt↔en) benefit much more from SCONES training than the rich-resource pair (de↔en).
[0072] Softmax-based models achieve (close to) optimal BLEU scores at a beam size of about 4. Most SCONES models can achieve similar or better BLEU scores with greedy search. Replacing beam-4 search with greedy search corresponds to a 3.9x speedup (2.76 → 10.64 sentences per second) on an entry-level NVIDIA Quadro P1000 GPU with a batch size of 4.8.
[0073] 7A to 7FBLEU scores as a function of decoding speed are shown for the six translation directions. Here, the SCONES results are all higher than the softmax results. Most of the speedup is due to the choice of a smaller cluster size, not to SCONES avoiding normalization of the entire vocabulary. When compared to models with larger vocabularies, it is expected that SCONES will be faster still further. Mitigating the beam search curse
[0074] As pointed out above, one of the most annoying drawbacks of traditional softmax-based NMT models is the “beam search curse”, where larger beam sizes improve the log probability of translations, but the translation quality becomes worse. This happens because the model prefers too short translations with larger beam sizes. This phenomenon is related to local normalization and poor model calibration in sequence models. It turns out that the patterns are often empty and indicate that the model’s inherent bias towards short translations is often masked by beam search errors. This lack of length may be due to the inherent uncertainty of the MT task.
[0075] Given that inherent uncertainty is explicitly accounted for using models trained with SCONES, an experiment was conducted to determine whether these models are more robust to the beam search curse than models trained with softmax. Figure 8 The BLEU scores are plotted as a function of the bundle size. The sharp drop in the softmax curve 802 for large bundle sizes reflects the beam search curse of the softmax baseline. The results show that SCONES is less affected when the bundle size is large, especially for small values of α: the BLEU score of SCONES (line 804) with α = 0.2 is stable for bundle sizes greater than 100.
[0076] The length ratio (hypothetical length divided by reference length) versus cluster size for German-English estimates is shown. Fig.9AThis suggests that the difference in BLEU trajectories is due to the length of the translations. The translations obtained using softmax (902) suddenly become shorter at larger cluster sizes, while for SCONES (904) with α = 0.2, there is no such a sharp decrease in length. To study the impact of α in the absence of beam search errors, the exact depth-first search algorithm of Stahlberg and Byrne (“On NMT search errors and model errors: Cat got your tongue?”, 2019, in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3356–636 3362, the entire disclosure of which is incorporated herein by reference) was run to find the translation with the global 309 highest probability. The appropriateness of the translation found by the exact search depends heavily on α, which is Fig. 9B Shown in a plot of German-English BLEU scores (softmax: 912; SCONES: 914) and length ratio (softmax 916; SCONES 918).
[0077] Small values of α produce good translations through precise searching, but α 1:0 performs similarly to the softmax baseline: the BLEU score drops because the hypothesis is too short. FIG. 10A to FIG. 10B Tables 1000 and 1020 in show that SCONES with α = 0.2 consistently outperforms the softmax baseline by a large margin using exact search. The mean and standard deviation (error bars) of the log-probability of the global highest probability translation (found using exact search) and the hole translation for German-English are shown. Fig. 10C It explains why SCONES with small α does not prefer hole translation. Here, 1002 shows softmax hole translation, 1004 shows softmax exact search translation, 1006 shows SCONES hole translation, and 1008 shows SCONES exact search translation.
[0078] The smaller α is, the larger the difference between the log probability of the exact search translation and the log probability of the hole translation is. This is due to the fact that the higher the log probability of the exact search translation, the smaller the variance. Intuitively, a small α reduces the above SCONES The negative loss component in the equation (x,y,i) ) and thus bias each binary classifier towards predicting the true label.
[0079] Fig.11 The percentage of beam search errors (for German-English as an example) is shown as a function of beam size, specifically the proportion of sentences for which beam search did not find the global best translation. Here, 1102 is a softmax graph, and 1104 is a SCONES graph for α=0.2. These test results confirm Stahlberg and Byrne's findings for the softmax model, specifically that the percentage of search errors remains relatively high at around 20% even for very large beam sizes. For SCONES with small values of α, increasing the beam size is most effective in reducing the number of search errors. However, a small α does not always produce the best overall BLEU score (see Figure 8 ). Taken together, these observations provide insights into model error in NMT: if one describes “model error” as the mismatch between the global most likely translation and a plausible translation, then a small α will simultaneously produce fewer search errors (see Fig.11 ) and fewer model errors (see FIG. 10A to FIG. 10B However, counterintuitively, the BLEU score peaks at slightly higher values of α (see Figure 6 Table 4 in ). Experiments using synthetic language pairs
[0080] The key motivation of SCONES is to enable the model to naturally represent inherent uncertainty, that is, there are multiple correct target sentences in the same source sentence. In order to verify the characteristic that SCONES changes along with uncertainty, the different synthetic language pairs of divergence degree are generated. For this reason, after using MGIZA to carry out subword segmentation, IBM-3 model is trained on German-English training data. MGIZA is described in Gao and Vogel in "Parallel implementations of word alignment tool (parallel implementation mode of word alignment tool)" on page 49 to 57 of "Software Engineering, Testing, and Quality Assurance for Natural Language Processing (software engineering, testing and quality assurance of natural language processing)" in 2008, and its full disclosure is incorporated into this paper by reference. IBM-3 is a generative symbolic model, which describes the translation process from one language to another language with a generative story, and has been used for finding word alignment for (based on phrase) statistical machine translation.
[0081] The generative story includes descriptions of the different steps of the translation process, such as warping (word reordering), multiplication (1:n word mapping), and lexical translation (word-to-word translation). The parameters of IBM-3 define the probability distribution of each step. In the experiments, IBM-3 was not used to find word alignments. Instead, for the original German sentences, synthetic English-like translations were sampled from the model at different sampling temperatures to control the degree of divergence in the translation task. Low sampling temperatures produce sentence pairs that are still able to capture some characteristics of MT (such as word reordering), but the mapping is mostly deterministic (i.e., the same source word-membrane is almost always translated into the same target word-membrane). High temperatures correspond to more stochasticity, that is, more intrinsic uncertainty. NMT models can be trained on synthetic corpora using softmax or SCONES.
[0082] FIG. 12A to FIG. 12B The table in contains more details about sampling from IBM-3. The parameters of the IBM-3 model consist of a set of reproduction probabilities n(·|∙), p0, p1, a set of translation probabilities t(·|∙), and a set of distortion probabilities d(·|∙). According to the IBM model 3, the following generative process produces a target language sentence y from a source language sentence x: 1. For each source word x indexed by i=1,2, ...,|x| i , using the probability Select the reproduction degree i . 2. Using the probability p1 and the sum of the multiplications from step 1, select the number of “fake” target words to generate from x0=NULL 0. 3. Let m = . 4. For each i=0,1,2, ..., |x| and each k=1,2,..., i , using the probability t ( ik |x i )Select a target word ik . 5. For each i=1,2, ..., |x| and each k=1,2, ..., i , with probability d ( ik |i, |x|, m) select the target position ik . 6. For each k=1,2, ..., 0, the remainder from 1,2, ...,m 0-k + 1 vacant positions to choose from 0k , the total probability is . 7. Output the target sentence, where the word ik Located at ik (0 i |x|, 1 k i )middle.
[0083] We can first use the MGIZA word alignment tool to estimate the IBM-3 model parameters. Then, following the generative story above, we sample English-like target sentences for the German source sentences. To control the degree of uncertainty in the synthetic translation task, the entropy of n(·|∙), t(·|∙), and d(·|∙) is adjusted by choosing different sampling temperatures. The distribution of . Different from direct sampling
[0084] From the classification distribution P(∙) of category C, the temperature sampled for each c C uses the following distribution:
[0085] Low temperatures amplify large differences in probability and thus produce lower entropy and less divergence.
[0086] Fig.13A A plot of BLEU scores for German to synthetic English translation using beam search (beam size 4) at different IBM-3 sampling temperatures is presented. The plot shows that softmax and SCONES perform similarly when using beam search. High IBM-3 sampling temperatures make the translation task less predictable and therefore produce lower BLEU scores. Fig. 13B As shown in Figure 1, the difference between the two methods becomes clear with the exact search. While the translation with the highest global probability is severely degraded at high IBM-3 sampling temperature for softmax (1300) and SCONES with α=1 (1302), the degradation is much smaller for SCONES with α=0.5 (1304) or α=0.2 (1306). Setting α to a low value enables the model to assign its highest probability to a good translation, even if the translation task is highly uncertain. This scaling factor is adjustable, for example to maximize translation performance
[0087] SCONES has one more parameter (α) than softmax to weigh positive and negative examples. This enables SCONES to achieve enhanced performance, as shown in the test results above. Fig.14 A sample implementation of SCONES written in JAX is provided. Example Compute Architecture
[0088] The models discussed herein can be trained on one or more tensor processing units (TPUs), CPUs, or other computing architectures to model divergence using SCONES based on the features disclosed herein. Fig.15A and Fig. 15B An example computing architecture is shown in FIG. Fig.15A and Fig. 15B 1502 can be implemented as a cloud-based server system. Databases 1504, 1506 and 1508 can store, for example, original source text (e.g., one or more groups of example source text sentences or other text groups), translated output text (e.g., one or more groups of target sentences or other target text groups) and / or machine translation models, respectively. The server system can access the database via network 1510. Client devices can include desktop computers 1512 and laptop or tablet PC 1514, for example, to provide original text or other content, and / or view output (e.g., the use of evaluation or other information in selected translation or translation services, applications or other programs).
[0089] like Fig. 15B As shown, each of the computing devices 1502 and 1512-1514 can include one or more processors, memory, data and instructions. The memory stores information that can be accessed by one or more processors, including instructions and data (e.g., models) that can be executed or otherwise used by the processor. The memory can be any type capable of storing information that can be accessed by the processor, including computing device readable media. The memory is a non-temporary medium such as a hard disk, a memory card, an optical disk, a solid state, etc. The system can include different combinations of the foregoing, thereby storing different parts of the instructions and data on different types of media. The instruction can be any instruction set (such as machine code) that will be directly executed by the processor or any instruction set (such as a script) that is indirectly executed. For example, the instruction can be stored as a computing device code on a computing device readable medium. In this regard, the terms "instruction", "module" and "program" can be used interchangeably herein. The instruction can be stored in an object code format for direct processing by the processor, or stored in any other computing device language, which includes a script or independent source code module set that is interpreted or compiled in advance on demand.
[0090] The processor may be any conventional processor, such as a commercially available CPU, TPU, graphics processing unit (GPU), etc. Alternatively, each processor may be a dedicated device, such as an ASIC or other hardware-based processor. Fig. 15B Processors, memory, and servers and other elements are functionally illustrated as being located within the same box, but such devices may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, memory may include a hard drive or other storage medium in a cloud computing system that is located in a different housing than the processor, such as server 1502. Therefore, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.
[0091] Input data (such as one or more sentences or array text content) can be operated by the model trained by SCONES to generate one or more translations, text quality assessment data, etc. Client devices can use such information in various applications or other programs to perform translation, dialogue modeling, spelling or grammatical error correction, quality assessment or other indicator analysis, recommendation, classification, search, etc. This may include assigning quality scores to different translations based on the results of the processing. Therefore, quality scores can be used to sort translations and give priority to good translations when serving (such as video streaming services with subtitles).
[0092] The computing device may include all components commonly used in connection with a computing device, such as the processor and memory described above, and a user interface subsystem for receiving input from a user and presenting information (e.g., text, images, and / or other graphical elements) to the user. The user interface subsystem may include one or more user input components (e.g., at least one front (user) camera, a mouse, a keyboard, a touch screen, and / or a microphone) and one or more display devices (e.g., a screen or monitor of any other electrical device with actionable information (e.g., text, images, and / or other graphical elements). Other output devices, such as speakers, may also provide information to the user.
[0093] The user-related computing devices (e.g., 1512-1514) can communicate with the backend computing system (e.g., server 1502) via one or more networks (such as network 1510). The network 910 and the intermediate nodes may include various configurations and protocols, including short-range communication protocols such as Bluetooth™, Bluetooth LE™, the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi, HTTP, etc., and various combinations of the foregoing. Such communications can be facilitated by any device capable of sending data to and from other computing devices, such as a modem and a wireless interface.
[0094] In one example, computing device 1502 may include one or more server computing devices having multiple computing devices, such as a load balancing server farm or a cloud computing system, which exchange information with different nodes of a network to receive, process, and send data to and from other computing devices. For example, computing device 1502 may include one or more server computing devices capable of communicating with any of computing devices 1512-1514 via network 1510.
[0095] Model information or other data derived from the SCONES method may be shared by the server and one or more of the client computing devices. Alternatively or additionally, the client devices may maintain their own databases, models, etc.
[0096] Fig.16 A method 1600 for resolving disagreements in neural machine translation according to aspects of the present technology is illustrated. At block 1602, a set of text examples are stored in a memory. At block 1604, an encoder module including an encoder neural network receives a given text example (x = <x1, ..., x |x|>). At block 1606, the encoder module generates an encoded representation of a given text example. At block 1608, the decoder module receives the encoded representation and a set of translation prefixes (y <i = <y 1, ..., y i-1 >). At block 1610, the decoder module outputs the unbounded function f(x, y <i ), the unbounded function corresponds to a set of tokens that correspond to a given text example x and the set of translation prefixes y <i , wherein each word-gram is assigned a probability between 0 and 1 in the vocabulary of a given text example at each time step. At box 1612, a log-odds module generates a corresponding bounded conditional probability for each word-gram based on an unbounded function, wherein the conditional probability is not normalized over the vocabulary at each time step. And at box 1614, a loss function module having a positive loss component and a scaled negative loss component identifies whether each target text in a set of target texts is a valid translation of a given text example.
[0097] Machine translation is a task with a high degree of intrinsic uncertainty: a source sentence can have multiple valid translations. It has been shown that NMT models (and specifically Transformer) can learn to model non-exclusive target sentences from single-label training data using the SCONES loss function. Instead of learning a single distribution over all target sentences, SCONES learns multiple binary classifiers that indicate whether a target sentence is a valid translation of a source sentence. SCONES produces improved translation quality over conventional softmax-based models for six different translation directions, or (alternatively), produces up to 3.9x speedup without any degradation in translation performance.
[0098] It has been shown that SCONES can be tuned to mitigate the beam search curse as well as the bad and empty pattern problems in standard NMT. Experiments on synthetic language translation show that, unlike models trained with softmax, the SCONES model is able to assign its highest probability to the correct translation even when the underlying task is highly diverse.
[0099] The SCONES loss function is easy to implement. Adapting a standard softmax-based sequence-to-sequence architecture (such as Transformers) only requires replacing the cross-entropy loss function with SCONES, and replacing the softmax with a sigmoid activation. The rest of the training and inference pipelines can remain unchanged. SCONES can be useful in dealing with uncertainty in a variety of divergent NLP problems beyond translation (such as generation and dialogue).
[0100] Although the technology has been described herein with reference to specific embodiments, it should be understood that these embodiments are only illustrative of the principles and applications of the technology. Therefore, it should be understood that various modifications may be made to the illustrative embodiments, and other arrangements may be designed without departing from the spirit and scope of the technology as defined by the appended claims.
Claims
1. A system configured for a machine translation model, the system comprising: a memory configured to store a set of text examples in a source language and a set of rewritten texts in one or more languages different from the source language; as well as one or more processing elements operatively coupled to the memory, the one or more processing elements implementing the machine translation model as a neural network having: The encoder module includes an encoder neural network configured to receive a given text example (x = <x1, ..., x |x| >) and generating an encoded representation of the given text example; A decoder module, the decoder module comprising a decoder neural network, the decoder neural network configured to receive the encoded representation and a set of translation prefixes (y <i = <y1, ..., y i-1 >) and outputs the unbounded function f(x, y <i ), the unbounded function corresponds to a set of tokens, the set of tokens corresponding to the given text example x and the set of translation prefixes y <i , wherein each word-gram is assigned a probability between 0 and 1 in the vocabulary of the given text example at each time step; a log-odds module configured to act on the unbounded function to generate a corresponding bounded conditional probability for each word-gram, wherein the conditional probability is not normalized over the vocabulary at each time step; as well as A loss function module having a positive loss component and a scaled negative loss component, wherein the loss function module is configured to identify whether each target text in a set of target texts is a valid translation of the given text example.
2. The system of claim 1, wherein the set of text examples comprises a set of input sentences, and the system is configured to learn, for each sentence pair (x, y), a binary classifier indicating whether y is a valid translation of x.
3. The system of claim 2, wherein intrinsic uncertainty is represented by setting the conditional probabilities of at least two correct translations y1 and y2 to maximum probabilities simultaneously.
4. The system of claim 2, wherein the log-odds module is configured to generate the probability of each translation using a separate binary classifier.
5. The system of claim 1, wherein the probability of a complete translation of the given text example is decomposed into a product of word-level probabilities.
6. The system of claim 1, wherein the log-odds module is configured to apply a sigmoid activation to the unbounded function at each time step.
7. The system of claim 1 , wherein the positive loss component applies a logarithmic function to the bounded conditional probability of each reference word-gram, and the scaled negative loss component applies a logarithmic function to the bounded conditional probability of each non-reference word-gram.
8. The system of claim 1, wherein during inference, the system is configured to search for a translation having a highest probability of being the translation of the text segment to be translated.
9. The system of claim 1, wherein the system is configured to express inherent uncertainty associated with a machine translation model.
10. The system of claim 1, wherein the loss function module is configured to adjust the scaling of the negative loss component to maximize translation performance.
11. The system of claim 1 , wherein the encoder module and the decoder module comprise a self-attention neural network encoder-decoder architecture.
12. The system of claim 1, wherein the encoder module and the decoder module comprise a sequence-to-sequence model architecture.
13. A machine translation method using a neural network, the method comprising: storing a set of text examples in a memory; The encoder module, which includes an encoder neural network, receives a given text example (x = <x1,..., x |x| >); generating, by the encoder module, an encoded representation of the given text example; The decoder module receives the encoded representation and a set of translation prefixes (y <i = <y1, ..., y i-1 >); The decoder module outputs an unbounded function f(x, y <i ), the unbounded function corresponds to a set of tokens, the set of tokens corresponding to the given text example x and the set of translation prefixes y <i , wherein each word-gram is assigned a probability between 0 and 1 in the vocabulary of the given text example at each time step; Generating, by a log-odds module, a corresponding bounded conditional probability for each word-gram based on the unbounded function, wherein the conditional probability is not normalized over the vocabulary at each time step; as well as A loss function module having a positive loss component and a scaled negative loss component is used to identify whether each target text in a set of target texts is a valid translation of the given text example.
14. The method of claim 13, wherein the set of text examples comprises a set of input sentences, and the method comprises learning, for each sentence pair (x, y), a binary classifier indicating whether y is a valid translation of x.
15. The method of claim 14, wherein intrinsic uncertainty is represented by simultaneously setting the conditional probabilities of at least two correct translations y1 and y2 to maximum probabilities.
16. The method of claim 14, wherein generating a corresponding bounded conditional probability for each word-gram comprises using a separate binary classifier to generate a probability for each translation.
17. The method of claim 13, wherein the probability of a complete translation of the given text example is decomposed into a product of word-level probabilities.
18. The method of claim 13, wherein generating a corresponding bounded conditional probability for each word-gram comprises applying a sigmoid activation to the unbounded function at each time step.
19. The method of claim 13, wherein the positive loss component applies a logarithmic function to the bounded conditional probability of each word-unit, and the scaled negative loss component applies a logarithmic function to the bounded conditional probability of each word-unit.
20. The method of claim 13, further comprising: During inference, the translation with the highest probability of being the translation of the text segment to be translated is searched.
21. The method of claim 13, further comprising: The scaling of the negative loss component is adjusted to maximize translation performance.
