System and method for bidirectional translation using sum-product networks

Abstract

The present invention describes a method and machine translation system for bidirectional translation of a text sequence between a first language and a second language. The machine translation system includes: a first autoencoder for receiving a vector representation of a first text sequence in the first language and encoding the vector representation of the first text sequence into a first sentence embedding; a sum-product network (SPN) for receiving the first sentence embedding and generating a second sentence embedding by maximizing a first conditional probability of the second sentence embedding given the first sentence embedding; and a second autoencoder for receiving the second sentence embedding, wherein the second autoencoder is trained to decode the second sentence embedding into a vector representation of a second text sequence in the second language.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of priority to U.S. patent application Ser. No. 16 / 900,481, filed on June 20, 2020, entitled “SYSTEM AND METHOD FOR BI-DIRECTIONAL TRANSLATION USING A SUM-PRODUCT NETWORK,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The embodiments described herein relate generally to machine translation of text sequences and, more particularly, to systems and methods for bidirectional translation of text sequences between at least two languages. Background Art

[0004] The processing of text sequences in human languages ​​(e.g., English, French, German, etc.) by computing devices is generally referred to as Natural Language Processing (NLP). In this context, a text sequence is typically a series of text or words that form a sentence or phrase, with a specific semantic meaning or narrative concept. One aspect of NLP is machine translation (MT), which uses a specially configured computer system (also called an MT system) that is configured to: receive a text sequence in a first (source) human language, process the source language text sequence, and predict a text sequence in a second (target) human language, where the text sequence in the second human language expresses the same or similar narrative concept as the text sequence in the first human language.

[0005] MT systems can implement a language model that uses a statistical probability distribution over text sequences to translate a text sequence in a source human language into a text sequence in a target human language. MT systems can implement Neural Machine Translation (NMT), which includes an artificial neural network trained to learn a single language model (e.g., a sequence-to-sequence model) that predicts a text sequence in a target language given a text sequence in a source language. The artificial neural network can be configured in an encoder-decoder architecture.

[0006] However, while the process of translating a text sequence from one language to another may be simple for skilled human translators, it presents a variety of technical challenges for traditional MT systems. These challenges become even more complex when the translation task requires bidirectional translation, where a text sequence in the source language is first translated into a text sequence in the target language, and then another text sequence is translated from the target language back to the source language. Summary of the Invention

[0007] The present invention relates to a machine translation system for bidirectional translation of text sequences between at least two languages.

[0008] In a broad aspect, described herein are machine translation systems and methods for bidirectional translation of text sequences (typically sentences) in a source language and a target language.

[0009] In some examples, the present invention describes a machine translation system for bidirectional translation of a text sequence between a first language and a second language. The system includes: a first autoencoder for receiving a vector representation of a first text sequence in the first language and encoding the vector representation of the first text sequence into a first sentence embedding; a sum-product network (SPN) for receiving the first sentence embedding and generating a second sentence embedding by maximizing a first conditional probability of the second sentence embedding given the first sentence embedding; and a second autoencoder for receiving the second sentence embedding and decoding the second sentence embedding into a vector representation of a second text sequence in the second language.

[0010] In some examples, the present invention describes a method for bidirectional translation of a text sequence between a first language and a second language. The method includes: a first autoencoder encoding a vector representation of a first text sequence in the first language into a first sentence embedding; a sum-product network (SPN) generating a second sentence embedding by maximizing a first conditional probability of the second sentence embedding given the first sentence embedding; and a second autoencoder decoding the second sentence embedding into a vector representation of a second text sequence in the second language.

[0011] In some examples, the present invention describes a computer-readable medium having machine-executable instructions stored thereon. When the instructions are executed by a processing device in a system, the instructions cause the system to perform the following operations: a first autoencoder encodes a vector representation of a first text sequence in a first language into a first sentence embedding; a sum-product network (SPN) generates a second sentence embedding by maximizing a first conditional probability of the second sentence embedding given the first sentence embedding; and a second autoencoder decodes the second sentence embedding into a vector representation of a second text sequence in the second language.

[0012] In any of the above examples, the second autoencoder is further used to encode the vector representation of the second text sequence in the second language into the second sentence embedding; the SPN is further used to generate the first sentence embedding by maximizing the second conditional probability of the first sentence embedding given the second sentence embedding; the first autoencoder is further used to decode the first sentence embedding into the vector representation of the first text sequence in the first language.

[0013] In any of the above examples, the first autoencoder includes a first encoder and a first decoder, and the second autoencoder includes a second encoder and a second decoder. The first encoder, the first decoder, the second encoder, and the second decoder can each independently include a recurrent neural network (RNN), a convolutional neural network (CNN), a deconvolutional neural network (D-CNN), or a transformer neural network.

[0014] In any of the above examples, the SPN may include multiple layers, and each layer may independently include multiple sum nodes or multiple product nodes.

[0015] In any of the above examples, the first autoencoder and the second autoencoder can be trained together in a first training stage to perform the following operations: on the first autoencoder and the second autoencoder side, the vector representations of the first training text sequence and the second training text sequence in the first language and the second language are encoded into corresponding first training sentence embeddings and second training sentence embeddings, respectively; on the first autoencoder and the second autoencoder side, the first training sentence embeddings and the second training sentence embeddings are decoded into corresponding vector representations of the first reconstructed text sequence and the second reconstructed text sequence in the first language and the second language, respectively; and the parameters of the first autoencoder and the second autoencoder are adjusted to minimize a loss function representing the reconstruction loss between the first reconstructed text sequence and the second reconstructed text sequence and the corresponding first training text sequence and the second training text sequence.

[0016] In any of the above examples, the SPN can be trained in a second training phase to perform the following operations: adjust the weights of the SPN to maximize the joint probability of the first training sentence embedding and the second training sentence embedding; and use the joint probability to calculate the maximum conditional probability.

[0017] In various examples, the present disclosure describes a sum-product network (SPN)-based system for bidirectional machine translation. In the disclosed examples, a single system is trained to perform translation from a first language to a second language and vice versa.

[0018] The disclosed example system includes two autoencoders, each trained on one of two languages. Both autoencoders support a mapping to a latent code space representing sentence embeddings for each respective language. An SPN can then be trained to compute a joint probability distribution between the two languages ​​in the latent code space. This joint probability distribution can be used to compute the conditional probability of achieving bidirectional translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Reference will now be made, by way of example, to the accompanying drawings which show exemplary embodiments of the present application, in which:

[0020] Figure 1 A high-level block diagram of the well-known architecture of a Sum-Product Network (SPN) for language modeling.

[0021] Figure 2A is a high-level functional block diagram of an exemplary SPN-based bidirectional translation system according to a non-limiting embodiment of the present invention;

[0022] Figure 2B According to the non-limiting embodiment provided by the present invention Figure 2A Detailed functional block diagram of the SPN-based bidirectional translation system in [1];

[0023] Figure 2C is another exemplary high-level functional block diagram of an exemplary SPN-based bidirectional translation system according to a non-limiting embodiment of the present invention;

[0024] Figure 3A is a non-limiting embodiment provided by the present invention for training Figure 2A Flowchart of an exemplary process of an automatic encoder module of an SPN-based bidirectional translation system;

[0025] Figure 3B is a non-limiting embodiment provided by the present invention for training Figure 2A A flowchart of an exemplary process of an SPN module of an SPN-based bidirectional translation system;

[0026] Figure 4 According to the non-limiting embodiment provided by the present invention, it can be used Figure 2A A high-level flowchart of an exemplary SPN-based bidirectional translation process performed by the SPN-based bidirectional translation system in FIG.

[0027] Figure 5 It can be used to implement Figure 2A A simplified block diagram of an exemplary computing system for an SPN-based bidirectional translation system in FIG.

[0028] Like reference numerals may be used in different drawings to identify like components. DETAILED DESCRIPTION

[0029] As mentioned above, various machine translation (MT) systems use neural machine translation (NMT) technology, which uses sequence-to-sequence learning to learn a model to predict a target language text sequence given a known source language text sequence. The above model is learned by training an artificial neural network arranged in an encoder-decoder architecture. The artificial neural network arranged in the encoder-decoder architecture can include a recurrent neural network (RNN), which acts as an "encoder" and a "decoder" respectively. For example, a text sequence in a first language (e.g., English) is mapped to a text sequence in a second language (e.g., Spanish) by an artificial neural network arranged in the encoder-decoder architecture.

[0030] Using this encoder-decoder architecture, the encoder can be implemented as an RNN that is trained to learn to map a source text sequence in a first language to a representation in a latent code space that captures the meaning of the source text sequence. Correspondingly, the decoder can also be implemented as an RNN that is trained to learn to map the representation provided by the latent code space to a target text sequence in a second language. Although the present invention describes that both the encoder and the decoder are implemented as trained RNNs, it should be understood that the encoder and the decoder can be implemented by other types of neural networks. For example, the encoder can be implemented as a convolutional neural network (CNN), and the decoder can be implemented as a deconvolutional neural network (D-CNN), or both the encoder and the decoder can be implemented as a Transformer neural network, and so on. In this way, the MT system can be trained to learn to effectively translate phrases and sentences from the first language to the second language based on the representation learned in the latent code space.

[0031] However, it should be understood that the training process of traditional RNN-based MT systems (i.e., MT systems with an encoder-decoder architecture in which the encoder and decoder are RNNs) has proven to be complex and / or inefficient (e.g., requiring frequent retuning of RNN hyperparameters, requiring a large amount of training data (e.g., labeled data including paired source text and corresponding translated text), consuming processing resources and time, etc.). Furthermore, RNN-based MT system processes are not amenable to parallelization techniques.

[0032] In addition, most RNN-based MT systems are trained to learn to translate text sequences from a first language to a second language, but not vice versa. In other words, typically, RNN-based MT systems provide translation in one direction, from a first language to a second language, but do not provide translation in the opposite direction (i.e., from a second language to a first language). Therefore, when translation of text content / sentences in the opposite direction (i.e., from a second language to a first language) is required, it may be necessary to independently train and configure a separate RNN to perform translation of text content / sentences in the opposite direction.

[0033] Machine Translation with Shared Latent Code Space

[0034] Recently, a model has been proposed that is feasible and has achieved a certain degree of success in performing bidirectional translation. (For example, see "Unsupervised Machine Translation Using Monolingual Corpora Only" published by Lample et al. in 2017 (hereinafter referred to as "Lample"), the entire content of which is incorporated by reference into this application). Lample discloses a model that extracts monolingual corpus properties from monolingual sentences in two different languages ​​and uses these properties to find and establish a common or "shared" latent code space. The shared latent code space includes the vector codes to which the denoised vector input sentence representations that the encoder has produced are mapped. With the help of the shared latent code space, the Lample model can be trained to reconstruct sentences in a given language based on noisy versions of these sentences, as well as to reconstruct the same sentence in another language given a noisy translation of the source sentence in one language, and vice versa.

[0035] However, it is understandable that in many cases, finding and establishing a shared latent code space between monolingual sentences in two languages ​​may not be achievable. This may be especially true where the two languages ​​are not from the same root or do not share the same alphabetic structure. It should also be understood that bidirectional MT operations facilitated by a shared latent code space (as described by Lample) may not fully reflect or even come close to achieving the capabilities of a skilled bilingual. For example, a skilled bilingual operates and translates as if they were a native speaker of both languages, being able not only to translate sentences bidirectionally but also to accurately and efficiently represent abstract nuances and contextual concepts in both languages.

[0036] Sum-Product Network (SPN) language model

[0037] Sum-Product Networks (SPNs) are a class of deep probabilistic models that have proven to provide a suitable framework for language modeling in speech recognition applications. SPNs consist of distributed leaves (which serve as network inputs) interwoven with layers of sum nodes and product nodes (which serve as internal nodes, or inner nodes). The sum nodes act as hidden variables, and the product nodes are used to capture interactions between the SPN inputs. SPNs are capable of modeling complex dependencies between words and use built-in structured learning algorithms to keep inference and training times relatively efficient and manageable.

[0038] In an existing SPN language modeling method, the above model is used to predict the first N words p(w i-n ,...,w i-1 ) to find the conditional probability of the next word in the given sentence, so as to facilitate the identification of complex correlations between words. (For example, see "Language Modeling with Sum-Product Networks" published by Wei-Chen Cheng et al. in 2014 (hereinafter referred to as "Cheng"), the entire contents of which are incorporated by reference into this application). Figure 1 An exemplary representation of the language modeling architecture 100 proposed by Cheng is shown.

[0039] As shown in the figure, the SPN language modeling architecture 100 proposed by Cheng includes a representative structure consisting of multiple layers 101 to 105 containing sum nodes (represented by the "+" symbol) and product nodes (represented by the "×" symbol). Among them, layers 101 to 105 are interconnected by edges, as shown in the figure. The sum node / product node layers 101 to 105 are all used to process the digital vector representations input from the lower layers and forward the resulting content to the subsequent higher layers. The leaves of the SPN language modeling architecture 100 are distribution functions, and each leaf is used to provide the unnormalized distribution of random variables according to a probability distribution (such as a Gaussian distribution or any suitable distribution capable of capturing and processing real vector values).

[0040] For example, as Figure 1 shown, the SPN language modeling architecture 100 receives the one-hot vector representation (V i-N ,..., V i-1 ) of the first N words through the first layer 101. The one-hot vector representation is a vector that has a value of "1" at the index corresponding to the word it represents and a value of "0" at all other indexes. The input of the first layer 101 is a fully connected convolutional layer, which is used to map the K-dimensional one-hot representation of N words in a sentence of the first language to a low-dimensional hidden space H il：iD of size D, where D << K and K represents the number of words in the dictionary. In this scheme, the output of the first layer 101 penetrates upward through the sum nodes and product nodes of the second to fifth layers 102 to 105 and thus undergoes their respective processes. After the fifth layer 1, the one-hot representation y i of the initial query word corresponding to the candidate word (w i ) (that is, if the query is for the i-th word, the variable y i has a value of "1", otherwise, it has a value of "0") is provided to the final sum node S to determine the conditional probability of the i-th word based on the first N words. Mathematically, the output of the SPN language modeling architecture is the value of the sum node S, that is, the conditional probability P(w i |w i-N ,..., w i-1 ).

[0041] The conditional probability information generated by the SPN language modeling architecture 100 can identify a wide range of correlations in word sequences, and these correlations can be used to provide one-way translations of these word sequences for speech recognition applications.

[0042] SPN-based bidirectional language translation system

[0043] As mentioned above, known MT shared latent code space methods have some shortcomings compared to the bidirectional translation of content provided by monolingual word sequences. It is also worth noting that while known SPN language modeling methods can determine the conditional probabilities of word sequences to affect unidirectional translation, they do not address the problem of bidirectional translation of word sequences or sentence content.

[0044] Thus, in at least one broad aspect, the systems and methods disclosed herein utilize, improve, and combine some features of the shared latent code space approach and the SPN language modeling approach to implement an SPN-based bidirectional translation system. In the present invention, a bidirectional translation system refers to a single machine translation system that can translate a text sequence from a first language to a second language (a first translation direction) and also translate a text sequence from a second language to a first language (a second translation direction opposite to the first direction). In both translation directions, the meaning and context of the text sequence are the same in both languages. In the present invention, a text sequence is a series of words that constitute a phrase or sentence, where the meaning and content are derived at least in part from the order of the words in the sequence.

[0045] Specifically, Figure 2A is a high-level block diagram of an exemplary SPN-based bidirectional translation system 200 according to a non-limiting embodiment of the present invention. As shown, the SPN translation system 200 employs a first autoencoder module 202 comprising a first encoder submodule 202A and a first decoder submodule 202B, a second autoencoder module 212 comprising a second encoder submodule 212A and a second decoder submodule 212B, and an SPN module 220. However, it should be understood that the SPN translation system 200 shown is merely an exemplary implementation of the present invention. Therefore, the following description is merely a description of an illustrative example of the present invention to facilitate understanding of the disclosed features and concepts, and is not intended to limit the scope of the present invention or to define its boundaries.

[0046] Considering the above structure of the SPN translation system 200, the SPN module 220 operates in the latent code space 210 (i.e., the set of all feasible sentence embeddings corresponding to the two languages ​​L1 and L2) provided by the first autoencoder module 202 and the second autoencoder module 212 to calculate the joint probability distribution P(L1, L2) of the sentence embeddings of the two languages. The joint probability distribution P(L1, L2) can be stored in the memory of the SPN translation system 200 (not in the memory). Figure 2A(shown in ). In addition, the joint probability distribution P(L1, L2) can also be used to calculate the conditional latent code space probability of each language when other languages ​​are known, namely P(L1|L2) and P(L2|L1), and these probabilities can also be stored in the memory of the SPN translation system 200. Generally speaking, sentence embedding refers to various technologies that can be used to represent text sequences as representation vectors of real numbers. The representation vector of real numbers output by the sentence embedding technology can also be called "sentence embedding". The distance between two sentence embeddings represents the similarity between the two text sequences represented by the sentence embeddings (for example, similarity in semantic sense or similarity in narrative concept). Specifically, the closer the distance between the two sentence embeddings, the greater the similarity between the two represented text sequences. It should be noted that although the present invention uses the term "sentence embedding", the text sequence represented by the sentence embedding is not necessarily a sentence (defined by grammar). For example, the text sequence represented by the sentence embedding can also be a phrase, a clause, a group of words, a single word or any other text sequence with some semantic meaning.

[0047] Using the conditional probabilities P(L1|L2) and P(L2|L1), the SPN translation system 200 implements bidirectional translation between two languages ​​by embedding the sentences provided by the first autoencoder module 202 and the second autoencoder module 212 through the corresponding first decoder submodule 202B and second decoder submodule 212B.

[0048] Back to Figure 2A , which shows the general functional flow of the system 200. Specifically, the sampled text sequence x is input into the trained first autoencoder module 202. In this example, the text sequence x is in the first language L1. For example, the text sequence x can be a sentence in a text corpus. Typically, a text corpus is a collection of texts (including words, phrases, sentences and / or entire documents). A monolingual text corpus includes texts in only one language; a multilingual text corpus includes texts in two or more languages, which are all translations of the same text (usually configured so that translations corresponding to the same text sequence can be aligned or matched together). The first autoencoder module 202 is used to receive the text sequence x as input to the trained first encoder submodule 202A to generate a sentence embedding z corresponding to the text sequence x. x And embed the sentence into z x is mapped into the latent code space 210. Accordingly, the sampled text sequence y represented in the second language L2 (e.g., sampled from the same or different text corpus as the text sequence x) is input into the trained second autoencoder module 212. The second autoencoder module 212 is configured to receive the text sequence y as input to the trained second encoder submodule 212A to generate a sentence embedding z for the text sequence yy And embed the sentence into z y is mapped into the latent code space 210. As mentioned above, the sentence embedding z x 、z y Features representing a text sequence x, y (e.g., words / phrases) are transformed into representation vectors of real numbers (also called digital vector codes, where the digital vector codes all have a predetermined number of bytes (e.g., 256 bytes, 512 bytes, etc.). The latent code space 210 is the set of all feasible sentence embeddings Z for the two languages ​​L1 and L2. The latent code space 210 is defined as the space that includes all potential sentence embeddings Z. The latent code space 210 is the space generated by the encoder submodules 202A, 212A in each autoencoder module 202, 212. Conceptually, in the context of translation, the latent code space 210 can be considered to be the sentence embedding z x 、z y Typically, the latent code space 210 is hidden (or latent) and therefore cannot be directly observed.

[0049] The latent code space 210 is not necessarily shared or common by the two autoencoder modules 202, 212. Each encoder sub-module 202A, 212A may generate its own latent code space 210, which may or may not overlap. Figure 2C An example of a system 200 is shown, similar to Figure 2A However, instead of a single latent code space 210 shared by both autoencoder modules 202, 212, a first encoder submodule 202A in the first autoencoder module 202 is configured to receive a text sequence x as input to the trained first encoder submodule 202A to generate a sentence embedding z corresponding to the text sequence x. x And embed the sentence into z x Mapped into the first latent code space 210A; the second encoder submodule 212A in the second autoencoder module 212 is used to receive the text sequence y as the input of the trained second encoder submodule 212A to generate a sentence embedding z of the text sequence y y And embed the sentence into z y is mapped into the latent code space 210B. It should be noted that the two latent code spaces 210A and 210B can be completely independent (as shown in the figure) or partially overlap. The operation of the system 200 does not depend on whether the latent code space is shared (for example, Figure 2A The potential code space 210 shown), independent (e.g., Figure 2CFor simplicity, the present disclosure refers to the singular potential code space 210, however, it should be understood that this is not intended to be limiting, and two potential code spaces 210A, 210B may exist.

[0050] Sentence Embedding z x 、z y is a vector of real numbers, which are samples (e.g., elements or terms) in the latent code space 210. x 、z y Input to SPN module 220. As mentioned above, SPN module 220 is used to calculate sentence embedding z of two languages ​​L1 and L2 x and z y The joint probability distribution P(z x , z y ). When calculating the joint probability distribution P(z x , z y ) After that, the SPN module 220 can further calculate the conditional probability p(z y |z x ) and p(z x |z y ), thereby inferring the distribution of target sentence embeddings given the source sentence embeddings, and vice versa. The sentence embeddings are then input into the corresponding language decoder submodule to achieve bidirectional translation between the two languages.

[0051] SPN-based two-way language translation system

[0052] Figure 2B is a detailed functional block diagram of a SPN-based bidirectional translation system 200 according to a non-limiting embodiment of the present invention. The first autoencoder module 202 and the second autoencoder module 212 can be trained using any suitable machine learning technique. For example, each autoencoder module 202, 212 can be trained using an unsupervised machine learning algorithm to learn the corresponding parameters (e.g., weights) of the neural network (e.g., RNN, CNN, D-CNN, or transformation neural network) implementing the corresponding autoencoder module 202, 212. In some contemplated embodiments, the SPN-based bidirectional translation system 200 can be trained in two stages, such as Figure 2B That is, in the first training phase, the first autoencoder module 202 and the second autoencoder module 212 (including the associated first encoder submodule 202A and second encoder submodule 212A and first decoder submodule 212A and second decoder submodule 212B) are trained to learn all feasible sentence embeddings z x 、z yIn the second training phase, the SPN module 220 is trained to learn sentence embeddings z in two languages. x and z y The joint probability distribution P(z x , z y ), in order to subsequently determine the conditional probability p(z y |z x ) and p(z x |z y ).

[0053] Consistent with the disclosed embodiments, in a first training phase, the first autoencoder module 202 can be trained by processing training samples comprising text sequences in a first language L1 selected from a multilingual text corpus (e.g., a text corpus comprising texts in at least languages ​​L1 and L2, which are translations of each other). Similarly, the second autoencoder module 212 can be trained by processing training samples comprising text sequences in a second language L2 selected from the same multilingual text corpus. In the first training phase, the text sequences used to train the first autoencoder module 202 and the second autoencoder module 212 do not need to be aligned (i.e., the text sequences in the two languages ​​L1 and L2 do not need to be parallel or translations of each other).

[0054] like Figure 2B As shown, the first encoder submodule 202A and the second encoder submodule 212A receive inputs of text sequences x and y in corresponding languages ​​L1 and L2, respectively, wherein each input text sequence x and y exhibits its own associated probability distribution P with respect to the multilingual text corpus. x 、P y . In some embodiments, each word / phrase in the input text sequence x, y is represented as a one-hot vector, where the vector has a value of "1" at the index corresponding to the word / phrase it represents and a value of "0" at all other indices. The present invention takes the one-hot vector representation as an example, however, other suitable techniques can also be used to process the input text sequences x, y into an acceptable format for input to the first sub-module 202A and the second sub-module 212A. After receiving the one-hot vector representation of the input text sequences x, y, the first encoder sub-module 202A and the second encoder sub-module 212A process and encode the corresponding one-hot vector representation of the input text sequences x, y to generate the corresponding sentence embedding z x and z yThe encoding process can be done using any neural network that models a language processing model, which can be an RNN, CNN, or a transformer neural network, as well as other possible neural network architectures. For example, in the case of an RNN-based MT system, the one-hot representation of the input text sequence x, y is passed to the word embedding layer and then input to the RNN, and the final hidden state of the RNN can be output as a sentence embedding. Sentence embedding z x and z y This is followed by samples (i.e., elements or terms) in the latent code space 210. The sentence is then embedded into z x 、z y Provided to SPN module 220. Figure 2B In the example shown in FIG, two instances of the SPN module 220 are shown, each receiving a sentence embedding z x 、z y The corresponding one in the output and the corresponding joint probability distribution P(z x , z y Although a single instance of the SPN module 220 is shown as a separate instance, it should be understood that a single instance of the SPN module 220 can be used to embed z according to an input sentence. x 、z y Output joint probability distribution P(z x , z y As detailed below, in the second training phase, the SPN module 220 is trained to process the sentence embedding z x 、z y , to determine the joint probability distribution P(z x , z y ).

[0055] In addition, in the first training phase, the first decoder submodule 202B and the second decoder submodule 212B are used to receive the sentence embedding z generated by the encoder submodules 202A and 212A. x 、z y , and predict the representative reconstruction z′ of the sentence embedding y 、z′ x. The first decoder submodule 202B and the second decoder submodule 212B may include any RNN, D-CNN or transformation neural network and other possible neural network architectures. In the case of an RNN-based decoder (e.g., a decoder implemented using RNN), the sentence embedding may be input to the first step of the RNN, and then the corresponding updated hidden state and the predicted word in this step may be obtained. The updated hidden state and the predicted word in each step may be input to the next recursive block (e.g., an RNN unit of an RNN) until the end-of-sentence symbol is reached, which indicates that the recursive decoding process should terminate. The decoder submodules 202B, 212B may be configured to output the sentence embedding z at each step. x 、z y The softmax representation is a vector-based probability distribution over the dictionary words corresponding to the language L1 or L2. The values ​​in the softmax representation sum to 1 (so that the values ​​in the softmax representation represent the probability of each word relative to all the words represented in the softmax vector), and the dictionary words include words from both languages ​​L1 and L2. This softmax representation therefore represents the probability distribution of the words in the dictionary that appear at the corresponding step.

[0056] It should be understood that in some embodiments, the autoencoder modules 202, 212 (including the constituent encoder submodules 202A / 202B / decoder submodules 212A / 212B) can be implemented using RNNs. In an alternative embodiment, the first encoder submodule 202A and the second encoder submodule 212A can be implemented as CNNs, while the first decoder submodule 202B and the second decoder submodule 212B can be implemented as deconvolutional neural networks (D-CNNs).

[0057] It should also be understood that any difference between the input one-hot vector representation of the text sequence x, y and the corresponding reconstructed output of the decoder submodule 202B, 212B can be calculated as (See Figure 2A) is quantified as a “reconstruction loss”. In various contemplated embodiments, the reconstruction loss can be minimized by implementing a “loss function” designed to optimize the training process of the first autoencoder module 202 and the second autoencoder module 212. The loss function can embody, for example, iterative gradient descent variant techniques that monitor the correlation between the input text sequence and the reconstructed output text sequence, and provide iterative feedback data to adjust and fine-tune various parameters (e.g., weights) of the encoder and decoder neural networks. That is, the first autoencoder module 202A and the second autoencoder module 212A can be iteratively trained to learn the latent code representation of the text sequence in the latent code space 210 until the quantized loss value determined by the loss function is reduced to a certain level, i.e., the reconstructed output The quality of is substantially similar to the input text sequence x, y, reaching an acceptable predetermined threshold level.

[0058] When this acceptable level is reached during training, the sentence embedding z x 、z y is used as input samples to the latent code space 210 for translation processing by the SPN module 220. Thus, the trained first autoencoder module 202 and the second autoencoder module 212 are then able to generate separate sentence embeddings z for the two languages ​​L1 and L2 x 、z y .

[0059] Figure 3A is a high-level flow chart of an exemplary process 300 for the first training phase of the SPN-based bidirectional translation system 200 for training the autoencoder modules 202, 212 according to a non-limiting embodiment of the present invention. It should be understood that the tasks in process 300 can be performed by one or more computing devices or entities and can be embodied in computer-executable instructions stored in a computer-readable medium (e.g., a non-transitory computer-readable medium). Some actions, tasks, or portions thereof may vary depending on the specific implementation without departing from the scope of the present invention.

[0060] Consistent with the disclosure of the first training phase above, process 300 begins at task block 302, where the autoencoder modules 202, 212 (including the constituent encoder submodules 202A / 202B / decoder submodules 212A / 212B) receive input training text sequences x, y, respectively, each of which is represented in the corresponding language L1 and L2 and has its own associated probability distribution P with respect to the multilingual text corpus. x 、P y In some examples, text sequences x, y are randomly sampled from a multilingual text corpus.

[0061] In task block 304, the encoder submodules 202A and 212A process and encode the training text sequences x and y, respectively, to generate the corresponding sentence embedding z in vector format. x and z y . Sentence embedding z x and z y is a sample (ie, an element or term) in the latent code space 210 .

[0062] In task block 308, the first decoder submodule 202B and the second decoder submodule 212B sample sentence embedding z from the latent code space 210. x and z y And process each sentence embedding to generate a representative reconstruction of the corresponding training text sequence x, y (referred to as the reconstructed text sequence and As described above, the decoding process may include applying a softmax or normalized exponential function (often referred to as a “softmax” function) to the sentence embedding at each prediction step for the output sentences in the two languages ​​L1 and L2, thereby obtaining a probability distribution over the entire vocabulary.

[0063] In task block 310, the reconstructed text sequence is quantified according to an acceptable predetermined level using a loss function and Is the reconstructed text sequence substantially related and similar to the input text sequences x and y? and The reconstruction loss between the original training text sequences x and y.

[0064] In decision block 312, process 300 determines whether to reconstruct the text sequence based on a predetermined level of acceptability of the entire training set. and Is the sentence z substantially related and similar to the input text sequences x and y? If so (yes), then process 300 proceeds to task block 314, where the sentence is embedded into z. x 、z y is used as a verified vector code selected from the latent code space 210 for translation between languages ​​L1 and L2. If not (No), the process 300 returns to task block 304, where the encoder submodules 202A, 212A and the decoder submodules 202B, 212B continue to be trained based on the iterative feedback data to adjust and fine-tune various decoder parameters (e.g., weights) of the neural network provided by the loss function information.

[0065] Back to Figure 2BAs mentioned above, the second training phase involves training the SPN module 220. As mentioned above, the SPN module 220 includes a sum-product network (SPN) that includes consecutive sum node processing layers and product node processing layers interconnected in a communication manner via edges. The SPN in the SPN module 220 is trained to learn pairwise aligned (i.e., parallel) sentence embeddings z for the two languages ​​L1 and L2. x 、z y The joint probability distribution of P(z x , z y ).

[0066] The SPN module 220 can be used to embed z in Gaussian distribution or sentence x 、z y , which is a set of parameters that can be adjusted iteratively during training. In this way, the parameters of the SPN module 220 can be iteratively adjusted in the second training phase to maximize the determination of the pairwise sentence embedding z of the aligned multilingual text corpus in languages ​​L1 and L2. x 、z y The joint probability distribution P(z x , z y ).

[0067] like Figure 2B As shown, the SPN module 220 receives a pair of sentence embeddings z generated by the encoder submodules 202A and 212A. x 、z y In the second training phase, a pair of sentence embeddings z is calculated by passing through the SPN module 220 from bottom to top through two SPN model calculations (pass-through). x 、z y That is, in the first model calculation, the SPN module 220 receives the input sentence embedding z x 、z y, and processes the input to calculate an output corresponding to the unnormalized probability. This output is obtained by calculating each layer and node and product node according to the SPN architecture. After calculating the unnormalized probability in the first model calculation, in the second model calculation, the SPN module 220 initializes all distribution leaves to the input value "1" and again processes the input through each layer and node and product node to output the normalization constant. It should be understood that initializing all distribution leaves to the input value "1" corresponds to integrating all random variables, so the SPN effectively calculates the normalization constant in the second model calculation. Then, by dividing the unnormalized probability by the normalization constant, the embedding z is calculated. x 、z y The normalized probability of .

[0068] Furthermore, much like the first training phase of the first and second autoencoder modules 202 and 212 described above, the joint probability distribution P(z x , z x ), the SPN module 220 can be trained in the second training phase. This maximization can adopt any iterative training or learning method that can iteratively adjust the parameters (i.e., weights) of the SPN in the SPN module 220, such as gradient ascent technology, maximum expectation value technology, Bayesian moment matching technology, collapsed variational Bayesian technology, etc. At this time, by maximizing the probability distribution SPNP (z x , z y ), the parameters (ie, weights) of the SPN in the SPN module 220 can be trained and adjusted.

[0069] In establishing the maximum probability distribution P(z x , z y ) then the conditional probability p(z y |z x ) and p(z x |z y ) can be calculated by and From p(z x , z y ) is derived from the above. It is worth noting that the joint probability distribution P(z x , z y ), we can calculate the conditional probability of the two translation directions. The conditional probability p(z y |z x )(i.e., given z x When z y The probability of ) represents the sentence embedding z in the latent code space 210x to sentence embedding z y The translation of the text sequence x in language L1 is translated into the text sequence y in language L2. Similarly, the conditional probability p(z x |z y )(i.e., given z y When z x The probability of ) represents the sentence embedding z in the latent code space 210 y to sentence embedding z x The translation of the text sequence y in language L2 is simultaneously realized as the text sequence x in language L1. Therefore, training a single SPN module 220 can realize bidirectional translation.

[0070] Figure 3B is a high-level flow chart of an exemplary process 350 for the second training phase of the SPN-based bidirectional translation system 200 for training the SPN module 220 according to a non-limiting embodiment of the present invention. It should be understood that the tasks in process 350 can be performed by one or more computing devices or entities and can be embodied in computer-executable instructions stored in a computer-readable medium (e.g., a non-transitory computer-readable medium). Some actions, tasks, or portions thereof may vary depending on the specific implementation without departing from the scope of the present invention.

[0071] Consistent with the disclosure of the second training phase above, process 350 begins at task block 352, where the SPN module 220 receives a pair of verified sentence embeddings z from the latent code space 210. x 、z y (generated after the first training phase).

[0072] In task block 354, the sentence embedding z is calculated within the bottom-up model calculation of the processing layer of the SPN module 220. x 、z y The unnormalized probability of .

[0073] Optionally, in task block 356, a second model calculation of the SPN module 220 is performed, wherein the input values ​​of the distribution leaves are all set to "1" to calculate the normalization constant, as described above. If the normalization constant has already been calculated (for example, in a previous iteration of training), task block 356 may not be necessary. The normalization constant is used to normalize the unnormalized probabilities calculated in task block 354 above. Dividing the unnormalized probabilities by the normalization constant yields the normalized joint distribution P(z x , z y ).

[0074] In task block 358, according to the joint distribution P(z x , z y)Calculate the conditional probability p(z y |z x ) and p(z x |z y ), as described above.

[0075] Process 350 can use different samples z in the latent code space 210 x 、z y This is repeated in multiple iterations to adjust the parameters (e.g., weights) of the SPN in the SPN module 220. For example, Hsu et al. (Hsu, W., Kalra, A., and Poupart, P. (2017) Online structure learning for sum-product networks with gaussian leaves, arXiv preprint (arXiv ID: 1701.05265), the entire contents of which are incorporated herein by reference) describe a method for learning parameters (e.g., weights) of each layer of an SPN.

[0076] After the autoencoder modules 202 and 212 and the SPN module 220 are trained, the trained SPN-based bidirectional translation system 200 can be used to perform bidirectional translation during the inference phase.

[0077] For example, given the sentence embedding z x , the most likely sentence embedding It can be calculated as: Corresponding to z x* The translated embedding of . Similarly, it is known that the sentence embedding z y , the most likely sentence embedding It can be calculated as Corresponding to z y* Post-translation embedding.

[0078] Figure 4 4 is a high-level flow chart of an exemplary SPN-based bidirectional translation process 400 according to a non-limiting embodiment of the present invention. It should be understood that the process 400 is configured to operate in conjunction with at least some of the architecture, structures, elements, and related functions of the SPN-based bidirectional translation system 200 disclosed above.

[0079] Furthermore, for some embodiments, it is assumed that the autoencoder modules 202, 212 (including the constituent encoder submodules 202A / 202B / decoder submodules 212A / 212B) have been trained to embed sentences into z in an acceptable manner (i.e., according to a predetermined threshold level). x 、z y and its reconstruction representation z′x 、z′ y In association, the reconstructed representation is based on the text sequence x, y in the corresponding languages ​​L1 and L2. Similarly, assume that the SPN module 220 has been trained to maximize the probability distribution P(z y , z x ), to derive the conditional probability p(z y , z x ) and p(z x |z y ).

[0080] thus, Figure 4 The bidirectional translation functionality of process 400 is illustrated by showing the translation of a text sequence from a first language L1 to a second language L2 as depicted in process 400A and the translation of a text sequence from the second language L2 to the first language L1 as depicted in process 400B.

[0081] In the case of translating a sentence from a first language L1 to a second language L2, as shown, process 400A begins at task block 402A, where the first encoder submodule 202A receives a vector representation (e.g., a one-hot vector representation) of a text sequence x in the first language L1 to be translated into a text sequence y in the second language L2. In task block 404A, the first encoder submodule 202A processes the vector representation of the text sequence x to generate a sentence embedding z as a sample in the latent code space 210. x .

[0082] In task block 406A, the SPN module 220 receives the sentence embedding z x , and by maximizing the conditional probability P(z y |z x ) to obtain the corresponding sentence embedding z y . Maximize P(z y |z x ) can be used to embed the sentence z y The gradient ascent technique is used to implement it, and it can be expressed as:

[0083] Finally, in task block 408A, the second decoder submodule 212B receives the sentence embedding z y and decode z y , to generate a vector representation (eg, a one-hot vector representation) of the text sequence y in the second language L2. Further processing may be performed to convert the vector representation of the text sequence y into a text representation.

[0084] Similarly, in the case of translating a sentence from a second language L2 to a first language L1, process 400B begins at task block 402B, where the second encoder submodule 212A receives a vector representation (e.g., a one-hot vector representation) of a text sequence y in a second language L2 to be translated into a text sequence x in the first language L1. In task block 404B, the second encoder submodule 212A processes the text sequence y to generate a sentence embedding z as a sample in the latent code space 210. y .

[0085] In task block 406B, the SPN module 220 receives the sentence embedding z y , and by maximizing the conditional probability P(z x |z y ) to obtain the corresponding sentence embedding z x . Maximize P(z x |z y ) can be used to embed the sentence z x The gradient ascent technique is used to implement it, and it can be expressed as:

[0086] Finally, in task block 408B, the first decoder submodule 202B receives the sentence embedding z x and decode z x , to generate a vector representation of the text sequence x in the first language L1. Further processing may be performed to convert the vector representation of the text sequence x into a text representation.

[0087] Figure 5 is a simplified block diagram of an exemplary computing system that can be used to implement the SPN-based bidirectional translation system and method disclosed herein. Other examples suitable for implementing the embodiments described herein may be used, and these examples may include components different from those described below. Although Figure 5 A single instance of each component is shown, but multiple instances of each component may be present in the processing system 500 .

[0088] The computing system 500 may include one or more processing devices 514, such as a processor, a microprocessor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuit, a dedicated artificial intelligence processor unit, or a combination thereof. The computing system 500 may also include one or more optional input / output (I / O) interfaces 516 that may support connection to one or more optional input devices 518 and / or optional output devices 520.

[0089] In the illustrated example, one or more input devices 518 (e.g., a keyboard, mouse, microphone, touch screen, and / or keypad) and one or more output devices 520 (e.g., a display, speakers, and / or printer) are shown as optional and external to computing system 500. In other examples, none of input device(s) 518 and output device(s) 520 may be present, in which case I / O interface(s) 516 may not be required.

[0090] The computing system 500 may include one or more network interfaces 522 for wired or wireless communication with a communication network. The one or more network interfaces 522 may include wired links (e.g., Ethernet cables) and / or wireless links (e.g., one or more antennas) for communicating within and / or outside the network.

[0091] The computing system 500 may also include one or more storage units 524, which may include mass storage units such as solid-state drives, hard disk drives, magnetic disk drives, and / or optical disk drives. The one or more storage units 524 may store data 526. For example, the stored data 526 may include text sequences to be processed by the SPN-based bidirectional translation system 200 or used as training data.

[0092] The computing system 500 may include one or more memories 528, which may include volatile or non-volatile memory (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The one or more non-transitory memories 528 may store instructions executed by the one or more processing devices 514, for example, to perform the examples described herein. The one or more memories 528 may include other software instructions, such as for implementing an operating system and other applications / functionality. In some examples, the one or more memories 528 may include software instructions executed by the processing device 514 to implement the SPN-based bidirectional translation system 200 (e.g., instructions for performing processes 300, 350, 400A, and / or 400B). In some examples, the computing system 500 may additionally or alternatively execute instructions from external memory (e.g., an external drive in wired or wireless communication with the server 500), or may have executable instructions provided by a transitory or non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable memory.

[0093] The computing system 500 may also include a bus to provide communication between components (including those described above) of the computing system 500. The bus may be any suitable bus architecture including, for example, a memory bus, a peripheral bus, or a video bus.

[0094] Thus, through the disclosed embodiments, the described systems, architectures, and methods enable bidirectional translation of sentences between two different languages, which overcomes at least some limitations and deficiencies of conventional machine translation systems and methods.

[0095] In various examples, the present invention provides methods and systems for implementing bidirectional translation using a SPN-based approach for computing a joint probability distribution in a latent code space to compute conditional probabilities for performing bidirectional translation.

[0096] The latent code space can be used to convert text sequences into sentence embeddings using an autoencoder map. After the SPN has been trained to output a maximum joint probability distribution, the decoder submodule in the trained autoencoder can be used to obtain a translation by decoding the sentence embeddings in the latent code space into a text sequence in the desired target language.

[0097] Although the present invention describes methods and processes by steps performed in a certain order, one or more steps in the methods and processes may be omitted or changed as appropriate. In appropriate cases, one or more steps may be performed in an order other than the order described.

[0098] Although the present invention has been described at least in part in terms of methods, it will be understood by those skilled in the art that the present invention is also directed to various components for performing at least some aspects and features of the methods, whether by hardware components, software, or any combination thereof. Accordingly, the technical solutions of the present invention may be embodied in the form of software products. Suitable software products may be stored in pre-recorded storage devices or other similar non-volatile or non-transient computer-readable media, including DVDs, CD-ROMs, USB flash drives, removable hard drives, or other storage media. The software product includes instructions tangibly stored thereon, which enable a processing device (e.g., a personal computer, a server, or a network device) to perform examples of the methods disclosed herein. Machine-executable instructions may be in the form of code sequences, configuration information, or other data that, when executed, cause a machine (e.g., a processor or other processing device) to perform steps in a method according to an example of the present invention.

[0099] The present invention may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are intended in all respects to be illustrative and non-restrictive. Selected features from one or more of the above-described embodiments may be combined to create alternative embodiments not explicitly described, and features suitable for such combinations are understood to be within the scope of the present invention.

[0100] All values ​​and subranges within the disclosed ranges are also disclosed. Furthermore, although the systems, devices, and processes disclosed and illustrated herein may include a specific number of elements / components, the systems, devices, and assemblies may be modified to include more or fewer of such elements / components. For example, although any disclosed element / component may be referenced as a single quantity, the embodiments disclosed herein may be modified to include a plurality of such elements / components. The subject matter described herein is intended to cover and encompass all suitable technical variations.

Claims

1. A machine translation system for bidirectional translation of a text sequence between a first language and a second language, characterized in that The machine translation system includes: a first autoencoder configured to receive a vector representation of a first text sequence in the first language and encode the vector representation of the first text sequence into a first sentence embedding, the first autoencoder comprising a first encoder and a first decoder; A sum-product network SPN is used to receive the first sentence embedding and determine a first conditional probability when the first sentence embedding is known, where the first conditional probability is a conditional probability corresponding to the second sentence embedding, and the second sentence embedding is a translated embedding of the first sentence embedding. The SPN includes multiple layers, each layer independently including multiple sum nodes or multiple product nodes; a second autoencoder for receiving the second sentence embedding, wherein the second autoencoder is trained to decode the second sentence embedding into a vector representation of a second text sequence in the second language, the second autoencoder comprising a second encoder and a second decoder; The second autoencoder is further configured to receive a vector representation of a third text sequence in the second language, and encode the vector representation of the third text sequence into a third sentence embedding; The sum-product network (SPN) is further configured to receive the third sentence embedding and determine a second conditional probability, where the second conditional probability is a conditional probability corresponding to a fourth sentence embedding, and the fourth sentence embedding is a translated embedding of the third sentence embedding, by knowing the third sentence embedding. The first autoencoder is further configured to receive the fourth sentence embedding and decode the fourth sentence embedding into a vector representation of a fourth text sequence in the first language.

2. The system according to claim 1, wherein: The first encoder, the first decoder, the second encoder and the second decoder independently include a recurrent neural network RNN, a convolutional neural network CNN or a transformation neural network.

3. The system according to claim 1, wherein: The first autoencoder and the second autoencoder are trained together in a first training phase to perform the following operations: On the first autoencoder and the second autoencoder side, encoding the vector representations of the first training text sequence and the second training text sequence in the first language and the second language into corresponding first training sentence embeddings and second training sentence embeddings, respectively; On the first autoencoder and the second autoencoder side, decoding the first training sentence embedding and the second training sentence embedding into corresponding vector representations of the first reconstructed text sequence and the second reconstructed text sequence in the first language and the second language, respectively; Parameters of the first autoencoder and the second autoencoder are adjusted to minimize a loss function representing a reconstruction loss between the first reconstructed text sequence and the second reconstructed text sequence and the corresponding first training text sequence and the second training text sequence.

4. The system according to claim 3, characterized in that The SPN is trained in the second training phase to perform the following operations: Adjusting the weights of the SPN to maximize the joint probability of the first training sentence embedding and the second training sentence embedding; The joint probability is used to calculate the maximum conditional probability.

5. A method for bidirectional translation of a text sequence between a first language and a second language, characterized in that The method comprises: A first autoencoder encodes a vector representation of a first text sequence in the first language into a first sentence embedding, the first autoencoder comprising a first encoder and a first decoder; A sum-product network SPN is used to receive the first sentence embedding and determine a first conditional probability when the first sentence embedding is known, where the first conditional probability is a conditional probability corresponding to the second sentence embedding, and the second sentence embedding is a translated embedding of the first sentence embedding. The SPN includes multiple layers, each layer independently including multiple sum nodes or multiple product nodes; A second autoencoder decodes the second sentence embedding into a vector representation of a second text sequence in the second language, the second autoencoder including a second encoder and a second decoder; The second autoencoder encodes the vector representation of the third text sequence in the second language into a third sentence embedding; The sum-product network (SPN) receives the third sentence embedding and determines a second conditional probability based on the third sentence embedding, where the second conditional probability is a conditional probability corresponding to a fourth sentence embedding, and the fourth sentence embedding is a translated embedding of the third sentence embedding. The first autoencoder receives the fourth sentence embedding and decodes the fourth sentence embedding into a vector representation of a fourth text sequence in the first language.

6. The method according to claim 5, characterized in that The first encoder, the first decoder, the second encoder and the second decoder independently include a recursive neural network RNN, a convolutional neural network CNN, a deconvolutional neural network D-CNN or a transformation neural network.

7. The method according to claim 5, characterized in that The method further comprises, before the encoding and the decoding, training the first autoencoder and the second autoencoder together in a first training phase, wherein the first training phase comprises: On the first autoencoder and the second autoencoder side, encoding the vector representations of the first training text sequence and the second training text sequence in the first language and the second language into corresponding first training sentence embeddings and second training sentence embeddings, respectively; On the first autoencoder and the second autoencoder side, decoding the first training sentence embedding and the second training sentence embedding into corresponding vector representations of the first reconstructed text sequence and the second reconstructed text sequence in the first language and the second language, respectively; The weights of the first autoencoder and the second autoencoder are adjusted to minimize a loss function representing a reconstruction loss between the first reconstructed text sequence and the second reconstructed text sequence and the corresponding first training text sequence and the second training text sequence.

8. The method according to claim 7, characterized in that The method further comprises, before the generating, training the SPN in a second training phase, wherein the second training phase comprises: Adjusting the weights of the SPN to maximize the joint probability of the first training sentence embedding and the second training sentence embedding; The joint probability is used to calculate the maximum conditional probability.

9. A computer-readable medium storing machine-executable instructions, characterized in that: When the machine-executable instructions are executed by a processing device in a system, the system is caused to perform the method according to any one of claims 5 to 8.

10. A computer program comprising machine-executable instructions, characterized in that When the machine-executable instructions are executed by a processing device in a system, the system is caused to perform the method according to any one of claims 5 to 8.

Images