A neural hybrid machine transliteration system for translation from a source language to a target language

Abstract

A neural hybrid machine transliteration system for translating from a source language into a target language, comprising: an input processing unit configured to receive the source word and retrieve the phoneme and graph sequence representation of the source word from a database consisting of a record of several named entities and English loanwords; a first encoder module connected to the input processing unit and configured to receive and encode the phoneme sequence representation of the source word to produce a first encoded representation, wherein the first encoder is configured to understand the phoneme sequence representation and produce an encoded representation of the entire sequence; a second encoding module configured to receive and encode a grapheme sequence representation of the source word to produce a second coded representation, wherein the second encoder is configured to understand the grapheme sequence representation and produce a coded representation of the entire sequence; an aggregation module configured to receive the first and second encoded representations and combine them to produce an aggregated representation; a decoder module configured to receive the aggregated representation from the aggregation module and generate a target word grapheme sequence in the target language based on the aggregated representation; and an output processing module with a user interface, wherein the output processing module is configured to receive the grapheme sequence of the target word in the target language and display the translated word on the user interface.

Description

AREA OF INVENTION

[0001] The present disclosure relates to a machine transliteration system, in particular a neural hybrid machine transliteration system for translating from a source language into a target language. More precisely, the invention relates to a hybrid neural transliteration system with multiple source languages ​​that automatically converts words of a source language into phonetically equivalent words of a target language. BACKGROUND OF THE INVENTION

[0002] Machine transliteration is the task of automatically converting a word from a source language into a phonetically equivalent word in a target language by taking the phonology of the target language into account. It is considered an important sub-problem of machine translation and cross-linguistic information retrieval, as it transliterates proper nouns, named entities, and loanwords. In multilingual societies with multiple writing systems, transliteration is used not only for named entities and loanwords but also in text processing at the sentence, paragraph, or document level.

[0003] Machine transliteration poses significant challenges for language pairs with different alphabets and sound systems. A direct one-to-one mapping of source to target graphemes may not be applicable for such language pairs. Differences in features, syllables, logograms, and alphabets lead to additional problems. General challenges include script specifications, phoneme deletion, phoneme insertion, and transliteration variants.

[0004] Machine transliteration systems can be broadly divided into three categories: direct models, pivot models, and hybrid models. Direct models are grapheme-based systems in which graphemes of the source language are directly transliterated into graphemes of the target language. Pivot models, also called phoneme-based models, first map the graphemes of the source language to the phonemes of the source language and then to the graphemes of the target language.

[0005] Grapheme- and phoneme-based models face several limitations. Pivoted models suffer from cascading errors due to the large number of mapping steps. Grapheme-based models often struggle to handle cases where a word's spelling differs significantly from its pronunciation. Furthermore, grapheme-based approaches require a sufficiently large parallel corpus, while phoneme-based approaches necessitate pronunciation dictionaries and bilingual pronunciation dictionaries. For resource-poor language pairs, neither large parallel corpora nor bilingual pronunciation dictionaries are readily available.

[0006] Hybrid models have the potential to overcome these limitations by utilizing both grapheme and phoneme features. Existing hybrid models combine grapheme and phoneme models using methods such as interpolation or grapheme-phoneme correspondence.

[0007] Current studies in machine transliteration are investigating neural methods of deep learning. Grapheme-based encoder-decoder models have proven superior for most language pairs. However, to date, no hybrid encoder-decoder-based transliteration system exists that utilizes a multi-source encoder architecture for the separate processing of grapheme and phoneme sequences.

[0008] There is a need for a hybrid transliteration system that effectively combines grapheme and phoneme features by using a multi-source neural architecture, especially for resource-poor language pairs with different writing and sound systems. SUMMARY OF THE INVENTION

[0009] This invention provides a novel neural hybrid system for machine transliteration that combines grapheme and phoneme representations of source words using a multi-source encoder-decoder framework. The system employs two separate encoders for the simultaneous processing of grapheme and phoneme sequences. Subsequently, an aggregation module intelligently combines these dual representations before they are fed to a decoder to generate transliterations in the target language. The invention supports both RNN-based and transformer-based architectures and introduces various aggregation methods, including concatenation, addition, and convolution, to effectively fuse the encoded representations, thereby significantly improving transliteration accuracy compared to conventional single-source models.

[0010] The present disclosure relates to a neural hybrid machine transliteration system for translating from a source language into a target language. The system comprises: an input processing unit that captures the source word and its phoneme and graph sequence representations from a database of several named entities and English loanwords; a first encoding module connected to the input processing unit that receives and encodes the phoneme sequence representation of the source word to produce a first encoded representation, the first encoding module understanding the phoneme sequence representation and producing an encoded representation of the entire sequence; a second encoding module that receives and encodes the grapheme sequence representation of the source word to produce a second encoded representation, the second encoding module understanding the grapheme sequence representation and producing an encoded representation of the entire sequence;an aggregation module that receives the first and second encoded representations and combines them into an aggregated representation; a decoder module configured to receive the aggregated representation from the aggregation module and generate a target word grapheme sequence in the target language based on the aggregated representation; and an output processing module with a user interface, wherein the output processing module is configured to receive the target word grapheme sequence in the target language and display the translated word on the user interface.

[0011] One aim of the present disclosure is to provide a neural hybrid machine transliteration system for translation from a source language into a target language.

[0012] Another objective of the present disclosure is to provide an improved machine transliteration system that effectively captures and combines both grapheme and phoneme features of source words to produce more accurate transliterations in the target language.

[0013] Another objective of the present disclosure is to provide a hybrid system based on RNN and transformer-based architectures.

[0014] Another objective of the present disclosure is to provide a multi-source encoder-decoder framework that overcomes the limitations of traditional hybrid models based on linear interpolation or statistical correspondence by utilizing neural network-based coding and aggregation techniques.

[0015] Another objective of the present disclosure is to provide a novel aggregation architecture for combining multiple coded representations that take into account different levels of importance of grapheme and phoneme sequences in the transliteration process.

[0016] However, another objective of the present disclosure is to provide a transliteration system that can be adapted to resource-poor languages ​​and achieves superior performance in the transliteration of proper names and loanwords across multiple language pairs.

[0017] To further clarify the advantages and features of the present disclosure, the invention is described in more detail with reference to specific embodiments illustrated in the accompanying drawings. It is understood that these drawings merely show typical embodiments of the invention and are therefore not to be understood as limiting its scope of protection. The invention is described and explained in more detail and with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE IMAGES

[0018] These and other features, aspects and advantages of the present disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings, in which identical symbols represent identical parts, wherein: Fig. Figure 1 shows a block diagram of a neural hybrid machine transliteration system for translating from a source language into a target language according to an embodiment of the present disclosure; Fig. Figure 2 illustrates the schematic diagram of the proposed multi-source RNN-based encoder-decoder hybrid transliteration model according to an embodiment of the present disclosure; and Fig. Figure 3 shows the schematic diagram of the proposed multi-source transformer-based encoder-decoder hybrid transliteration model with serial attention combination according to an embodiment of the present disclosure.

[0019] Furthermore, those skilled in the art will recognize that the elements in the drawings are simplified and not necessarily drawn to scale. For example, the flowcharts illustrate the process by highlighting the main steps to facilitate understanding of this disclosure. With regard to the construction of the device, one or more components may be represented in the drawings by conventional symbols. The drawings may show only those specific details relevant to understanding the embodiments of this disclosure, so as not to clutter the drawings with details that are already apparent to those skilled in the art from the description contained herein. DETAILED DESCRIPTION:

[0020] To facilitate understanding of the principles of the invention, reference is made below to the embodiment illustrated in the drawings, which is described using specific terms. It is understood, however, that this does not limit the scope of protection of the invention. Rather, modifications and further developments of the illustrated system, as well as further applications of the inventive principles depicted therein, are conceivable, insofar as they would typically occur to a person skilled in the art in the field of the invention.

[0021] It will be clear to those skilled in the art that the foregoing general description and the following detailed description are exemplary and explanatory of the invention and are not to be understood as a limitation thereof.

[0022] References to “an aspect”, “another aspect”, or similar phrases in this description mean that a particular feature, structure, or property described in connection with the embodiment is included in at least one embodiment of the present disclosure. Therefore, phrases such as “in one embodiment”, “in another embodiment”, and similar expressions in this description may, but do not necessarily, all refer to the same embodiment.

[0023] The terms "includes," "comprehensive," or similar expressions denote non-exclusive inclusion. Thus, a procedure or method containing a list of steps does not only include those steps but may also include further steps not explicitly listed or inherent in the procedure or method. Likewise, the statement "includes..." for one or more devices, subsystems, elements, structures, or components, without further limitations, does not preclude the existence of other devices, subsystems, elements, structures, or components.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meanings generally known to those skilled in the art in the field to which this invention belongs. The systems, methods, and examples described herein serve only for illustration and are not to be understood as limiting.

[0025] Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0026] The functional units described in this specification are referred to as devices. A device may be implemented in programmable hardware such as processors, digital signal processors, central processing units, FPGAs, PALs, PLDs, cloud processing systems, or similar. Devices may also be implemented in software for execution by various processor types. An identified device may contain executable code and, for example, comprise one or more physical or logical blocks of computer instructions, which may be organized as an object, procedure, function, or other construct. However, the executable files of an identified device need not be physically related; they may consist of different instructions stored in different locations that, when logically combined, constitute the device and fulfill its purpose.

[0027] The executable code of a device or module can consist of a single instruction or multiple instructions and can even extend across different code sections, applications, and storage media. Similarly, operational data within the device can be identified and represented, and can exist in any suitable form and be organized in any data structure. The operational data can be captured as a single data record or distributed across various storage media and may exist, at least partially, as electronic signals within a system or network.

[0028] References to “a selected embodiment”, “an embodiment”, or “an embodiment” in this description mean that a particular feature, structure, or property described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter. Therefore, the phrases “a selected embodiment”, “in an embodiment”, or “in an embodiment” appearing at different points in this description do not necessarily refer to the same embodiment.

[0029] Furthermore, the described features, structures, or properties can be combined in one or more embodiments in any suitable manner. The following description contains numerous specific details to enable a comprehensive understanding of the embodiments of the disclosed subject matter. However, a person skilled in the art will recognize that the disclosed subject matter can also be realized without one or more of the specific details or with other methods, components, materials, etc. In other cases, known structures, materials, or processes are not presented or described in detail so as not to obscure aspects of the disclosed subject matter.

[0030] According to the exemplary embodiments, the disclosed computer programs or modules can be executed in a variety of ways, for example, as an application running in the memory of a device or as a hosted application running on a server and communicating with the device application or browser via various standard protocols such as TCP / IP, HTTP, XML, SOAP, REST, JSON, and other suitable protocols. The disclosed computer programs can be written in programming languages ​​that run either in the device's memory or on a hosted server, such as BASIC, COBOL, C, C++, Java, Pascal, or scripting languages ​​such as JavaScript, Python, Ruby, PHP, Perl, or other suitable programming languages.

[0031] Some of the described embodiments involve data transmission over a network, such as the transmission of various inputs or files. The network may include, for example, the internet, wide area networks (WANs), local area networks (LANs), analog or digital wired and wireless telephone networks (e.g., PSTN, ISDN, cellular networks, and xDSL), radio, television, cable, satellite, and / or other transmission or tunneling mechanisms for data. It may include multiple networks or subnetworks, each of which may, for example, have a wired or wireless data path. The network may include a circuit-switched voice network, a packet-switched data network, or another network for transmitting electronic data. For example, it may be based on the Internet Protocol (IP) or Asynchronous Transfer Mode (ATM) and support voice communication using VoIP, Voice over ATM, or similar protocols.In one embodiment, the network comprises a mobile network configured for the exchange of text or SMS messages.

[0032] Examples of networks include Personal Area Networks (PAN), Storage Area Networks (SAN), Home Area Networks (HAN), Campus Area Networks (CAN), Local Area Networks (LAN), Wide Area Networks (WAN), Metropolitan Area Networks (MAN), Virtual Private Networks (VPN), Enterprise Private Networks (EPN), the Internet, Global Area Networks (GAN), and so on.

[0033] Fig. Figure 1 shows a block diagram of a neural hybrid machine transliteration system for translating from a source language into a target language according to an embodiment of the present disclosure.

[0034] The system according to Fig. 1 comprises: an input processing unit (102) configured to capture the source word and retrieve its phoneme and graph sequence representation from a database (102a) consisting of a dataset of several named entities and English loanwords; a first encoding module (104) connected to the input processing unit (102) and configured to receive and encode the phoneme sequence representation of the source word to produce a first encoded representation, wherein the first encoding module (104) is configured to understand the phoneme sequence representation and produce an encoded representation of the entire sequence;a second encoding module (106) configured to receive and encode a grapheme sequence representation of the source word to produce a second encoded representation, wherein the second encoding module (106) is configured to understand the grapheme sequence representation and produce an encoded representation of the entire sequence; an aggregation module (108) configured to receive the first and second encoded representations and combine them to produce an aggregated representation; a decoder module (110) configured to receive the aggregated representation from the aggregation module (108) and generate a target word grapheme sequence in the target language based on the aggregated representation;and an output processing module (112) with a user interface (114), wherein the output processing module (112) is configured to receive the target word grapheme sequence in the target language and display the translated word on the user interface (114).

[0035] In one embodiment, the first encoder module (104) and the second encoder module (106) each use a recurrent neural network (RNN), wherein each RNN is configured to update a hidden state at each time step based on a previous hidden state and a current input, and wherein the first and second encoded representations are hidden states of the respective RNNs after processing the entire input sequences, wherein the use of multiple encoders with multiple input sequences, i.e., phoneme sequence and grapheme sequence of the input word in the source language, enables the system to utilize the properties of both the source phoneme and source grapheme representations in predicting the target grapheme.

[0036] In one embodiment, the RNN comprises at least one Long Short-Term Memory (LSTM) cell and / or one Gated Recurrent Unit (GRU) cell, wherein the architecture framework consists of at least one of the following encoder-decoder frameworks: a single-layer unidirectional RNN for each encoder with a single-layer RNN decoder, a two-layer stacked unidirectional RNN on each encoder combined with a single-layer RNN on the decoder, or a single-layer bidirectional RNN on each encoder combined with a single-layer RNN on the decoder.

[0037] In one embodiment, the aggregation module (108) is configured to aggregate the outputs of multi-source RNN-based encoders. The aggregation module (108) combines the first and second encoded representations using one of the following methods: a concatenation approach, in which the hidden states of the encoders are simply concatenated to initialize the hidden states of the decoder; an addition approach, in which the hidden states of each encoder are subjected to linear transformations with parameter matrices and then additively combined; and a convolution approach, in which a convolutional neural network (CNN) is used to aggregate the hidden states of the encoders. Applying one of these methods yields the aggregated encoder output, which is then fed to the decoder module.

[0038] In one embodiment, the aggregation module (108), which uses the convolution approach to aggregate the first and second coded representations, is configured to apply a convolution operation to a stacked matrix of the first and second coded representations, using multiple filters with a ReLU activation function to obtain a convolution output. A max-pooling layer performs max-pooling on the convolution output to obtain pooled vectors, and a fully connected dense layer generates the aggregated representation from the pooled vectors.

[0039] In one embodiment, the system (100) further comprises: a multi-encoder attention mechanism (116) configured to generate a first context vector from the first encoder and a second context vector from the second encoder in each decoder time step; associates the first and second context vectors with a corresponding decoder hidden state; and computes a final decoder state vector using a parameter matrix and a hyperbolic tangent activation function, wherein the attention weights are computed using the current decoder hidden state and all encoder states, and subsequently the context vector is computed as a weighted average of all encoder hidden states based on the attention weights, and wherein the decoder is further configured to generate the target word grapheme sequence based on the final decoder state vector.

[0040] In one embodiment, the first encoder module (104) and the second encoder module (106) are also configured to use a transformer architecture. Each transformer architecture comprises a stack of several identical layers, each layer including a multi-head attention sublayer and a positional feedforward neural network. Both the multi-head and feedforward sublayers are coupled to a normalization layer and a residual connection. The multi-head attention sublayer is configured to compute self-attention weights for each token within an input sequence. The decoder module (110) includes a modified transformer decoder configured to process the outputs of both the first encoder module (104) and the second encoder module (106).The decoder module has a similar architectural structure to the transformer-based encoder, with the transformer-based decoder featuring multiple stacked identical layers. Each layer consists of a multi-head attention circuit followed by a feedforward network, each coupled with a layer normalizer and a residual connection.

[0041] In one embodiment, the modified transformed decoder module comprises two multi-head attention sublayers within the decoder, consisting of a decoder-masked multi-head attention sublayer and an encoder-decoder attention sublayer. The decoder-masked multi-head attention layer considers previous predictions and masks future positions. In the encoder-decoder attention sublayer, keys and values ​​are derived from the encoder output, while the queries originate from the preceding decoder layer. This allows each position in the decoder to consider all positions in the input sequence.

[0042] In one embodiment of the transformation-based hybrid transliteration model, the two encoder modules encode the phoneme and grapheme sequence features separately. The architecture of the encoder modules is similar to that of RNN-based encoder modules, and the decoder module is modified to capture input from multiple sources. A serial model is configured to combine grapheme and phoneme input sequences in the transformation-based hybrid transliteration model, with serial attention computing encoder-decoder attention serially for each encoder.

[0043] In one embodiment, the system (100) further comprises: a grapheme-to-phoneme conversion module (118) with a transformer-based grapheme-to-phoneme conversion toolkit trained on an English pronunciation dictionary, wherein the grapheme-to-phoneme conversion module is configured to convert English source words into phoneme representations using the transformer-based grapheme-to-phoneme conversion toolkit.

[0044] In one embodiment, the input processing unit (102), the first encoder module (104), the second encoder module (106), the aggregation module (108), the decoder module (110), the output processing module (112) with user interface (114), the multi-encoder attention mechanism (116) and the grapheme-to-phoneme conversion module (118) can be implemented in programmable hardware devices such as processors, digital signal processors, central processing units, field-programmable gate arrays, programmable array logic, programmable logic devices, cloud processing systems or the like.

[0045] The present invention relates to a system for hybrid machine transliteration from a source language to a target language. It comprises a grapheme-phoneme conversion module, a first and a second encoder, an aggregation module, a decoder, and a computing device for executing a transliteration application. The system transforms the transliteration task as a sequence-to-sequence problem. The model consists of two key components: encoders and decoders. For modeling hybrid transliteration from multiple source languages, both architectures based on recurrent neural networks and transformer-based architectures are investigated.

[0046] In one embodiment, a hybrid transliteration system based on multi-source encoders and decoders is presented, which successfully integrates grapheme and phoneme features. The system is based on an RNN and transformer architecture and aggregates the outputs of multi-source RNN-based encoders using concatenation, addition, and convolution.

[0047] Fig. Figure 2 shows a schematic diagram of the proposed multi-source RNN-based encoder-decoder hybrid transliteration model according to an embodiment of the present disclosure.

[0048] Fig. Figure 2 shows a schematic diagram of the proposed hybrid transliteration model with multiple RNN sources and encoder-decoder. In this model, two different source sequences are received via two encoders (Encoder 1 and Encoder 2). The aggregation module then combines the outputs of the two encoders. The output of the aggregation module is then passed to the decoder module.

[0049] The proposed neural multi-source encoder-decoder model transforms the transliteration task into a sequence-to-sequence problem. The model consists of two key components: an encoder and a decoder.

[0050] In the multi-source RNN-based model, both the encoder and decoder are interconnected recurrent neural networks (RNNs). The encoder's task is to understand the input sequence—which represents either the grapheme or phoneme sequence of the source word—and generate an encoded representation of the entire sequence. At each time step, the hidden state of the encoder RNN is updated based on the previous hidden state and the current input. The update function is a nonlinear function that can be either a GRU or an LSTM network. The previous hidden state refers to the state of the previous time step. The encoded representation h is the hidden state of the RNN after reading the entire input sequence.

[0051] The encoded representation is then passed to the decoder RNN to generate the output sequence, which represents the grapheme sequence of the target word. The output sequence is determined sequentially by predicting the output symbol at each time using the current hidden state of the decoder. The prediction is performed using a softmax function applied to a transformation function, which generates a vector whose size corresponds to the number of graphemes in the target language.

[0052] The decoder's hidden state is updated based on the previous state, the encoded representation, and the previously predicted output symbol. The model's training goal is to minimize entropy loss, calculated over a set of transliteration pairs containing lists of source word sequences and their corresponding target word sequences.

[0053] The basic RNN-based encoder-decoder framework is extended with multiple encoders to enable the proposed model to learn from multiple input sequences. The multi-source encoder-decoder architecture was originally introduced for machine translation and is now being adapted for machine transliteration to integrate grapheme and phoneme features.

[0054] The proposed multi-source encoder-decoder model processes multiple input sequences. The phoneme and grapheme sequences of the input word in the source language are fed to two different encoders. The phoneme sequence is processed by a first encoder, and the grapheme sequence by a second encoder. The outputs of the two encoders are then combined using an aggregation function. The output of this function is then fed to the decoder to obtain the grapheme sequence of the target word.

[0055] The use of multi-encoders serves to leverage the properties of both the source phonemes and grapheme representations when predicting the target grapheme. The training data consists of tuples containing the grapheme sequence of the source word, the phoneme sequence of the source word, and the corresponding grapheme sequence of the target word. The objective function of the proposed model minimizes the cross-entropy loss calculated using this training dataset.

[0056] In the RNN architecture, the correct initialization of the decoder's hidden state affects overall performance. Therefore, in the present invention, three different aggregation techniques are implemented by the system, each technique taking into account the different significance of individual source sequences and comparing them with the proposed method.

[0057] The aggregation module combines the encoded representations generated by the first and second encoders. Since each input sequence, consisting of the phoneme and grapheme sequences of the source word, is processed by two separate encoders, the resulting representations differ. The grapheme representation is generally more robust, and the system considers the phoneme representation, especially when the pronunciation of the word deviates significantly from its spelling. Therefore, robust aggregation of the two representations is essential before the combined representation is sent to the decoder. Previous approaches do not explicitly model the role of the individual source sequences. Apart from the two aggregation methods, "Basic" and "Child-Sum," most studies focus on improving the multi-source attention mechanism.The system is configured such that, within the architecture of the recurrent neural network, the correct initialization of the decoder's hidden state influences overall performance. The system implements various aggregation methods that take into account the different significance of the individual source sequences.

[0058] The first and second encoded representations are generated by the first and second encoders, respectively. These two representations are combined using the aggregation functions implemented in the aggregation module. For frameworks based on Long Short-Term Memory (LSTM), the decoder cell states are added element-wise if the first and second cell states are the cell states of the first and second encoders, respectively, at the end of the input sequence. The aggregation module implements several of the aggregation methods described below.

[0059] Basics: In the first aggregation technique, a linear transformation using a parameter matrix is ​​applied to the composition of the first and second coded representations, followed by a hyperbolic tangent activation function.

[0060] Chaining: A second aggregation technique applies a hyperbolic tangent activation function to a simple chaining of the hidden states of the encoders to initialize the hidden state of the decoder.

[0061] Supplement: A third aggregation method is based on additive attention. In contrast to the first method, the hidden states of each encoder, consisting of the first and second encoded representations, are subjected to separate linear transformations. The first and second parameter matrices, respectively, are used to better account for the differences in the properties of the source sequences. The transformed representations are then additively combined.

[0062] Convolution: A fourth aggregation technique is based on Convolutional Neural Networks (CNNs). The aggregation module uses a CNN to aggregate the hidden states of the encoders. These states comprise the first and second coded representations, where the number of hidden encoder units is defined. The input to the CNN is a matrix created by stacking the first and second coded representations. The aggregation module then performs the following operations to obtain the aggregated encoder output: First, convolution is applied to the stacked matrix. This involves using multiple filters with the same padding option and an RLU activation function to obtain a convolutional output. Then, max-pooling with a fixed pool size is applied to the convolutional output to obtain pooled vectors.The system uses a predefined number of filters to obtain a corresponding number of convolution matrices and pooled vectors. Finally, a fully connected dense network is applied to the flattened, pooled vectors to obtain the aggregated encoder output, which serves as the decoder's initialized hidden state. This aggregated encoder output, obtained from one of the four methods, is then provided to the decoder component to generate the target language's transliteration sequence.

[0063] In one embodiment, the system includes a multi-encoder attention mechanism based on a global attention mechanism, configured to process the outputs of both source encoders concurrently. The multi-encoder attention mechanism employs a concatenation-based attention method for the RNN-based multi-source models. In each decoder time step, the two encoders generate a first and a second context vector. The multi-encoder attention mechanism concatenates the first and second context vectors with the corresponding decoder hidden state to compute a final decoder state vector using a parameter matrix and a hyperbolic tangent activation function. The computation of the first and second context vectors is performed for both encoders using the same procedure.The multi-encoder attention mechanism calculates the attention weights based on the current decoder hidden state and all encoder hidden states. The context vector is then calculated as a weighted average of all encoder hidden states based on the calculated attention weights, where each encoder hidden state is multiplied by its corresponding attention weight and summed over all positions in the source sequence.

[0064] Fig. Figure 3 shows a schematic diagram of the proposed multi-source transformer-based encoder-decoder hybrid transliteration model with serial attention combination.

[0065] The Transformer model uses positional embedding to encode word order in a word sequence such as a sentence without using recurrent layers. The encoder consists of several identical layers. Each layer contains a multi-head attention sublayer followed by a positionally forward-facing neural network. Both the multi-head and forward sublayers are coupled to a normalization layer and a residual connection.

[0066] The multi-head attention sublayer calculates self-attention weights for each token within a sequence, including the token itself. Self-attention links different positions within the input sequence. The attention mechanism processes query, key, and value matrices. The value matrix contains all value vectors of the sequence, each with dimension dv, while the query and key matrices contain key and query vectors, respectively, with dimension dk.

[0067] In this implementation, the multi-head attention computes self-attention h times, where h is the number of heads. The vectors representing the queries, keys, and values ​​are linearly transformed into h projections. Attention is computed independently in each head, and the outputs are then chained and projected back to the original dimension using parameter matrices. Each multi-head attention layer of the encoder uses the outputs of the previous layer as inputs to account for all positions of the previous layer. The decoder has a similar architecture to the encoder and consists of several identical layers. Each layer comprises a multi-head attention layer followed by a feedforward network, each coupled with layer normalization and residual connections.

[0068] The decoder contains two multi-head attention sublayers: first, a decoder-masked multi-head attention layer, and second, an encoder-decoder attention layer. The decoder-masked multi-head attention layer considers previous predictions and masks future positions. In the second decoder multi-head attention sublayer, the encoder-decoder attention layer, keys and values ​​come from the encoder output, while the queries originate from the previous decoder layer. This allows each position in the decoder to consider all positions in the input sequence.

[0069] The proposed transformer-based hybrid transliteration model is similar to the multi-source RNN-based model. It uses two encoders—a first and a second—to separately encode the features of phoneme and grapheme sequences. The architecture of each encoder corresponds to that of a conventional transformer-encoder. However, the decoder structure has been modified to process inputs from multiple sources. Various forms of multi-source transformer models have been proposed in the literature. Different variants, such as serial, parallel, flat, and hierarchical multi-source transformers, achieve comparable results. Based on these findings, the proposed hybrid model uses the serial model to combine grapheme and phoneme sequences. The serial attention combination computes the encoder-decoder attention serially for each encoder.The query set for the first encoder-decoder attention layer consists of the context vectors from the preceding decoder attention layer. However, the key and value sets are obtained from one of the source encoders, specifically the second encoder. The query set of the subsequent encoder-decoder attention layer is the output of the preceding sublayer, while the key-value pairs originate from the other encoder, specifically the first encoder. Similar to other sublayers, these encoder-decoder attention layers are interconnected via residual connections.

[0070] In one embodiment, we investigate the performance of the proposed hybrid models using English Manipuri transliteration in two different resource scenarios. A detailed description of the experimental setups follows.

[0071] The system uses a dataset containing 3402 named entities and 2633 English loanwords. It includes a preprocessing module that converts plural forms of named entities into their corresponding singular forms by removing morphological plurals. The processed dataset is randomly split into three subsets: a training, a development, and a test dataset. Additionally, the system includes a module to extend the test dataset, adding multiple spellings in the target language for each word in the source language. The extended test dataset contains an average of 3.21 reference transliterations for each English word. Furthermore, the system is configured to support extremely resource-constrained scenarios by randomly selecting 2000 transliterated word pairs as training data and 500 pairs as the development set from the original training data.

[0072] The system includes a toolkit for converting English graphemes into phonemes, configured to generate phoneme representations of source words for the transliteration model. The toolkit consists of a three-layer transformer model with 256 hidden units, trained on the CMU English pronunciation dictionary.

[0073] The system includes an evaluation module configured to measure performance using two standard evaluation metrics: word accuracy and character accuracy. The word accuracy module is configured to measure the correctness of the system's predicted transliterations by comparing them to reference transliterations in the test dataset. The character accuracy module is configured to measure the number of insertions, deletions, and substitutions of characters between the predicted transliterated word and the original transliteration, based on an edit distance algorithm.

[0074] The system comprises RNN-based encoder-decoder models configured with three different architectural frameworks: The first framework includes a single-layer unidirectional RNN for each encoder with a single-layer RNN decoder. The second framework consists of two layers of stacked unidirectional RNNs on each encoder, combined with a single-layer RNN on the decoder. The third framework consists of a single-layer bidirectional RNN on each encoder, combined with a single-layer RNN on the decoder. Each framework is configured to use either Long Short-Term Memory (LSTM) cells or Gated Recurrent Unit (GRU) cells. The RNN-based models are configured for optimization with the Adam optimizer, a learning rate of 0.001, and a batch size of 32. Both recurrent dropout and regular dropout are set to 0.2.The hidden layer of the RNN in the decoder consists of 512 units, and the embedding dimension is configured to 256.

[0075] The system comprises transformer-based models with a modified transformer architecture to implement the proposed hybrid transliteration model. The transformer-based models are optimized using an Adam optimizer with an initial learning rate of 0.2. The system is configured with Noam learning rate decay parameters and 4000 warm-up steps. The dropout value is set to 0.1. The multi-head attention layer is configured to use eight heads. The batch size is set to 32, and the hidden layer dimension in the feedforward layer is configured to 512. The system is configured to test models in two different model dimensions: 256 and 512. Furthermore, the system is configured to investigate three different settings for the encoder and decoder layers: 2-layer, 4-layer, and 6-layer configurations.

[0076] In one embodiment, the LSTM, GRU, and Transformer-based models are compared with other neural Seq2Seq models, namely RNNs and CNNs, using a graph EME-based transliteration setup with moderate data sizes. For the RNN, LSTM, and GRU, a single-layer encoder configuration is used for comparison. For the Transformer, a two-layer encoder and decoder configuration (model dimension = 256) is considered. The remaining parameters remain unchanged. In the case of the CNN, the Convolution Seq2Seq transliteration model is used. This employs a ten-layer encoder and decoder configuration with 512 hidden units, an embedding dimension of 256, and a kernel width of 3 per layer. The experimental results show that LSTM, GRU, and Transformer significantly outperform the RNN- and CNN-based models.

[0077] The proposed model is investigated on both RNN-based and transformer-based encoder-decoder frameworks. Furthermore, various aggregation methods are proposed to better combine incoming information from multiple sources in RNN-based models. These transliteration models are then compared with their phoneme- and grapheme-based counterparts. Finally, the performance of the proposed models is evaluated using the resource-poor English-Manipuri language pair for the transliteration of proper names and loanwords. The results from the various experimental setups clearly demonstrate that the multi-source encoder-decoder transliteration model can effectively integrate the features of phoneme and grapheme sequences and outperforms its phoneme- and grapheme-based counterparts.An evaluation to determine the effectiveness of the proposed model on four additional language pairs (English-Chinese, English-Thai, English-Persian and English-Hindi) to determine its generalizability showed that the proposed hybrid models outperformed their respective phoneme- and grapheme-based base models in all language pairs.

[0078] The drawings and the preceding description illustrate embodiments. Those skilled in the art will recognize that one or more of the described elements can be combined to form a single functional element. Alternatively, certain elements can be divided into several functional elements. Elements of one embodiment can be added to another. For example, the process flows described here can be modified and are not limited to the manner described herein. Furthermore, the actions of a flowchart need not be performed in the sequence shown; nor do all actions necessarily need to be carried out. Actions that do not depend on other actions can be performed in parallel with the other actions. The scope of protection of the embodiments is in no way limited by these specific examples. Numerous variations, whether explicitly stated in the description or not, such as...Differences in structure, dimensions, and materials are possible. The scope of protection of the embodiments is at least as comprehensive as described by the following claims.

[0079] The advantages, other benefits, and problem solutions have been described above with reference to specific embodiments. However, the advantages, benefits, problem solutions, and any components that can effect or enhance an advantage, benefit, or solution are not to be construed as critical, necessary, or essential features or components of the claims. REFERENCES 100 A Neural Hybrid Transliteration System for the Translation of a Source Language into a Target Language. 102 Input processing unit 102a Database 104 First encoder module 106 Second encoder module 108 Aggregation module 110 decoder module 112 Output processing module 114 User interface 116 Multi-encoder attention mechanism 118 Module for Converting Graphemes to Phonemes 202 Input of the phoneme string 204 Input grapheme string 206 Encoder 1 206a Encoder 2 208 Aggregation module 210 decoders 212 Output grapheme string 302 Input of the phoneme string 304 Input Embedding and Position Encoding 306 Input grapheme string 308 Output grapheme string 310 Softmax 312 Linear 314 decoders 316 Serial Number Attention Combination

Claims

A neural hybrid machine transliteration system for translating from a source language to a target language, comprising: an input processing unit configured to receive the source word and retrieve the phoneme and graph sequence representation of the source word from a database consisting of a dataset of multiple named entities and English loanwords; a first encoder module connected to the input processing unit and configured to receive and encode the phoneme sequence representation of the source word to produce a first coded representation, the first encoder being configured to understand the phoneme sequence representation and produce a coded representation of the entire sequence;a second encoding module configured to receive and encode a grapheme sequence representation of the source word to produce a second coded representation, wherein the second encoder is configured to understand the grapheme sequence representation and produce a coded representation of the entire sequence; an aggregation module configured to receive the first and second coded representations and combine them to produce an aggregated representation; a decoder module configured to receive the aggregated representation from the aggregation module and generate a target word grapheme sequence in the target language based on the aggregated representation; and an output processing module with a user interface, wherein the output processing module is configured to receive the grapheme sequence of the target word in the target language and display the translated word on the user interface. System according to claim 1, wherein the first and second encoder modules each use a recurrent neural network (RNN), wherein each RNN is configured to update a hidden state at each time step based on a previous hidden state and a current input, and wherein the first and second encoded representations are hidden states of the respective RNNs after processing the entire input sequences, wherein the use of multiple encoders with multiple input sequences, i.e., phoneme sequences and grapheme sequences of the input word in the source language, enables the system to utilize the properties of both the source phoneme and source grapheme representations in predicting the target grapheme. System according to claim 2, wherein the RNN comprises at least one Long Short-Term Memory (LSTM) cell and / or one Gated Recurrent Unit (GRU) cell, wherein the architectural framework consists of at least one of the following encoder-decoder frameworks: a single-layer unidirectional RNN for each encoder with a single-layer RNN decoder, a two-layer stacked unidirectional RNN on each encoder combined with a single-layer RNN on the decoder, or a single-layer bidirectional RNN on each encoder combined with a single-layer RNN on the decoder. System according to claim 1, wherein the aggregation module is configured to aggregate the outputs of multi-source RNN-based encoders, wherein the aggregation module is configured to combine the first and second coded representations by means of one of the following methods: a concatenation approach in which a simple concatenation of the hidden states of the encoders is performed to initialize the hidden states of the decoder; an addition approach in which the hidden states of each encoder are subjected to linear transformations with parameter matrices and subsequently combined additively; and a convolution approach in which a convolutional neural network (CNN) is used to aggregate the hidden states of the encoders, wherein by applying one of the aforementioned methods the aggregated encoder output is obtained and subsequently fed to the decoder module. System according to claim 4, wherein the aggregation module, which uses the convolution approach to aggregate the first and second coded representations, is configured to apply a convolution operation to a stacked matrix of the first and second coded representations using a plurality of filters with a ReLU activation function to obtain a convolution output, wherein a max-pooling layer applies max-pooling to the convolution output to obtain pooled vectors, and wherein a fully connected dense layer generates the aggregated representation from the pooled vectors. System according to claim 1, further comprising: a multi-encoder attention mechanism configured to generate a first context vector from the first encoder and a second context vector from the second encoder in each decoder time step; associates the first and second context vectors with a corresponding decoder hidden state; and computes a final decoder state vector using a parameter matrix and a hyperbolic tangent activation function, wherein the attention weights are computed using the current decoder hidden state and all encoder states, and subsequently the context vector is computed as a weighted average of all encoder hidden states based on the attention weights, and wherein the decoder is further configured to generate the target word grapheme sequence based on the final decoder state vector. System according to claim 1, wherein the first and second encoder modules are also configured to utilize a transformer architecture, each transformer architecture comprising a stack of several identical layers, each layer comprising a multi-head attention sublayer and a positional feedforward neural network, both the multi-head and feedforward sublayers being coupled with a normalization layer and a residual connection, the multi-head attention sublayer being configured to compute self-attention weights for each token within an input sequence, and the decoder module comprising a modified transformer decoder configured to process outputs from both the first and second encoder modules, the decoder module having the same architectural structure as the transformer-architecture encoder.wherein the decoder with transformer architecture has several stacked identical layers, each layer consisting of a multi-head attention followed by a feedforward network, each coupled with a layer normalization and a residual connection. System according to claim 7, wherein the modified transformed decoder module comprises two multi-head attention sublayers in the decoder, including: a decoder-masked multi-head attention sublayer and an encoder-decoder attention sublayer, wherein the decoder-masked multi-head attention layer takes into account the previous predictions and masks the future positions, wherein in the encoder-decoder attention sublayer, keys and values ​​are taken from the output of the encoder, while the requests are taken from the previous decoder layer, whereby each position in the decoder can take into account all positions in the input sequence. System according to claim 7, wherein in the transformation-based hybrid transliteration model the two encoder modules encode the phoneme and grapheme sequence features separately, wherein the architecture of the encoder modules is similar to the RNN-based encoder modules and the decoder module is modified to be able to capture inputs from multiple sources, wherein a serial model is configured to combine grapheme and phoneme input sequences in the transformation-based hybrid transliteration model, wherein the serial attention computes the encoder-decoder attention serially for each encoder. System according to claim 1, further comprising a grapheme-to-phoneme conversion module with a transformer-based grapheme-to-phoneme conversion toolkit trained on an English pronunciation dictionary, wherein the grapheme-to-phoneme conversion module is configured to convert English source words into phoneme representations using the transformer-based grapheme-to-phoneme conversion toolkit.

Images