A method and system for multilingual multi-domain neural machine translation of an adapter

Abstract

This invention relates to an adapter-based multilingual, multi-domain neural machine translation method and system, belonging to the field of natural language processing technology. Firstly, in the encoder-decoder neural machine translation model, a language adapter and a domain adapter are inserted internally to mine linguistic and domain knowledge, respectively, thereby decoupling the linguistic and domain feature spaces and enabling cross-language sharing of domain knowledge. Through cross-language sharing of target domain knowledge, for a given target language pair, even without labeled corpus in the target domain, the model can still achieve high translation quality in the target domain. This invention achieves multilingual sharing of target domain knowledge through cross-language transfer, improving the translation performance of the neural machine translation model in the target domain and reducing the difficulty of deployment and implementation.

Description

Technical Field

[0001] This invention relates to an adapter-based multilingual and multi-domain neural machine translation method and system, belonging to the field of natural language processing technology. Background Technology

[0002] Machine translation is an important branch of natural language processing (NLP) technology. Most existing machine translation methods are built on an encoder-decoder model structure, utilizing end-to-end learning methods. The source language sentence and its corresponding target language sentence are directly input into the model, which then learns the correspondence. However, when the text to be translated differs in style from the model's training corpus, the performance of neural machine translation models often falls short. In such applications, domain-adaptive methods are needed to improve the model's translation quality within the target domain.

[0003] For the domain adaptation problem, the mainstream and effective approach is fine-tuning, which involves using existing, limited target domain data to train the translation model and improve its performance in that domain. However, parallel corpora in the target domain are extremely scarce in most scenarios, and collecting such corpora is time-consuming and expensive. In some resource-rich language pairs, a certain amount of target domain parallel corpus exists; these are called auxiliary language pairs. Therefore, mining and transferring domain-specific knowledge from auxiliary language pairs to improve the translation quality of target language pairs in specific target domains is essential to address the resource scarcity problem in current domain adaptation tasks. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and address the problems of low translation quality in specific domains and the scarcity of parallel corpora in specific domains in current neural machine translation methods. This invention creatively proposes an adapter-based multilingual, multi-domain neural machine translation method and system. This invention utilizes an adapter to decouple language and domain feature spaces, and improves the translation performance of neural machine translation models in the target domain through cross-language transfer of domain knowledge.

[0005] The innovation of this invention lies in its proposal of a cross-language knowledge sharing method combining a language adapter and a domain adapter, specifically for application scenarios where the target model is a multilingual, multi-domain translation model. First, within the encoder-decoder neural machine translation model, a language adapter and a domain adapter are inserted to mine linguistic and domain knowledge, respectively, thereby decoupling the linguistic and domain feature spaces and achieving cross-language sharing of domain knowledge. Through this cross-language sharing of target domain knowledge, for a given target language pair, even without labeled corpus in the target domain, the model can still achieve high translation quality in the target domain.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] An adapter-based multilingual, multi-domain neural machine translation method includes the following steps:

[0008] Step 1: Perform data preprocessing.

[0009] First, different word segmentation tools are used to segment the source language corpus and the target language corpus, respectively, according to different languages.

[0010] Then, Byte Pair Encoding (BPE) is used to encode the segmented training corpus and obtain a vocabulary.

[0011] Finally, the corpus is indexed, replacing words in the corpus with the indexes from the vocabulary, which means processing the text into an index to facilitate computer processing.

[0012] Step 2: Use an encoder to encode the source sentence into an intermediate representation.

[0013] Specifically, it includes the following steps:

[0014] Step 2.1: The source language sequence is mapped through the embedding layer to obtain the source language sequence embedding representation.

[0015] Specifically, it includes the following steps:

[0016] Step 2.1.1: The source language sequence is processed through word embeddings to obtain word embedding representation vectors.

[0017] Step 2.1.2: The source language sequence is processed through position embeddings to obtain position embedding representation vectors.

[0018] Step 2.1.3: Add the word embedding representation vector and the position embedding representation vector to obtain the source sentence embedding representation.

[0019] Step 2.2: The source language sequence embedding representation is input into the encoder to calculate the context representation C of the source language sequence.

[0020] Specifically, it includes the following steps:

[0021] Step 2.2.1: The multi-head self-attention layer processes the source language sequence embedding representation and calculates the output Self-Attn. i

[0022] Step 2.2.2: Self_Attn iThe input is fed into the feedforward network layer, and the output Enc_FFN is calculated. i

[0023] Step 2.2.3: Enc_FFN i The input is fed into the source language's corresponding language adapter and a shared language adapter. The weighted sum of the two adapters yields the current output, Enc_. THE i

[0024] Step 2.2.4: Enc_LA i The input is fed into the domain adapter corresponding to the source sentence and the shared domain adapter. The weighted sum of the two adapters yields the current output Enc_DA. i

[0025] Step 2.2.5: Enc_DA i The input is fed into the next sub-layer of the Transformer model encoder, and steps 2.2.1 to 2.2.4 are repeated for the calculation process.

[0026] Step 2.2.6: After calculation by N sub-layers of the Transformer encoder, the context representation C of the source language sequence is obtained.

[0027] Step 3: Decode the context representation C of the source language sentence into the target language using a decoder.

[0028] Specifically, it includes the following steps:

[0029] Step 3.1: The decoder has generated the target language sequence and mapped it through the embedding layer to obtain the target language sequence embedding representation.

[0030] Specifically, it includes the following steps:

[0031] Step 3.1.1: The generated target language sequence is passed through the word embedding layer to obtain the word embedding representation vector.

[0032] Step 3.1.2: The generated target language sequence is passed through the position embedding layer to obtain the position embedding representation vector.

[0033] Step 3.1.3: Add the word embedding representation vector and the position embedding representation vector to obtain the generated target language sequence embedding representation.

[0034] Step 3.2: Input the target language sequence embedding representation and context representation C into the decoder to generate the word at the current time step.

[0035] Specifically, it includes the following steps:

[0036] Step 3.2.1: The target language sequence embedding representation is processed by a masked multi-head self-attention layer, and the output Mask_Attn is calculated. i .

[0037] Step 3.2.2: Mask_Attn i The input is fed into the encoder-decoder attention layer, and the output EncDec_Attn is calculated. i .

[0038] Step 3.2.3: EncDec_Attn i The input is fed into the feedforward network layer, and the output Dec_FFN is calculated. i .

[0039] Step 3.2.4: Dec_FFN i The input is fed into the target language's corresponding language adapter and a shared language adapter. The weighted sum of the two adapters yields the current output, Dec_LA. i .

[0040] Step 3.2.5: Dec_LA i The input is fed into the domain adapter corresponding to the target language sentence and the shared domain adapter. The weighted sum of the two adapters yields the current output Dec_DA. i .

[0041] Step 3.2.6: Dec_DA i The input is fed into the next sub-layer of the Transformer model decoder, and the calculation process from steps 3.2.1 to 3.2.6 is repeated.

[0042] Step 3.2.7: After calculation by N sub-layers of the Transformer decoder, an intermediate representation is obtained. This representation is then input into the generation layer to obtain the probability distribution of candidate words. Based on the probability distribution, the word for the current time step is generated.

[0043] Step 3.3: Repeat steps 3.1 and 3.2 until the decoder generates the termination tag, i.e., "[EOS]" (End of Sentence).

[0044] Beneficial effects

[0045] Compared with existing technologies, this invention realizes domain knowledge sharing in multilingual translation models, proposes a combination of language adapters and domain adapters to decouple language and domain feature spaces, realize multilingual sharing of target domain knowledge, and improve the translation quality of the model in multiple languages ​​to the target domain.

[0046] Experiments on WMT news corpora and TED talk corpora demonstrate the effectiveness of the current method, significantly improving model performance in specific domains. Furthermore, compared to global fine-tuning methods that maintain a model for each task, this method greatly reduces model maintenance costs and lowers deployment difficulty in resource-constrained application scenarios. Brief Description of the Drawings

[0047] Figure 1 is the flowchart of the method according to an embodiment of the present invention;

[0048] Figure 2 is the schematic diagram of the system architecture according to an embodiment of the present invention. Detailed Description of the Preferred Embodiments

[0049] The present invention will be further described in detail below with reference to the accompanying drawings.

[0050] Embodiment

[0051] Taking English-Chinese translation as an example, it includes an English sentence (source language) and a Chinese sentence (target language):

[0052] English sentence: I am a student.

[0053] Chinese sentence: 我是学生。

[0054] As Figure 1 shown, a multi-language and multi-domain neural machine translation method based on an adapter includes the following steps:

[0055] Step 1: Data preprocessing.

[0056] Step 1.1: According to different languages, different word segmentation tools are used to segment the source language corpus and the target language corpus respectively.

[0057] For English sentences, word segmentation usually separates punctuation marks from words. For example, for the sentence "I am a student.", after word segmentation, it is "I am a student.".

[0058] For Chinese sentences, the Chinese word segmentation tool Jieba (https: / / github.com / fxsjy / jieba) can be used to segment Chinese words. For example, for the sentence "我是学生。", after word segmentation, it is "我-是-学生-。". Here, for the convenience of expression, "-" represents a space.

[0059] Step 1.2: Use Byte Pair Encoding (BPE) to encode the segmented training corpus and obtain a vocabulary.

[0060] Encoding using BPE essentially divides words into finer-grained subwords. Taking the English word "playing" as an example, the BPE algorithm divides it into "play" and "ing". After performing BPE processing on the entire corpus, a vocabulary is obtained. Then, for each sample E, the sentence is split according to the vocabulary obtained by BPE encoding.

[0061] Step 1.3: Corpus indexing, which replaces words in the corpus with the indexes in the vocabulary, that is, processing the text into an index to facilitate computer processing.

[0062] In the vocabulary, each subword has a corresponding index, such as "the" having an index of "9". For each subword sequence, the subwords in the sequence are replaced with their corresponding indices to generate an index sequence, which facilitates further processing by the model.

[0063] For example, for the English sentence "I am a student", if the word index sequence corresponding to the subwords in the sequence is "20,915,68,6250", then this index sequence will be used as the input of the model.

[0064] Step 2: Encode the source sentence into an intermediate representation using an encoder.

[0065] Specifically, it includes the following steps:

[0066] Step 2.1: The source language sequence is mapped through the embedding layer to obtain the source language sequence embedding representation.

[0067] Step 2.1.1: The source language sequence is passed through the word embedding layer to obtain the word embedding representation vector.

[0068] Specifically, the source language sequence, after being indexed, is input into the word embedding layer. This layer is implemented as a lookup table. Let the vocabulary length be |V| and the model dimension be dim. This lookup table is essentially a two-dimensional matrix of |V|×dim. By inputting the current indexed sequence, the word vector representation of that sequence can be obtained. After the above steps, the source language sequence is encoded into an n×dim continuous vector representation, where n is the sequence length.

[0069] Step 2.1.2: The source language sequence is passed through the position embedding layer to obtain the position embedding representation vector.

[0070] Similar to the implementation of the word embedding layer, we first define the maximum sequence length N, then define an N×dim two-dimensional parameter matrix, and obtain the corresponding positional encoding by indexing. This parameter matrix, along with other parameters in the model, is optimized using the gradient descent algorithm. After the above steps, the positional information of the source language sequence is encoded into an n×dim continuous vector representation, where n is the sequence length.

[0071] Step 2.1.3: Add the word embedding representation vector and the position embedding representation vector to obtain the source sentence embedding representation.

[0072] The word embedding representation vector and the position embedding representation vector are added together to obtain an n×dim continuous vector representation, which is then passed to the next layer of the model for further feature extraction and calculation.

[0073] Step 2.2: The source language sequence embedding representation is input into the encoder to calculate the context representation C of the source language sequence.

[0074] Step 2.2.1: The multi-head self-attention layer processes the source language sequence embedding representation and calculates the output Self_Attn. i .

[0075] First, each word vector in the source language sequence is mapped to a query value Q, a key value K, and a value V using a matrix:

[0076] Q = W Q ·e (1)

[0077] K = W K ·e (2)

[0078] V = W V ·e (3)

[0079] Among them, W Q W K W V represents the query value Q, key value K, and value V, respectively; e represents the input word vector.

[0080] Then, for each word, its attention score *s* relative to all other words is calculated using a self-attention mechanism:

[0081]

[0082] Where, d k Let K be the dimension. T represents the transpose.

[0083] Then, using the attention score as the weight, the corresponding attention score is multiplied by V and summed to obtain the corresponding hidden layer state representation vector.

[0084]

[0085] Where s represents the attention score.

[0086] Here, the representations generated by each attention head are concatenated:

[0087]

[0088] Among them, h ′ This represents the concatenated hidden layer state representation vector. This is represented as the representation generated by the nth head.

[0089] Finally, two linear layers with ReLU activation function are used:

[0090] h = FFN(h) ′ ) = max(0, h ′ W1+b1)W2+b2 (7) where FFN(·) represents a linear unit, W1 and W2 represent the training weights, and b1 and b2 represent the bias terms corresponding to W1 and W2, respectively. This yields the encoding sequence of the context sentence, where h is the hidden layer representation of the i-th layer of the model, Self_Attn. i .

[0091] The source language sentence is encoded using a self-attention layer as described above.

[0092] Step 2.2.2: Self_Attn i The input is fed into the feedforward network layer, and the output Enc_FFN is calculated. i :

[0093] Enc_FFN i =FFN(Self_Attn) i (8)

[0094] Step 2.2.3: Enc_FFN i The input is fed into the source language's corresponding language adapter and the shared language adapter. The weighted sum of the two adapters yields the current output, Enc_LA. i :

[0095]

[0096] Among them, LA i Represents language L in the i-th layer of the Transformer j adapter Shared adapter with The weighted sum, where λ is used as a weighting coefficient to control the proportion of a specific language adapter. Further simplified to: Enc_LA i =LA i .

[0097] Step 2.2.4: Enc_LA i The input is fed into the domain adapter corresponding to the source sentence and the shared domain adapter. The weighted sum of the two adapters yields the current output Enc_DA. i :

[0098]

[0099] Among them, DA i Represents the domain D in the i-th layer of the Transformer. k adapter Shared adapter with The weighted sum of , where μ is the weight coefficient. Further simplified to: Enc_DA i =DA i .

[0100] Step 2.2.5: Enc_DA i Input into the next sub-layer of the Transformer model encoder and repeat steps 2.2.1 to 2.2.4 of the calculation process.

[0101] Step 2.2.6: After calculation by N sub-layers of the Transformer encoder, the context representation C of the source language sequence is obtained.

[0102] Step 3: Decode the context representation C of the source language sentence into the target language using a decoder.

[0103] Specifically, it includes the following steps:

[0104] Step 3.1: The decoder has generated the target language sequence and mapped it through the embedding layer to obtain the target language sequence embedding representation.

[0105] Step 3.1.1: The generated target language sequence is passed through the word embedding layer to obtain the word embedding representation vector.

[0106] Specifically, the source language sequence, after being indexed, is input into the word embedding layer. This layer is implemented as a lookup table. Let the vocabulary length be |V| and the model dimension be dim. This lookup table is essentially a two-dimensional matrix of |V|×dim. By inputting the current indexed sequence, the word vector representation of that sequence can be obtained. After the above steps, the source language sequence is encoded into an n×dim continuous vector representation, where n is the sequence length.

[0107] Step 3.1.2: The generated target language sequence is passed through the position embedding layer to obtain the position embedding representation vector.

[0108] Similar to the implementation of the word embedding layer, we first define the maximum sequence length N, then define an N×dim two-dimensional parameter matrix, and obtain the corresponding positional encoding by indexing. This parameter matrix, along with other parameters in the model, is optimized using the gradient descent algorithm. After the above steps, the positional information of the source language sequence is encoded into an n×dim continuous vector representation, where n is the sequence length.

[0109] Step 3.1.3: Add the word embedding representation vector and the position embedding representation vector to obtain the generated target language sequence embedding representation.

[0110] Step 3.2: Input the target language sequence embedding representation and context representation C into the decoder to generate the word at the current time step.

[0111] Step 3.2.1: The target language sequence embedding representation is processed by a masked multi-head self-attention layer, and the output Mask_Attn is calculated. i .

[0112] The self-attention calculation mechanism of the decoder is similar to that of the encoder, and its specific calculation process is shown in formulas 1 to 7.

[0113] Step 3.2.2: Mask_Attn i The input is fed into the encoder-decoder attention layer, and the output EncDec_Attn is calculated. i .

[0114] Step 3.2.3: EncDec_Attn i The input is fed into the feedforward network layer, and the output Dec_FFN is calculated. i .

[0115] The decoding mechanism in this step is similar to that of the encoder.

[0116] Step 3.2.4: Dec_FFN i The input is fed into the target language's corresponding language adapter and a shared language adapter. The weighted sum of the two adapters yields the current output, Dec_LA. i .

[0117] The decoding mechanism in this step is similar to that of the encoder.

[0118] Step 3.2.5: Dec_LA i The input is fed into the domain adapter corresponding to the target language sentence and the shared domain adapter. The weighted sum of the two adapters yields the current output Dec_DA. i .

[0119] The decoding mechanism in this step is similar to that of the encoder.

[0120] Step 3.2.6: Dec_DA i The input is fed into the next sub-layer of the Transformer model decoder, and the calculation process from steps 3.2.1 to 3.2.6 is repeated.

[0121] Step 3.2.7: After calculation by N sub-layers of the Transformer decoder, an intermediate representation is obtained. This representation is then input into the generation layer to obtain the probability distribution of candidate words. Based on the probability distribution, the word for the current time step is generated.

[0122] Step 3.3: Repeat steps 3.1 and 3.2 until the decoder generates the termination tag "[EOS]" (End of Sentence).

[0123] This invention decouples language and domain space by combining domain and language adapters, thereby enabling cross-language sharing of domain knowledge. This significantly improves the model's translation quality in the target domain without affecting translation performance in other domains. Compared to the baseline model, this method achieves a 4.58 BLEU improvement on the TED spoken language test set, enhancing the model's translation quality in the spoken language domain.

[0124] On the other hand, this invention proposes an adapter-based multilingual, multi-domain neural machine translation system, including an embedding layer, an encoder module, a decoder module, and a generator module. For example... Figure 2 As shown.

[0125] The embedding layer converts words into corresponding word vector sequences, including word embedding layer and position embedding layer.

[0126] Word embedding is a word representation method that is jointly learned with neural network models in specific natural language processing tasks (such as language modeling and machine translation). It's important to note that, for ease of explanation, this layer is referred to as the word embedding layer; however, the units encoded in this layer are not strictly words, but rather sub-word tokens processed by Word Embedding (BPE). Essentially, word embedding is a technique that maps individual tokens to vector representations. Each token is represented by a real-valued vector, typically with hundreds (or thousands) of dimensions.

[0127] Embedding layers are typically implemented as lookup tables. Let the vocabulary length be |V| and the model dimension be dim. This lookup table is essentially a two-dimensional matrix of |V|×dim. By inputting the index of the current token, its specified word vector representation can be obtained. In natural language processing tasks, the model input is usually a discrete sequence of length n. The word embedding layer looks up the representation of each token in the sequence based on its index. The discrete sequence is then encoded into a continuous vector representation of n×dim, which is passed to the next layer of the model for further feature extraction and computation.

[0128] Corresponding to word embedding layers, positional embeddings encode the positional information of the input sequence. Word position and order are fundamental components of language, encompassing both the basic grammar of a sentence and defining its actual semantics. Recurrent neural networks inherently possess temporal dependencies, naturally considering sentence position and order information. However, to better support parallel computation, the Transformer abandons temporal dependency mechanisms and instead extracts sentence features through self-attention. But during Transformer computation, each word in the sequence is processed in parallel by the encoder; the model itself is unaware of the word position and order. Therefore, the Transformer needs a way to encode the positional information of words in the sequence into the model.

[0129] The encoder module encodes the source language to obtain a contextual representation of the source language sentence, which serves as part of the decoder input. The encoder consists of N layers of encoder components with the same structure, including multi-head self-attention layers and feedforward layers.

[0130] Among them, the multi-head attention layer is responsible for fusing information from the input word vectors or semantic vectors to generate corresponding semantic information; the feedforward layer is responsible for linearly mapping the input encoded information.

[0131] The decoder module decodes the obtained encoding information to generate decoded information.

[0132] The decoder module consists of multiple decoder layers stacked together, each layer containing a mask self-attention layer, an encoder-decoder attention layer, and a feedforward layer.

[0133] Among them, the masked self-attention layer is responsible for information fusion of word vectors of the generated translation; the encoder-decoder attention layer helps the decoder pay more attention to the representation of relevant words in the input sequence during the decoding process, which is consistent with human translation cognition and helps the model generate more accurate translations; the feedforward layer is responsible for linear mapping of the encoded information of the input.

[0134] The generator module is responsible for mapping the decoded information to the target vocabulary, generating the translation, and presenting the result to the user.

[0135] The connection relationships of the above components are as follows:

[0136] The embedding layer is responsible for mapping words to corresponding word vector representations, which serve as input to the encoder / decoder. The encoder module receives the input from the embedding layer, performs calculations, and outputs a contextual representation of the source language sequence. Then, the encoded sequence is input to the decoder module; the decoder module outputs a decoded sequence; and the generator module generates the translated sentence based on the decoded sequence.

Claims

1. An adapter-based multi-language multi-domain neural machine translation method, characterized in that, The method comprises the following steps: Step 1: data preprocessing; First, according to different languages, different word segmentation tools are used to segment the source language corpus and the target language corpus respectively; Then, the segmented training corpus is encoded by using byte pair encoding to obtain a word table; Finally, the corpus is indexed, and the index in the word table is used to replace the words in the corpus to process the text into an index; Step 2: encode the source sentence into an intermediate representation using an encoder; Step 2.1: the source language sequence is mapped through an embedding layer to obtain a source language sequence embedding representation; Step 2.1.1: the source language sequence is mapped through a word embedding layer to obtain a word embedding representation vector; After the source language sequence is indexed, it is input into the word embedding layer; the layer implementation is a lookup table, assuming that the length of the word table is |V| and the model dimension is dim, the lookup table is essentially a |V|xdim two-dimensional matrix, which obtains the word vector representation of the current index sequence by inputting the index sequence; After the above steps, the source language sequence is encoded into an nxdim continuous vector representation, where n is the sequence length; Step 2.1.2: the source language sequence is mapped through a position embedding layer to obtain a position embedding representation vector; First, define the maximum sequence length N, then define an Nxdim two-dimensional parameter matrix, and obtain the corresponding position encoding through indexing; the parameter matrix is optimized together with other parameters in the model through the gradient descent algorithm; after the above steps, the position information of the source language sequence is encoded into an nxdim continuous vector representation, where n is the sequence length; Step 2.1.3: add the word embedding representation vector and the position embedding representation vector to obtain the source sentence embedding representation; The word embedding representation vector and the position embedding representation vector are added to obtain an nxdim continuous vector representation, which is input into the next layer of the model for further feature extraction and calculation; Step 2.2: input the source language sequence embedding representation into the encoder to calculate the context representation C of the source language sequence; Step 2.2.1: Multi-head self-attention layer processes source language sequence embedding representation, computes output Self-Attn i ; First, map each word vector of the source language sequence to a query value Q, a key value K and a value V through a matrix: Q = W Q • e (1) K = W K • e (2) V = W V • e (3) where W Q , W K , and W V represent the query value Q, the key value K, and the value V, respectively; e represents an input word vector; Then, for each word, the attention score s of all other words is calculated through a self-attention mechanism: where d k is the dimension of K; T denotes the transpose; Then, the attention score is used as a weight, and the corresponding attention score is multiplied by V and summed up to obtain the corresponding hidden layer state representation vector Where s represents the attention score; Here, the representation generated by each attention head is concatenated: where h' denotes the concatenated hidden layer state representation vector, denotes the representation produced by the nth head. Finally, pass through two linear layers with ReLU as the activation function: h=FFN(h′)=max(0,h′W1+b1)W2+b2 (7) Wherein, FFN(·) represents a linear unit, W1, W2 represent trained weights, b1, b2 represent bias terms corresponding to W1, W2 respectively; thus obtaining the encoding sequence of the context sentence, h is the hidden layer representation of the i-th layer of the model Self_Attn i ; In the above manner, the source language sentence is encoded through the self-attention layer; Step 2.2.2: Self_Attn i input to the feed forward network layer, the output Enc_FFN i is computed Enc_FFN i = FFN(Self_Attn i ) (8) Step 2.2.3: Enc_FFN i Input to source language corresponding language adapter and shared language adapter, both adapters are weighted summed to get current output Enc_LA i : wherein, LA i represents the adapter for language L j in the i-th layer of the Transformer and its shared adapter with a weighted sum, λ as a weight coefficient to control the proportion of a specific language adapter; further simplified as: Enc_LA i = LA i ; Step 2.2.4: Enc_LA i Input to the domain adapter corresponding to the source sentence and the shared domain adapter, the weighted sum of the two adapters gives the current output Enc DA i : where DA i represents the adapter of the i-th layer of the Transformer in the domain D k of the i-th layer of the Transformer in the domain D and the weighted sum of the adapters that share it, where μ is a weight coefficient; further simplified as: Enc_DA = DA i = DA i ; Step 2.2.5: Enc DA i The input to the next sublayer of the Transformer model encoder, the process of steps 2.2.1 to 2.2.4 is repeated; Step 2.2.6: after the calculation of the N sub-layers of the Transformer encoder, the context representation C of the source language sequence is obtained; Step 3: decode the context representation C of the source language sentence into the target language through a decoder; Step 3.1: the generated target language sequence is mapped through an embedding layer to obtain a target language sequence embedding representation; Step 3.1.1: the generated target language sequence is mapped through a word embedding layer to obtain a word embedding representation vector; After the source language sequence is indexed, it is input into the word embedding layer; the layer is implemented as a lookup table, with the length of the vocabulary table being |V| and the model dimension being dim. The lookup table is essentially a |V|xdim two-dimensional matrix, and the word vector representation of the current index sequence can be obtained by inputting the current index sequence. After the above steps, the source language sequence is encoded into a continuous vector representation of nxdim, where n is the sequence length. Step 3.1.2: The generated target language sequence passes through the position embedding layer to obtain a position embedding representation vector. First, define the maximum sequence length N, then define a two-dimensional parameter matrix of Nxdim, and obtain the corresponding position encoding through indexing. This parameter matrix is optimized together with other parameters in the model through the gradient descent algorithm. After the above steps, the position information of the source language sequence is encoded into a continuous vector representation of nxdim, where n is the sequence length. Step 3.1.3: Add the word embedding representation vector and the position embedding representation vector to obtain the embedding representation of the generated target language sequence. Step 3.2: Input the target language sequence embedding representation and the context representation C into the decoder to generate the word at the current time step. Step 3.2.1: Masked multi-head self-attention layer processes the target language sequence embedding representation, computing an output Mask_Attn i ; Step 3.2.2: Mask_Attn i Input to the encoder-decoder attention layer, the output EncDec_Attn is computed i ; Step 3.2.3: EncDec_Attn i Input to the feedforward network layer, compute output Dec_FFN i ; Step 3.2.4: Dec_FFN i The input is sent to the language adapter corresponding to the target language and to the shared language adapter. The two adapters are summed with weights to obtain the current output Dec_LA i ; Step 3.2.5: Dec_LA i The input is passed to the domain adapter corresponding to the target language sentence and the shared domain adapter. The two adapters are weighted and summed to get the current output Dec_DA i ; Step 3.2.6: Dec DA i Input to the next sublayer of the Transformer model decoder, repeat the process of Step 3.2.1 through Step 3.2.6; Step 3.2.7: After the calculation of the N sub-layers of the Transformer decoder, the intermediate representation is obtained, which is input into the generation layer to obtain the candidate word probability distribution, and the word at the current time step is generated according to the probability distribution. Step 3.3: Repeat steps 3.1 and 3.2 until the decoder generates a termination tag.

2. A multi-language multi-domain neural machine translation system implementing the adapter of the method of claim 1, characterized in that, It includes an embedding layer, an encoder module, a decoder module, and a generator module. The embedding layer converts words into corresponding word vector sequences, including a word embedding layer and a position embedding layer. The word embedding layer looks up the representation of each token in the sequence according to its index. The discrete sequence is encoded into a continuous vector representation of nxdim. The model dimension is dim, and n is the sequence length. The input is passed to the next layer of the model for further feature extraction and calculation. Position embedding is an encoding of the position information of the input sequence. The encoder module is used to encode the source language to obtain the context representation of the source language sentence as part of the decoder input. The encoder is composed of N layers of encoder components with the same structure, including a multi-head self-attention layer and a feedforward layer. The multi-head attention layer is responsible for information fusion on the input word vector or semantic vector to generate corresponding semantic information. The feedforward layer is responsible for linear mapping of the input encoding information. The decoder module decodes the obtained encoding information to generate decoding information. The decoder module is composed of multiple decoder layers, each containing a masked self-attention layer, an encoder-decoder attention layer, and a feedforward layer. The masked self-attention layer is responsible for information fusion on the word vector of the generated translation. The encoder-decoder attention layer helps the decoder focus more on the representation of related words in the input sequence during decoding. This process is consistent with the cognitive process of human translation, which helps the model generate more accurate translations. The feedforward layer is responsible for linear mapping of the input encoding information. The generator module is responsible for mapping the decoding information into the target language vocabulary to generate the translation and present the result to the user. The connection relationship of the above components is as follows: The embedding layer is responsible for mapping words into corresponding word vector representations, serving as the input of the encoder / decoder; the encoder module receives the input of the embedding layer, and outputs the context representation of the source language sequence after calculation; then, the encoded sequence is input into the decoder module; the decoder module outputs the decoding sequence; the generator module generates the translation sentence according to the decoding sequence.

Images