Multi-source machine translation method and apparatus
By adding additional parameters to the encoder and decoder in the multi-source machine translation model and combining it with transfer learning, the problems of high training cost and poor performance of multi-source machine translation are solved, and efficient translation is achieved under the condition of zero multi-parallel data.
Patent Information
- Application Number
- CN202211612530.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-14
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-12-14
AI Technical Summary
Existing multi-source machine translation methods have high training costs and poor training results, especially under conditions of zero or multiple parallel data samples.
By adding encoder additional parameters and decoder additional prefix parameters to the multi-source machine translation model, fixing other parameters, and training only these additional parameters, combined with transfer learning methods, the model is trained using a bilingual parallel dataset with single-source annotations.
It effectively reduces the number of training parameters, lowers training costs, and improves translation performance and keyword recall under conditions of zero multivariate parallel data.
Smart Images

Figure CN116187352B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine translation technology, and in particular to a multi-source machine translation method and apparatus. Background Technology
[0002] Traditional machine translation tasks translate a single source sentence into a single target sentence, completing the translation task from one source language to one target language. Multi-source machine translation, on the other hand, aims to utilize multiple synonymous source sentences to translate into a single target sentence, thus completing translation tasks from multiple source languages to one target language, or other similar tasks. Multi-source machine translation models not only effectively utilize the complementarity and noise resistance of multiple source sentences, greatly improving translation performance, but also provide significant assistance for other tasks, such as translation with limiting words and automatic post-editing.
[0003] To better accomplish multi-source translation tasks, some existing technologies concatenate multiple source sentences in multi-source machine translation, treating it as a simple sequence-to-sequence problem, and directly train the model using all its parameters. With the development of pre-training techniques, some existing technologies have also begun to utilize pre-trained models for transfer learning, thereby improving the performance of multi-source machine translation.
[0004] However, the existing methods generally have the following two problems: First, when training the model, all parameters need to be fine-tuned, the parameter scale is extremely large, and the training cost is extremely high; second, training requires a massive amount of multivariate parallel data, otherwise the training effect is extremely poor, but the actual amount of multivariate parallel data is very small, or even non-existent.
[0005] Therefore, how to solve the problems of high training cost and poor training effect of existing multi-dimensional translation methods is an important issue that urgently needs to be addressed in the field of machine translation technology. Summary of the Invention
[0006] This invention provides a multi-source machine translation method and apparatus to overcome the shortcomings of existing machine translation methods, such as high training cost and poor training effect. It can effectively reduce the number of training parameters and achieve multi-source machine translation tasks under the condition of zero multivariate parallel data samples.
[0007] On one hand, the present invention provides a multi-source machine translation method, comprising: based on a pre-trained multi-source machine translation model, obtaining encoder input data according to the multi-source sentence to be translated, and combining the pre-trained encoder additional parameters corresponding to each source; and sequentially encoding and decoding the encoder input data according to the encoder-decoder in the multi-source machine translation model to obtain the target sentence; wherein, the multi-source machine translation model is obtained by improving the model parameters in a pre-trained multilingual sequence-to-sequence model, including encoder additional parameters corresponding to the encoder and additional prefix parameters corresponding to the decoder.
[0008] Furthermore, based on the pre-trained multi-source machine translation model, the step of obtaining encoder input data according to the multi-source sentences to be translated, combined with the pre-trained encoder additional parameters corresponding to each source, includes: adding language tag suffixes and sentence-end tag suffixes to each source sentence to be input and then concatenating them to obtain a concatenated sentence; inputting the concatenated sentence into the pre-trained word embedding layer of the multi-source machine translation model to obtain the corresponding word embedding tensor; and adding the pre-trained encoder additional parameters corresponding to each source to the word embedding tensor to obtain the encoder input data.
[0009] Furthermore, the encoder includes a self-attention layer and a feedforward network layer, with a gating unit following the self-attention layer. According to the encoder-decoder in the multi-source machine translation model, the encoder input data is sequentially encoded and decoded, including: after inputting the encoder input data into the self-attention layer of the encoder, the gating unit performs gating processing on the encoder additional parameters corresponding to each source end in the latent state output by the self-attention layer to obtain the target latent state; the target latent state is then input into the feedforward network layer to obtain the encoder output data.
[0010] Furthermore, the expression formula for the target hidden state is as follows:
[0011] H i =[H X;i-1 ;σ(P w;i )⊙H P;i-1 +P b;i
[0012] Among them, H X;i-1 The tensor representing the word embeddings of sentences from multiple sources, σ is the attention scoring function, and H is the word embedding tensor. P;i-1 P w;i and P b;i These are all learnable encoder additional parameters, where i is the layer number.
[0013] Furthermore, the decoder includes a cross-attention layer; according to the encoder-decoder in the multi-source machine translation model, the encoder input data is sequentially encoded and decoded, including: inputting the encoder output data into the cross-attention layer of the decoder, combining it with pre-trained additional prefix parameters added to the cross-attention layer, performing decoding processing to obtain decoder output data; and performing beam search on the decoder output data to obtain the target statement.
[0014] Furthermore, the encoder input data is represented as follows:
[0015] H0 = [Embed(x); P0] = [H X;0 H P;0 ]
[0016] Where H0 is the encoder input data, Embed(x) and H X;0 For word embedding tensors, P0 and H P;0 Add parameters to the encoder.
[0017] Furthermore, training the encoder appended parameters corresponding to the encoder and the appended prefix parameters corresponding to the decoder specifically includes: acquiring and merging multiple bilingual parallel datasets with single-source annotations to obtain a hybrid multi-source training dataset; and training the encoder appended parameters and the appended prefix parameters based on the hybrid multi-source training dataset.
[0018] Secondly, the present invention also provides a multi-source machine translation device, comprising: an encoder input data acquisition module, used to acquire encoder input data based on a pre-trained multi-source machine translation model, according to the multi-source sentence to be translated, and in combination with pre-trained encoder additional parameters corresponding to each source; and an encoding / decoding processing module, used to sequentially encode and decode the encoder input data according to the encoder-decoder in the multi-source machine translation model to obtain the target sentence; wherein the multi-source machine translation model is obtained by improving the model parameters in a pre-trained multilingual sequence-to-sequence model, including encoder additional parameters corresponding to the encoder and additional prefix parameters corresponding to the decoder.
[0019] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-source machine translation method as described above.
[0020] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-source machine translation method as described above.
[0021] The multi-source machine translation method provided by this invention is based on a pre-trained multi-source machine translation model. It obtains encoder input data from the multi-source sentence to be translated, combined with pre-trained encoder appended parameters corresponding to each source. Then, according to the encoder-decoder in the multi-source machine translation model, the encoder input data is sequentially encoded and decoded to obtain the target sentence. This method adds encoder appended parameters and corresponding decoder prefix parameters to the encoder-decoder of the multi-source machine translation model. During training, other parameters in the multi-source machine translation model are fixed, and only the encoder appended parameters and prefix parameters need to be trained. This effectively reduces the number of training parameters and lowers training costs. Furthermore, by improving the composability of the appended parameter module, the translation effect of multi-source machine translation can be effectively improved. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0023] Figure 1 A flowchart illustrating the multi-source machine translation method provided by this invention;
[0024] Figure 2 This is a schematic diagram of the encoding and decoding of the multi-source machine translation method provided by the present invention;
[0025] Figure 3 A schematic diagram of the multi-source machine translation model provided by this invention;
[0026] Figure 4 This is a schematic diagram of the structure of the multi-source machine translation device provided by the present invention;
[0027] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0029] Existing multi-source translation methods suffer from high training costs and poor training results. To address this, a method of efficient parameter fine-tuning can be applied. By adding some parameters and training only these additional parameters while fixing the other parameters of the pre-trained model, high performance can be achieved while reducing training costs.
[0030] Furthermore, the relationship between multi-source machine translation and traditional single-source machine translation should be considered. Transfer learning methods can be used to leverage massive amounts of binary parallel data to solve the problem of scarce multivariate parallel data, and even achieve multi-source machine translation with zero multivariate parallel data samples without utilizing any multivariate parallel data. This invention considers how to combine efficient parameter fine-tuning with transfer learning and further improve the capabilities of multi-source machine translation models.
[0031] Specifically, Figure 1 A flowchart illustrating the multi-source machine translation method provided by this invention is shown. Figure 1 As shown, the method includes:
[0032] S110, based on a pre-trained multi-source machine translation model, obtains encoder input data by combining the pre-trained encoder additional parameters corresponding to each source sentence to be translated.
[0033] It should be noted that the multi-source machine translation method provided in this embodiment of the invention is achieved by improving the existing pre-trained multilingual sequence-to-sequence model (Multilingual Denoising Pre-training for Neural Machine Translation). That is, it is achieved through a multi-source machine translation model. The improvement lies in adding an additional parameter module to the pre-trained multilingual sequence-to-sequence model. Specifically, corresponding additional parameters are added to the encoder module and decoder module of the model, namely the encoder additional parameters corresponding to the encoder and the additional prefix parameters corresponding to the decoder.
[0034] Each source end has its own encoder additional parameters and additional prefix parameters, and the encoder additional parameters and additional prefix parameters are different for different source ends.
[0035] For example, in one specific embodiment, parameter P0 is added to the encoder for each source end. (k) ,P w ( ; k i ) ,P b ( ; ki ) , where (k) represents the kth source end, and (k) will be omitted below.
[0036] It should also be noted that the network model used in this embodiment is a multi-source machine translation model. Except for the newly added additional parameter module, the other modules are the same as those used in the pre-trained multilingual sequence translation model.
[0037] The sequence models have the same structure, and all other modules are pre-trained, so inference can be performed directly.
[0038] Understandably, when determining the multiple source statements that need to be translated, after each source statement...
[0039] Add tag suffixes, such as language tag suffixes and sentence end tag suffixes, and then concatenate all source statements with tag suffixes to get a complete concatenated statement.
[0040] Among them, multi-source statements are combinations of multiple language types corresponding to different countries, or they can be single-language statements and target language restriction words, which are not specifically limited here.
[0041] In a specific embodiment, there are m source statements w1, w2, ..., w m ,spell
[0042] The resulting concatenation statement is: w1 <lan1> <eos> ;w2 <lan2> <eos>5;…;in m <lan m > <eos>.
[0043] Furthermore, the concatenated sentence is input into the pre-trained word embedding layer of the multi-source machine translation model to convert the concatenated sentence into a word embedding tensor. Based on this, pre-trained encoder additional parameters are appended to the word embedding tensor to obtain the corresponding encoder input data.
[0044] 0. Among these, the encoder additional parameters are learnable soft cue parameters used in high-dimensional encoding.
[0045] The representations of each input source are separated in the space, allowing the model to distinguish the information from each source and perform different processing on the source, such as extracting keywords and extracting semantics to disambiguate.
[0046] S120, based on the encoder-decoder in the multi-source machine translation model, sequentially encodes and decodes the encoder input data to obtain the target statement.
[0047] It should be noted that the encoder-decoder refers to the encoder-decoder in the above multi-source machine translation model. The encoder includes newly added encoder additional parameters, and the decoder includes newly added additional prefix parameters. Pre-training refers to the pre-training of the encoder additional parameters and additional prefix parameters.
[0048] Understandably, based on the previous step S110, which obtains encoder input data by combining the pre-trained encoder additional parameters corresponding to each source end with the multi-source end sentence to be translated, the encoder input data is further processed by using the pre-trained encoder-decoder in the multi-source machine translation model to obtain the target end sentence.
[0049] Specifically, the encoder input data is first input into the encoder, and the encoder additional parameters are used for encoding. Then, the encoder output is input into the decoder, and the encoder output is used for decoding by combining the additional prefix parameters in the decoder, thereby obtaining the corresponding target statement.
[0050] The target statement can be one of the various language types corresponding to different countries, and is not specifically limited here. For example, in a specific embodiment, the target statement is English.
[0051] In this embodiment, based on a pre-trained multi-source machine translation model, encoder input data is obtained according to the multi-source sentence to be translated, combined with the pre-trained encoder additional parameters corresponding to each source. Then, according to the encoder-decoder in the multi-source machine translation model, the encoder input data is sequentially encoded and decoded to obtain the target sentence. This method adds encoder additional parameters and corresponding additional prefix parameters to the encoder-decoder of the multi-source machine translation model. During training, other parameters in the multi-source machine translation model are fixed, and only the encoder additional parameters and additional prefix parameters need to be trained. This effectively reduces the number of training parameters and lowers the training cost. Furthermore, by improving the composability of the additional parameter module, the translation effect of multi-source machine translation can be effectively improved.
[0052] Based on the above embodiments, further, based on a pre-trained multi-source machine translation model, encoder input data is obtained according to the multi-source sentences to be translated, combined with the pre-trained encoder additional parameters corresponding to each source. This includes: adding language tag suffixes and sentence-end tag suffixes to each source sentence to be input and then concatenating them to obtain a concatenated sentence; inputting the concatenated sentence into the pre-trained word embedding layer of the multi-source machine translation model to obtain the corresponding word embedding tensor; and adding the pre-trained encoder additional parameters corresponding to each source after the word embedding tensor to obtain encoder input data.
[0053] Understandably, based on the multi-source sentences to be translated, and combined with the pre-trained encoder additional parameters corresponding to each source, the encoder input data is obtained. Specifically, first, the multi-source sentences to be translated are determined, then language tag suffixes and sentence-end tag suffixes are added to each source sentence, and then all source sentences with added tag suffixes are concatenated to obtain the concatenated sentence.
[0054] Taking the source statements as Chinese, Spanish, and French as an example, there are three source statements w1, w2, and w3, corresponding to Chinese, Spanish, and French statements respectively. The concatenated input x is w1. <zh> <eos> ;w2 <es> <eos> ;w3 <fr> <eos>.
[0055] After obtaining the concatenated sentence, it is input into the pre-trained word embedding layer of the multi-source machine translation model to obtain the corresponding output, namely the word embedding tensor Embed(x). Then, the pre-trained encoder additional parameters P0 corresponding to each source end are added to the word embedding tensor to obtain the encoder input data H0.
[0056] The encoder input data H0 is represented as follows:
[0057] H0 = [Embed(x); P0] = [H X;0 H P;0 ]
[0058] Where H0 is the encoder input data, Embed(x) and H X;0 For word embedding tensors, P0 and H P;0 Add parameters to the encoder.
[0059] After acquiring the encoder input data, the encoder input data is encoded and decoded using a pre-trained encoder-decoder to obtain the corresponding target statement.
[0060] Specifically, the encoder includes a self-attention layer and a feedforward network layer, with a gating unit following the self-attention layer. Based on the encoder-decoder in the multi-source machine translation model, the encoder input data is sequentially encoded and decoded, including: after inputting the encoder input data into the encoder's self-attention layer, the gating unit performs gating processing on the encoder additional parameters corresponding to each source end in the latent state output by the self-attention layer to obtain the target latent state; the target latent state is then input into the feedforward network layer to obtain the encoder output data.
[0061] It is understandable that the encoder includes a self-attention layer and a feedforward network layer, and a gating unit is provided after the self-attention layer.
[0062] The encoder input data is encoded by the encoder. Specifically, the encoder input data is input into the self-attention layer of the encoder to obtain the hidden state. Then, the gating unit after the attention layer will perform gating processing on the hidden state to obtain the target hidden state.
[0063] The formula for expressing the hidden state of the target is as follows:
[0064] H i =[H X;i-1 ;σ(P w;i )⊙H P;i-1 +P b;i
[0065] Among them, H X;i-1 The tensor representing the word embeddings of sentences from multiple sources, σ is the attention scoring function, and H is the word embedding tensor. P;i-1 P w;i and P b;i These are all learnable encoder additional parameters, where i is the layer number.
[0066] It should be noted that the gating unit after the self-attention layer in each encoder layer can reduce the mutual interference of different soft cue parameters, i.e., different encoder additional parameters in the self-attention layer, and filter the additional parameters of each encoder after the self-attention layer. This can greatly enhance the combination ability of the additional parameters of each encoder in the scenario of zero multi-source sample dataset, and assist the decoding process, improve the quality and effect of translation, as well as the keyword recall rate and other indicators.
[0067] After obtaining the target hidden state, the target hidden state is input into the feedforward network layer. After passing through l layers of encoder, the encoder output data H can be obtained. l .
[0068] It should be noted that the encoder in the multi-source machine translation model has multiple layers, and the processing of each layer is the same as above, which will not be elaborated here.
[0069] After encoding the encoder input data, a pre-trained decoder is needed to decode the encoder output data.
[0070] Specifically, the decoder includes a cross-attention layer; based on the encoder-decoder in the multi-source machine translation model, the encoder input data is sequentially encoded and decoded, including: inputting the encoder output data into the decoder's cross-attention layer, combining the pre-trained additional prefix parameters added to the cross-attention layer, performing decoding processing to obtain the decoder output data; and performing beam search on the decoder output data to obtain the target sentence.
[0071] Understandably, the decoder includes a cross-attention layer that decodes the encoder output data. Specifically, the encoder output data is input into the decoder's cross-attention layer, where the hidden states K and V are pre-trained prefix parameters, which are the attention weight matrix W in the decoder's cross-attention layer. K W V Augmented learnable additional prefix parameter k Prefix v The additional prefix parameter is independent of each source end. The cross-attention layer combines the additional prefix parameter to decode the encoder output data. After passing through the l-layer decoder, the corresponding decoder output data D can be obtained. l .
[0072] It should be noted that the calculation of the cross-attention layer in the above decoder can be expressed as:
[0073] Attention(yW Q H l [Prefix k W K ],H l [Prefix v W V ])
[0074] Where y represents the hidden state input of the cross-attention layer in the decoder, and H l W represents the hidden state output of the last layer of the encoder. Q W K W V These are the corresponding attention weight matrices.
[0075] It should also be noted that the decoder and encoder each have multiple layers, and the processing of each layer is the same as above, so it will not be repeated here.
[0076] After obtaining the decoder output data, a beam search is performed on the decoder output data to obtain the translation result that is closest to the meaning of the multi-source statement, i.e., the target statement.
[0077] in addition, Figure 2 A schematic diagram illustrating the encoding and decoding of the multi-source machine translation method provided by this invention is shown. Figure 2 As shown, Figure 2 The left side is the encoder module, whose input includes a word embedding tensor obtained by concatenating sentences from multiple sources, such as... Figure 2 The source sentences 1 and 2 are used, and pre-trained encoder parameters are added after the word embedding tensor. Figure 2 The input data consists of two prompts: prompt 1 corresponds to source sentence 1, and prompt 2 corresponds to source sentence 2. The word embedding tensor and encoder additional parameters together constitute the encoder input data. Then, the encoder input data is input to the self-attention layer in the encoder. The gating unit after the self-attention layer will perform gating processing on the encoder additional parameters corresponding to each source end in the hidden state output by the self-attention layer to obtain the corresponding target hidden state. Then, the target hidden state is input to the feedforward network layer to obtain the encoder output data.
[0078] Figure 2 On the right is the decoder module, which inputs the encoder output data into the decoder's cross-attention layer, i.e. Figure 2 In the cross-attention layer, it can also be seen that the hidden states K and V of the cross-attention layer are pre-trained additional prefix parameters added to complete the decoding process.
[0079] In addition, according to Figure 2 The encoder and decoder include a training phase and a prediction phase. During the training process, only the newly added encoder additional parameters and prefix additional parameters are trained. The original model parameters of the pre-trained multilingual sequence to sequence model are pre-trained and directly used for inference. They do not participate in the training of the relevant additional parameters in the encoder-decoder, which can effectively reduce the number of training parameters and reduce training costs.
[0080] In this embodiment, based on a pre-trained multi-source machine translation model, encoder input data is obtained according to the multi-source sentence to be translated, combined with the pre-trained encoder additional parameters corresponding to each source. Then, according to the encoder-decoder in the multi-source machine translation model, the encoder input data is sequentially encoded and decoded to obtain the target sentence. This method adds encoder additional parameters and corresponding additional prefix parameters to the encoder-decoder of the multi-source machine translation model. During training, other parameters in the multi-source machine translation model are fixed, and only the encoder additional parameters and additional prefix parameters need to be trained. This effectively reduces the number of training parameters and lowers the training cost. Furthermore, by improving the composability of the additional parameter module, the translation effect of multi-source machine translation can be effectively improved.
[0081] Based on the above embodiments, further, the encoder additional parameters corresponding to the encoder and the additional prefix parameters corresponding to the decoder are trained, specifically including: acquiring and merging multiple bilingual parallel datasets with single-source annotations to obtain a hybrid multi-source training dataset; and training the encoder additional parameters and additional prefix parameters according to the hybrid multi-source training dataset.
[0082] It is understandable that before performing multi-source machine translation, the encoders and decoders mentioned in the above embodiments need to be trained, that is, the encoder additional parameters added to the encoder and the corresponding additional prefix parameters of the decoder need to be trained.
[0083] Specifically, firstly, multiple bilingual parallel datasets with single-source annotations are obtained. Then, these bilingual parallel datasets with single-source annotations are merged and randomly mixed to ensure that the source end corresponding to each mini-batch dataset is consistent. The source end corresponding to each mini-batch dataset is recorded. The dataset obtained by random mixing is used as the training set, that is, the mixed multi-source training dataset.
[0084] The encoder appended parameters and appended prefix parameters are trained based on the mixed multi-source training dataset. Specifically, each mini-batch dataset is first extracted from the mixed multi-source training dataset for training. The source end corresponding to the mini-batch dataset is determined. Then, only the corresponding encoder appended parameters are appended to the encoder, and only the corresponding appended prefix parameters are appended to the decoder. During training, only the corresponding encoder appended parameters and the corresponding appended prefix parameters of the decoder are trained.
[0085] in addition, Figure 3 A schematic diagram of the multi-source machine translation model provided by this invention is shown. Figure 3 As shown, the overall process includes two phases: a training phase and a prediction phase. In the training phase, a multi-source machine translation model is trained using a bilingual parallel dataset with single-source end annotations. Specifically, taking Chinese and Japanese to English translation as an example, when the input training dataset is Chinese, encoder additional parameters are added to the multi-source machine translation model. Figure 3 The prompt (in Chinese); when the input training dataset is Japanese, add the corresponding encoder additional parameters to the multi-source machine translation model, i.e. Figure 3 The prompt (in Japanese).
[0086] It should be noted that, in addition to the encoder additional parameters mentioned above, the multi-source machine translation model also adds corresponding additional prefix parameters from the source end. Figure 3 not explicitly stated in .
[0087] In the prediction phase, the multi-source sentence to be translated is input into a pre-trained multi-source machine translation model, which then translates it into the target sentence. It should be noted that during the prediction phase, because the input is a multi-source sentence, the multi-source machine translation model simultaneously adds encoder additional parameters and prefix additional parameters corresponding to each source source, such as... Figure 3 The prompts (in Chinese) and prompts (in Japanese) are used to translate the corresponding English sentences.
[0088] It should also be noted that, Figure 3 The "zero sample" in this context refers to a zero multi-source parallel dataset.
[0089] In this embodiment, by acquiring and merging multiple bilingual parallel datasets with single-source annotations, a hybrid multi-source training dataset is obtained. Based on the hybrid multi-source training dataset, the encoder's additional parameters and additional prefix parameters are trained, which greatly improves the composability of the additional parameter module and can also effectively alleviate the problem of sparsity of parallel data for annotated multilingual languages.
[0090] Figure 4 A schematic diagram of the structure of the multi-source machine translation device provided by the present invention is shown. Figure 4 As shown, the device includes: an encoder input data acquisition module 410, used to acquire encoder input data based on a pre-trained multi-source machine translation model, according to the multi-source sentences to be translated, and in combination with the pre-trained encoder additional parameters corresponding to each source; and an encoding / decoding processing module 420, used to encode and decode the encoder input data sequentially according to the encoder-decoder in the multi-source machine translation model to obtain the target sentence; wherein, the multi-source machine translation model is obtained by improving the model parameters in the pre-trained multilingual sequence-to-sequence model, including the encoder additional parameters corresponding to the encoder and the additional prefix parameters corresponding to the decoder.
[0091] In this embodiment, the encoder input data acquisition module 410, based on a pre-trained multi-source machine translation model, acquires encoder input data according to the multi-source sentence to be translated, combined with the pre-trained encoder additional parameters corresponding to each source. Then, the encoding / decoding processing module 420 sequentially encodes and decodes the encoder input data according to the encoder-decoder in the multi-source machine translation model to obtain the target sentence. This device adds encoder additional parameters and corresponding additional prefix parameters to the encoder-decoder of the multi-source machine translation model. During training, other parameters in the multi-source machine translation model are fixed, and only the encoder additional parameters and additional prefix parameters need to be trained. This effectively reduces the number of training parameters and lowers the training cost. Furthermore, by improving the composability of the additional parameter module, the translation effect of multi-source machine translation can be effectively improved.
[0092] The multi-source machine translation apparatus provided in this embodiment of the invention can be referred to in correspondence with the multi-source machine translation method described above, and will not be repeated here.
[0093] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a multi-source machine translation method. This method includes: based on a pre-trained multi-source machine translation model, obtaining encoder input data according to the multi-source sentence to be translated, combined with the pre-trained encoder additional parameters corresponding to each source; and sequentially encoding and decoding the encoder input data according to the encoder-decoder in the multi-source machine translation model to obtain the target sentence; wherein the multi-source machine translation model is obtained by improving the model parameters in a pre-trained multilingual sequence-to-sequence model, including encoder additional parameters corresponding to the encoder and additional prefix parameters corresponding to the decoder.
[0094] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0095] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-source machine translation method provided by the above methods. This method includes: based on a pre-trained multi-source machine translation model, obtaining encoder input data according to the multi-source sentence to be translated, combined with pre-trained encoder additional parameters corresponding to each source; and sequentially encoding and decoding the encoder input data according to the encoder-decoder in the multi-source machine translation model to obtain the target sentence; wherein the multi-source machine translation model is obtained by improving the model parameters in a pre-trained multilingual sequence-to-sequence model, including encoder additional parameters corresponding to the encoder and additional prefix parameters corresponding to the decoder.
[0096] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / eos> < / fr> < / eos> < / es> < / eos> < / zh> < / eos> < / eos> < / lan2> < / eos> < / lan1>
Claims
1. A multi-source machine translation method, characterized in that, include: Based on a pre-trained multi-source machine translation model, encoder input data is obtained by combining the pre-trained encoder additional parameters corresponding to each source with the multi-source sentence to be translated. Based on the encoder-decoder in the multi-source machine translation model, the encoder input data is sequentially encoded and decoded to obtain the target statement. The multi-source machine translation model is obtained by improving the model parameters of the multilingual denoising pre-trained model for neural machine translation. The model parameters include encoder additional parameters corresponding to the encoder and additional prefix parameters corresponding to the decoder. The encoder includes a self-attention layer and a feedforward network layer, and a gating unit is provided after the self-attention layer; The process of sequentially encoding and decoding the encoder input data includes: After the encoder input data is input to the self-attention layer of the encoder, the gating unit performs gating processing on the hidden state output by the self-attention layer, combined with the encoder additional parameters corresponding to each source end, to obtain the target hidden state. The target hidden state is input into the feedforward network layer to obtain encoder output data.
2. The multi-source machine translation method according to claim 1, characterized in that, The acquisition of encoder input data includes: Each source statement in the input is appended with a language tag suffix and a sentence-end tag suffix, and then concatenated to obtain the concatenated statement. The concatenated sentence is input into the pre-trained word embedding layer of the multi-source machine translation model to obtain the corresponding word embedding tensor; The encoder input data is obtained by adding pre-trained encoder parameters corresponding to each source end to the word embedding tensor.
3. The multi-source machine translation method according to claim 1, characterized in that, The formula for expressing the target hidden state is as follows: ]; in, The word embedding tensor representing the multi-source sentences, For the Sigmoid function, , as well as These are all learnable encoder additional parameters. The number of floors.
4. The multi-source machine translation method according to claim 3, characterized in that, The decoder includes a cross-attention layer; The decoding process includes: The encoder output data is input into the cross-attention layer of the decoder, and the pre-trained additional prefix parameters added to the cross-attention layer are combined to perform decoding processing to obtain the decoder output data. A beam search is performed on the decoder output data to obtain the target statement.
5. The multi-source machine translation method according to claim 2, characterized in that, The encoder input data is represented as follows: ; in, Input data to the encoder, and Word embedding tensor, and Add parameters to the encoder.
6. The multi-source machine translation method according to any one of claims 1-5, characterized in that, The training steps of the multi-source machine translation model include: Multiple bilingual parallel datasets with single-source annotations are obtained and merged to obtain a hybrid multi-source training dataset; The encoder's additional parameters and the additional prefix parameters are trained based on the hybrid multi-source training dataset.
7. A multi-source machine translation device, characterized in that, include: The encoder input data acquisition module is used to acquire encoder input data based on the pre-trained multi-source machine translation model, according to the multi-source sentences to be translated, and combined with the pre-trained encoder additional parameters corresponding to each source. The encoding and decoding processing module is used to encode and decode the encoder input data sequentially according to the encoder-decoder in the multi-source machine translation model to obtain the target statement; The multi-source machine translation model is obtained by improving the model parameters of the multilingual denoising pre-trained model for neural machine translation. The model parameters include encoder additional parameters corresponding to the encoder and additional prefix parameters corresponding to the decoder. The encoder includes a self-attention layer and a feedforward network layer, and a gating unit is provided after the self-attention layer; The process of sequentially encoding and decoding the encoder input data includes: After the encoder input data is input to the self-attention layer of the encoder, the gating unit performs gating processing on the hidden state output by the self-attention layer, combined with the encoder additional parameters corresponding to each source end, to obtain the target hidden state. The target hidden state is input into the feedforward network layer to obtain encoder output data.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multi-source machine translation method as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the multi-source machine translation method as described in any one of claims 1 to 6.
