Method and apparatus for multi-language processing

Abstract

Embodiments of the present disclosure relate to methods, devices, equipment, storage media, and program products for multilingual processing. The method described herein includes: generating a text representation of the second language through a translation model based on a text representation of a first language and a second language tag; generating a text representation of a mixed language and a markup language tag through a translation model based on a set of language tags and a text representation of a second language, wherein a set of language tags at least includes a third language tag of a third language different from the first language and the second language, and the markup language tag is used to indicate parallel corpus data across multiple languages ​​associated with the first language, the second language, and the third language; and using the text representation of the first language and the text representation of the mixed language as inputs to the translation model to update the parameters of the translation model, including parallel corpus data across multiple languages. Thus, a context-dependent multilingual parallel processing model can be obtained, which greatly improves the translation speed and quality.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of natural language processing, and more specifically, to methods, devices, equipment, media, and program products for multi-language processing. Background Art

[0002] Multilingual neural translation (MNMT) technology can train language models to handle translation tasks across multiple languages. MNMT has two significant advantages. One is that it can provide multilingual translation services through a single model, greatly reducing the cost of online services. Second, multilingual training enables language models to transfer knowledge from high-resource languages ​​to low-resource languages, which helps improve the translation quality of low-resource language pairs.

[0003] The MNMT-based system adds a dedicated decoder for each target language, without learning the alignment information of cross-language representation. In addition, because it uses an autoregressive model, decoding needs to be performed in sequence when translating the source language into the target language. Therefore, there is still room for improvement in the decoding speed and cross-language representation of the current multilingual conversion technology. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, apparatus, device, medium, and program product for multi-language processing.

[0005] In a first aspect of the present disclosure, a method for multilingual processing is provided. The method includes: generating a text representation of a second language through a translation model based on a text representation of a first language and a second language tag; obtaining a text representation of a mixed language and a markup language tag through a translation model based on a set of language tags and the text representation of the second language, wherein the set of language tags at least includes a third language tag of a third language different from the first language and the second language, and the markup language tag is used to indicate parallel corpus data across multiple languages ​​associated with the first language, the second language, and the third language; and using the text representation of the first language and the text representation of the mixed language as inputs to the translation model to update parameters of the translation model, the parameters including the parallel corpus data across multiple languages.

[0006] In a second aspect of the present disclosure, a device for multilingual processing is provided. The device includes: a generation module configured to generate a text representation of a second language based on a text representation of a first language and a second language tag; an acquisition module configured to acquire a text representation of a mixed language and a markup language tag based on a set of language tags and the text representation of the second language, wherein the set of language tags at least includes a third language tag of a third language different from the first language and the second language, and the markup language tag is used to indicate parallel corpus data across multiple languages ​​associated with the first language, the second language, and the third language; and an update module configured to use the text representation of the first language and the text representation of the mixed language as inputs of a translation model to update parameters of the translation model, the parameters including the parallel corpus data across multiple languages.

[0007] In a third aspect of the present disclosure, a method for multilingual processing is provided. The method includes: obtaining original text data of a source language and multiple target language tags; encoding the original text data into a source text representation of the source language; based on multiple target language tags and pre-configured parallel corpus data across multiple languages, decoding the source text representation in parallel into multiple target text representations of multiple target languages ​​indicated by multiple target language tags; and decoding the multiple target text representations of the multiple target languages ​​in parallel into multiple target text data of the multiple target languages.

[0008] In a fourth aspect of the present disclosure, a device for multilingual processing is provided. The device includes: an encoder configured to: obtain original text data and multiple target language tags in a source language; and encode the original text data into a source text representation in the source language; and a decoder deployed with a translation model, the translation model having parallel corpus data across multiple languages, the decoder configured to decode the source text representation in parallel into multiple target text representations in multiple target languages ​​indicated by multiple target language tags based on multiple target language tags and pre-configured parallel corpus data across multiple languages; and decode the multiple target text representations in multiple target languages ​​in parallel into multiple target text data in multiple target languages.

[0009] In a fifth aspect of the present disclosure, an electronic device is provided, comprising: a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to the first aspect or the third aspect.

[0010] In a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein one or more computer instructions are stored on the computer-readable storage medium, wherein the one or more computer instructions are executed by a processor to implement the method according to the first aspect or the third aspect.

[0011] In a seventh aspect of the present disclosure, a computer program product is provided, wherein the computer program product comprises one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the method according to the first aspect or the third aspect.

[0012] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0014] Figure 1 A block diagram of a multi-language processing system according to some embodiments of the present disclosure is shown;

[0015] Figure 2 A block diagram illustrating an example GLAT-based architecture according to certain embodiments of the present disclosure is shown;

[0016] Figure 3 A schematic diagram showing self-enhancement learning of a multi-language processing model according to certain embodiments of the present disclosure is shown;

[0017] Figure 4A A schematic diagram showing the difference between a multi-language processing system according to certain embodiments of the present disclosure and a traditional multi-language conversion system in terms of cross-language mark representation;

[0018] Figure 4B A schematic diagram showing translation performance of a multi-language processing system according to certain embodiments of the present disclosure;

[0019] Figure 5 A flowchart of a method for training a multi-language processing model according to certain embodiments of the present disclosure is shown;

[0020] Figure 6 A flowchart showing a method for multi-language processing according to certain embodiments of the present disclosure is shown;

[0021] Figure 7 A block diagram showing an apparatus for training a multi-language processing model according to certain embodiments of the present disclosure; and

[0022] Figure 8 A block diagram of a computing system is shown in which one or more embodiments of the present disclosure may be implemented.

[0023] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0025] The term "language" used in this disclosure refers to the type of language defined in linguistics, also known as a language, such as English, Chinese, French, German, etc. The term "corpus" used in this disclosure refers to the form of presenting a language, such as a text presented in words, which has ideological content and meaning and can be understood by users who master the language. Corpus can also be information or data of a certain nature. Examples of types of information or data include, but are not limited to, voice, video, text, pictures or documents, etc. The term "corpus" used in this disclosure refers to a collection of corpora, and multiple corpora may also be referred to as a "corpus collection".

[0026] The term "representation" used in this disclosure refers to mapping a corpus into a corresponding low-dimensional vector (e.g., a word embedding vector) for easy processing by a computing system. The corpus can be mapped into a representation using known techniques such as word2vec, one-hot, etc. Of course, other existing or to-be-developed methods can also be used to map the corpus into a representation, and this disclosure does not limit this. The term "token" used in this disclosure refers to a unit with a specific meaning obtained by segmenting the corpus, such as a word or several consecutive words. Tags can be used to analyze the content and meaning of text information. For example, the text "The weather is good today" includes tags ["The", "weather", "is", "good", "today"], while the text "The weather is good today" includes tags ["today", "weather", "good"].

[0027] The term "conversion" as used in this article refers to the conversion between any two types of information or data. Examples of conversion include, but are not limited to, translation between two languages, conversion between speech and text, conversion between text and pictures, and the like. In the context of this disclosure, for the purpose of convenience of discussion, the translation process between different languages ​​is mainly used as an example of the conversion process. Generally, the conversion process can be implemented with the aid of a corresponding conversion model or translation model. Therefore, the term "model" or "layer" will sometimes be used in the description of this article to refer to the corresponding conversion process.

[0028] The terms "training" or "learning" used in this article refer to the process of updating configuration parameters and optimizing system performance using experience or training data. For example, a machine translation system can gradually optimize translation performance, such as improving translation accuracy, through a training or learning process. The training or learning process can end based on certain convergence conditions. In the context of the present disclosure, for the purpose of convenience of discussion, the terms "training" or "learning" can be used interchangeably. The terms "derivation" or "inference" used in this article refer to the process of performing specific tasks on data in the real world using a model or system that has been trained or has learned capabilities. It should be understood that the training and reasoning of the system may occur in a specific order or simultaneously.

[0029] The term "multilingual processing method / model" used in this article refers to a method / model established based on prior knowledge associated with the syntax, grammar, morphology, etc. of a specific language, which can be used to generate conversion results during the conversion process. The conversion result may include the generated corpus in the target language, and may also include the representation of the generated corpus in the target language. The representation of the corpus in the target language can continue to be used by other subjects and continue to be used for other tasks, such as classification tasks, labeling tasks, etc.

[0030] The term "including" and its variations used in this document are open-ended, i.e., "including but not limited to". The term "based on" used in this document means "based at least in part on". The terms "one embodiment" and "embodiment" used in this document mean "at least one embodiment"; the term "another embodiment" used in this document means "at least one additional embodiment". The relevant definitions of other terms will be given in the following description.

[0031] In traditional natural language processing models, corpora (e.g., text) in different languages ​​are mapped into low-dimensional vectors, which are then converted from vectors to text after a series of processing. Taking the MNMT system as an example, given a source language sentence X = {x1, x2, ..., x M}, the target language sentence is Y = {y1, y2, ..., y N}, where M and N represent the sentence lengths. MNMT reuses the standard bilingual neural translation model and uses language tags to expand the source and target inputs, that is, X becomes X′={src, x1, x2, ..., x M}, and transform Y into Y′={tgt, y1, y2, ..., y N}. Typically, MNMT builds a model from X' to Y' through a transformer. The transformer consists of stacked encoder and decoder layers, where the encoder layer is a self-attention block followed by a position-wise feedforward block. Based on this architecture, the decoding layer has an additional encoder-decoder attention block. The encoder and decoder are jointly trained to maximize the conditional probability of Y' given X'. The conditional probability P(Y'|X') can be determined, for example, according to the following formula (1):

[0032]

[0033] where θ represents the parameters of the trainable model.

[0034] Translation efficiency and translation quality are important indicators for considering the performance of machine translation. Traditional multilingual machine translation systems generally use autoregressive models. This type of model generates translations step by step, and the target language words translated at each step depend on the previous translation results. Therefore, the translation quality is good, but the translation speed is slow. If the text to be translated is large, a lot of processing time will be required. In addition, when performing multilingual processing tasks, whether the vector representations of corpora in different languages ​​with related (for example, the same) semantics are accurate and aligned will significantly affect the conversion results and translation quality. If the cross-multilingual representations are not aligned, it may sometimes even cause the converted corpus to lose semantics, contain repeated words, or miss translations, etc.

[0035] The inventors have recognized that the traditional multilingual machine translation system is limited by the model structure and training method, and it is impossible to reach a compromise between translation performance and translation speed. Therefore, a non-autoregressive multilingual processing system is provided in the embodiments of the present disclosure. The system architecture has parallel processing capabilities and can perform parallel decoding on tags at various positions in the source text, greatly improving the translation efficiency. During the training phase, the translation model learns aligned vector representations across different languages ​​by incrementally creating context-dependent code conversion sentences. Therefore, the multilingual processing system exhibits high processing performance and translation quality in multilingual processing tasks.

[0036] In the following description, some embodiments will be discussed with reference to language translation processes, for example, English, Chinese, etc. However, it should be understood that this is only to enable ordinary technicians in the field to better understand the principles and ideas of the embodiments of the present disclosure, and is not intended to limit the scope of the present disclosure in any way.

[0037] Figure 1 1 shows a block diagram of a multi-language processing system 100 according to some embodiments of the present disclosure. The multi-language processing system 100 may be a computing system, a translation system, or any other device capable of performing a language conversion task. It should be understood that Figure 1 The illustrated system 100 is merely exemplary and should not constitute any limitation on the functionality and scope of the implementations described in the present disclosure.

[0038] like Figure 1 As shown, the components of the multi-language processing system 100 may include, but are not limited to, an encoder 110 and a decoder 120. The encoder 110 and the decoder 120 may each include one or more processors or processing units, a memory, one or more communication units, one or more input devices, and one or more output devices (not shown). In addition, the multi-language processing system 100 is deployed with a translation model, which can be trained to obtain a parallel multi-language translation (PCSS) capability based on context-dependent self-conversion, so that it can be used to perform multi-language processing tasks. In the context of the present disclosure, the multi-language processing system 100 and the PCSS system can be used interchangeably.

[0039] The training of the translation model of the system 100 may include two stages. In the first stage of training, the translation model may be trained using original corpus data across multiple language pairs, where the original corpus data includes original parallel data. For example, given a parallel corpus dataset containing L input-output language pairs, Where i represents the i-th language pair, X represents the text data of the input language, which can also be called the source language, and Y represents the text data of the output language, which can also be called the target language. For example, the translation model can be trained based on the following formula (2), aiming to maximize the sum of the logarithmic probabilities of the true value output given the input:

[0040]

[0041] where θ M Represents the parameters of the translation model, α is used to control the loss function L stage1 and factors of relative importance. The i-th parallel corpus in N i parallel statements, expressed as A specific training criterion can be used to calculate Represents a parallel corpus The length prediction loss of the k-th pair in .

[0042] The following discussion takes the translation model using the Glancing Translator (GLAT) architecture as an example, but it should be understood that the embodiments of the present disclosure are not limited to the GLAT architecture, but are applicable to any language conversion model. GLAT is a non-autoregressive translation (NAT) architecture that can achieve an 8 to 15 times speed increase in machine translation. Figure 2 2 shows a block diagram of an example GLAT-based architecture 200 according to certain embodiments of the present disclosure. Figure 2 As shown, the GLAT architecture 200 includes an encoding module 201, a parallel decoding module 202, an adoption module 203, and a parallel decoding module 204. Of course, the GLAT architecture 200 may also include any other components, modules, elements, etc. required to perform processing tasks.

[0043] During the learning process, the GLAT architecture 200 performs two-step decoding. Specifically, assuming that the source language sentence input to the encoding module 201 is X, and the target language sentence is Y, given the sentence from the encoder F e (ie, the encoding module 201) outputs the collected decoder F d (ie, parallel decoding module 202) input is And Y can be predicted according to the following formula (3):

[0044]

[0045] Where θ represents the parameters of the trainable model. The parallel decoding module 200 calculates Y and The module 203 adopts a corresponding glacing sampling strategy to sample the subset of Y based on the calculated distance, and obtains in Denotes the subset remaining after removing the sampled tokens from sentence Y in the target language.

[0046] In the second parallel decoding, the GLAT architecture 200 can be based on the subset according to the following formula (4): And the source language sentence X to predict the target sentence Y:

[0047]

[0048] in Represents the updated decoded input.

[0049] It can be calculated according to the GLAT-based training standard as shown in formula (4): and then, can be determined as follows:

[0050]

[0051] in represents a one-hot vector that characterizes the optimal target length distribution, and is based on the output of the encoding module 201 and source language embedding E src and the target language embedding E tgt Here, an operator (e.g., softmax) of a normalization indicator function is applied for length prediction, and the normalized output corresponds to a set of possible target lengths.

[0052] In the same way, the GLAT module uses the parallel corpus dataset The corpus of all L language pairs in is trained until the convergence condition is met. At this point, the translation model has balanced translation performance for the L language pairs and can enter the second stage of training.

[0053] In the second stage of training, the decoder 120 may perform two-stage decoding to perform context-dependent code conversion on the input text representation. Figure 1 As shown, assuming that the first language as the source language is English En, and the second language as the target language is German De, the decoder 120 obtains the text representation of the first language [The, cat, is, very, cute] and the second language label Lang=De from the input end.

[0054] In the first stage decoding 121, the decoder 120 generates a text representation of the second language [Die, Katze, ist, sehr, süβ] based on the text representation of the first language and the second language tag. Then, the decoder 120 obtains a mixed language text representation [Die, chat, est, muy, lindo] and a markup language tag based on a set of language tags Lang = De, Fr, Es and the text representation of the second language. Specifically, after the first training phase is completed, the decoder 120 can perform a parallel corpus dataset. Sampling is performed to obtain a parallel corpus data subset of length T in Know denote the source language tag and the target language tag of the j-th language pair respectively. Then, in the first stage decoding 121, the decoder 120 decodes the source language tag and the target language tag of the j-th language pair at a given ratio P M For text data Y in the target language i Mask it and get Thereafter, the decoder 120 uses the markup language tag to The positions of the masks in are decoded into randomly sampled languages. Therefore, the final decoded text sequence is Includes mixed-language markup language tags, which can be used to indicate parallel corpus data across multiple languages.

[0055] In the second stage decoding 122, the decoder 120 can convert the mixed language text sequence As the source side input, the source language text sequence X is used as the target side input, and the synthetic parallel corpus is obtained by decoding

[0056] Based on a similar approach, using code-converted corpus data and parallel corpus data The second stage of training the translation model is performed iteratively in the following way:

[0057]

[0058] This cross-iterative training method can achieve self-enhancement of the translation model. Ultimately, the decoder 120 can obtain the markup language tags across L language pairs. The markup language tags are used to distinguish parallel corpus data in different languages, which are word tags at the same position in the text representation. For example, the multilingual processing system 100 may include K stacked encoder layers and decoder layers. Specific language tags are added to the first layer input and the last layer output at each position, as shown below:

[0059]

[0060] in represents the first encoder layer input at position i, represents the first decoder layer input at position j. Correspondingly, represents the output of the last encoder layer at position i, and Represents the last decoder layer output at position j. E src and E tgt Represents the source language tag and the target language tag respectively. Thus, you can use To update the text representation y in formula (4) t .

[0061] It should be understood that the decoder 120 may include independent decoding units for performing the first level decoding 121 and the second level decoding 122 respectively, or may include a single decoder supporting both the first level decoding 121 and the second level decoding 122, and the present disclosure is not limited in this respect.

[0062] Additionally or alternatively, as the second stage training is iteratively performed, the translation model can gradually improve the mask ratio P based on the training results. M and the number of mixed languages. Figure 3Schematic diagram of self-enhancement learning of a multi-language processing model according to certain embodiments of the present disclosure is shown. Given a step number of 0.1, the mask ratio P per epoch M The value of is iterated from 0.1 to 0.5, which can be expressed as follows:

[0063] P M =(((Epoch÷I)mod 5)+1)÷10 (8)

[0064] Epoch represents the current round number.

[0065] like Figure 3 As shown, in P M In the first iteration of , the number of mixed languages ​​is set to 1. Thereafter, the number of mixed languages ​​is increased to one third of the total number of languages. During the iterations, a large number of code-switched sentences are generated. This helps the translation model learn context-dependent aligned cross-lingual representations, and thus provides better translation performance.

[0066] Additionally or alternatively, in the case of training a translation model via a deep neural network, an annealed dropout strategy can be applied to gradually reduce the number of neurons that are randomly reset to zero during training. For example, a linear annealing procedure for a given mini-batch (e.g., 64,000 tokens) can be used as follows:

[0067]

[0068] Where t represents the training update, N represents the total annealing amount, and P d [0] represents the initial abandonment rate (e.g., 0.3). The annealing abandonment strategy can stabilize training and improve translation quality. In particular, for acoustic models, applying the annealing abandonment strategy can significantly reduce the word error rate of the model.

[0069] In the training phase of the translation model, only the decoder 120 of the translation system 100 may be involved. In the inference phase, the translation model performs multi-language processing tasks through the encoder 110 and the decoder 120.

[0070] In the inference phase, the trained PCSS model showed significantly enhanced translation performance, both in terms of translation speed and translation quality. Table 1 below shows the comparison results of the translation performance of the traditional translation model and the multilingual processing model obtained based on the training method of the present disclosure when performing English, German, and French translation tasks, where the bilingual substitution evaluation average score (BLEU) is used as a performance parameter to measure translation performance.

[0071] Table 1. Performance comparison results of various translation models

[0072]

[0073]

[0074] Among them, Transformer and GLAT are bilingual translation models, M-transformer, GLSR and Adaptor are multilingual translation models, and MNAT and PCSS proposed in this disclosure are multilingual NAT models. As shown in Table 1, compared with M-transformer, PCSS has a translation speed of 6.1 times and an average score of +1.7 BLEU.

[0075] Figure 4A and Figure 4B The translation performance of the multi-language processing system according to some embodiments of the present disclosure is intuitively illustrated. Figure 4A The figure shows a schematic diagram of the mark representation obtained by using the conventional multilingual conversion system of the prior art and the PCSS-based translation system proposed in the present disclosure according to the bilingual words in the English-German dictionary, wherein blue represents English words and red represents German words. Figure 4A As shown in Figure (A), there is a clear boundary in the cross-language tag representations obtained by learning in the traditional multilingual conversion system, while the cross-language tag representations obtained by learning in the PCSS-based translation system are well aligned.

[0076] Figure 4B A schematic diagram showing the performance of a PCSS-based translation system according to certain embodiments of the present disclosure is shown, wherein each German word is shown in green, and if the similarity of the English word paired with the corresponding German word is greater than a given threshold (e.g., 0.8), the English word is shown in blue, and if the similarity is less than the given threshold, the English word is shown in yellow. Figure 4B As shown, the number of English words shown in blue is much higher than the number of English words shown in yellow. In other words, the PCSS-based translation system can generate fully aligned cross-language vector representations, which will greatly improve the translation quality.

[0077] Figure 5 A flowchart of a method 500 for training a multilingual processing model according to some embodiments of the present disclosure is shown. The method 500 may be implemented by the multilingual translation system 100, for example, may be implemented at the encoder 110 and the decoder 120 of the translation system 100.

[0078] At block 501, the decoder 120 generates a text representation in a second language through a translation model based on the text representation in a first language and a second language tag. As an example, the translation model may be based on a GLAT language model.

[0079] Additionally or alternatively, in some embodiments, the method 500 further includes: before generating the text representation of the second language by the translation model, training the translation model with parallel corpus data until the translation model has balanced translation performance for multiple language pairs. The parallel corpus data may include corpus data for multiple language pairs. This allows the training translation model to have similar translation performance for multiple language pairs.

[0080] In some embodiments, the method may further include: determining a plurality of sampling factors for the original corpus data of the plurality of language pairs, each sampling factor being associated with the original corpus data of a corresponding language pair in the plurality of language pairs. The method samples the original corpus data of the plurality of language pairs based on the plurality of sampling factors to obtain parallel corpus data for training the translation model.

[0081] Additionally or alternatively, in some embodiments, the training method further comprises determining a plurality of sampling factors for the original corpus data of the plurality of language pairs, and then sampling the original corpus data of the plurality of language pairs based on the plurality of sampling factors to obtain parallel corpus data for training the translation model.

[0082] Additionally or alternatively, in some embodiments, the training method may determine a sampling ratio parameter based on the amount of corpus data of each language pair in the original corpus data and the total corpus data. The training method further includes: applying an adjustment coefficient associated with the importance of the corresponding language pair to the sampling ratio parameter to obtain a plurality of sampling factors.

[0083] At block 502, a text representation of a mixed language and a markup language tag are obtained through a translation model based on a set of language tags and a text representation of a second language. The set of language tags at least includes a third language tag of a third language different from the first language and the second language. The markup language tag is used to indicate parallel corpus data across multiple languages ​​associated with the first language, the second language, and the third language.

[0084] Additionally or alternatively, in some embodiments, the training method may include: the decoder 120 samples word representations in the text representation of the second language based on the first ratio. Based on a set of language tags, the sampled word representations of the first ratio are converted into word representations corresponding to a set of languages. The encoder 120 determines a markup language tag associated with the converted word representations of the first ratio. Then, the encoder 120 generates a text representation of the mixed language based on the converted word representations of the first ratio and the remaining word representations in the text representation of the second language.

[0085] Additionally or alternatively, in some embodiments, the training method may include the decoder 120 generating a target text representation in at least one target language through an updated translation model based on the source text representation in the source language and the markup language tag. The decoder 120 determines a distance parameter between the target text representation and the source text representation. The decoder 120 updates the first ratio based on the distance parameter.

[0086] At block 503, the encoder 120 uses the text representation of the first language and the text representation of the mixed language as inputs of the translation model to update parameters of the translation model. As an example, the parameters may include parallel corpus data across multiple languages.

[0087] Additionally or alternatively, in some embodiments, the encoder 120 may perform the following operations at least once: input the text representation of the mixed language as source data for training and the text representation of the first language as target data for training into the translation model; and the encoder 120 obtains another text representation of the mixed language and updated markup language tags based on another set of language tags, wherein the another set of language tags includes at least a fourth language tag different from the set of language tags, and the updated markup language tags are used to indicate parallel corpus data across multiple languages ​​associated with the first language, the second language, the third language, and the fourth language.

[0088] Additionally or alternatively, in some embodiments, the translation model obtained by training method 500 can be used to perform multi-language processing tasks.

[0089] Figure 6 A flowchart of a method 600 for multi-language processing according to some embodiments of the present disclosure is shown. The method 600 may be implemented by the multi-language translation system 100, for example, may be implemented at the encoder 110 and the decoder 120 of the translation system 100.

[0090] At block 601 , the encoder 110 obtains original text data in a source language and a plurality of target language tags.

[0091] At block 602 , the encoder 110 encodes the original text data into a source text representation in a source language. The encoder 110 may then output the source text representation to the decoder 120 .

[0092] At block 603 , the decoder 120 decodes the source text representation in parallel into multiple target text representations in multiple target languages ​​indicated by the multiple target language tags based on the multiple target language tags and preconfigured parallel corpus data across multiple languages.

[0093] At block 604 , the decoder 120 decodes the plurality of target text representations in the plurality of target languages ​​into a plurality of target text data in the plurality of target languages ​​in parallel.

[0094] Figure 7 A block diagram of an apparatus 700 for training a multilingual processing model according to certain embodiments of the present disclosure is shown. The apparatus includes a generation module 701, an acquisition module 702, and an update module 703. The generation module 701 is configured to generate a text representation of a second language based on a text representation of a first language and a second language tag. The acquisition module 702 is configured to acquire a text representation of a mixed language and a markup language tag based on a set of language tags and a text representation of a second language, wherein the set of language tags at least includes a third language tag of a third language different from the first language and the second language, and the markup language tag can be used to indicate parallel corpus data across multiple languages ​​associated with the first language, the second language, and the third language. The update module 703 is configured to use the text representation of the first language and the text representation of the mixed language as inputs to a translation model to update parameters of the translation model, the parameters including parallel corpus data across multiple languages.

[0095] According to an embodiment of the present disclosure, a multi-language processing device is provided. The multi-language processing device adopts a context-dependent non-autoregressive translation model, and can learn cross-language representations between multiple language pairs. The multi-language processing device can perform multi-language translation tasks in parallel, thereby significantly speeding up the translation speed. In addition, good translation quality can be obtained through aligned cross-language tag representations.

[0096] Figure 8 A block diagram of a computing system 800 is shown in which one or more embodiments of the present disclosure may be implemented. Figure 5 The method 500 and Figure 6 The illustrated method 600 may be implemented by a computing system 800 . Figure 8 The computing system 800 shown is merely an example and should not be construed as limiting the functionality and scope of use of the implementations described herein.

[0097] like Figure 8 As shown, computing system 800 is in the form of a general purpose computing device. Components of computing system 800 may include, but are not limited to, one or more processors or processing units 800, memory 820, one or more input devices 830, one or more output devices 840, storage 850, and one or more communication units 860. Processing unit 800 may be an actual or virtual processor and is capable of performing various processes according to the program stored in memory 820. In a multi-processing system, multiple processing units execute computer executable instructions to increase processing capabilities.

[0098] The computing system 800 typically includes a plurality of computer media. Such media may be any available media accessible to the computing system 800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 820 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 850 may be removable or non-removable and may include a machine-readable medium, such as a flash drive, a disk, or any other medium that may be capable of storing information and may be accessed within the computing system 800.

[0099] The computing system 800 may further include additional removable / non-removable, volatile / non-volatile computer system storage media. Figure 8 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus 18 by one or more data media interfaces. The memory 820 may include at least one program product having (e.g., at least one) set of program modules that are configured to perform the functions of the various embodiments described herein.

[0100] A program / utility tool 822 having a collection of one or more execution modules 824 may be stored, for example, in memory 820. Execution modules 824 may include, but are not limited to, an operating system, one or more application programs, other program modules, and operational data. Each example or specific combination of these examples may include an implementation of a networking environment. Execution modules 824 generally perform the functions and / or methods of embodiments of the subject matter described herein, such as method 200.

[0101] Input unit 830 can be one or more various input devices. For example, input unit 839 can include user equipment, such as a mouse, keyboard, tracking ball, etc. Communication unit 860 implements communication to other computing entities on a communication medium. Additionally, the functions of the components of computing system 800 can be implemented with a single computing cluster or multiple computing machines, which can communicate through a communication connection. Therefore, computing system 800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another general network node. For example, but not limited to, the communication medium includes wired or wireless networking technology.

[0102] The computing system 800 may also communicate with one or more external devices (not shown) as needed, such as storage devices, display devices, etc., with one or more devices that enable a user to interact with the computing system 800, or with any device that enables the computing system 800 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communications may be performed via an input / output (I / O) interface (not shown).

[0103] The functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0104] The program code for implementing the methods of the subject matter described herein can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, causes the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0105] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the subject matter described herein. Certain features described in the context of a separate implementation can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0107] Some example implementations of the present disclosure are listed below.

[0108] In certain embodiments of the first aspect, a method for multilingual processing is provided. The method includes: generating a text representation of the second language through a translation model based on a text representation of a first language and a second language tag; obtaining a text representation of a mixed language and a markup language tag through a translation model based on a set of language tags and the text representation of the second language, wherein the set of language tags includes at least a third language tag of a third language different from the first language and the second language, and the markup language tag is used to indicate parallel corpus data across multiple languages ​​associated with the first language, the second language, and the third language; and using the text representation of the first language and the text representation of the mixed language as inputs to the translation model to update parameters of the translation model, the parameters including the parallel corpus data across multiple languages.

[0109] In some embodiments, the method further includes: before generating the text representation of the second language through the translation model, training the translation model with parallel corpus data until the translation model has balanced translation performance with respect to multiple language pairs, the parallel corpus data including corpus data of the multiple language pairs.

[0110] In some embodiments, training a translation model using parallel corpus data includes: determining multiple sampling factors for original corpus data of multiple language pairs, each sampling factor being associated with the original corpus data of a corresponding language pair in the multiple language pairs; and sampling the original corpus data of the multiple language pairs based on the multiple sampling factors to obtain parallel corpus data for training the translation model.

[0111] In some embodiments, determining the plurality of sampling factors includes: determining a sampling ratio parameter based on the amount of corpus data of each language pair in the original corpus data and the total corpus data; and applying an adjustment coefficient associated with the importance of the corresponding language pair to the sampling ratio parameter to obtain the plurality of sampling factors.

[0112] In some embodiments, obtaining a text representation of a mixed language and markup language tags includes: sampling word representations in a text representation of a second language based on a first ratio; converting the sampled word representations of the first ratio into word representations corresponding to a set of languages ​​based on a set of language tags; determining markup language tags associated with the converted word representations of the first ratio; and generating a text representation of the mixed language based on the converted word representations of the first ratio and the remaining word representations in the text representation of the second language.

[0113] In some embodiments, the method further includes: generating a target text representation in at least one target language through an updated translation model based on the source text representation in the source language and the markup language tag; determining a distance parameter between the target text representation and the source text representation; and updating the first ratio based on the distance parameter.

[0114] In some embodiments, updating the first ratio includes: if the distance parameter exceeds the distance threshold, updating the first ratio to a second ratio that is smaller than the first ratio; and if the distance parameter does not exceed the distance threshold, updating the first ratio to a third conversion ratio that is larger than the first ratio.

[0115] In some embodiments, the method further comprises: if the distance parameter exceeds the distance threshold, decreasing the number of tags in the set of language tags; and if the distance parameter does not exceed the distance threshold, increasing the number of tags in the set of language tags.

[0116] In some embodiments, updating the translation model includes performing the following operations at least once: inputting the text representation of the mixed language as source data for training and the text representation of the first language as target data for training into the translation model; and obtaining, through the translation model, another text representation of the mixed language and updated markup language tags based on another set of language tags, wherein the another set of language tags includes at least a fourth language tag different from the set of language tags, and the updated markup language tags are used to indicate parallel corpus data across multiple languages ​​associated with the first language, the second language, the third language, and the fourth language.

[0117] In some embodiments, the method further includes: determining a performance parameter of the updated translation model; and stopping updating the translation model if the performance parameter exceeds a threshold parameter, wherein the performance parameter includes a bilingual substitution evaluation score.

[0118] In some embodiments, at least a portion of the translation model is based on a Glancing language model.

[0119] In certain embodiments, the method further includes causing the updated translation model to be deployed for multi-language parallel translation tasks.

[0120] In certain embodiments of the second aspect, a method for multilingual processing is provided. The method includes: obtaining original text data in a source language and multiple target language tags; encoding the original text data into a source text representation in the source language; based on the multiple target language tags and preconfigured parallel corpus data across multiple languages, decoding the source text representation in parallel into multiple target text representations in multiple target languages ​​indicated by the multiple target language tags; and decoding the multiple target text representations in the multiple target languages ​​in parallel into multiple target text data in the multiple target languages.

[0121] In some embodiments, the method of the first aspect is executed to train the translation model of the apparatus of the second aspect.

[0122] In an embodiment of the third aspect, a device for multilingual processing is provided. The device includes: a generation module configured to generate a text representation of a second language based on a text representation of a first language and a second language tag; an acquisition module configured to acquire a text representation of a mixed language and a markup language tag based on a set of language tags and the text representation of the second language, wherein the set of language tags at least includes a third language tag of a third language different from the first language and the second language, and the markup language tag is used to indicate parallel corpus data across multiple languages ​​associated with the first language, the second language, and the third language; and an update module configured to use the text representation of the first language and the text representation of the mixed language as inputs of a translation model to update parameters of the translation model, wherein the parameters include parallel corpus data across multiple languages.

[0123] In some embodiments, the device also includes: a training module configured to train the translation model using parallel corpus data before generating a text representation in a second language through the translation model until the translation model has balanced translation performance with respect to multiple language pairs, wherein the parallel corpus data includes corpus data of the multiple language pairs.

[0124] In some embodiments, the training module is configured to: determine a plurality of sampling factors for original corpus data of a plurality of language pairs, each sampling factor being associated with the original corpus data of a corresponding language pair in the plurality of language pairs; and; and sample the original corpus data of the plurality of language pairs based on the plurality of sampling factors to obtain parallel corpus data for training a translation model.

[0125] In some embodiments, determining the plurality of sampling factors includes: determining a sampling ratio parameter based on the amount of corpus data of each language pair in the original corpus data and the total corpus data; and applying an adjustment coefficient associated with the importance of the corresponding language pair to the sampling ratio parameter to obtain the plurality of sampling factors.

[0126] In some embodiments, the acquisition module is configured to: sample word representations in a text representation of a second language based on a first ratio; convert the sampled word representations of the first ratio into word representations corresponding to a set of languages ​​based on a set of language tags; determine markup language tags associated with the converted word representations of the first ratio; and generate a text representation of a mixed language based on the converted word representations of the first ratio and the remaining word representations in the text representation of the second language.

[0127] In some embodiments, the generation module is further configured to: generate a target text representation in at least one target language through an updated translation model based on the source text representation in the source language and the markup language tag, and the update module is further configured to: determine a distance parameter between the target text representation and the source text representation; and update the first ratio based on the distance parameter.

[0128] In some embodiments, the update module is configured to: if the distance parameter exceeds the distance threshold, update the first ratio to a second ratio that is smaller than the first ratio; and if the distance parameter does not exceed the distance threshold, update the first ratio to a third conversion ratio that is larger than the first ratio.

[0129] In some embodiments, the update module is further configured to: decrease the number of tags in the set of language tags if the distance parameter exceeds the distance threshold; and increase the number of tags in the set of language tags if the distance parameter does not exceed the distance threshold.

[0130] In some embodiments, the update module is further configured to perform the following operations at least once: input the text representation of the mixed language as source data for training and the text representation of the first language as target data for training into the translation model; and based on another set of language tags, obtain another text representation of the mixed language and updated markup language tags through the translation model, wherein the other set of language tags includes at least a fourth language tag different from the set of language tags, and the updated markup language tags are used to indicate parallel corpus data across multiple languages ​​associated with the first language, the second language, the third language, and the fourth language.

[0131] In some embodiments, the apparatus further comprises a determination module configured to: determine a performance parameter of the updated translation model; and stop updating the translation model if the performance parameter exceeds a threshold parameter, wherein the performance parameter comprises a bilingual substitution evaluation score.

[0132] In some embodiments, at least a portion of the translation model is based on a Glancing language model.

[0133] In some embodiments, the apparatus further includes an execution module configured to: enable the updated translation model to be deployed for multi-language parallel translation tasks.

[0134] In an embodiment of the fourth aspect, a device for multilingual processing is provided. The device includes: an encoder configured to: obtain original text data of a source language and multiple target language tags; and encode the original text data into a source text representation of the source language; and a decoder deployed with a translation model, the translation model having parallel corpus data across multiple languages, and the decoder configured to: based on multiple target language tags and pre-configured parallel corpus data across multiple languages, decode the source text representation in parallel into multiple target text representations of multiple target languages ​​indicated by multiple target language tags; and decode the multiple target text representations of multiple target languages ​​in parallel into multiple target text data of multiple target languages.

[0135] In some embodiments, the method of the first aspect is executed to train the translation model of the apparatus of the fourth aspect.

[0136] In an embodiment of the fifth aspect, an electronic device is provided, comprising: a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to the first aspect or the second aspect.

[0137] In an embodiment of the sixth aspect, a computer-readable storage medium is provided, wherein one or more computer instructions are stored on the computer-readable storage medium, wherein the one or more computer instructions are executed by a processor to implement the method according to the first aspect or the second aspect.

[0138] In an embodiment of the seventh aspect, a computer program product is provided, wherein the computer program product comprises one or more computer instructions, wherein when the one or more computer instructions are executed by a processor, the method according to the first aspect or the second aspect is implemented.

[0139] Although the disclosure has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for multilingual processing, comprising: Based on the text representation of the first language and the second language label, a text representation of the second language is generated through a translation model; Based on a set of language tags and the text representation of the second language, obtaining a mixed language text representation and a markup language tag through the translation model, wherein the set of language tags at least includes a third language tag of a third language different from the first language and the second language, and the markup language tag is used to indicate parallel corpus data across multiple languages ​​associated with the first language, the second language, and the third language; as well as The text representation of the first language and the text representation of the mixed language are used as inputs of the translation model to update parameters of the translation model, wherein the parameters include the parallel corpus data across multiple languages.

2. The method according to claim 1, further comprising: Before generating the text representation of the second language by the translation model, the translation model is trained using parallel corpus data, where the parallel corpus data includes corpus data of the multiple language pairs.

3. The method according to claim 2, wherein training the translation model using the parallel corpus data comprises: Determine a plurality of sampling factors for the original corpus data of the plurality of language pairs, each sampling factor being associated with the original corpus data of a corresponding language pair in the plurality of language pairs; as well as The original corpus data of the multiple language pairs are sampled based on the multiple sampling factors to obtain the parallel corpus data for training the translation model.

4. The method of claim 3, wherein determining the plurality of sampling factors comprises: Determining a sampling ratio parameter based on the amount of corpus data of each language pair in the original corpus data and the total amount of corpus data; as well as An adjustment coefficient associated with the importance of the corresponding language pair is applied to the sampling ratio parameter to obtain the plurality of sampling factors.

5. The method according to claim 1, wherein obtaining the text representation of the mixed language and the markup language tag comprises: sampling the word representations in the text representation of the second language based on the first ratio; Based on the set of language tags, converting the sampled word representations of the first proportion into word representations corresponding to the set of languages; determining a markup language tag associated with the converted word representation of the first ratio; as well as The mixed-language text representation is generated based on the converted word representations of the first ratio and the remaining word representations in the second language text representation.

6. The method according to claim 5, further comprising: Based on the source text representation in the source language and the markup language tags, generating a target text representation in at least one target language by means of the updated translation model; determining a distance parameter between the target text representation and the source text representation; The first ratio is updated based on the distance parameter.

7. The method according to claim 6, further comprising: If the distance parameter exceeds a distance threshold, updating the first ratio to a second ratio, wherein the second ratio is smaller than the first ratio; as well as If the distance parameter does not exceed the distance threshold, the first ratio is updated to a third conversion ratio, and the third conversion ratio is greater than the first ratio.

8. The method according to claim 6, further comprising: If the distance parameter exceeds a distance threshold, reducing the number of tags in the set of language tags; as well as If the distance parameter does not exceed the distance threshold, the number of tags in the set of language tags is increased.

9. The method of claim 1, wherein updating the translation model comprises performing the following operations at least once: inputting the text representation in the mixed language as source data for training and the text representation in the first language as target data for training into the translation model; and Based on another set of language tags, another text representation of the mixed language and updated markup language tags are obtained through the translation model, wherein the another set of language tags includes at least a fourth language tag different from the set of language tags, and the updated markup language tags are used to indicate parallel corpus data across multiple languages ​​associated with the first language, the second language, the third language, and the fourth language.

10. The method according to claim 1, further comprising: determining an updated performance parameter of the translation model; as well as If the performance parameter exceeds a threshold parameter, then the updating of the translation model is stopped, wherein the performance parameter includes a bilingual substitution evaluation score.

11. The method of claim 1, wherein at least a portion of the translation model is based on a Glancing language model.

12. The method according to claim 1, further comprising: The updated translation model is deployed for multi-language parallel translation tasks.

13. A device for multi-language processing, comprising: A generating module configured to generate a text representation in a second language based on the text representation in the first language and the second language label; an acquisition module configured to acquire a mixed-language text representation and a markup language tag based on a set of language tags and the text representation of the second language, wherein the set of language tags at least includes a third language tag of a third language different from the first language and the second language, and the markup language tag is used to indicate cross-multilingual parallel corpus data associated with the first language, the second language, and the third language; as well as An updating module is configured to use the text representation of the first language and the text representation of the mixed language as inputs of a translation model to update parameters of the translation model, wherein the parameters include the parallel corpus data across multiple languages.

14. A method for multi-language processing, comprising: Obtaining original text data in a source language and multiple target language tags; encoding the original text data into a source text representation in the source language; Based on the multiple target language tags and preconfigured parallel corpus data across multiple languages, the source text representation is decoded in parallel into multiple target text representations in multiple target languages ​​indicated by the multiple target language tags, wherein the parallel corpus data across multiple languages ​​includes multiple groups of parallel sentences, and each parallel sentence in the multiple groups of parallel sentences includes multiple text representations corresponding to the multiple target languages; as well as The plurality of target text representations in the plurality of target languages ​​are decoded in parallel into a plurality of target text data in the plurality of target languages.

15. An apparatus for multi-language processing, comprising: The encoder is configured as: Obtaining original text data in a source language and multiple target language tags; as well as encoding the original text data into a source text representation in the source language; as well as A decoder is deployed with a translation model having parallel corpus data across multiple languages, and the decoder is configured to: Based on the multiple target language tags and preconfigured parallel corpus data across multiple languages, the source text representation is decoded in parallel into multiple target text representations in multiple target languages ​​indicated by the multiple target language tags, wherein the parallel corpus data across multiple languages ​​includes multiple groups of parallel sentences, and each parallel sentence in the multiple groups of parallel sentences includes multiple text representations corresponding to the multiple target languages; as well as The plurality of target text representations in the plurality of target languages ​​are decoded in parallel into a plurality of target text data in the plurality of target languages.

16. An electronic device, comprising: Memory and processor; The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 12 and claim 14.

17. A computer-readable storage medium having one or more computer instructions stored thereon, wherein the one or more computer instructions are executed by a processor to implement the method according to any one of claims 1 to 12 and claim 14.

18. A computer program product comprising one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the method according to any one of claims 1 to 12 and claim 14.

Images