Training method of language model for multilingual translation and translation method

Abstract

The disclosure provides a language model training method and a translation method for multilingual translation, and relates to the technical field of artificial intelligence, in particular to the fields of natural language processing, deep learning, machine translation and the like. The language model training method for multilingual translation comprises: obtaining a sample text pair, the sample text pair comprising a first text in a first language and a second text in a second language, the semantics of the first text being the same as the semantics of the second text; inputting the first text and the second language into a language model to obtain a first translated text in the second language output by the language model; inputting the second text and the second language into the language model to obtain a second translated text in the second language output by the language model; determining a loss value of the language model based on a first difference between the first translated text and the second text and a second difference between the first translated text and the second translated text; and adjusting parameters of the language model based on the loss value.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of natural language processing, deep learning, machine translation and the like, and specifically relates to a training method and device of a language model for multilingual translation, a translation method and device, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] Machine translation is a process of translating a natural language text (source language text) into another natural language text (target language text) with the help of a machine, and is an important research field of natural language processing and one of the commonly used services on the Internet.

[0003] A large language model (LLM, also known as a large model) is a deep learning model trained using a large amount of text data, which can understand the meaning of natural language text or generate natural language text. At present, large language models have shown amazing potential in machine translation tasks. Using a large language model, multilingual text can be translated.

[0004] The methods described in this section can not necessarily be the methods that have been previously conceived or adopted. Unless otherwise indicated, nothing in this section should be assumed to be prior art merely because of its inclusion in this section. Similarly, issues mentioned in this section should not be assumed to have been admitted to be prior art in any jurisdiction unless otherwise indicated. SUMMARY

[0005] The present disclosure provides a training method and device of a language model for multilingual translation, a translation method and device, an electronic device, a computer readable storage medium and a computer program product.

[0006] According to an aspect of the present disclosure, a training method of a language model for multilingual translation is provided, comprising: obtaining a sample text pair, wherein the sample text pair comprises a first text in a first language and a second text in a second language, and the semantics of the first text is the same as the semantics of the second text; inputting the first text and the second language into the language model to obtain a first translation text in the second language output by the language model; inputting the second text and the second language into the language model to obtain a second translation text in the second language output by the language model; determining a loss value of the language model based on a first difference between the first translation text and the second text and a second difference between the first translation text and the second translation text; and adjusting parameters of the language model based on the loss value.

[0007] According to an aspect of the present disclosure, a translation method is provided, comprising: obtaining source text to be translated and a target language; and inputting the source text and the target language into a trained language model to obtain target text in the target language output by the language model, wherein the language model is trained according to the training method described above.

[0008] According to an aspect of the present disclosure, a training device for a language model for multilingual translation is provided, comprising: an obtaining module configured to obtain a sample text pair, wherein the sample text pair comprises first text in a first language and second text in a second language, and the semantics of the first text are the same as the semantics of the second text; a first input module configured to input the first text and the second language into the language model to obtain first translated text in the second language output by the language model; a second input module configured to input the second text and the second language into the language model to obtain second translated text in the second language output by the language model; a determination module configured to determine a loss value of the language model based on a first difference between the first translated text and the second text and a second difference between the first translated text and the second translated text; and an adjustment module configured to adjust parameters of the language model based on the loss value.

[0009] According to an aspect of the present disclosure, a translation device is provided, comprising: an obtaining module configured to obtain source text to be translated and a target language; and an input module configured to input the source text and the target language into a trained language model to obtain target text in the target language output by the language model, wherein the language model is trained according to the training device described above.

[0010] According to an aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any of the above aspects.

[0011] According to an aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions being used to cause a computer to perform the method of any of the above aspects.

[0012] According to an aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, implement the method of any of the above aspects.

[0013] According to one or more embodiments of the present disclosure, the multilingual translation quality of the language model can be improved.

[0014] It is to be understood that the details set forth herein do not limit the key or critical features of the present disclosure. Other features of the present disclosure will be apparent from the following specification, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, and together with the description serve to explain exemplary implementations of the present disclosure. The illustrated embodiments are merely examples of the present disclosure and are not intended to restrict the scope of the claims in any way.

[0016] Figure 1 shows a schematic diagram of an exemplary system in which the various methods described herein can be implemented according to embodiments of the present disclosure;

[0017] Figure 2 shows a flowchart of a training method of a language model for multilingual translation according to embodiments of the present disclosure;

[0018] Figure 3 shows a schematic diagram of a training process of a language model according to embodiments of the present disclosure;

[0019] Figure 4 shows a flowchart of a translation method according to embodiments of the present disclosure;

[0020] Figure 5 shows a structural block diagram of a training apparatus of a language model for multilingual translation according to embodiments of the present disclosure;

[0021] Figure 6 shows a structural block diagram of a translation apparatus according to embodiments of the present disclosure; and

[0022] Figure 7 shows a structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, and together with the description serve to explain exemplary implementations of the present disclosure. The illustrated embodiments are merely examples of the present disclosure and are not intended to restrict the scope of the claims in any way. In the drawings:

[0024] In the present disclosure, the terms "first", "second", and the like are used to describe various elements only for the purpose of distinguishing one element from another, and do not intend to imply a relative position relationship, a temporal sequence relationship, or a relative importance between the elements. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0025] The terms used in the description of various described examples in the present disclosure are only for the purpose of describing specific examples, and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in the present disclosure encompasses any one of the listed items and all possible combinations. "Multiple" refers to two or more.

[0026] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information involved comply with relevant laws and regulations and do not violate public order and good customs.

[0027] In the related art, parallel corpus, i.e., a pair of texts composed of texts with the same semantics in two different languages, is needed to train a large language model to have translation capabilities.

[0028] The translation quality of a large language model for a certain language direction (hereinafter referred to as "language direction") depends on the amount of parallel corpus for the language direction. Generally, the translation quality of a large language model for a language direction with more parallel corpus (i.e., supervised language direction) is better, but the translation effect for a language direction with less parallel corpus (i.e., low-resource language direction) and a language direction without parallel corpus (i.e., zero-resource language direction) is not good. The translation quality in the zero-resource language direction is particularly unstable and has a large gap with the translation quality in the supervised language direction.

[0029] For example, English is a language widely used around the world, and a large amount of parallel corpus of English and other languages can be conveniently obtained, and the large language model can be trained using these parallel corpus, so that the large language model can achieve good translation effect in the English-related language direction. For language directions not involving English, such as Arabic to Czech, it is difficult to obtain parallel corpus for the language direction, resulting in poor translation effect of the large language model for the language direction.

[0030] To solve the above problems, the embodiments of the present disclosure provide a training method of a language model for multilingual translation. The loss value used in the training process of the language model can represent the semantic consistency of text generation across language directions. Adjusting the model parameters based on the loss value can enable the language model to map texts with the same semantics in different languages to similar positions in the latent space, learn better multilingual semantic representations, and thus improve the translation quality of the language model in different language directions, especially in low-resource language directions and zero-resource language directions.

[0031] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0032] Figure 1 A schematic diagram of an example system 100 in which various methods and apparatus described herein can be implemented according to embodiments of the present disclosure is shown. Referring to Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more application programs.

[0033] In embodiments of the present disclosure, the client devices 101, 102, 103, 104, 105, and 106 and the server 120 can run one or more services or software applications that enable the execution of a training method or a translation method of a language model for multilingual translation.

[0034] In certain embodiments, the server 120 can also provide other services or software applications, which can include non-virtual and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, for example, to users of the client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0035] In Figure 1 In the illustrated configuration, the server 120 can include one or more components implementing the functionality performed by the server 120. These components can include software components, hardware components, or a combination thereof, executable by one or more processors. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 can in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that various different system configurations are possible, which can differ from the system 100. Thus, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0036] Client devices 101, 102, 103, 104, 105, and / or 106 can provide an interface that enables a user of the client device to interact with the client device. The client device can also output information to the user via the interface. Although Figure 1 Only six client devices are depicted, but those skilled in the art will understand that the present disclosure can support any number of client devices.

[0037] Client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service kiosk devices, service robots, in-vehicle devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems; or including various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular telephones, smartphones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices can include head-mounted displays, such as smart glasses, and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices are capable of executing a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications), short message service (SMS) applications, and can use various communication protocols.

[0038] Network 110 can be any type of network familiar to those skilled in the art, which can support data communications using any of a variety of available protocols, including without limitation TCP / IP, SNA, IPX, etc. As examples only, one or more of networks 110 can be a LAN, an Ethernet network, a Token Ring network, a WAN, the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a local area network (LAN), a wide area network (WAN), a wireless network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a Bluetooth network), and / or any combination of these and / or other networks.

[0039] The server 120 can include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other appropriate arrangement and / or combination. The server 120 can include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for the servers). In various embodiments, the server 120 can run one or more services or software applications that provide the functionality described below.

[0040] The computing units in the server 120 can run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. The server 120 can also run any of a variety of additional server applications and / or mid-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0041] In some embodiments, the server 120 can include one or more applications to analyze and consolidate data feeds and / or event updates from users of the client devices 101, 102, 103, 104, 105, and / or 106. The server 120 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client devices 101, 102, 103, 104, 105, and / or 106.

[0042] In some embodiments, the server 120 can be a server of a distributed system, or a server that incorporates blockchain. The server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services.

[0043] The system 100 can also include one or more databases 130. In certain embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store information such as audio files and video files. The databases 130 can reside in various locations. For example, databases used by the server 120 can be local to the server 120 or can be remote from the server 120 and can communicate with the server 120 via a network- based or dedicated connection. The databases 130 can be of different types. In certain embodiments, databases used by the server 120 can be, for example, relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

[0044] In certain embodiments, one or more of the databases 130 can also be used by applications to store application data. Databases used by applications can be different types of databases, such as key-value stores, object stores, or regular stores supported by file systems.

[0045] Figure 1 The system 100 of FIG. 1 can be configured and operated in various ways to enable the various methods and apparatuses described in accordance with the present disclosure to be applied.

[0046] According to some embodiments, the server 120 can perform the training method of the language model for multilingual translation of the embodiments of the present disclosure to obtain a trained language model. Further, the server 120 can perform the translation method of the embodiments of the present disclosure using the trained language model to provide a multilingual translation service for the client devices 101-106. For example, a user can specify source language text (e.g., Chinese text) to be translated and a target language (e.g., English) through the client device 101-106. The client device 101-106 sends the user-specified source language text and target language to the server 120. The server 120 translates the source language text into target language text by invoking the trained language model and returns it to the client device 101-106.

[0047] According to some embodiments, the server 120 can perform the training method of the language model for multilingual translation of the embodiments of the present disclosure to obtain a trained language model. The trained language model can be deployed to the client devices 101-106. The client devices 101-106 can perform the translation method of the embodiments of the present disclosure using the locally deployed trained language model to provide a multilingual translation service for the user.

[0048] According to some embodiments, the client devices 101-106 can also perform the training method of the language model for multilingual translation of embodiments of the present disclosure to obtain a trained language model. This generally requires the client devices 101-106 to have a higher hardware configuration and computing power.

[0049] Figure 2 A flowchart of the training method 200 of the language model for multilingual translation according to embodiments of the present disclosure is shown. The execution subject of each step of the method 200 is generally a server, such as the server 120 shown in Figure 1 In some embodiments, the execution subject of the method 200 can also be a client device, such as the client devices 101-106 shown in Figure 1

[0050] As shown in Figure 2 The method 200 includes steps S210-S250.

[0051] In step S210, a sample text pair is obtained. The sample text pair includes a first text in a first language and a second text in a second language. The semantics of the first text is the same as the semantics of the second text.

[0052] In step S220, the first text and the second language are input into the language model to obtain a first translated text in the second language output by the language model.

[0053] In step S230, the second text and the second language are input into the language model to obtain a second translated text in the second language output by the language model.

[0054] In step S240, a loss value of the language model is determined based on a first difference between the first translated text and the second text and a second difference between the first translated text and the second translated text.

[0055] In step S250, the parameters of the language model are adjusted based on the loss value.

[0056] According to embodiments of the present disclosure, the first difference can represent the semantic consistency of the text generated by the language model in the cross-language direction with the original text, and the second difference can represent the semantic consistency of the text generated by the language model in the cross-language direction with the text generated within a single language. Determining the loss value of the language model based on the first difference and the second difference and adjusting the model parameters accordingly can enable the language model to map texts with the same semantics in different languages to similar positions in the latent space, learn better multilingual semantic representations, and thus improve the translation quality of the language model in different language directions, especially in low-resource language directions and zero-resource language directions.

[0057] ​According to some embodiments, the language model to be trained can be a large language model of a decoder structure (Decoder-Only). The number of model parameters of the large language model can reach hundreds of billions or even trillions. The large language model of the decoder structure only includes a decoder in a transformer structure, but does not include an encoder. It has been tested that the large language model of the decoder structure has good performance on the text generation task. Text translation is a cross-language text generation task, and thus the use of the large language model of the decoder structure can achieve good translation results.

[0058] In step S210, a sample text pair is obtained. There are usually multiple sample text pairs. Each sample text pair is a parallel corpus, which includes two texts of different languages with the same semantics, i.e., a first text in a first language and a second text in a second language. For example, the sample text pair can be a Chinese-English text pair (today the weather is good, The weather is good today), an English-Japanese text pair (The weather is good today, Today is a good weather), etc.

[0059] According to some embodiments, the method 200 further includes pre-training the language model by using a first corpus of the first language and a second corpus of the second language, to obtain a pre-trained language model. The pre-trained language model has the ability to understand texts in the first language and the second language.

[0060] In the case where multiple sample text pairs involve multiple languages (i.e., there are multiple first languages and / or multiple second languages), the language model needs to be pre-trained by using the corpus of each language to obtain a pre-trained language model. In this way, the language model can have the ability to understand multiple languages.

[0061] According to some embodiments, the language model can be pre-trained in an unsupervised manner. That is, the language model is pre-trained by using a large amount of unlabeled corpus of different languages.

[0062] On the basis of the pre-trained language model, the pre-trained language model is fine-tuned by using the sample text pairs in steps S210-S250, so that the fine-tuned language model can be used to perform the downstream task of multilingual machine translation. By fine-tuning the language model for the multilingual translation task, the language model can have excellent machine translation capability on the basis of reusing the understanding ability of the language model for multiple languages (obtained by pre-training), and the convergence speed and generalization of the language model can be improved.

[0063] According to some embodiments, in steps S220 and S230, a preset prompt template can be utilized to guide the language model to generate the first translation text and the second translation text.

[0064] The prompt template is used to guide the language model to translate the source text in the source language into the target text in the target language. The prompt template can include multiple slots to be filled, for example, including multiple slots for filling the source language, the source text, the target language, and the target text respectively. By filling the respective data in the sample text pair into the corresponding slots, the query text (e.g., the first query text and the second query text below) can be generated. By inputting the query text into the language model, the translation text (e.g., the first translation text and the second translation text) output by the language model can be obtained.

[0065] Several examples of the prompt template are given below for reference. In the following prompt templates, {} represents a slot to be filled, and \n represents a line break.

[0066] Prompt template 1: {target language} {source text}\n{target text}

[0067] Prompt template 2: {source language} {target language}: {source text}\n{target text}

[0068] Prompt template 3: Translate this from {source language} into {target language}:\n{source language}: {source text}\n{target language}: {target text}

[0069] According to some embodiments, step S220 can include steps S221-S223.

[0070] In step S221, a preset prompt template is obtained. The prompt template is used to guide the language model to translate the source text in the source language into the target text in the target language. The prompt template can be any one of the prompt templates 1-3 described above, or other prompt templates other than the prompt templates 1-3 described above.

[0071] In step S222, the first language is taken as the source language, the second language is taken as the target language, and the first text is taken as the source text to generate the first query text. That is, the first language, the second language, and the first text are filled into the slots of the source language, the target language, and the source text in the prompt template respectively to obtain the first query text.

[0072] For example, the sample text pair is a Chinese-English text pair (today the weather is good, The weather is good today). Using the prompt template 3 described above, the first query text is generated as follows:

[0073]

[0074] The weather is good today

[0075]

[0076] In step S223, the generated first query text is input into the language model to obtain a first translation text in the second language output by the language model.

[0077] According to the above embodiment, the translation quality of the language model can be improved by using the prompt template to guide the language model to perform text translation.

[0078] According to some embodiments, step S230 can include steps S231-S233.

[0079] In step S231, a preset prompt template is obtained. The prompt template is used to guide the language model to translate the source text in the source language into the target text in the target language. The prompt template may, for example, be any one of the above prompt templates 1-3, or other prompt templates other than the above prompt templates 1-3.

[0080] In step S232, the second language is taken as the source language and the target language, and the second text is taken as the source text to generate a second query text. That is, the second language is filled into the slots of the source language and the target language in the prompt template, and the second text is filled into the slot of the source text to obtain the second query text.

[0081] For example, the sample text pair is a Chinese-English text pair (The weather is good today). Using the above prompt template 3, the second query text is generated as follows:

[0082]

[0083] The weather is good today

[0084]

[0085] In step S233, the generated second query text is input into the language model to obtain a second translation text in the second language output by the language model.

[0086] According to the above embodiment, the translation quality of the language model can be improved by using the prompt template to guide the language model to perform text translation. ​​​​

[0087] In step S240, a loss value of the language model is determined based on the first difference between the first translation text and the second text and the second difference between the first translation text and the second translation text.

[0088] According to some embodiments, the first translation text can be represented as a first probability distribution. It can be understood that the language model generates each token in the first translation text in turn until an end of sentence (eos) token is generated. When generating each token in the first translation text, a probability of the token being each token in a preset second language vocabulary can be calculated, and then the token with the highest probability is taken as the current token in the first translation text. Therefore, each token in the first translation text corresponds to a probability distribution. The first probability distribution of the first translation text can be a combination of the probability distributions of each token, or a probability vector composed of the maximum probability of each token.

[0089] Similarly, the second translation text can be represented as a second probability distribution. It can be understood that the language model generates each token in the second translation text in turn until an end of sentence (eos) token is generated. When generating each token in the second translation text, a probability of the token being each token in a preset second language vocabulary can be calculated, and then the token with the highest probability is taken as the current token in the second translation text. Therefore, each token in the second translation text corresponds to a probability distribution. The second probability distribution of the second translation text can be a combination of the probability distributions of each token, or a probability vector composed of the maximum probability of each token.

[0090] The second text can be represented as one-hot encoding, that is, the second text can be represented as a vector whose dimension is the same as the number of tokens included in the second language vocabulary, and each element in the vector corresponds to a token in the second language vocabulary. In the vector, the value of the element corresponding to the token in the second text is 1, and the values of the elements corresponding to other tokens are 0.

[0091] It can be understood that the first translation text and the second translation text are both prediction values output by the language model. The second text is the true value corresponding to the first translation text and the second translation text.

[0092] The first difference between the first translated text and the second text can represent semantic consistency of the text generated by the language model in the cross-lingual direction (i.e., the first translated text) with the original text (i.e., the first text or the second text, which have the same semantics). According to some embodiments, the first difference between the first translated text and the second text can be a cross-entropy of the first probability distribution with the one-hot encoding.

[0093] The second difference between the first translated text and the second translated text can represent semantic consistency of the text generated by the language model in the cross-lingual direction (i.e., the first translated text) with the text generated within a single language (i.e., the second translated text). According to some embodiments, the second difference between the first translated text and the second translated text can be a KL divergence of the first probability distribution with the second probability distribution.

[0094] The loss value of the language model can be a weighted sum of the first difference and the second difference.

[0095] According to some embodiments, the loss value of the language model can be calculated according to the following equations (1)-(3):

[0096] L(θ) = L CE (θ) + aL KL (θ) (1)

[0097]

[0098] L KL (θ) = KL(f(x, y; θ) || f(y, y; θ)) (3)

[0099] In the above equation (1), L(θ) is the loss value of the language model, L CE (θ), L KL (θ) are the cross-entropy loss (i.e., the first difference) and the KL divergence loss (i.e., the second difference), respectively, θ is a parameter of the language model, and a is a preset constant (which is a hyperparameter of the language model).

[0100] In the above equation (2), E() represents calculating an expectation, (x, y) represents a sample text pair composed of a first text x in a first language and a second text y in a second language, S is a set of sample text pairs, l() is a cross-entropy calculation function, f(x, y; θ) is the first probability distribution, is a one-hot representation of the second text.

[0101] In the above equation (3), KL() is a KL divergence calculation function, f(x, y; θ), f(y, y; θ) are the first probability distribution, the second probability distribution, respectively.

[0102] According to an embodiment of the present disclosure, the regularization of cross-lingual semantic consistency is introduced in the training process of the language model, which can make the language model map the texts with the same semantics in different languages to the similar positions in the latent space, learn better multilingual semantic representations, and thus improve the translation quality of the language model in different language directions, especially in low-resource language directions and zero-resource language directions.

[0103] According to an embodiment of the present disclosure, the translation quality in zero-resource language directions can be improved. Specifically, the first language and the second language in the sample text pair can be any two languages in a preset language set, and the language set includes at least three languages. Generally, the language set includes many languages (sometimes as many as 200 or more), and the number of language directions corresponding to the language set is also large, A(n, 2), where A represents the arrangement number, and n is the number of languages included in the language set. The number of language directions involved in the sample text pair N is usually much smaller than the number of language directions of the language set, i.e., N << A(n, 2).

[0104] The trained language model can be used to translate a third text in a third language into a fourth text in a fourth language. Wherein the third language and the fourth language both belong to the language set, and the language direction from the third language to the fourth language is different from the language direction from the first language to the second language. That is, the language direction from the third language to the fourth language never appears in the sample text pair, which is a zero-resource language direction. According to an embodiment of the present disclosure, the trained language model can still achieve good translation results for the zero-resource language direction.

[0105] According to some embodiments, the language model can be pre-trained using the corpus of each language in the language set to obtain a pre-trained language model. The pre-trained language model has the ability to understand each language in the language set.

[0106] On the basis of the pre-trained language model, the pre-trained language model is fine-tuned by steps S210-S250 using the sample text pair, so that the fine-tuned language model can specifically perform the downstream task of multilingual machine translation, and the translation quality of the language model for the zero-resource language direction is improved.

[0107] For example, the language set includes Chinese, English, German, Russian, and Czech. The language directions of the sample text pair include According to an embodiment of the present disclosure, after training the language model using the above sample text pair, the translation quality in the zero-resource language direction from Czech to Russian can also be improved.

[0108] Figure 3 A schematic diagram of the training process of the language model 300 according to an embodiment of the present disclosure is shown. As Figure 3As shown, the language model 300 adopts a decoder structure (Decoder-Only), which includes a transform decoder (Transformer Decoder) 310 and a regression layer (Softmax) 320.

[0109] The sample text pair is a Chinese-English text pair (The weather is good today). Using the prompt template: Translate this from {source language} into {target language}:\n{source language}:{source text}\n{target language}:{target text}, the sample text pair is generated into a first query text 301 and a second query text 302. The first query text 301 is:

[0110] Translate this from Chinese into English:

[0111] English: The weather is very good today

[0112] English:

[0113] The second inquiry text 302 is:

[0114] Translate this from English into English:

[0115] English:The weather is good today

[0116] English:

[0117] Input the first query text 301 and the second query text 302 into the language model 300 respectively, and obtain the first probability distribution f(x, y; θ) (i.e., the first translation text) and the second probability distribution f(y, y; θ) (i.e., the second translation text) output by the language model 300. Based on the first probability distribution f(x, y; θ) and the second text The weather is good today <eos>cross-entropy loss value L CE Based on the KL divergence of the first probability distribution f(x, y; θ) and the second probability distribution f(y, y; θ), the KL divergence loss value L KL is calculated. CE The L KL and L KL are weighted and summed to obtain the overall loss value L of the model. Further, based on L, the error is back propagated to adjust the values of the parameters of the language model 300.

[0118] The effect of the method 200 of the embodiments of the present disclosure is tested based on an open source multilingual large language model and an open source translation data set. The test conditions are as follows.

[0119] Training languages: Czech, German, Russian, Chinese, English.

[0120] Training data: WMT17-20 English-centric data sets; Flores-200dev data sets corresponding to the above languages.

[0121] Test data: WMT22 test set (for supervised language direction); Flores-200 devtest data set (for zero-resource language direction).

[0122] Test indicators: SacreBLEU and COMET based on wmt22-comet-da

[0123] Multilingual large language base model used: ALMA-7B-Pretrain (Haoran Xu and Young Jin Kim and Amr Sharaf and Hany Hassan Awadalla, "A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models", arXiv:2309.11674.)

[0124] Prompt template used for testing:

[0125] 1. T-ENC: {target language} {source sentence}\n{target sentence}

[0126] 2. S-ENC-T-ENC: {source language} {target language}: {source sentence}\n{target sentence}

[0127] 3. ALMA:Translate this from{source language}into{target language}:\n{source language}:{source sentence}\n{target language}:{target sentence}

[0128] Test results:

[0129]

[0130] As can be seen from the above table, for zero-resource direction, the test index values under different prompt templates are all improved after using the loss function of the present disclosure, which shows that the loss function of the present disclosure can generally improve the translation effect of zero-resource direction (regardless of which prompt template is used).

[0131] For supervised direction, the evaluation indexes of each prompt template are all slightly reduced after using the loss function of the present disclosure. This reduction is caused by the difference in the model training process (for example, the initial values of the randomly generated model parameters in different training processes are different), and is not caused by the loss function. Therefore, it can be considered that the translation effect of the supervised direction is basically flat before and after using the loss function of the present disclosure.

[0132] According to an embodiment of the present disclosure, a translation method is also provided. The method translates source text in a source language into target text in a target language by calling a trained language model.

[0133] Figure 4 A flowchart of a translation method 400 according to an embodiment of the present disclosure is shown. The execution subject of the method 400 is usually a server. In some embodiments, the execution subject of the method 400 can also be a client device, which usually requires the client device to have a higher hardware configuration and computing power. As shown in the figure, Figure 4 The method 400 includes steps S410 and S420.

[0134] In step S410, the source text to be translated and the target language are obtained.

[0135] In step S420, the source text and the target language are input into the trained language model to obtain the target text in the target language output by the language model. The language model is trained according to the training method of the language model for multilingual translation according to an embodiment of the present disclosure.

[0136] According to an embodiment of the present disclosure, the translation quality of different language directions (especially low-resource language directions and zero-resource language directions) can be improved by using the trained language model.

[0137] According to some embodiments, in step S420, the source text and the target language can be constructed into a query text by using a preset prompt template. The query text is input into the trained language model to obtain a translation result output by the language model, i.e., a target text in the target language.

[0138] Examples of the prompt template and the generation manner of the query text can refer to steps S220 and S230 described above. Details are not described herein again.

[0139] According to an embodiment of the present disclosure, a training device for a language model for multilingual translation is also provided. Figure 5 A structural block diagram of the training device 500 for the language model for multilingual translation according to an embodiment of the present disclosure is shown. As shown in the figure, Figure 5 The device 500 includes an acquisition module 510, a first input module 520, a second input module 530, a determination module 540, and an adjustment module 550.

[0140] The acquisition module 510 is configured to acquire a sample text pair, where the sample text pair includes a first text in a first language and a second text in a second language, and the semantics of the first text is the same as the semantics of the second text.

[0141] The first input module 520 is configured to input the first text and the second language into the language model to obtain a first translation text in the second language output by the language model.

[0142] The second input module 530 is configured to input the second text and the second language into the language model to obtain a second translation text in the second language output by the language model.

[0143] The determination module 540 is configured to determine a loss value of the language model based on a first difference between the first translation text and the second text and a second difference between the first translation text and the second translation text.

[0144] The adjustment module 550 is configured to adjust parameters of the language model based on the loss value.

[0145] According to an embodiment of the present disclosure, the first difference can represent semantic consistency of the text generated by the language model in the cross-language direction with the original text, and the second difference can represent semantic consistency of the text generated by the language model in the cross-language direction with the text generated in the single language. Determining the loss value of the language model based on the first difference and the second difference and adjusting the model parameters accordingly can enable the language model to map texts with the same semantics in different languages to similar positions in the latent space, learn better multilingual semantic representations, and thus improve the translation quality of the language model in different language directions, especially in low-resource language directions and zero-resource language directions.

[0146] According to some embodiments, the apparatus 500 further comprises: pre-training the language model using a first corpus of the first language and a second corpus of the second language to obtain a pre-trained language model, wherein the sample text is used to fine-tune the pre-trained language model.

[0147] According to some embodiments, the inputting the first text and the second language into the language model to obtain the first translated text of the second language output by the language model comprises: obtaining a preset prompt template, wherein the prompt template is used to guide the language model to translate source text in a source language into target text in a target language; generating a first inquiry text by taking the first language as the source language, the second language as the target language, and the first text as the source text; and inputting the first inquiry text into the language model to obtain the first translated text output by the language model.

[0148] According to some embodiments, the inputting the second text and the second language into the language model to obtain the second translated text of the second language output by the language model comprises: obtaining a preset prompt template, wherein the prompt template is used to guide the language model to translate source text in a source language into target text in a target language; generating a second inquiry text by taking the second language as the source language and the target language, and the second text as the source text; and inputting the second inquiry text into the language model to obtain the second translated text output by the language model.

[0149] According to some embodiments, the first translated text is represented as a first probability distribution, the second translated text is represented as a second probability distribution, and the second text is represented as a one-hot encoding; the first difference is a cross-entropy of the first probability distribution and the one-hot encoding; and the second difference is a KL divergence of the first probability distribution and the second probability distribution.

[0150] According to some embodiments, the first language and the second language are any two languages ​​in a preset language set, wherein the language set includes at least three languages; the trained language model is used to translate a third text in a third language into a fourth text in a fourth language, wherein the third language and the fourth language both belong to the language set, and a language direction from the third language to the fourth language is different from a language direction from the first language to the second language.

[0151] According to some embodiments, the apparatus 500 further comprises: pre-training the language model using corpus of each language in the language set to obtain a pre-trained language model, wherein the sample text is used to fine-tune the pre-trained language model.

[0152] According to some embodiments, the language model is a large language model of a decoder structure.

[0153] It should be understood that Figure 5 The modules and units of the apparatus 500 shown in FIG. 5 can be compared with those of the reference Figure 2 The steps in the method 200 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 200 are also applicable to the apparatus 500 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0154] According to an embodiment of the present disclosure, a translation device is also provided. Figure 6 FIG. 6 shows a structural block diagram of a translation device 600 according to an embodiment of the present disclosure. Figure 6 As shown, the apparatus 600 includes an acquisition module 610 and an input module 620 .

[0155] The acquisition module 610 is configured to acquire a source text and a target language to be translated.

[0156] The input module 620 is configured to input the source text and the target language into a trained language model to obtain a target text in the target language output by the language model, wherein the language model is trained according to the apparatus 500 of the embodiment of the present disclosure.

[0157] According to the embodiments of the present disclosure, the trained language model can be used to improve the translation quality of different languages ​​(especially low-resource languages ​​and zero-resource languages).

[0158] It should be understood that Figure 6 The modules and units of the apparatus 600 shown in FIG. 6 can be compared with those of the reference Figure 4 The steps in the method 400 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 400 are also applicable to the apparatus 600 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0159] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various modules discussed herein may be separated into multiple modules, and / or at least some functionality of multiple modules may be combined into a single module.

[0160] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 5 、 Figure 6 The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program code / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of modules 510-550, 610-620 can be implemented together in a system on chip (SoC). SoC can include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and can optionally execute the received program code and / or include embedded firmware to perform a function.

[0161] According to an embodiment of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the language model training method and / or translation method for multilingual translation of the embodiment of the present disclosure.

[0162] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided. The computer instructions are used to enable a computer to execute the language model training method and / or translation method for multilingual translation of the embodiment of the present disclosure.

[0163] According to an embodiment of the disclosure, a computer program product is also provided, comprising computer program instructions which, when executed by a processor, implement the training method and / or the translation method of the language model for multilingual translation according to the embodiments of the disclosure.

[0164] Reference Figure 7 The structure block diagram of the electronic device 700 which can be applied to various aspects of the disclosure will now be described, which is an example of a hardware device that can be applied to the present disclosure. The electronic device is intended to represent a variety of forms of digital electronic computing devices such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computing devices. The electronic device can also represent a variety of forms of mobile devices such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0165] As Figure 7 shown, the electronic device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0166] Multiple components within electronic device 700 are connected to I / O interface 705, including an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. Input unit 706 can be any type of device capable of inputting information into electronic device 700. Input unit 706 can receive input numeric or character information and generate key signal input related to user settings and / or function control of the electronic device. It can include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 707 can be any type of device capable of presenting information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 708 can include, but is not limited to, a magnetic disk or an optical disk. Communication unit 709 allows electronic device 700 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks. It can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or chipset, such as a Bluetooth device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communication device, and / or the like.

[0167] The computing unit 701 may be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as method 200 or method 400. For example, in some embodiments, method 200 and method 400 may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the methods 200 and 400 described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the method 200 or the method 400 in any other appropriate manner (eg, by means of firmware).

[0168] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0169] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0170] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0171] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0172] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0173] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is generally established by computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers combined with a blockchain.

[0174] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0175] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalents. Various elements in the embodiments or examples can be omitted or replaced by their equivalents. In addition, the steps can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that as technology evolves, many of the elements described here can be replaced by equivalent elements that appear after the present disclosure.< / eos>

Claims

1. A method for training a language model for multilingual translation, comprising: obtaining a sample text pair, wherein the sample text pair comprises a first text in a first language and a second text in a second language, the first text having a same semantic meaning as the second text; inputting the first text as a source text, the first language as a source language, and the second language as a target language into the language model to obtain a first translated text in the second language output by the language model; inputting the second text as a source text, the second language as a source language and a target language into the language model to obtain a second translated text in the second language output by the language model; determining a loss value of the language model based on a first difference between the first translated text and the second text and a second difference between the first translated text and the second translated text, the second difference representing semantic consistency of the first translated text generated by the language model in a cross-language direction and the second translated text generated within a single language; and adjusting parameters of the language model based on the loss value.

2. The method of claim 1, further comprising: pre-training the language model using a first corpus in the first language and a second corpus in the second language to obtain a pre-trained language model, wherein the sample text pair is used to fine-tune the pre-trained language model.

3. The method of claim 1 or 2, wherein, the inputting the first text and the second language into the language model to obtain a first translated text in the second language output by the language model comprises: obtaining a preset prompt template, wherein the prompt template is used to guide the language model to translate a source text in a source language into a target text in a target language; generating a first query text with the first language as the source language, the second language as the target language, and the first text as the source text; and inputting the first query text into the language model to obtain the first translated text output by the language model.

4. The method of claim 1, wherein, the inputting the second text and the second language into the language model to obtain a second translated text in the second language output by the language model comprises: obtaining a preset prompt template, wherein the prompt template is used to guide the language model to translate a source text in a source language into a target text in a target language; generating a second query text with the second language as the source language and the target language, and the second text as the source text; and inputting the second query text into the language model to obtain the second translated text output by the language model.

5. The method of claim 1, wherein: the first translated text is represented as a first probability distribution, the second translated text is represented as a second probability distribution, and the second text is represented as a one-hot encoding; the first difference is a cross-entropy between the first probability distribution and the one-hot encoding; the second difference is a KL divergence between the first probability distribution and the second probability distribution.

6. The method of claim 1, wherein: The first language and the second language are any two languages in a preset language set, wherein the language set includes at least three languages; The trained language model is used to translate third text in a third language into fourth text in a fourth language, wherein the third language and the fourth language both belong to the language set, and a language direction from the third language to the fourth language is different from a language direction from the first language to the second language.

7. The method of claim 6, further comprising: pre-training the language model using a corpus of each language in the language set to obtain a pre-trained language model, wherein the sample text pair is used to fine-tune the pre-trained language model.

8. The method of claim 1, wherein, The language model is a large language model of a decoder structure.

9. A translation method, comprising: obtaining source text to be translated and a target language; and inputting the source text and the target language into a trained language model to obtain target text in the target language output by the language model, wherein the language model is trained according to the method of any one of claims 1-8.

10. A training device for a language model for multilingual translation, comprising: an obtaining module configured to obtain a sample text pair, wherein the sample text pair includes first text in a first language and second text in a second language, and the semantics of the first text are the same as the semantics of the second text; a first input module configured to input the first text as source text, the first language as a source language, and the second language as a target language into the language model to obtain first translated text in the second language output by the language model; a second input module configured to input the second text as source text, the second language as a source language and a target language into the language model to obtain second translated text in the second language output by the language model; a determination module configured to determine a loss value of the language model based on a first difference between the first translated text and the second text and a second difference between the first translated text and the second translated text, the second difference representing semantic consistency of the first translated text generated by the language model in a cross-language direction and the second translated text generated within a single language; and an adjustment module configured to adjust parameters of the language model based on the loss value.

11. The device of claim 10, further comprising: a first pre-training module configured to pre-train the language model using a first corpus of the first language and a second corpus of the second language to obtain a pre-trained language model, wherein the sample text pair is used to fine-tune the pre-trained language model. The first input module includes:

12. The apparatus of claim 10 or 11, wherein, a first obtaining unit configured to obtain a preset prompt template, wherein the prompt template is used to guide the language model to translate source text in a source language into target text in a target language; ​ The first generating unit is configured to generate a first query text by taking the first language as the source language, the second language as the target language, and the first text as the source text. The first input unit is configured to input the first query text into the language model to obtain the first translation text output by the language model.

13. The apparatus of claim 10, wherein, The second input module includes: The second obtaining unit is configured to obtain a preset prompt template, where the prompt template is used to guide the language model to translate source text in a source language into target text in a target language; The second generating unit is configured to generate a second query text by taking the second language as the source language and the target language, and the second text as the source text; and The second input unit is configured to input the second query text into the language model to obtain the second translation text output by the language model.

14. The apparatus of claim 10, wherein: the first translation text is represented as a first probability distribution, the second translation text is represented as a second probability distribution, and the second text is represented as a one-hot encoding; the first difference is a cross-entropy of the first probability distribution and the one-hot encoding; the second difference is a KL divergence of the first probability distribution and the second probability distribution.

15. The apparatus of claim 10, wherein: the first language and the second language are any two languages in a preset language set, where the language set includes at least three languages; the trained language model is used to translate third text in a third language into fourth text in a fourth language, where the third language and the fourth language both belong to the language set, and a language direction from the third language to the fourth language is different from a language direction from the first language to the second language.

16. The apparatus of claim 15, further comprising: a second pre-training module configured to pre-train the language model by using a corpus of each language in the language set to obtain a pre-trained language model, wherein the sample text pair is used to fine-tune the pre-trained language model.

17. The apparatus of claim 10, wherein, The language model is a large language model of a decoder structure.

18. A translation apparatus, comprising: an obtaining module configured to obtain source text to be translated and a target language; and an input module configured to input the source text and the target language into a trained language model to obtain target text in the target language output by the language model, wherein the language model is trained by the apparatus of any one of claims 10-17.

19. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8. ​ ​ 20. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are for causing a computer to perform the method of any one of claims 1-8.

21. A computer program product comprising computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the method of any one of claims 1-8.

Images