Machine translation model training method and apparatus, device, medium, and product

Abstract

The application relates to a machine translation model training method and device, equipment, medium and product, the method comprising: obtaining a data set comprising a plurality of parallel corpora; constructing a plurality of variant models of a machine translation model, determining a single variant model as a student model and the rest as teacher models; training each teacher model to a convergent state using parallel corpora in the data set; constructing a knowledge distillation network, inputting parallel corpora in the data set into the knowledge distillation network to implement training, and jointly supervising the student model training to a convergent state through a plurality of teacher models. Based on the machine translation model, a plurality of variant models with different advantage reasoning capabilities are prepared in advance as teacher models, and then the advantage reasoning capabilities of the plurality of teacher models are transmitted to the same student model through the knowledge distillation mode, so that the student model has more comprehensive translation reasoning capability and is suitable for translating commodity information of various commodity categories.

Description

Technical Field

[0001] This application relates to machine translation technology, and more particularly to a machine translation model training method, apparatus, device, medium, and product thereof. Background Technology

[0002] In cross-border e-commerce, translating product information serves as a crucial bridge between sellers and buyers. Accurate translations play a vital connecting role in product display, search, and recommendation processes. High-quality translated product information not only significantly enhances buyers' understanding of the products but also increases the exposure of suitable products to suitable buyers, thereby boosting sales and conversion rates on cross-border e-commerce websites.

[0003] The translation of product information is typically carried out using machine translation models. However, the parallel corpora currently used to train these models are mainly general-domain corpora, which are of low quality and quite disorganized. Product information contains a large number of proper nouns, including product terms, locations, and popular terms, which can be summarized as proper nouns. Machine translation models trained using currently available parallel corpora often fail to accurately translate these proper nouns, sometimes even resulting in inaccurate translations of entire sentences. Creating parallel corpora specifically for cross-border e-commerce scenarios would, predictably, be extremely costly and inefficient.

[0004] One improved approach is to pre-select proper nouns during machine translation model training, preventing the model from translating the selected parts. After the model obtains the translation results, the selected parts are then restored to proper nouns in the corresponding language. However, this approach is still limited by the number of parallel corpora for different product categories, which prevents the machine translation model from effectively generalizing and results in inaccurate translations of product information for some categories.

[0005] Knowledge distillation can, to some extent, compensate for the deficiencies caused by the poor quality of parallel corpora. In knowledge distillation, a teacher model is first trained using the existing parallel corpus, then used to train a student model, which is finally deployed in practical applications. In traditional knowledge distillation, the teacher and student models use the same network structure and interact one-to-one. This results in little difference in translation quality between the trained student and teacher models, leading to homogenization. Using multiple student models with different product category adaptability to provide translation services will negatively impact training costs, deployment costs, and response efficiency, resulting in unsatisfactory practical results.

[0006] It is evident that traditional machine translation models, due to the uneven quality of their corpora, even when improved using traditional knowledge distillation training methods, still fail to achieve high-quality translation results when applied to e-commerce product information translation scenarios due to the presence of numerous specialized terms in the product information. Summary of the Invention

[0007] The purpose of this application is to solve the above-mentioned problems by providing a machine translation model training method and corresponding apparatus, devices, non-volatile readable storage media, and computer program products.

[0008] According to one aspect of this application, a method for training a machine translation model is provided, comprising the following steps:

[0009] Obtain the dataset, which includes multiple parallel corpora;

[0010] Construct multiple variant models of the machine translation model, and determine one of the variant models as the student model and the rest as the teacher model;

[0011] Each teacher model is trained to convergence using parallel corpora from the dataset.

[0012] A knowledge distillation network is constructed, and the knowledge distillation network is trained by inputting parallel corpora from the dataset. The student model is trained to convergence by joint supervision of multiple teacher models.

[0013] Optionally, each of the teacher models is trained to a convergent state using parallel corpora from the dataset, including:

[0014] Obtain a single parallel corpus from the dataset, and replace part or all of the proper terms with the same meaning in the two language corpora with placeholders of the corresponding type.

[0015] The parallel corpus is synchronously input into multiple teacher models to obtain the second language result generated by each teacher model based on the first language corpus in the parallel corpus. The second language result is supervised using the second language corpus in the parallel corpus to obtain the corresponding model loss value.

[0016] Based on the model loss value, determine whether the corresponding teacher model has reached convergence. If it has not reached convergence, perform gradient update on the corresponding teacher model and continue iterative training until the corresponding teacher model is trained to convergence and no longer participates in the iteration.

[0017] Optionally, a single parallel corpus from the dataset is obtained, and proper nouns with the same meaning in the two languages ​​are partially or completely replaced with placeholders of the corresponding type, including:

[0018] The named entity model is used to identify different types of named entities in each parallel corpus of the dataset. The named entities are proper terms used to describe product information.

[0019] A mapping table is established between the proper nouns and their corresponding placeholders. The mapping table is used to store the mapping relationship data between placeholders and proper nouns, so as to restore the placeholders in the second language result.

[0020] Optionally, before training the knowledge distillation network using the parallel corpus as input, the following steps are included:

[0021] One or more converged teacher models are used to infer translation results based on first language corpora in any parallel corpora in the dataset. Before being input into the teacher models, proper terms in the first language corpora are replaced with placeholders of their corresponding types.

[0022] Based on the mapping relationship data between placeholders and proper nouns in the preset mapping table, the placeholders in each translation result are restored to the corresponding type of proper nouns to obtain the second language result;

[0023] Each of the aforementioned second language results is used as a second language corpus and its corresponding first language corpus is recombined into a new parallel corpus, which is then added to the dataset to expand the dataset.

[0024] Optionally, several variant models of the machine translation model can be constructed, including:

[0025] Obtain configuration information for a machine translation model, wherein the machine translation model includes an encoding network and a decoding network, and the configuration information includes type information and / or quantity information for defining the network structure of the encoding network;

[0026] Modify the type information and / or quantity information in the configuration information to obtain multiple derived configuration information, and modify the network structure of the machine translation model according to each derived configuration information to obtain multiple variant models of the machine translation model;

[0027] The variant model with the smallest network structure was selected as the student model in the knowledge distillation network, and the remaining variant models were selected as the teacher models in the knowledge distillation network.

[0028] Optionally, the knowledge distillation network is trained by inputting parallel corpora from the dataset, including:

[0029] Use a single parallel corpus from the dataset as a training sample;

[0030] The first language corpus in the training samples is synchronously input into all teacher models and student models to obtain the second language results generated by the reasoning of each model.

[0031] Determine the comprehensive loss value corresponding to the second language results of all teacher models and the true loss value of the student model, and correct the true loss value of the student model according to the comprehensive loss value to obtain the distillation loss value;

[0032] If the student model fails to converge, the student model is updated with gradients based on the distillation loss value and training continues iteratively until the student model reaches convergence.

[0033] Optionally, a knowledge distillation network is constructed, and training is performed using parallel corpora from the dataset as input to the knowledge distillation network. After the student model is trained to a convergent state under the joint supervision of multiple teacher models, the process includes:

[0034] In response to the translation execution request, retrieve the first language text carried in the request;

[0035] The converged student model is invoked to translate the first language text to obtain the second language text;

[0036] In response to the translation execution request, push the second language text.

[0037] According to another aspect of this application, a machine translation model training apparatus is provided, comprising:

[0038] The data acquisition module is configured to acquire datasets, which include multiple parallel corpora.

[0039] The variant construction module is set up to construct multiple variant models of the machine translation model, and determine one of the variant models as the student model and the rest as the teacher model.

[0040] The teacher training module is configured to train each teacher model to a convergent state using parallel corpora in the dataset.

[0041] The student training module is configured to construct a knowledge distillation network, input the knowledge distillation network with parallel corpora from the dataset for training, and train the student model to convergence through joint supervision by multiple teacher models.

[0042] According to another aspect of this application, a machine translation model training device is provided, including a central processing unit and a memory, wherein the central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the machine translation model training method described in this application.

[0043] According to another aspect of this application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the machine translation model training method in the form of computer-readable instructions, wherein the computer program, when invoked by a computer, executes the steps included in the method.

[0044] According to another aspect of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0045] Compared to existing technologies, this application primarily focuses on model construction and training mechanisms. It constructs multiple teacher models and a single student model, first training each teacher model to convergence. Since these teacher models are different variants of the machine translation model with different network structures, even when trained using the same parallel corpus, their learned reasoning abilities will vary. For example, they may each have advantages in translating product information across different product categories. Based on this, multiple converged teacher models are used to jointly supervise the training of the same student model, implementing knowledge distillation. The reasoning abilities of the multiple teacher models are then integrated and passed on to the student model. This ensures that the reasoning ability learned by the student model is a comprehensive result of the superior reasoning abilities learned by the multiple teacher models. Therefore, the student model can achieve high-quality translation results for product information across different product categories. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the network architecture of an exemplary application environment for this application;

[0048] Figure 2 This is a schematic diagram of the architecture of an exemplary knowledge distillation network of this application;

[0049] Figure 3 This is a flowchart illustrating one embodiment of the machine translation model training method of this application;

[0050] Figure 4 This is a schematic diagram of the process for training the teacher model in an embodiment of this application;

[0051] Figure 5 This is a schematic diagram illustrating the process of expanding the dataset using a teacher model in an embodiment of this application;

[0052] Figure 6 This is a schematic diagram illustrating the process of constructing multiple variant models in an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of the process of implementing knowledge distillation training in the embodiments of this application;

[0054] Figure 8 This is a schematic diagram illustrating the process of performing machine translation services in an embodiment of this application;

[0055] Figure 9 This is a schematic diagram of the machine translation model training device of this application;

[0056] Figure 10 This is a schematic diagram of the structure of a machine translation model training device used in this application. Detailed Implementation

[0057] The models cited or potentially cited in this application, including traditional machine learning models or deep learning models, can be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly, unless explicitly specified in the text. In some embodiments, when running on the client, the corresponding intelligence can be obtained through transfer learning in order to reduce the requirements on the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0058] Please see Figure 1 The network architecture adopted in an exemplary application scenario of this application includes a terminal device 80, an independent website server 81, and an application server 82. The application server 82 can be used to deploy machine translation services. The machine translation service trains a student model by running a computer program product implemented according to the machine translation model training method of this application, and then deploys the student model to the application server to run and open corresponding interfaces to provide services. The independent website server 81 can be used to deploy and open online stores for e-commerce services. Users on the terminal device 80 can submit first language text to the independent website server 81 on the page of the online store. The independent website server 81 further calls the corresponding interface provided by the machine translation service of the application server 82 to submit a translation request for the first language text, obtains the translated second language text, and returns it to the terminal device 80.

[0059] The computer program product implemented according to the machine translation model training method of this application can run on any computer device with sufficient computing power, execute the various steps of the method, obtain a student model as the final machine translation model, and then deploy it to... Figure 1In the exemplary network architecture shown, application server 82 provides machine translation services to a massive number of independent website servers. It is evident that the training environment and deployment environment of the student model are decoupled and do not depend on each other.

[0060] In other exemplary application scenarios, the student model can also be deployed on any terminal device 80 or server with sufficient computing power, for example, in Figure 1 In the network architecture shown, the student model can also be deployed to the independent website server 81 for operation, without relying on the centralized service of the application server 82.

[0061] Please see Figure 2 The example knowledge distillation network is an exemplary network architecture used to train a student model in this application. The knowledge distillation network includes multiple variant models that act as teacher models and one variant model that acts as student models. Each variant model is obtained by modifying the network structure based on the same neural network model suitable for performing machine translation tasks. Therefore, each variant model can be derived from the same machine translation model and they can have different network structures from each other. Of course, it is also allowed that some variant models have exactly the same network structure.

[0062] The prototype of the machine translation model typically includes an encoding network and a decoding network. The encoding network extracts deep semantic information from the first-language text, obtaining a serialized text feature vector. This vector is then input into the decoding network, which performs time-step autoregressive decoding to obtain a vector sequence corresponding to the second-language text as the translation result. The second-language result can then be obtained by querying the machine translation model's knowledge dictionary based on this vector sequence. The machine translation model used in this application can be based on recurrent neural networks, convolutional neural networks, or self-attention models. Self-attention models have shown good performance in practice; therefore, this application will focus on introducing such models. A typical example of a self-attention model is the Transformer model. Figure 2 Each variant model in the knowledge distillation network shown can be constructed by configuring and adjusting the type information of each key layer of its encoding network based on the Transformer model as a prototype, thereby creating variant models with different network structures.

[0063] Please see Figure 3 According to a machine translation model training method provided in this application, in one embodiment, the method includes the following steps:

[0064] Step S1100: Obtain the dataset, which includes multiple parallel corpora;

[0065] To implement the training of this application, a dataset is prepared, which stores a massive amount of parallel corpus sufficient to train any variant model of the machine translation model of this application to a convergent state. Based on the exemplary application scenario of this application, and adapting to the need to use the machine translation model to translate product information in the e-commerce field, the parallel corpus in the dataset all adopt product information describing products in the e-commerce field, where the product information mainly refers to text information.

[0066] In one embodiment, the product information material in the parallel corpus can be product titles. The parallel corpus can be constructed by collecting product titles in different languages ​​from online stores on the same independent website server. For example, an independent website may publish the same product in both Chinese and English versions, providing corresponding Chinese and English product titles. One language version can typically be manually translated to ensure semantic correspondence, or it can be obtained from other excellent machine translation models. Such product titles with both Chinese and English versions can be collected to construct a pair of parallel corpora. Therefore, the parallel corpus consists of corpora in two languages, such as Chinese and English. Product titles have a certain uniqueness, typically containing multiple words representing the product's functions, characteristics, effects, and properties, and may also include popular terms, place names, industry names, product models, etc. Some words in this dataset may not require translation, while others are prone to ambiguity, preventing machine translation models from obtaining accurate results. Still others have specific corresponding translations. In some embodiments of this application, such words can be treated as proper nouns and processed specially to improve the accuracy of machine translation models in translating similar texts. Therefore, for the parallel corpora in the dataset, proper nouns can also be pre-annotated for later use.

[0067] Step S1200: Construct multiple variant models of the machine translation model, and determine one variant model as the student model and the rest as the teacher model;

[0068] The exemplary machine translation model in this application is based on the Transformer model, which includes an encoding network and a decoding network, and incorporates an attention mechanism. The encoding network typically consists of an encoding layer, an attention layer, and a normalization layer. Each attention layer corresponds to a normalization layer, forming an attention unit. Multiple attention units can be stacked to build the encoding network, and the same applies to the decoding network.

[0069] The encoding layer in an encoding network encodes the input text, obtaining corresponding encoded feature information, which is then input into the attention unit for sequential extraction of semantic features. The encoding layer can employ various known encoding algorithms to encode the text; that is, it can have multiple different encoding types. By specifying the corresponding type information, the appropriate encoding algorithm can be determined. For example, when encoding the input text, the encoding layer can use any of the following algorithms to construct diverse encoding layers, such as sine coding, cosine coding, or relative position coding, to encode the positional information.

[0070] The attention layer in the attention unit primarily extracts deep semantic information from the text based on the context of the encoded feature information, obtaining corresponding text feature information. The text feature information of the last attention unit can be used by the decoding network for time-step autoregressive decoding. The normalization layer in the attention unit can normalize the results of the preceding attention layers. The attention layer can employ various attention algorithms, such as multi-head attention and talking-head attention, to construct diverse attention layers. The normalization algorithm used in the normalization layer can employ pre-normalization and post-normalization algorithms to construct diverse normalization layers. Correspondingly, different algorithms for the attention layer and different algorithms for the normalization layer can be distinguished using type information.

[0071] Therefore, based on the prototype configuration of the machine translation model, combining the types adopted by each layer of the encoding network can define a corresponding network structure. When the type information of corresponding layers is specified differently among multiple network structures, different network structures are obtained. Different network structures lead to different encoding networks, which in turn lead to different machine translation model instances, thus becoming a variant model of the prototype machine translation model. By specifying different combinations of type information between layers in the encoding network, multiple variant models can be obtained.

[0072] In one embodiment, during the construction of the variant model, the decoding network in the variant model can be configured to adapt to the customization of the layer type information used in the encoding model. This allows the corresponding layers of the decoding network to be defined using the same type information as the encoding network. For example, when the attention layer in the encoding network uses a conversational attention layer, the type of the attention layer in the decoding network can also be defined as a conversational attention layer; the same applies to the normalization layer. This allows the encoding and decoding networks to use the same algorithm, thereby improving the consistency of their semantic understanding and ensuring high training efficiency for the corresponding variant model.

[0073] In another embodiment, when constructing a variant model, a differentiated configuration can be achieved by modifying the number of attention units in the encoding network and / or decoding network of the variant model. For example, three attention units are used in the encoding network of one variant model, while five attention units are used in the encoding network of another variant model. Other configuration information of the two variant models can remain consistent.

[0074] As can be seen, based on the prototype of the machine translation model, different configuration schemes can be generated by modifying various relevant configuration information in its network architecture. Different variant models can be obtained based on different configuration schemes, resulting in multiple variant models. Because these variant models vary in complexity, even when trained with the same corpus, they may still acquire different reasoning abilities. Higher-complexity variant models are more likely to extract deep semantic information from the text to be translated, thus gaining a stronger reasoning ability in a particular area. Lower-complexity variant models, on the other hand, are more likely to extract shallow semantic information from the text to be translated, thus gaining a stronger reasoning ability in another area. For example, it is possible that the first variant model translates the text of the first product category more accurately than the second variant model, while the second variant model translates the text of the second product category more accurately than the first variant model.

[0075] Based on obtaining multiple variant models, and to meet the needs of subsequent knowledge distillation training in this application, one variant model can be designated as the student model, and the remaining variant models can be designated as teacher models. Generally, the selection of the student model can be flexibly chosen by those skilled in the art.

[0076] Step S1300: Train each of the teacher models to convergence using parallel corpora in the dataset;

[0077] In order to perform knowledge distillation, each of the teacher models needs to be trained to a convergent state first.

[0078] In one embodiment, each teacher model can be trained synchronously using the same parallel corpus in the dataset. That is, through a synchronous training task, the same parallel corpus is synchronously input into each teacher model for training, thereby approximately compressing the total time for training each teacher model individually into the training time of a single teacher model.

[0079] In another embodiment, the individual teacher models can still be trained synchronously, but the parallel corpora they use can be randomly obtained, so that the training sample sets that each teacher model ultimately relies on are different, thereby further expanding the degree of differentiation in the reasoning ability learned by each teacher model.

[0080] In one embodiment, when training the teacher model, training can be performed based on the original parallel corpus in the dataset. That is, the first language corpus in the parallel corpus is directly input into the corresponding teacher model without processing, and the teacher model predicts the corresponding second language result. Then, the second language corpus in the parallel corpus is used to supervise the second language result to realize the gradient update of the teacher model.

[0081] In another embodiment, before inputting the first language corpus from the parallel corpus into the teacher model, proper nouns with the same meaning in the first and second language corpora are replaced with corresponding placeholder labels. This instructs the teacher model not to translate the placeholder labels, thus obtaining the corresponding second language result. The second language corpus with the replaced placeholder labels is then used to supervise the second language result to update the gradient of the teacher model. It is easy to understand that the teacher model trained in this way is more accurate in translating text containing proper nouns than in translating other text content besides proper nouns, thereby ensuring more accurate translation of the entire sentence and making it more suitable for translating product information.

[0082] It is easy to understand that by using sufficient parallel corpora to iteratively train each teacher model, each teacher model can be trained to convergence. Since the training sample sets and network structures of each teacher model may be different, the reasoning abilities learned by each teacher model will be different, and their translation abilities will each have their own advantages.

[0083] Step S1400: Construct a knowledge distillation network, input the knowledge distillation network with parallel corpora from the dataset for training, and train the student model to convergence through joint supervision by multiple teacher models.

[0084] Based on the convergence of multiple teacher models, a more converging model can be constructed. Figure 2The example knowledge distillation network connects a selected student model to the network, and then trains the network using parallel corpora from the dataset. Multiple teacher models and the student model perform synchronous reasoning on the same parallel corpora and generate corresponding translation results. The translation results of the student model are then jointly supervised by the translation results of multiple teacher models to train the student model to a convergent state.

[0085] The knowledge distillation network can be trained using either offline distillation or semi-supervised distillation. It utilizes the translation results of all teacher models to jointly supervise the translation results of the student models, obtaining the student model's distillation loss value and updating its gradient. During supervised training using the combined translation results of all teacher models, for each iteration, the best translation results from all teacher models can be selected for supervising the student model, or the combined results from all teacher models can be used. In short, during each iteration, the combined translation results of all teacher models are used to supervise the student model's training, enabling the student model to learn the superior reasoning abilities of all teacher models.

[0086] As can be seen from the above embodiments, this application mainly focuses on model construction and training mechanisms. By constructing multiple teacher models and a single student model, each teacher model is first trained to convergence. Since each teacher model is a different variant of the machine translation model with a different network structure, even when trained using the same parallel corpus, their learned reasoning abilities will differ. For example, they may each have advantages in translating product information for different product categories. Based on this, multiple converged teacher models are used to perform joint supervised training on the same student model, implementing knowledge distillation. The reasoning abilities of the multiple teacher models are then integrated and transmitted to the student model, ensuring that the reasoning ability learned by the student model is a comprehensive result of the superior reasoning abilities learned by the multiple teacher models. This allows the student model to obtain high-quality translated texts for product information across different product categories.

[0087] Based on any embodiment of this application, please refer to Figure 4 The teacher models are trained to convergence using parallel corpora from the dataset, including:

[0088] Step S1310: Obtain a single parallel corpus from the dataset, and replace part or all of the proper terms with the same meaning in the corpus of the two languages ​​with placeholders of the corresponding type.

[0089] During the iterative training of each teacher model, for each parallel corpus obtained from the dataset in each iteration, both the first-language corpus and the second-language corpus may contain mutually translatable proper nouns expressing the same meaning. There may be one or more of these proper nouns, which can be pre-annotated manually or through named entity recognition. For these proper nouns that are not intended to be translated by the teacher model, based on their entity types, these proper nouns in the parallel corpus can be replaced with placeholder labels corresponding to their entity types, thereby achieving preprocessing of the parallel corpus.

[0090] To guide the teacher model in recognizing these placeholder labels, in one embodiment, the same proper nouns in the knowledge dictionary corresponding to the second language on which the teacher model relies are replaced with placeholder labels corresponding to their entity types. Subsequently, when the teacher model obtains the corresponding translation results, it can retain the corresponding placeholder labels in the second language results based on the knowledge dictionary.

[0091] In an exemplary embodiment, for parallel corpora that have not been manually annotated with proper nouns, the following steps can be used to preprocess the parallel corpora retrieved from the dataset, namely:

[0092] Step S1311: Use a named entity model to identify different types of named entities in each parallel corpus of the dataset. The named entities are proper terms used to describe product information.

[0093] A pre-trained, convergent named entity recognition model, suitable for identifying entity types corresponding to proper nouns in product information, is used to perform named entity recognition on the invoked parallel corpus. This yields multiple named entities, each indicated by its corresponding entity type, thus clarifying the various proper nouns in the parallel corpus. Of course, for parallel corpora in different languages, named entity models trained in the corresponding languages ​​can be used for corresponding identification. Further manual review can be used to ensure that proper nouns with the same meaning in the same parallel corpus corresponding to each other in both languages ​​are correctly identified.

[0094] Step S1312: Establish a mapping table between the proper nouns and their corresponding placeholders. The mapping table is used to store the mapping relationship data between placeholders and proper nouns, so as to restore the placeholders in the second language result.

[0095] To facilitate the subsequent reconstruction of the complete second language result using the translation results of the teacher model when necessary, a temporary mapping table can be established for the replaced proper nouns in the called parallel corpus. This mapping table stores the mapping relationship data between the first language version, the second language version, and their placeholder labels of the proper nouns, thus constructing the mapping table.

[0096] After obtaining the translation results corresponding to the first language corpus in the parallel corpus called by the teacher model, and restoring the placeholder labels in the knowledge dictionary to obtain the second language results, the placeholder labels can be used to query the mapping table to determine the second language version of the proper term corresponding to the placeholder label, and replace the placeholder label in the translation results to obtain the translation results fully expressed in the second language.

[0097] As can be seen from the above embodiments, using a named entity recognition model to identify proper terms in parallel corpora and establish corresponding mapping tables can improve the efficiency of corpus processing and ensure that the corresponding variant models are suitable for processing text with placeholders when they are trained, so that the corresponding variant models can be trained to accurately translate text with placeholders.

[0098] In one embodiment of this application, when replacing proper nouns in parallel corpora with placeholders of the corresponding type, a random number of replacements can be performed on all proper nouns in a single parallel corpus. That is, some proper nouns retain their original text, while others are replaced with placeholders of the corresponding type. This processing ensures that some proper nouns with the same or related semantics in the parallel corpora are retained in their original text, providing adequate context for the training of the teacher model. This makes it easier for the teacher model to refer to this context for accurate translation, enabling the teacher model to more easily generalize features during training and improving training efficiency.

[0099] Step S1320: Simultaneously input the parallel corpus into multiple teacher models to obtain the second language result generated by each teacher model based on the first language corpus in the parallel corpus. Use the second language corpus in the parallel corpus to supervise the second language result and obtain the corresponding model loss value.

[0100] When training the multiple teacher models, a synchronous training task can be enabled, in which the same parallel corpus is simultaneously input into the multiple teacher models to perform translation, thereby performing synchronous training on all teacher models and improving overall training efficiency.

[0101] For each teacher model, the first language corpus, input through the encoding layer of the encoding network, is encoded and then fed into the attention unit. The attention unit obtains text feature information, which is then decoded by the decoding network to produce a sequenced vector as the translation result. Each text feature vector stores the probability distribution of each word in the knowledge dictionary. The corresponding second language result can be obtained by looking up the word in the knowledge dictionary based on the translation result. For each second language result, the teacher model can calculate the model loss value using second language corpus with placeholder labels replaced.

[0102] For the same parallel corpus input synchronously, all teacher models receiving the parallel corpus perform model inference synchronously according to the above process, so that each teacher model can obtain its corresponding model loss value.

[0103] Step S1330: Determine whether the corresponding teacher model has reached convergence based on the model loss value. If it has not reached convergence, perform gradient update on the corresponding teacher model and continue iterative training until the corresponding teacher model is trained to convergence and no longer participates in iteration.

[0104] To determine whether each teacher model has reached convergence, a threshold can be preset. The model loss value obtained in each iteration is then compared to this threshold. When the model loss value reaches the threshold, it means that the corresponding teacher model has reached convergence and can exit training, no longer participating in iterations, and no longer processing parallel corpora. When the model loss value does not reach the threshold, it indicates that the corresponding teacher model has not yet reached convergence. Therefore, gradient updates can be performed on the teacher model based on the model loss value to further approach convergence. Then, it continues to participate in subsequent iterations, receiving input from the next parallel corpus, and so on, until the teacher model reaches convergence and no longer participates in iterations. It is easy to understand that all teacher models will be trained to convergence successively in the synchronous task.

[0105] In one embodiment, although the teacher models can be trained through synchronous tasks, the parallel corpora referenced by each teacher model in each iteration of training can be randomly processed to differentiate the training conditions of each teacher model. Random processing of the parallel corpora can include randomly selecting a parallel corpus from the dataset and providing it to a specific teacher model, or it can include replacing a random number of proper nouns in the parallel corpus with placeholders of the corresponding type. According to this embodiment, the teacher models are inherently heterogeneous. Furthermore, the differentiation of the parallel corpora used further exacerbates the differentiation of the training sample sets between different teacher models, resulting in a greater differentiation in the abilities learned by each heterogeneous teacher model. This allows all teacher models to be combined to have a more comprehensive reasoning ability. Taking the translation of product information as an example, it is expected to cover different types of differentiated product information and obtain accurate translation results for all.

[0106] As can be seen from the above embodiments, the above embodiments achieve multiple technical advantages, including but not limited to:

[0107] First, when training each teacher model, by preprocessing the parallel corpus and replacing some or all of the proper nouns with placeholders of the corresponding type, the teacher model can gain the reasoning ability to partially translate the text, making it more suitable for translating product information containing proper nouns.

[0108] Secondly, since proper nouns in parallel corpora can be partially replaced by placeholder labels, and can be adapted to different teacher models to perform different degrees of partial replacement of proper nouns in the same parallel corpus, the differences in parallel corpora on which each teacher model relies can be generalized, and the degree of differentiation in the superior reasoning ability learned by each teacher model can be aggravated.

[0109] Furthermore, since the teacher models are inherently heterogeneous, even if they are trained synchronously to improve training efficiency, regardless of whether the same training sample set is used to train all teacher models, each teacher model can still acquire differentiated reasoning abilities due to its heterogeneous characteristics. For example, after training, each teacher model can acquire the superior reasoning ability to more accurately translate product information corresponding to one or more product categories, thereby obtaining multiple teacher models with their own strengths. This prepares the technical foundation for training student models to transfer the superior reasoning abilities of each teacher model to the student models.

[0110] Furthermore, the above embodiments not only differentiate the training samples relied upon by each teacher model through parallel corpus preprocessing, but also differentiate the network structures of the teacher models themselves. From the two dimensions of training data and network structure, each teacher model exhibits obvious heterogeneous characteristics, jointly promoting the distinguishing advantages of each teacher model. All the teacher models obtained in this way have their own strengths. When the superior reasoning abilities of these teacher models are combined to train the student model, the student model taught will inevitably obtain reasoning abilities that are significantly better than any single teacher model. This enables the student model to provide accurate translation results universally. For example, it can adapt to product information corresponding to a wide range of product categories and provide accurate translation text for product information of these categories.

[0111] To further leverage the superior reasoning abilities learned by each teacher model, more parallel corpora can be prepared using the teacher models that have been trained to convergence, enriching the dataset and generalizing the features of the parallel corpora in the dataset, thus making the training of the student model more effective. Therefore, based on any embodiment of this application, please refer to... Figure 5 Before training the knowledge distillation network using the parallel corpus, the following steps are included:

[0112] Step S2100: Using one or more converged teacher models, the translation result is obtained by reasoning based on the first language corpus in any parallel corpus in the dataset. Before being input into the teacher model, the proper terms in the first language corpus are replaced with placeholders of the corresponding type.

[0113] When preparing parallel corpora using converged teacher models, the existing parallel corpora in the dataset can be used as a basis. To do this, any parallel corpus can be retrieved from the dataset, and the first language corpus within it can be input into one or more of the converged teacher models. The corresponding teacher models then perform inference to obtain the corresponding translation results.

[0114] In this embodiment, before the first language corpus is input into the corresponding teacher model, the proprietary terms in it can be partially or completely replaced, as described in the relevant embodiments above, so as to retain some of the text to provide sufficient reference semantics for model inference, and then input into the corresponding teacher model.

[0115] In one embodiment, for multiple teacher models used to prepare parallel corpora, a synchronization task can be used to synchronously input the same parallel corpus from the dataset into the multiple teacher models, so as to synchronously obtain the translation results input by the multiple teacher models respectively.

[0116] Step S2200: Based on the mapping relationship data between placeholders and proper nouns in the preset mapping table, restore the placeholders in each translation result to the corresponding type of proper nouns to obtain the second language result;

[0117] As mentioned earlier, the mapping table stores mapping data between proper nouns in the second language version and their corresponding entity type placeholders in a parallel corpus used as training samples. Therefore, for the translation results obtained by each teacher model from the first language corpus in the same parallel corpus, the placeholders can be used to retrieve the corresponding proper nouns from the mapping table and replace them in the translation results to obtain the corresponding second language results. Thus, the second language results are the final second language text translated by the teacher models.

[0118] Step S2300: The second language results are combined with their corresponding first language corpora to form new parallel corpora, which are then added to the dataset to expand the dataset.

[0119] For a training sample, multiple second language results are obtained through inference by multiple teacher models. Each second language result is treated as a corresponding second language corpus and recombine with the first language corpus in the training sample to form new parallel corpora. These parallel corpora are actually translated by teacher models with accurate translation capabilities based on the corresponding first language corpus. Each teacher model obtains its own second language corpus according to its inherent strengths. Thus, the constructed batch of parallel corpora implicitly contains the generalization features resulting from the combined superior inference capabilities of each teacher model. Therefore, storing this batch of parallel corpora in the dataset can expand the parallel corpus of the dataset, multiplying the data volume and providing more sufficient preparation for training student models.

[0120] Based on the above embodiments, it is easy to understand that by using each converged teacher model to translate the parallel corpus in the original dataset, the parallel corpus can be expanded and used for knowledge distillation training. The expanded parallel corpus itself carries the characteristics corresponding to the superior reasoning abilities of each teacher model. Therefore, using these parallel corpora during knowledge distillation training can generalize features, making the student model more likely to converge and improving the training efficiency in the knowledge distillation process.

[0121] Based on any embodiment of this application, please refer to Figure 6 Several variant models for constructing machine translation models are included:

[0122] Step S1210: Obtain the configuration information of the machine translation model, wherein the machine translation model includes an encoding network and a decoding network, and the configuration information includes type information and / or quantity information for defining the network structure of the encoding network;

[0123] When constructing the machine translation model, multiple variant models can be obtained by setting the configuration information of the machine translation model. To set the configuration information, the configuration file of the prototype of the machine translation model can be read first to obtain its configuration information, and then modified based on this information to derive multiple configuration schemes.

[0124] The type information in the configuration information is used to specify the algorithm type used by each component in the encoding network and / or decoding network, including any one or more of the following: type information corresponding to the encoding algorithm of the encoding layer, type information corresponding to the attention algorithm of the attention layer, and type information corresponding to the normalization algorithm of the normalization layer.

[0125] The quantity information is used to specify the number of attention units in the encoding network and / or decoding network, and may therefore include the number of attention units used by the encoding network and / or decoding network.

[0126] Step S1220: Modify the type information and / or quantity information in the configuration information to obtain multiple derived configuration information, and modify the network structure of the machine translation model according to each derived configuration information to obtain multiple variant models of the machine translation model;

[0127] In one embodiment, multiple heterogeneous variant models can be constructed by individually modifying the type information in the configuration information and combining different type information. Taking the Transformer model of this application as an example of the machine translation model, multiple different schemes can be combined by specifying the type information corresponding to the encoding algorithm of the encoding layer, the attention algorithm of the attention layer, and the normalization algorithm of the normalization layer to obtain the configuration information corresponding to each variant model. In one embodiment, the specification of the type information of the attention algorithm and normalization algorithm of the encoding network of the machine translation model can also be applied to the decoding network of the machine translation model, and those skilled in the art can adjust it according to actual needs.

[0128] For example, in one variant model, the encoding algorithm can be a relative position encoding algorithm, the attention algorithm can be a multi-head attention algorithm, and the normalization algorithm can be a pre-normalization algorithm; in another variant model, the encoding algorithm can be a sine encoding algorithm, the attention algorithm can be a multi-head attention algorithm, and the normalization algorithm can be a pre-normalization algorithm; in yet another embodiment, the encoding algorithm can be a cosine encoding algorithm, the attention algorithm can be a conversational attention algorithm, and the normalization algorithm can be a post-normalization algorithm. And so on, these can be flexibly combined and configured.

[0129] In another embodiment, multiple heterogeneous variant models can be constructed by individually modifying the quantity information in the configuration information, specifying the use of different numbers of attention units. The quantity information primarily indicates the number of attention units in the encoding and decoding networks; therefore, the corresponding stacking quantity of each attention unit in the encoding and / or decoding networks can be modified in the configuration information to further differentiate the variant models.

[0130] In another embodiment, the type information and quantity information can be further combined in a differentiated manner to modify the configuration information to obtain a further differentiated variant model. In this way, it is not difficult to understand that as long as the type information and / or quantity information are modified based on the configuration information of the prototype of the machine translation model, a differentiated variant model can be obtained. Thus, multiple variant models can be constructed by setting different derived configuration information.

[0131] In practice, after determining the configuration scheme corresponding to each variant model based on the above principles, corresponding modifications can be made to the configuration information of the prototype to obtain the corresponding derived configuration information. Then, the network structure of the machine translation model is constructed based on the derived configuration information to obtain the corresponding variant model. It is easy to understand that by adapting to multiple derived configuration information, multiple variant models can be obtained.

[0132] Step S1230: Select the variant model with the smallest network structure as the student model in the knowledge distillation network, and the remaining variant models as the teacher models in the knowledge distillation network.

[0133] For the multiple variant models constructed in the above process, in order to make the final student model lightweight, the variant model with the smallest network structure can be preferred as the student model in the knowledge distillation network, and the remaining variant models can be used as the teacher model in the knowledge distillation network.

[0134] As can be seen from the above embodiments, multiple variant models can be obtained by modifying the configuration information based on the same machine translation model. After constructing multiple variant models, these variant models can be optimized, and student models and teacher models can be reasonably set. The student model can be set as a model with a minimal network structure to be more suitable for deployment. Although the teacher model has slightly higher network complexity, it is beneficial to obtain better reasoning capabilities. Subsequently, through knowledge distillation training, these superior reasoning capabilities can be transferred to the lightweight student model, taking into account the advantages of each component in the knowledge distillation network, making the entire training process more efficient, while ensuring that the student model maintains reasonable deployment and operating costs.

[0135] Based on any embodiment of this application, please refer to Figure 7 The knowledge distillation network is trained by inputting parallel corpora from the dataset, including:

[0136] Step S1410: Use a single parallel corpus from the dataset as a training sample;

[0137] When it is necessary to perform knowledge distillation training on the knowledge distillation network, parallel corpora can be called from the dataset as training samples. As can be seen from the various embodiments described above, in a preferred case, the dataset can further include more parallel corpora obtained by expanding the original parallel corpus using the converged teacher model, thereby making sufficient data preparation for knowledge distillation.

[0138] During each iteration of the knowledge distillation network, a single parallel corpus is called from the dataset as a training sample. Throughout the training process, the entire dataset can be traversed, and each parallel corpus can be called one by one for training.

[0139] Step S1420: Simultaneously input the first language corpus in the training samples into all teacher models and student models to obtain the second language results generated by the inference of each model;

[0140] according to Figure 2 As shown in the network architecture, for each training iteration, the first language corpus contained in a training sample is synchronously input into all teacher models and student models. Each teacher model and student model performs model inference according to its own computation process. After passing through the encoding network and decoding network, the corresponding translation result is obtained. The corresponding second language result is obtained by looking up words in the knowledge dictionary referenced by each model.

[0141] In one embodiment, since the teacher model already possesses the translation ability corresponding to some of the translated texts, it is not necessary to replace the proper terms in the first language corpus of the training samples during the knowledge distillation process, nor is it necessary to restore the corresponding translation results. Similarly, the student model can learn some of the translation abilities possessed by each teacher model through knowledge distillation.

[0142] In another embodiment, referring to the training process of each teacher model, some or all of the proper nouns in the first language corpus of the training samples can still be replaced with placeholders of the corresponding type before they are input. After each model in the knowledge distillation network has obtained the corresponding translation results, the placeholders can be replaced back with their corresponding proper nouns. This processing can generally further enhance the translation capabilities of the student model.

[0143] Step S1430: Determine the comprehensive loss value corresponding to the second language results of all teacher models and the true loss value of the student model, and correct the true loss value of the student model according to the comprehensive loss value to obtain the distillation loss value;

[0144] The overall loss value corresponding to all teacher models can be obtained by calculating the individual loss values ​​corresponding to the translation results of the student models based on the translation results of each teacher model. In one embodiment, the individual loss values ​​corresponding to the translation results of each teacher model and the student model are calculated, and then the one with the largest individual loss value is selected as the overall loss value. Thus, in each training iteration, the one with the largest individual loss value is selected as the overall loss value to determine the distillation loss value, and the teacher model with the greatest difference between its translation result and the student model can be used to guide the training of the student model. In another embodiment, the average of the individual loss values ​​of all teacher models can be taken as the overall loss value, and the average ability of all teacher models can be used to guide the training of the student model.

[0145] The true loss value of the student model can be obtained by calculating the translation result of the student model, i.e., the second language result, using the second language corpus of the training samples.

[0146] Furthermore, after correcting the true loss value with the comprehensive loss value, the distillation loss value can be obtained, which is used as the basis for implementing gradient updates on the student model.

[0147] It can be seen that all teacher models are used together to supervise the training of the student model, and the way they are combined is reflected in the way the comprehensive loss value is determined.

[0148] Step S1440: When the student model has not converged, perform gradient updates on the student model based on the distillation loss value and continue iterative training until the student model reaches a convergent state.

[0149] To determine whether the student model has reached convergence, a threshold can be preset. The distillation loss value is compared with the threshold. When the distillation loss value reaches the threshold, it indicates that the student model has reached convergence, and training of the student model can be terminated. When the distillation loss value does not reach the threshold, it indicates that the student model has not yet reached convergence. In this case, gradient updates can be performed on the student model based on the distillation loss value, and the next training sample can be called from the dataset to continue iterative training of the student model.

[0150] It should be noted that during the knowledge distillation process described above, the weights of each teacher-student model are no longer updated.

[0151] As can be seen from the above embodiments, by using a knowledge distillation network to expand the parallel corpus dataset and by using multiple teacher models to jointly supervise the training of the student model, the resulting student model not only gains the advantage of generalization features on the dataset, but also inherits the superior reasoning abilities of each teacher model. Therefore, the student model is easier to train to a convergent state and has more comprehensive superior reasoning abilities, making it capable of translating various types of information. For example, it can perform excellent translations of product information for different product categories on e-commerce platforms.

[0152] Based on any embodiment of this application, please refer to Figure 8 Construct a knowledge distillation network, train the knowledge distillation network using parallel corpora from the dataset, and train the student model to convergence under the joint supervision of multiple teacher models, including:

[0153] Step S3100: Respond to the translation execution request and obtain the first language text carried in the request;

[0154] After the student model is trained to convergence, it can be deployed to a server, for example, to a platform such as... Figure 1 The application server shown provides translation services to various independent website servers by opening corresponding interfaces.

[0155] When a user of the independent website server needs to translate product information, such as the product title, they submit it to the independent website server. The independent website server then sends a corresponding translation execution request to the application server, providing the product title in the request.

[0156] The machine translation service deployed in the application server receives the translation execution request in response to a call to its open interface, and obtains the first language text carried in the request, such as the product title.

[0157] Step S3200: Call the converged student model to translate the first language text to obtain the second language text;

[0158] Based on the obtained first language text, the machine translation service calls the already deployed student model, inputs the first language text, such as the product title, into the student model, and the student model translates the first language text to obtain the corresponding second language result as the second language text.

[0159] Step S3300: Respond to the translation execution request and push the second language text.

[0160] The machine translation service can then push the second language text to the independent website server, which will then return it to the terminal device, allowing the user to obtain accurately translated second language text, such as a product title generated in the second language.

[0161] As can be easily understood from the above embodiments, the student model of this application has practical value and can be used to deploy machine translation services, providing translation services for product information to a large number of independent websites of cross-border e-commerce. With its ability to accurately translate product information of different product categories, it can optimize the service experience of cross-border e-commerce platforms.

[0162] Please see Figure 9 According to one aspect of this application, a machine translation model training apparatus includes a data acquisition module 1100, a variant construction module 1200, a teacher training module 1300, and a student training module 1400, wherein: the data acquisition module 1100 is configured to acquire a dataset including multiple parallel corpora; the variant construction module 1200 is configured to construct multiple variant models of the machine translation model, determining one variant model as a student model and the rest as teacher models; the teacher training module 1300 is configured to train each teacher model to a convergent state using the parallel corpora in the dataset; and the student training module 1400 is configured to construct a knowledge distillation network, input the knowledge distillation network with the parallel corpora in the dataset for training, and jointly supervise the student model training to a convergent state through multiple teacher models.

[0163] Based on any embodiment of this application, the teacher training module 1300 includes: a corpus processing unit, configured to acquire a single parallel corpus in the dataset, and partially or completely replace proper nouns with the same meaning in the corpora of the two languages ​​with placeholders of the corresponding type; a synchronous training unit, configured to synchronously input the parallel corpus into multiple teacher models, obtain the second language result generated by each teacher model based on the first language corpus in the parallel corpus, supervise the second language result using the second language corpus in the parallel corpus, and obtain the corresponding model loss value; and an iterative processing unit, configured to determine whether the corresponding teacher model has reached a convergence state based on the model loss value, and if it has not reached a convergence state, perform gradient update on the corresponding teacher model and continue iterative training until the corresponding teacher model is trained to convergence and no longer participates in the iteration.

[0164] Based on any embodiment of this application, the corpus processing unit includes: an entity recognition subunit, configured to use a named entity model to identify different types of named entities in each parallel corpus of the dataset, wherein the named entities are proper terms used to describe product information; and a table construction subunit, configured to establish a mapping table between the proper terms and placeholders corresponding to their types, wherein the mapping table is used to store mapping relationship data between placeholders and proper terms for restoring placeholders in the second language result.

[0165] Based on any embodiment of this application, the machine translation model training device of this application, preceding the student training module 1400, includes: a corpus translation module, configured to use one or more converged teacher models to infer translation results based on first language corpora in any parallel corpora in the dataset, wherein proper nouns in the first language corpora are replaced with placeholders of corresponding types before being input into the teacher models; a corpus restoration module, configured to restore the placeholders in each translation result to proper nouns of corresponding types according to the mapping relationship data between placeholders and proper nouns in a preset mapping table, thereby obtaining second language results; and a corpus expansion module, configured to recombine each of the second language results as second language corpora with their corresponding first language corpora to form new parallel corpora, and add them to the dataset to expand the dataset.

[0166] Based on any embodiment of this application, the variant construction module 1200 includes: a configuration acquisition unit, configured to acquire configuration information of a machine translation model, wherein the machine translation model includes an encoding network and a decoding network, and the configuration information includes type information and / or quantity information for defining the network structure of the encoding network; an adjustment construction unit, configured to modify the type information and / or quantity information in the configuration information to obtain multiple derived configuration information, modify the network structure of the machine translation model according to each derived configuration information, and obtain multiple variant models of the machine translation model; and a selection processing unit, configured to select the variant model with the smallest network structure as the student model in the knowledge distillation network, and the remaining variant models as the teacher models in the knowledge distillation network.

[0167] Based on any embodiment of this application, the student training module 1400 includes: a sample retrieval unit, configured to retrieve a single parallel corpus from the dataset as a training sample; a synchronous inference unit, configured to synchronously input the first language corpus from the training sample into all teacher models and the student model to obtain the second language results generated by the inference of each model; a result calculation unit, configured to determine the comprehensive loss value corresponding to the second language results of all teacher models and the true loss value of the student model, and correct the true loss value of the student model according to the comprehensive loss value to obtain a distillation loss value; and an iterative decision unit, configured to perform gradient updates on the student model according to the distillation loss value and continue iterative training when the student model has not converged, until the student model reaches a convergent state.

[0168] Based on any embodiment of this application, the machine translation model training device of this application, following the student training module 1400, includes: a request response module, configured to respond to a translation execution request and obtain the first language text carried by the request; a translation execution module, configured to call the converged student model to translate the first language text and obtain the second language text; and a result push module, configured to respond to the translation execution request and push the second language text.

[0169] Another embodiment of this application also provides a machine translation model training device. For example... Figure 10 The diagram shows the internal structure of a machine translation model training device. This device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database can store information sequences, and when executed by the processor, these computer-readable instructions enable the processor to implement a machine translation model training method.

[0170] The processor of this machine translation model training device provides computational and control capabilities, supporting the operation of the entire device. The memory of the device can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the machine translation model training method of this application. The network interface of the device is used for communication with a terminal.

[0171] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application, and does not constitute a limitation on the machine translation model training device on which the solution of this application is applied. The specific machine translation model training device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0172] In this embodiment, the processor is used to execute... Figure 9 The specific functions of each module are described, and the memory stores the program code and various data required to execute the above modules or sub-modules. The network interface is used to realize data transmission between user terminals or servers. In this embodiment, the non-volatile readable storage medium stores the program code and data required to execute all modules in the machine translation model training device of this application, and the server can call the server's program code and data to execute the functions of all modules.

[0173] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the machine translation model training method of any embodiment of this application.

[0174] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0175] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0176] In summary, this application, based on a machine translation model, pre-prepares multiple variant models with different reasoning ability advantages as teacher models. Then, through knowledge distillation, the superior reasoning abilities of the multiple teacher models are transferred to the same student model, enabling the student model to have a more comprehensive translation reasoning ability. This is applicable to the translation of product information corresponding to various product categories on e-commerce platforms.

Claims

1. A method for training a machine translation model, characterized in that, include: Obtain the dataset, which includes multiple parallel corpora, and the parallel corpora are product information in the e-commerce field; Construct multiple variant models of a machine translation model, wherein the multiple variant models are obtained by modifying the type information and / or quantity information in the configuration information of the machine translation model, and are configured to have different network structures so that they can obtain different superior reasoning abilities after training, and determine that a single variant model is used as a student model and the rest are used as teacher models. Training each of the teacher models to convergence using parallel corpora in the dataset includes: acquiring a single parallel corpus from the dataset, partially or completely replacing proper nouns with the same meaning in the two languages ​​with placeholders of the corresponding type; synchronously inputting the parallel corpus into multiple teacher models to obtain the second language result generated by each teacher model based on the first language corpus in the parallel corpus; supervising the second language result using the second language corpus in the parallel corpus to obtain the corresponding model loss value; determining whether the corresponding teacher model has reached convergence based on the model loss value; if it has not reached convergence, performing gradient updates on the corresponding teacher model and continuing iterative training until the corresponding teacher model is trained to convergence and no longer participates in iteration. One or more converged teacher models are used to infer translation results based on the first language corpus in the parallel corpus of the dataset. Before being input into the teacher model, the proper nouns in the first language corpus are replaced with placeholders of the corresponding types. According to the mapping relationship data between placeholders and proper nouns in the preset mapping table, the placeholders in each translation result are restored to proper nouns of the corresponding types to obtain the second language result. Each of the second language results is used as the second language corpus and combined with its corresponding first language corpus to form a new parallel corpus, which is added to the dataset to expand the dataset. A knowledge distillation network is constructed, and the knowledge distillation network is trained by inputting parallel corpora from the dataset. The student model is trained to convergence by joint supervision of multiple teacher models.

2. The machine translation model training method according to claim 1, characterized in that, Obtain a single parallel corpus from the dataset, and replace part or all of the proper nouns with the same meaning in the two language corpora with placeholders of the corresponding type, including: The named entity model is used to identify different types of named entities in each parallel corpus of the dataset. The named entities are proper terms used to describe product information. A mapping table is established between the proper nouns and their corresponding placeholders. The mapping table is used to store the mapping relationship data between placeholders and proper nouns, so as to restore the placeholders in the second language result.

3. The machine translation model training method according to claim 1 or 2, characterized in that, Several variant models for constructing machine translation models are included: Obtain configuration information for a machine translation model, wherein the machine translation model includes an encoding network and a decoding network, and the configuration information includes type information and / or quantity information for defining the network structure of the encoding network; Modify the type information and / or quantity information in the configuration information to obtain multiple derived configuration information, and modify the network structure of the machine translation model according to each derived configuration information to obtain multiple variant models of the machine translation model; The variant model with the smallest network structure was selected as the student model in the knowledge distillation network, and the remaining variant models were selected as the teacher models in the knowledge distillation network.

4. The machine translation model training method according to claim 1 or 2, characterized in that, Training the knowledge distillation network using parallel corpora from the dataset includes: Use a single parallel corpus from the dataset as a training sample; The first language corpus in the training samples is synchronously input into all teacher models and student models to obtain the second language results generated by the reasoning of each model. Determine the comprehensive loss value corresponding to the second language results of all teacher models and the true loss value of the student model, and correct the true loss value of the student model according to the comprehensive loss value to obtain the distillation loss value; If the student model fails to converge, the student model is updated with gradients based on the distillation loss value and training continues iteratively until the student model reaches convergence.

5. The machine translation model training method according to claim 1 or 2, characterized in that, Constructing a knowledge distillation network, training the network using parallel corpora from the dataset, and training the student model to convergence under the joint supervision of multiple teacher models includes: In response to the translation execution request, retrieve the first language text carried in the request; The converged student model is invoked to translate the first language text to obtain the second language text; In response to the translation execution request, push the second language text.

6. A machine translation model training device, characterized in that, include: The data acquisition module is configured to acquire a dataset, which includes multiple parallel corpora, and the parallel corpora are product information in the e-commerce field; The variant construction module is configured to construct multiple variant models of the machine translation model, wherein the multiple variant models are obtained by modifying the type information and / or quantity information in the configuration information of the machine translation model, and are configured to have different network structures so that they can obtain different superior reasoning abilities after training, and determine that a single variant model is used as a student model and the rest are used as teacher models. The teacher training module is configured to train each of the teacher models to convergence using parallel corpora in the dataset, including: a corpus processing unit, configured to acquire a single parallel corpus in the dataset, and partially or completely replace proper nouns with the same meaning in the two languages ​​with placeholders of the corresponding type; a synchronous training unit, configured to synchronously input the parallel corpus into multiple teacher models, obtain the second language result generated by each teacher model based on the first language corpus in the parallel corpus, supervise the second language result using the second language corpus in the parallel corpus, and obtain the corresponding model loss value; and an iterative processing unit, configured to determine whether the corresponding teacher model has reached convergence based on the model loss value, and if it has not reached convergence, perform gradient updates on the corresponding teacher model and continue iterative training until the corresponding teacher model is trained to convergence and no longer participates in the iteration. The corpus translation module is configured to use one or more converged teacher models to infer translation results based on first language corpora in any parallel corpora in the dataset. Before inputting the first language corpora into the teacher models, proper nouns in the first language corpora are replaced with placeholders of the corresponding types. The corpus restoration module is configured to restore the placeholders in each translation result to proper nouns of the corresponding types according to the mapping relationship data between placeholders and proper nouns in a preset mapping table, thereby obtaining second language results. The corpus expansion module is configured to recombine each of the second language results as second language corpora with its corresponding first language corpora to form new parallel corpora, and add them to the dataset to expand the dataset. The student training module is configured to construct a knowledge distillation network, input the knowledge distillation network with parallel corpora from the dataset for training, and train the student model to convergence through joint supervision by multiple teacher models.

7. The machine translation model training device according to claim 6, characterized in that, The corpus processing unit includes: The entity recognition subunit is configured to use a named entity model to identify different types of named entities in each parallel corpus of the dataset, wherein the named entities are proper terms used to describe product information; The table construction subunit is configured to establish a mapping table between the proper nouns and their corresponding placeholders. The mapping table is used to store the mapping relationship data between placeholders and proper nouns for use in restoring the placeholders in the second language result.

8. The machine translation model training apparatus according to claim 6 or 7, characterized in that, The variant construction module includes: The configuration acquisition unit is configured to acquire configuration information of a machine translation model, wherein the machine translation model includes an encoding network and a decoding network, and the configuration information includes type information and / or quantity information for defining the network structure of the encoding network; Adjust the construction unit to modify the type information and / or quantity information in the configuration information to obtain multiple derived configuration information, and modify the network structure of the machine translation model according to each derived configuration information to obtain multiple variant models of the machine translation model; The selection processing unit is configured to select the variant model with the smallest network structure as the student model in the knowledge distillation network, and the remaining variant models as the teacher models in the knowledge distillation network.

9. A machine translation model training device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 5.

10. A non-volatile readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 5, which, when invoked by a computer, executes the steps included in the corresponding method.

Images