Training of a translation model, translation method, apparatus, device and storage medium

Abstract

Embodiments of the present application provide a translation model training method, a translation method, a device, equipment and a storage medium to improve the accuracy of the translation model. The method comprises: determining at least one training task of a translation model; obtaining training data corresponding to the at least one training task, the training data comprising at least one sentence; inputting the training data into the translation model for processing to obtain a corresponding output result; determining loss information of the output result according to the training task; and adjusting the translation model according to the loss information to obtain a trained translation model. The translation model can complete the required training of the training task, and the accuracy of the translation model is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a training method and apparatus for a translation model, a translation method and apparatus, an electronic device, and a storage medium. Background Technology

[0002] Machine translation (MT), also known as automatic translation, is the process of using computers to convert one natural language (source language) into another natural language (target language).

[0003] Currently, sentence machine translation (Sent-MT) has achieved relatively good translation results, but some problems still exist in context-aware translation. Document machine translation (Doc-MT), which aims to improve translation quality by utilizing source sentences and context, has also received increasing attention in recent years.

[0004] Despite the wide range of practical applications of document machine translation, its development has lagged behind. Compared to sentence machine translation, which has been extensively studied, document machine translation presents several challenges due to the introduction of additional contextual information. For example, the current scarcity of document-level corpora and the difficulty in effectively utilizing contextual information can both lead to inaccurate translation models. Summary of the Invention

[0005] This application provides a method for training a translation model to improve the accuracy of the translation model.

[0006] Accordingly, embodiments of this application also provide a translation model training device, a translation method and apparatus, an electronic device, and a storage medium to ensure the implementation and application of the above methods.

[0007] To address the aforementioned issues, this application discloses a method for training a translation model. The method includes: determining at least one training task for the translation model; acquiring training data corresponding to the at least one training task, wherein the training data includes at least one sentence; inputting the training data into the translation model for processing to obtain a corresponding output result; determining loss information of the output result based on the training task; and adjusting the translation model based on the loss information to obtain a trained translation model.

[0008] This application also discloses a translation method, the method comprising: obtaining a document to be translated; inputting the document to be translated into a document translation model for translation processing to obtain a corresponding translated document, wherein the document translation model is trained according to at least one training task; and outputting the translated document.

[0009] This application also discloses a training device for a translation model, the device comprising: a task determination module for determining at least one training task of the translation model; a data acquisition module for acquiring training data corresponding to the at least one training task, the training data including at least one sentence; a model processing module for inputting the training data into the translation model for processing to obtain a corresponding output result; a loss determination module for determining loss information of the output result based on the training task; and an adjustment feedback module for adjusting the translation model based on the loss information to obtain a trained translation model.

[0010] This application also discloses a translation device, comprising: a document acquisition module for acquiring a document to be translated; a translation module for inputting the document to be translated into a document translation model for translation processing to obtain a corresponding translated document, wherein the document translation model is trained according to at least one training task; and an output module for outputting the translated document.

[0011] This application also discloses an electronic device, characterized in that it includes: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform one or more methods as described in the embodiments of this application.

[0012] This application also discloses one or more machine-readable media storing executable code thereon, which, when executed, causes a processor to perform one or more methods as described in this application.

[0013] Compared with the prior art, the embodiments of this application have the following advantages:

[0014] In this embodiment, at least one training task for the translation model can be determined. The training task can be set based on various problems, objectives, etc. of the translation model. Therefore, training data corresponding to the at least one training task can be obtained. The training data includes at least one sentence. Then, the translation model is trained based on the training task. The training data is input into the translation model for processing to obtain the corresponding output result. The loss information of the output result is determined according to the training task. The translation model is adjusted according to the loss information to obtain a trained translation model, so that the translation model can complete the training required for the training task and improve the accuracy of the translation model. Attached Figure Description

[0015] Figure 1 This is a schematic diagram illustrating a training example of a translation model according to an embodiment of this application;

[0016] Figure 2 This is a flowchart illustrating the steps of an embodiment of a translation model training method according to this application;

[0017] Figure 3 This is a schematic diagram illustrating another example of translation model training in accordance with an embodiment of this application;

[0018] Figure 4 This is a flowchart illustrating the steps of another embodiment of the training method for the translation model of this application;

[0019] Figure 5A This is a flowchart illustrating the steps of one embodiment of the translation method of this application;

[0020] Figure 5B This is a schematic diagram illustrating a translation service scenario example according to an embodiment of this application;

[0021] Figure 6 This is a structural block diagram of an embodiment of a translation model training device according to this application;

[0022] Figure 7 This is a structural block diagram of one embodiment of the translation device of this application;

[0023] Figure 8 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0024] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] This application's embodiments can be applied to document translation scenarios, translating source language documents into target language documents. Specifically, Document Machine Translation (Doc-MT) aims to improve translation quality by utilizing source sentences and context. It can be applied to various translation scenarios with translation needs, such as various translation servers, translation websites, and translation platforms. It can also be integrated with corresponding translation services to provide document translation services to those requesting translation, such as translation agencies and service providers corresponding to databases of documents in various languages.

[0026] This application embodiment can address the problems existing in Doc-MT (Doc Machine Translation) by setting training tasks and training the translation model based on these tasks, thereby solving the corresponding problems and improving the accuracy of the model's translation.

[0027] Reference Figure 1 The diagram shows a training example of a translation model according to an embodiment of this application.

[0028] Step 102: Determine at least one training task for the translation model.

[0029] Step 104: Obtain training data corresponding to the at least one training task, wherein the training data includes at least one sentence.

[0030] This application implements a configurable training task, whereby the training task refers to the task of training the translation model. The training task can be set based on information such as the problem, objective, and requirements for model training. The training task includes training data, training objectives, and other data. Based on the training objectives of the training task, the corresponding training data can be determined. In the embodiments of this application, for a document translation model, each sample of the training data can consist of at least one sentence, specifically determined based on the training objectives.

[0031] After setting the training objectives and collecting the training data for the training task, training can be performed based on that training task. Specifically, at least one training task for the translation model can be obtained, and the corresponding training data can be acquired for each training task.

[0032] In this embodiment, the training task can be divided into intralingual training tasks and cross-lingual training tasks based on language, or it can be divided based on training objectives, training data, etc., such as cross-sentence translation tasks, inter-sentence generation tasks, and parallel sentence translation tasks based on training objectives. The models divided in the above different ways can be independent or mutually inclusive. For example, cross-sentence translation tasks and parallel sentence translation tasks can be regarded as cross-lingual training tasks, and inter-sentence generation tasks can be regarded as intralingual training tasks, etc.

[0033] The training task includes at least one of the following: cross-sentence translation task, inter-sentence generation task, and parallel sentence translation task.

[0034] The goal of the Cross Sentence Translation (CST) task is to generate a target sentence based on contextual information when there is no source sentence. The training data for CST is referred to as the first training data. The first training data includes multiple samples, each of which is a source training sequence that masks the source sentence. The source training sequence includes the contextual information (or surrounding information) of the source sentence, such as the source sentence being x. i Then the context statements of the source statement must include at least x. i-1 and x i+1Of course, it may also include more contextual statements, which can be determined according to needs, and this application embodiment does not limit this. Each sample corresponds to label data, which includes the standard target statement of the source statement. The standard target statement is the standard statement of the target statement corresponding to the source statement. The cross-sentence translation task (CST) enables the translation model to make full use of contextual information. In the cross-sentence translation task (CST), the source statement and the target statement belong to different languages. The source statement belongs to the source language, and the target statement belongs to the target language. The source language is the language used before translation, and the target language is the language used after translation. For example, in English to Chinese translation, the source language is the application language, and the target language is Chinese. The standard statement can be understood as the correctly translated statement.

[0035] The goal of the Inter-Sentence Generation (ISG) task is to predict inter-sentence relationships based on contextual information. The training data for the ISG task is referred to as the second training data. This second training data includes multiple samples, each of which is a target training sequence that masks the target sentence. The target training sequence includes the contextual sentences (or contextual information) of the target sentence. For example, if the target sentence is y... i Then the context statements of the target statement must include at least y i-1 and y i+1 Of course, it may also include more contextual statements, which can be determined according to needs. This application embodiment does not limit this. Each sample corresponds to label data, which includes the standard target statement of the target statement. The standard target statement is the standard statement of the target statement. Cross-sentence translation task (CST) can capture the interaction between multiple sentences in a document in order to model discourse phenomena. Cross-sentence translation task (CST) can be regarded as discourse language modeling, injecting cross-sentence dependencies in the document into the decoder of the translation model. Here, discourse (discourse / text) refers to the actual language unit used, which is a series of continuous segments or sentences in the communication process that constitute the language whole. Discourse phenomena include coreference resolution, lexical cohesion, and lexical ambiguity. Coreference resolution is one of the key technologies in the fields of natural language processing, machine translation, information extraction, and information retrieval, and plays the role of hyperlinks in natural language. Coreference refers to the phenomenon that pronouns, appellations, and abbreviations are used to refer to the full names of the aforementioned entities in order to avoid repetition.

[0036] The goal of the Parallel Sentence Translation (PST) task is to achieve knowledge transfer from rich sentence-level parallel data to document-level parallel data, thereby alleviating the shortcomings of document-level bilingual databases. The training data for PST is called the third training data. The second training data includes multiple samples, each a source sentence, and each sample corresponds to labeled data. This labeled data includes the standard target sentence of the source sentence, which is the standard statement corresponding to the target sentence. The Parallel Sentence Translation (PST) task can be understood as a context-free sentence translation task that only requires sentence-level parallel data.

[0037] The above discussion uses several training tasks as examples. In actual processing, training tasks can also be set based on specific needs, goals, and problems to be solved. This application does not impose any restrictions on this.

[0038] For example, to address the problem of scarce document-level bilingual corpus data, a parallel sentence translation task (PST) can be set up; to address the problem of not being able to fully utilize the context, or to achieve the goal of effectively utilizing the context, a cross-sentence translation task (CST) can be set up; and to achieve the goal of fully utilizing cross-sentence dependencies, a cross-sentence translation task (CST) can be set up.

[0039] This allows you to select the training task required for model training, obtain the training data for that task, and then train the translation model based on that training data.

[0040] In some cases, the semantic relationship between the input and output of the target may be loose. Such samples are generally considered to be noisy data that may lead to misleading gradient signals. In order to solve this problem, the embodiments of this application may implement a data selection strategy based on pointwise mutual information (PMI).

[0041] The method also includes a step of filtering training data: constructing word pairs based on samples and labels in the training data; calculating the relevance information of the word pairs; and filtering the samples in the training data based on the relevance information.

[0042] Given the source output x and the target output y, the correlation between x and y is:

[0043]

[0044] Where |x| and |y| represent the lengths of x and y, respectively, and PMI(x,y) is the pointwise mutual information of word pairs (x,y), which is calculated based on word co-occurrence.

[0045]

[0046] Here, c(x,y) is the co-occurrence count of the word pair (x,y). Similar to c(x) and c(y), c(x) is the number of times x appears, and c(y) is the number of times y appears. C is a fixed parameter defined as the logarithmic count of all possible word pairs. In practice, all instances whose relevance exceeds the threshold τ can be selected for model training.

[0047] S={(x,y)|corr(x,y)>τ,(x,y)∈D} (11)

[0048] Where τ is an adjustable hyperparameter, and D and S represent the original dataset and the selected dataset, respectively. Upright x and y represent sentences, and italic x and y represent words.

[0049] Step 106: Input the training data into the translation model for processing to obtain the corresponding output results.

[0050] Training data can be input into the translation model, and after processing by the translation model, the corresponding processing result can be output. From the processing result, the output result corresponding to the training task can be determined.

[0051] This involves obtaining samples from the training data, sequentially inputting these samples into the translation model for output, obtaining the corresponding processing results, and then extracting the output result corresponding to the training task from the processing results. Sequence-to-sequence learning can be performed based on the translation model.

[0052] The translation model includes an encoder and a decoder. The encoder has M identical layers that map the source sequence x to a distributed representation. Based on this, a decoder with N identical layers generates the target sequence. Each encoder layer consists of a multi-head self-attention module and a feedforward network, which work in conjunction with skip connections and normalization layers, following this workflow:

[0053] C i =LN(MHA(H i-1 H i-1 H i-1 )+H i-1 (1)

[0054] H i =LN(FFN(C i )+C i (2)

[0055] Among them, Hi is the hidden representation of the i-th encoder layer, FFN represents the feed-forward network, and LN represents the normalization layer. MHA(a, b, c) can be understood as the multi-head attention operation with a as the query, b as the key, and c as the value.

[0056] Different from the encoder, each decoder layer of the translation model is additionally equipped with a cross-attention module. Among them, the hidden representation H of the i-th decoder layer i is:

[0057] C i = LN(MHA(H i-1 , H i-1 , H i-1 ) + H i-1 ) (3)

[0058]

[0059]

[0060] Among them, ε is the hidden representation of the last encoder layer. By inputting the hidden representation of the last decoder layer into a normalized softmax classifier for calculation, the final output distribution of each target token is predicted.

[0061] The training task includes a cross-sentence translation task; the first training data of the cross-sentence translation task includes: masking the source training sequence of the source sentence;

[0062] The goal of sequence-to-sequence learning is to maximize the log-likelihood of the target sequence y = (y1,..., y n ), given the source sentence x = (x1,..., x m ), where n and m are the lengths of y and x respectively, and the training loss corresponding to the instance (x, y) is defined as:

[0063]

[0064] Among them, y<t identifies the sequence (y1,..., y t ) of processing the token y t-1 at the t-th time step. During the inference process, the model output is defined as the sequence with the highest likelihood, and each selected token is selected as:

[0065]

[0066] Among them, ν represents the target vocabulary. Multiple candidates can be generated to improve the translation quality.

[0067] The embodiments of this application enable pre-training of the model based on training tasks, thereby improving model performance. The translation model can be trained based on training data from each training task. Examples include cross-sentence translation tasks, inter-sentence generation tasks, and parallel sentence translation tasks.

[0068] The training task includes a cross-sentence translation task; the first training data sample for the cross-sentence translation task includes: a source training sequence that masks the source sentence; the step of inputting the training data into the translation model for processing to obtain the corresponding output result includes: inputting the training sequence that masks the source sentence into the translation model for processing, and taking the output target sentence as the output result, wherein the source sentence and the target sentence belong to different languages.

[0069] Among them, source statement x i With the target statement y i Semantically they are tightly coupled, while existing translation models typically focus on the source sentence x. i This leads to the neglect of the source context, i.e., the contextual information of the source statement, c = (x i-1 ,x i+1 To effectively utilize contextual information c, this application employs a cross-sentence translation task (CST), which encourages the model to more effectively utilize valuable information, such as contextual information c. The training data samples for this CST task are masked source sentences x. i After obtaining the source training sequence, the target statement y can be obtained based on its context information c. i This application embodiment can mask the source statements and package them together with context information to form an input sequence. It can also use a token to indicate the end of each statement. To distinguish between different languages, language identifiers can be added to the end of both the source and target output sequences. For example, the language identifier for English is... <en>The language identifier for German is <de>The language identifier for Chinese is <ch>Similarly, in the cross-sentence translation task CST, such as an English-to-German translation example, the source training sequence that masks the source sentence can be used as the source input sequence, which can be represented as: x i-1 <mask> x i+1 <en>Inputting this into a translation model yields the corresponding processing results. The target output, or output result, is obtained from these processing results and is represented as: y i <de>The corresponding loss information can be determined as shown in formula (6).

[0070] The training task includes an inter-sentence generation task; the second training data sample for the inter-sentence generation task includes the target training word order that masks the target sentence; the step of inputting the training data into the translation model for processing to obtain the corresponding output result includes: inputting the training word order that masks the target sentence into the translation model for processing, and taking the output target sentence as the output result.

[0071] Cross-sentence relevance in documents is a crucial feature of machine translation, effectively improving translation quality. Decoders can perform contextual modeling of historical information to enhance coherence and lexical cohesion during translation. Based on these features, this application embodiment sets up an inter-sentence generation task (ISG) to capture inter-sentence dependencies between target outputs. The ISG task, based on its contextual preceding statement y... i-1 and subsequent statement y i To predict the target statement y i Translation models can be trained based on ISG tasks to capture the interactions between sentences in a document. Furthermore, ISG tasks can utilize monolingual document corpora of the target language, effectively alleviating the lack of document-level parallel data in document machine translation. For example, in the English-to-German translation example above, the target training word order that masks the target sentence, as the source input sequence, can be represented as: y i-1 <mask> y i+1 <de>By inputting it into the translation model, the corresponding processing results can be obtained. The target output, or output result, can be obtained from the processing results and can be represented as y. i <de>.

[0072] The training task includes a parallel sentence translation task, and the third training data sample for the parallel sentence translation task includes: source sentences; the step of inputting the training data into the translation model for processing to obtain the corresponding output results includes: inputting the source sentences into the translation model for processing, and taking the output target sentences as the output results.

[0073] Existing sentence-level parallel corpora are often much larger than document-level parallel corpora. Therefore, this application's embodiments set up a Parallel Sentence Translation Task (PST), which aims to transfer knowledge from abundant sentence-level data to limited document-level data. The PST task can be understood as sentence translation without context, requiring only sentence-level parallel data, thereby further reducing the lack of document-level parallel data in document machine translation. As in the English-to-German example above, the source sentences as the source input sequence can be represented as: <none> <x i > <none> <en>By inputting it into the translation model, the corresponding processing results can be obtained. The target output, or output result, can be obtained from the processing results and can be represented as y. i <de>.

[0074] This allows for model training based on training data from each training task.

[0075] Step 108: Determine the loss information of the output result based on the training task.

[0076] Step 110: Adjust the translation model based on the loss information to obtain the trained translation model.

[0077] Then, the labels corresponding to the samples can be determined based on each training task, and the loss information can be calculated based on the output results and labels, such as by using formula (6) to calculate the loss. Alternatively, other methods can be used to calculate the loss, and this application embodiment does not limit this.

[0078] Based on this loss information, it can be backpropagated to the translation model, and the model parameters in the translation model can be adjusted. This process can be iteratively executed until the loss information meets the requirements. These requirements can be determined based on needs, training objectives, etc., so that the translation model that meets the requirements can be used as the trained translation model. The trained translation model can then be applied to document translation scenarios.

[0079] The embodiments of this application can perform contextual interaction pre-training, effectively alleviating the data shortage caused by document machine translation. At the same time, it effectively utilizes valuable text information through cross-sentence translation and determines the cross-sentence dependencies of the document through inter-sentence generation.

[0080] In this embodiment of the application, the strategy of masking the entire sentence has a significant impact on the performance of the pre-trained model. For example, the CST task and ISG task mentioned above predict the target sentence based on the source language and the target language, respectively.

[0081] In this embodiment, to avoid the problem of forgetting the trained translation model during application, the trained translation model can be fine-tuned. This further improves the model's performance. Normalization can be performed based on the fine-tuning to avoid catastrophic forgetting.

[0082] The process of fine-tuning the trained translation model, which involves setting weights for each training task and fine-tuning the model based on those weights, includes: determining the weights of the training tasks and determining the fine-tuning loss information based on the training tasks and weights; and fine-tuning the trained translation model based on the fine-tuning loss information.

[0083] For example, one form of weight-based fine-tuning is Elastic Weight Consolidation (EWC). A trained translation model can be used as an initial model for subsequent fine-tuning on various downstream datasets.

[0084] The input to the translation model is a sequence x consisting of the source sentence and contextual information. i-1 <x i >x i+1 <en>The expected output is the target statement y. i <de>This application's embodiments can improve model performance by using EWC to prevent forgetting the training task. Specifically, weights can be regularized according to their importance to the training task, enabling the translation model to remember the training task and effectively reducing overfitting to limited downstream data. The EWC regularization calculation is as follows:

[0085]

[0086] Where λ is a weighted hyperparameter representing the importance of the old LM task relative to the new MT task, and each parameter can be labeled. The training loss is used for fine-tuning. Defined as the original negative log-likelihood With the newly introduced EWC regularization The sum of these is used for the final training loss in fine-tuning. Represented as:

[0087]

[0088] This allows for further improvements in the performance of the translation model through fine-tuning.

[0089] In summary, at least one training task for the translation model can be determined. The training task can be set based on various problems and objectives of the translation model. Therefore, training data corresponding to the at least one training task can be obtained. The training data includes at least one sentence. Then, the translation model is trained based on the training task. The training data is input into the translation model for processing to obtain the corresponding output result. The loss information of the output result is determined according to the training task. The translation model is adjusted according to the loss information to obtain a trained translation model. This enables the translation model to complete the training required for the training task and improves the accuracy of the translation model.

[0090] Based on the above embodiments, this application proposes a method for training a translation model that can train the translation model in conjunction with a training task.

[0091] Reference Figure 2 The diagram illustrates a flowchart of the steps in an embodiment of a translation model training method according to this application.

[0092] Step 202: Determine at least one training task for the translation model.

[0093] Step 204: Obtain training data corresponding to the at least one training task, wherein the training data includes at least one sentence.

[0094] Step 206: Input each sample of the training data into the translation model for processing in sequence.

[0095] Step 208: Obtain the output processing result.

[0096] Step 210: Determine the output result corresponding to the sample from the processing results.

[0097] Step 212: Obtain the label of the sample, and determine the loss information based on the label and the output result.

[0098] Step 214: Adjust the translation model based on the loss information to obtain the trained translation model.

[0099] Training tasks can be set based on information such as the problem, objectives, and requirements for model training. The samples and labels for these training tasks can be determined, and the translation model can be trained based on these tasks to ensure it meets the requirements and solves the corresponding problems, thereby improving its accuracy. For example, to address the problem of insufficient bilingual corpora, training tasks based on monolingual corpora can be set up, and the model can then be trained based on these tasks, thus solving the aforementioned problem and improving accuracy.

[0100] Based on the above embodiments, the parallel execution of three tasks—cross-sentence translation, inter-sentence generation, and parallel sentence translation—is discussed as an example. After executing the above tasks, fine-tuning can be performed to further improve the model performance.

[0101] Reference Figure 3 The diagram illustrates another example of translation model training according to an embodiment of this application.

[0102] Reference Figure 4 The diagram illustrates a step flowchart of an embodiment of a training method for another translation model of this application.

[0103] Step 402: Determine at least one training task for the translation model.

[0104] The training task includes at least one of the following: cross-sentence translation task, inter-sentence generation task, and parallel sentence translation task.

[0105] Step 404: Obtain training data corresponding to the at least one training task, wherein the training data includes at least one sentence.

[0106] The first sample of the first training data for the cross-sentence translation task includes: a source training sequence that masks the source sentence. The second sample of the second training data for the inter-sentence generation task includes: a target training word order that masks the target sentence. The third sample of the third training data for the parallel sentence translation task includes: the source sentence.

[0107] Step 406: Input the training sequence of the masked source statement into the translation model for processing, and take the output target statement as the first output result, wherein the source statement and the target statement belong to different languages.

[0108] Step 408: Determine the first loss information of the first output result based on the cross-sentence translation task.

[0109] Step 410: Adjust the translation model based on the first loss information.

[0110] Step 412: Input the training word order of the masked target sentence into the translation model for processing, and use the output target sentence as the second output result.

[0111] Step 414: Determine the second loss information of the second output result based on the inter-sentence generation task.

[0112] Step 416: Adjust the translation model based on the second loss information.

[0113] Step 418: Input the source statement into the translation model for processing, and use the output target statement as the third output result.

[0114] Step 420: Determine the third loss information of the third output result based on the parallel sentence translation task.

[0115] Step 422: Adjust the translation model based on the third loss information.

[0116] Step 424: Determine the completed translation model.

[0117] Step 426: Determine the weights of the training task, and determine the fine-tuned loss information based on the training task and the weights.

[0118] Step 428: Fine-tune the trained translation model based on the fine-tuned loss information.

[0119] The pre-training process described above effectively completes model conversion and improves document translation performance. This task effectively utilizes the contextual information of the source sentences, extracts valuable information from the context, and through document-level modeling of the target language, better captures cross-sentence dependencies in the document, contributing to improved machine translation quality. Parallel sentence translation tasks also effectively promote knowledge transfer.

[0120] Based on the above embodiments, this application also provides a translation method that can use a translation model pre-trained and fine-tuned in the above manner to translate documents, thereby effectively translating documents and improving the translation quality.

[0121] Reference Figure 5A The diagram illustrates a flowchart of one embodiment of the translation method of this application.

[0122] Step 502: Obtain the document to be translated.

[0123] Step 504: Input the document to be translated into the document translation model for translation processing to obtain the corresponding translated document, wherein the document translation model is trained based on at least one training task.

[0124] The document translation model can be pre-trained and fine-tuned based on the methods described in the above embodiments.

[0125] Step 506: Output the translated document.

[0126] The above can be applied to various document translation scenarios, providing translation services for documents. The translated document can be output to the party requesting the translation, whose identity depends on the translation scenario. For example, in the translation of legal documents, the requesting party could be a legal service agent, a legal-related institution, or a relevant enterprise or legal service provider. In other scenarios, the requesting party could be a translation agency, which can provide machine translation of the document or proofread the translated document, thereby improving translation efficiency. In still other scenarios, the requesting party could be an educational institution such as a school.

[0127] For example, in a Software-as-a-Service (SaaS) scenario, this translation model can be used to generate translation services and set up service interfaces. Each translation requester can subscribe to this service to translate documents. Specifically, the document to be translated can be obtained through this service interface (e.g., the first interface), then the translation service can be invoked to perform the translation, and finally, the translated document can be output through another service interface (e.g., the second interface). The first and second interfaces can be the same or different interfaces. Furthermore, a corresponding translation page can be provided to users, allowing them to access the service interfaces more conveniently.

[0128] like Figure 5B This example illustrates a translation service scenario. The translation page provides translation controls, such as language selection controls, document upload controls, and proofreading controls. The language selection controls allow users to choose the source language and / or the target language for translation. The document upload controls retrieve the document to be translated from the corresponding address, call the service interface to upload it to the server for translation, and then provide the translated document back based on the service interface. The translated document can also be edited using the proofreading controls (e.g., ...). Figure 5B The "Adjust" control, along with the corresponding proofreading results (such as the location and content of the proofreading), can also be sent back to the translation service so that the translation model can be adjusted subsequently to improve translation accuracy.

[0129] The translation processing of the above translation model can also be applied to various translation servers, translation websites and translation platforms, and this application embodiment does not limit this.

[0130] Based on the above embodiments, the pre-training of the translation model based on training tasks and the fine-tuning of the translation model after training can also be integrated into a corresponding pre-training service. Specifically, a pre-training service can be set up for each training task, or the pre-training service can include multiple training tasks, executing the corresponding training processing based on the user's selection. This provides a corresponding service interface to the user, who can upload the translation model, training data, etc., through the service interface. Based on the selected training task, the pre-training process of the above embodiments and the fine-tuning of the translation model after training are executed, and then the translation model is returned to the user. In some scenarios, the above data filtering can also be integrated into a filtering service. Users can call the filtering service to filter the training data before selecting a training task to execute pre-training, and then call the training service for training based on the filtered data. All the above services can be integrated into one service, including multiple sub-items for the user to choose from, or they can be integrated into individual services, which users can call separately or select and use together. The above services can also provide corresponding service pages for users to select services and upload data, etc.

[0131] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0132] Based on the above embodiments, this embodiment also provides a training device for a translation model, which is applied in an electronic device of a server.

[0133] Reference Figure 6 The diagram shows a structural block diagram of an embodiment of a training device for a translation model according to this application, which may specifically include the following modules:

[0134] Task determination module 602 is used to determine at least one training task for the translation model.

[0135] The data acquisition module 604 is used to acquire training data corresponding to the at least one training task, wherein the training data includes at least one sentence.

[0136] The model processing module 606 is used to input the training data into the translation model for processing and to obtain the corresponding output results.

[0137] The loss determination module 608 is used to determine the loss information of the output result based on the training task.

[0138] The adjustment feedback module 610 is used to adjust the translation model based on the loss information to obtain the trained translation model.

[0139] In summary, at least one training task for the translation model can be determined. The training task can be set based on various problems and objectives of the translation model. Therefore, training data corresponding to the at least one training task can be obtained. The training data includes at least one sentence. Then, the translation model is trained based on the training task. The training data is input into the translation model for processing to obtain the corresponding output result. The loss information of the output result is determined according to the training task. The translation model is adjusted according to the loss information to obtain a trained translation model. This enables the translation model to complete the training required for the training task and improves the accuracy of the translation model.

[0140] The training task includes at least one of the following: cross-sentence translation task, inter-sentence generation task, and parallel sentence translation task.

[0141] When the training task includes a cross-sentence translation task, the first training data for the cross-sentence translation task includes: a source training sequence that masks the source sentence; and a model processing module 606, used to input the masked source sentence training sequence into the translation model for processing, and to output the target sentence as the output result, wherein the source sentence and the target sentence belong to different languages. The cross-sentence translation task (CST) enables the translation model to fully utilize contextual information.

[0142] When the training task includes an inter-sentence generation task, the second training data for the inter-sentence generation task includes: a target training word order that masks the target sentence; and a model processing module 606, used to input the training word order that masks the target sentence into the translation model for processing, and to output the target sentence as the output result. The Cross-Sentence Translation Task (CST) can capture the interactions between multiple sentences in a document.

[0143] When the training task includes a parallel sentence translation task, the third training data for the parallel sentence translation task includes: source sentences; and a model processing module 606, used to input the source sentences into the translation model for processing, and to output the target sentences as the output results. This achieves knowledge transfer from rich sentence-level parallel data to document-level parallel data, thereby alleviating the shortcomings of document-level bilingual libraries.

[0144] It also includes a fine-tuning module, used to fine-tune the trained translation model.

[0145] The fine-tuning module is used to determine the weights of the training task and to determine the fine-tuning loss information based on the training task and the weights; and to fine-tune the trained translation model based on the fine-tuning loss information.

[0146] It also includes a data filtering module, used to filter training data.

[0147] The data filtering module is used to construct word pairs based on the samples and labels of the training data; calculate the relevance information of the word pairs; and filter the samples of the training data according to the relevance information.

[0148] The pre-training process described above effectively completes model conversion and improves document translation performance. This task effectively utilizes the contextual information of the source sentences, extracts valuable information from the context, and through document-level modeling of the target language, better captures cross-sentence dependencies in the document, contributing to improved machine translation quality. Parallel sentence translation tasks also effectively promote knowledge transfer.

[0149] Based on the above embodiments, this embodiment also provides a translation device that can be applied to electronic devices such as servers and user equipment.

[0150] Reference Figure 7 The diagram shows a structural block diagram of an embodiment of a translation device according to this application, which may specifically include the following modules:

[0151] The document acquisition module 702 is used to acquire documents to be translated.

[0152] The translation module 704 is used to input the document to be translated into a document translation model for translation processing to obtain a corresponding translated document, wherein the document translation model is trained based on at least one training task.

[0153] The output module 706 is used to output the translated document. This translated document can be sent to the party requesting the translation.

[0154] The translation model, which has been pre-trained and fine-tuned in the above manner, can be used for translation, thereby effectively translating documents and improving the quality of document translation.

[0155] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0156] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0157] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices, servers (clusters), etc. Figure 8 An exemplary apparatus 800 is schematically shown that can be used to implement the various embodiments described in this application.

[0158] In one embodiment, Figure 8 An exemplary device 800 is shown, which includes one or more processors 802, a control module (chipset) 804 coupled to at least one of the processors 802, a memory 806 coupled to the control module 804, a non-volatile memory (NVM) / storage device 808 coupled to the control module 804, one or more input / output devices 810 coupled to the control module 804, and a network interface 812 coupled to the control module 804.

[0159] Processor 802 may include one or more single-core or multi-core processors, and processor 802 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 800 can serve as a terminal device, server (cluster), or other device as described in the embodiments of this application.

[0160] In some embodiments, apparatus 800 may include one or more computer-readable media (e.g., memory 806 or NVM / storage device 808) having instructions 814 and one or more processors 802 that are combined with the one or more computer-readable media and configured to execute the instructions 814 to implement the module and thus perform the actions described in this disclosure.

[0161] In one embodiment, the control module 804 may include any suitable interface controller to provide any suitable interface to at least one of the processors 802 and / or any suitable device or component communicating with the control module 804.

[0162] The control module 804 may include a memory controller module to provide an interface to the memory 806. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0163] Memory 806 may be used, for example, to load and store data and / or instructions 814 for device 800. In one embodiment, memory 806 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 806 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0164] In one embodiment, the control module 804 may include one or more input / output controllers to provide an interface to the NVM / storage device 808 and (one or more) input / output devices 810.

[0165] For example, NVM / storage device 808 may be used to store data and / or instructions 814. NVM / storage device 808 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0166] NVM / storage device 808 may include storage resources that are physically part of a device on which device 800 is mounted, or that are accessible to the device but do not necessarily have to be part of the device. For example, NVM / storage device 808 may be accessed via a network via one or more input / output devices 810.

[0167] One or more input / output devices 810 may provide an interface for device 800 to communicate with any other suitable device. Input / output devices 810 may include communication components, audio components, sensor components, etc. A network interface 812 may provide an interface for device 800 to communicate via one or more networks. Device 800 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof.

[0168] In one embodiment, at least one of the processors 802 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 804. In one embodiment, at least one of the processors 802 may be logically packaged with one or more controllers of the control module 804 to form a system-in-package (SiP). In one embodiment, at least one of the processors 802 may be integrated with the logic of one or more controllers of the control module 804 on the same die. In one embodiment, at least one of the processors 802 may be integrated with the logic of one or more controllers of the control module 804 on the same die to form a system-on-a-chip (SoC).

[0169] In various embodiments, device 800 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop, handheld computing device, tablet, netbook, etc.). In various embodiments, device 800 may have more or fewer components and / or different architectures. For example, in some embodiments, device 800 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0170] The detection device may use a main control chip as a processor or control module, and sensor data, position information, etc. may be stored in a memory or NVM / storage device. The sensor group may be used as an input / output device, and the communication interface may include a network interface.

[0171] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0172] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0173] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0177] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0178] The foregoing has provided a detailed description of a translation model training method and apparatus, a translation method and apparatus, an electronic device, and a storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.< / de> < / en> < / de> < / en> < / none> < / none> < / de> < / de> < / mask> < / de> < / en> < / mask> < / ch> < / de> < / en>

Claims

1. A method for training a translation model, characterized in that, The method includes: The training task of the translation model is defined as at least one, including a cross-sentence translation task, and further including at least one of the following: an inter-sentence generation task and a parallel sentence translation task. The objective of the cross-sentence translation task is to generate a target sentence based on contextual information in the absence of a source sentence. The objective of the inter-sentence generation task is to predict inter-sentence relationships based on contextual information. The objective of the parallel sentence translation task is to achieve knowledge transfer from sentence-level parallel data to document-level parallel data to alleviate the inadequacy of document-level bilingual databases. When the training task includes a cross-sentence translation task, the first training data for the cross-sentence translation task includes: a source training sequence with the source sentence masked, wherein the source sentence is masked and packaged with contextual information to form an input sequence, and language identifiers are added to the ends of the source training sequence and the target output to distinguish between different languages. Obtain training data corresponding to the at least one training task, wherein the training data includes at least one sentence; The training data is input into the translation model for processing to obtain the corresponding output results. The translation model includes an encoder and a decoder. The encoder has M layers, and the encoder layer consists of a multi-head self-attention module and a feedforward network. The decoder has N layers, and the decoder layer consists of a multi-head self-attention module, a cross-attention module, and a feedforward network. The loss information of the output result is determined based on the training task; The translation model is adjusted based on the loss information to obtain the trained translation model.

2. The method according to claim 1, characterized in that, When the training task includes a cross-sentence translation task, the step of inputting the training data into the translation model for processing to obtain the corresponding output results includes: The training sequence of the masked source statement is input into the translation model for processing, and the output target statement is taken as the output result, wherein the source statement and the target statement belong to different languages.

3. The method according to claim 1, characterized in that, When the training task includes an inter-sentence generation task, the second training data for the inter-sentence generation task includes: the target training word order that masks the target sentence. The step of inputting the training data into the translation model for processing to obtain the corresponding output results includes: The training word order of the masked target sentence is input into the translation model for processing, and the output target sentence is used as the output result.

4. The method according to claim 1, characterized in that, In the case where the training task includes a parallel sentence translation task, the third training data for the parallel sentence translation task includes: source sentences; The step of inputting the training data into the translation model for processing to obtain the corresponding output results includes: The source statement is input into the translation model for processing, and the output target statement is taken as the output result.

5. The method according to claim 1, characterized in that, It also includes the step of fine-tuning the trained translation model: Determine the weights of the training task, and determine the fine-tuning loss information based on the training task and the weights; The trained translation model is then fine-tuned based on the fine-tuned loss information.

6. The method according to claim 1, characterized in that, It also includes the step of filtering the training data: Word pairs are constructed based on the samples and labels in the training data; Calculate the relevance information of the word pairs; The training data samples are filtered based on the correlation information.

7. A translation method, characterized in that, The method includes: Get the document to be translated; The document to be translated is input into a document translation model for translation processing to obtain the corresponding translated document. The document translation model is trained based on at least one training task, which includes a cross-sentence translation task and at least one of the following: an inter-sentence generation task and a parallel sentence translation task. The objective of the cross-sentence translation task is to generate a target sentence based on contextual information in the absence of a source sentence. The objective of the inter-sentence generation task is to predict inter-sentence relationships based on contextual information. The objective of the parallel sentence translation task is to achieve knowledge transfer from sentence-level parallel data to document-level parallel data. To alleviate the shortcomings of document-level bilingual libraries; when the training task includes a cross-sentence translation task, the first training data for the cross-sentence translation task includes: a source training sequence with the source sentences masked, wherein the source sentences are masked and packaged with context information to form an input sequence, and language identifiers are added to the ends of the source training sequence and the target output to distinguish between different languages; the document translation model includes an encoder and a decoder, the encoder has M layers, the encoder layer consists of a multi-head self-attention module and a feedforward network, and the decoder has N layers, the decoder layer consists of a multi-head self-attention module, a cross-attention module and a feedforward network; Output the translated document.

8. The method according to claim 7, characterized in that, The output of the translated document includes: The translated document is sent to the party requesting the translation.

9. A training device for a translation model, characterized in that, The device includes: A task determination module is used to determine at least one training task for the translation model. The training task includes a cross-sentence translation task, and further includes at least one of the following: an inter-sentence generation task and a parallel sentence translation task. The cross-sentence translation task aims to generate a target sentence based on contextual information in the absence of a source sentence. The inter-sentence generation task aims to predict inter-sentence relationships based on contextual information. The parallel sentence translation task aims to achieve knowledge transfer from sentence-level parallel data to document-level parallel data to alleviate the inadequacy of a document-level bilingual database. When the training task includes a cross-sentence translation task, the first training data for the cross-sentence translation task includes: a source training sequence with the source sentence masked. The source sentence is masked and packaged with contextual information to form an input sequence, and language identifiers are added to the ends of the source training sequence and the target output to distinguish between different languages. The data acquisition module is used to acquire training data corresponding to the at least one training task, wherein the training data includes at least one sentence; The model processing module is used to input the training data into the translation model for processing and obtain the corresponding output results. The translation model includes an encoder and a decoder. The encoder has M layers, and the encoder layer consists of a multi-head self-attention module and a feedforward network. The decoder has N layers, and the decoder layer consists of a multi-head self-attention module, a cross-attention module, and a feedforward network. The loss determination module is used to determine the loss information of the output result based on the training task; The adjustment feedback module is used to adjust the translation model based on the loss information to obtain the trained translation model.

10. A translation device, characterized in that, The device includes: The document acquisition module is used to acquire documents to be translated. The translation module is used to input the document to be translated into a document translation model for translation processing to obtain the corresponding translated document. The document translation model is trained based on at least one training task, which includes a cross-sentence translation task and at least one of the following: an inter-sentence generation task and a parallel sentence translation task. The objective of the cross-sentence translation task is to generate a target sentence based on contextual information in the absence of a source sentence. The objective of the inter-sentence generation task is to predict inter-sentence relationships based on contextual information. The objective of the parallel sentence translation task is to achieve knowledge transfer from sentence-level parallel data to document-level parallel data. The document translation model employs a knowledge transfer mechanism to alleviate the shortcomings of document-level bilingual libraries. When the training task includes a cross-sentence translation task, the first training data for that task includes a source training sequence with the source sentences masked. The source sentences are masked and packaged with contextual information to form an input sequence. Language identifiers are added to the ends of both the source training sequence and the target output to distinguish between different languages. The document translation model includes an encoder and a decoder. The encoder has M layers, each consisting of a multi-head self-attention module and a feedforward network. The decoder has N layers, each consisting of a multi-head self-attention module, a cross-attention module, and a feedforward network. The output module is used to output the translated document.

11. An electronic device, characterized in that, include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the method as described in one or more of claims 1-8.

12. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the method as described in one or more of claims 1-8.

Images