Translation evaluation model training method, translation evaluation method, equipment and medium

Abstract

The present application provides a training method, a translation evaluation method, a device and a medium for a translation evaluation model. The training method for the translation evaluation model comprises: obtaining a training sample set and identification information of each training text pair therein, wherein the training text pair comprises a translation and at least one type of reference information, the reference information comprises at least one of an original text and a reference translation, and the identification information is used to indicate the type of reference information in the training text pair; training a translation evaluation model according to the identification information of each training text pair in the training sample set to obtain a trained translation evaluation model, thereby realizing translation evaluation in multiple input scenarios through a unified model framework, improving the convenience of translation evaluation, and improving the application scope and flexibility of the translation evaluation model.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a training method for a translation evaluation model, a translation evaluation method, a device and a medium. Background Art

[0002] Machine translation, also known as automatic translation, is a process that uses computers to convert text in a source language into text in a target language. With the increase in the application scenarios of machine translation, the evaluation of translation quality has also received widespread attention.

[0003] Machine Translation Metric is mainly measured by measuring the consistency between the translation result and the language expression of the given part. Common translation quality evaluation methods are mainly based on BLEU (Bilingual Evaluation Understudy, a bilingual translation quality evaluation auxiliary tool) and BERT (Bidirectional Encoder Representations from Transformers, Tansformer's bidirectional encoder representation) models. The above models can only process evaluation based on source text or reference text, that is, they only support one input mode, such as inputting the translation and its original text, or inputting the translation and its reference text. The model has a limited scope of application and poor flexibility. Summary of the invention

[0004] The present application provides a translation evaluation model training method, a translation evaluation method, a device and a medium, provides a translation evaluation model that can be used in a variety of input scenarios, and improves the application scope and flexibility of the model.

[0005] In a first aspect, the present application provides a method for training a translation evaluation model, the method comprising:

[0006] Acquire a training sample set and identification information of each training text pair in the training sample set, wherein the training text pair includes a translation and at least one type of reference information, the reference information includes at least one of an original text and a reference translation, and the identification information is used to indicate the type of reference information in the training text pair;

[0007] The translation evaluation model is trained according to the identification information of each training text pair in the training sample set to obtain a trained translation evaluation model.

[0008] In a second aspect, the present application provides a translation evaluation method, comprising:

[0009] Acquire a text pair to be evaluated and identification information, wherein the text pair to be evaluated includes a translation to be evaluated and at least one type of reference information, the reference information includes at least one of an original text and a reference translation of the translation to be evaluated, and the identification information is used to indicate the type of the reference information;

[0010] Determining a score of the translation to be evaluated based on the text pair to be evaluated, the identification information, and the translation evaluation model;

[0011] Wherein, the translation evaluation model is trained based on the method provided in the first aspect of this application.

[0012] In a third aspect, the present application provides another translation evaluation method, comprising:

[0013] Obtain translations to be evaluated of multiple original texts translated by translation tools;

[0014] Generating each pair of texts to be evaluated and identification information thereof according to each of the translations to be evaluated and reference information thereof, wherein the reference information includes at least one of the original text and the reference translation of the translation to be evaluated, and the identification information is used to indicate the type of the reference information;

[0015] For each text pair to be evaluated, determining the score of the translation to be evaluated in the text pair to be evaluated according to the translation evaluation model, the text pair to be evaluated and its identification information; wherein the translation evaluation model is obtained by training based on the method provided in the first aspect of the present application;

[0016] The evaluation result of the translation tool is generated according to the scores of the translations to be evaluated.

[0017] In a fourth aspect, the present application provides a training device for a translation evaluation model, comprising:

[0018] A training set acquisition module, used to acquire identification information of a training sample set and each training text pair therein, wherein the training text pair includes a translation and at least one type of reference information, the reference information includes at least one of an original text and a reference translation, and the identification information is used to indicate the type of reference information in the training text pair;

[0019] The model training module is used to train the translation evaluation model according to the identification information of each training text pair in the training sample set to obtain a trained translation evaluation model.

[0020] In a fifth aspect, the present application provides a translation evaluation device, comprising:

[0021] A text pair acquisition module, used to acquire a text pair to be evaluated and identification information, wherein the text pair to be evaluated includes a translation to be evaluated and at least one type of reference information, the reference information includes at least one of an original text and a reference translation of the translation to be evaluated, and the identification information is used to indicate the type of the reference information;

[0022] A score determination module is used to determine the score of the translation to be evaluated based on the text pair to be evaluated, the identification information and the translation evaluation model; wherein the translation evaluation model is trained based on the method provided in the first aspect of the present application.

[0023] In a sixth aspect, the present application provides another translation evaluation device, comprising:

[0024] A translation acquisition module is used to acquire translations to be evaluated of multiple original texts translated by a translation tool;

[0025] A text pair generation module, used to generate each to-be-evaluated text pair and its identification information according to each to-be-evaluated translation and its reference information, wherein the reference information includes at least one of the original text and the reference translation of the to-be-evaluated translation, and the identification information is used to indicate the type of the reference information;

[0026] A score determination module, for determining, for each text pair to be evaluated, the score of the translation to be evaluated in the text pair to be evaluated according to the translation evaluation model, the text pair to be evaluated and its identification information; wherein the translation evaluation model is obtained by training based on the method provided in the first aspect of the present application;

[0027] The evaluation result generating module is used to generate the evaluation result of the translation tool according to the score of each translation to be evaluated.

[0028] In a seventh aspect, the present application provides an electronic device, including:

[0029] A processor, and a memory communicatively connected to the processor;

[0030] The memory stores computer-executable instructions;

[0031] The processor executes the computer-executable instructions stored in the memory to implement the method provided in the first aspect, the second aspect or the third aspect of the present application.

[0032] In an eighth aspect, the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the method provided in the first aspect, the second aspect or the third aspect of the present application.

[0033] In a ninth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method provided in the first aspect or the second aspect of the present application.

[0034] The training method, translation evaluation method, device and medium of the translation evaluation model provided in the present application are aimed at translation evaluation in multiple input scenarios, such as the input scenarios are original text scenarios, reference translation scenarios, original text and reference translation scenarios. The current input scenario is identified by identification information, and the score of the translation in the text pair is determined according to the text pair input in the current input scenario by the designed translation evaluation model, thereby realizing translation evaluation in multiple input scenarios through the same translation evaluation model, improving the application scope and flexibility of the translation evaluation model, and improving the convenience of translation evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application;

[0037] Figure 2 A flowchart of a method for training a translation evaluation model provided in an embodiment of the present application;

[0038] Figure 3 For this application Figure 2 A schematic diagram of the structure of the translation evaluation model in the illustrated embodiment;

[0039] Figure 4 A flowchart of another translation evaluation method provided in an embodiment of the present application;

[0040] Figure 5 A schematic diagram of a user interaction interface provided by an embodiment of the present application;

[0041] Figure 6 A flowchart of a method for training a translation evaluation model provided in an embodiment of the present application;

[0042] Figure 7 For this application Figure 6 A schematic diagram of the structure of the translation evaluation model provided in the illustrated embodiment;

[0043] Figure 8 For this application Figure 6 A flowchart of a method for determining a pseudo-scoring label in the illustrated embodiment;

[0044] Fig. 9 A flowchart of a translation tool evaluation method provided in one embodiment of the present application;

[0045] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0046] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] The method provided in the embodiments of the present application can be applied to the evaluation of translation quality in a variety of scenarios, such as cross-border e-commerce scenarios, online translation scenarios, translation education scenarios, corporate office scenarios, subtitle translation scenarios, etc.

[0049] In the cross-border e-commerce scenario, the two or more parties to the transaction communicate through chat software, audio and video software, trading platforms, etc., and can translate texts in other languages ​​(original texts) into texts in a specified language (translation) based on the translation plug-ins set in the software or platform. After obtaining the translation, the translation output by the translation plug-in and the original text of the translation can be input into a translation evaluation model in an online or offline manner, and the translation score is output based on the translation evaluation model, so that the user can judge whether to accept the translation, and the provider of the translation plug-in can maintain the translation plug-in based on the score to improve the translation quality.

[0050] In an enterprise office scenario, such as an online meeting in a multilingual scenario, the text data (original text) of the recognized conference audio is translated into a translation in the set language through a translation plug-in set in the conference software. The translation and the original text of the translation are input into a translation evaluation model to obtain a score for the translation, so that the user can judge whether to refer to the original text or re-translate it with the help of other translation tools based on the score, so as to improve the smoothness of conference communication and enable users to better understand the content of the meeting.

[0051] In cross-border e-commerce scenarios or enterprise office scenarios, this application provides a translation evaluation method for a translation plug-in or a translation plug-in evaluation method, including:

[0052] Obtain an original text to be evaluated translated by the translation plug-in; generate a text pair to be evaluated and its identification information according to the original text and its translation to be evaluated; determine a score of the translation to be evaluated in the text pair to be evaluated according to the translation evaluation model, the text pair to be evaluated and its identification information; display the score of the translation to be evaluated to the user; and / or, based on the scores of multiple translations to be evaluated, generate an evaluation result of the translation plug-in.

[0053] After the evaluation result is generated, the evaluation result can be sent to a terminal corresponding to the translation plug-in, such as a terminal set by the translation plug-in, a maintenance terminal of the translation plug-in, etc., so that the evaluation result can be displayed through the terminal to facilitate maintenance of the translation plug-in based on the evaluation result.

[0054] In the online translation scenario, the provider of the translation tool needs to evaluate the translation quality of the translation tool regularly or irregularly in order to improve the translation quality. The translation tools, such as robots, handheld devices, etc., and the translations and reference texts, such as the original text and reference translations of the translations, are input into the translation evaluation model, and the translation quality of the translation tool is evaluated based on the translation evaluation model. When the translation quality of the translation tool is poor, the translation tool can be maintained in time to improve the translation accuracy of the translation tool, thereby improving the user experience and loyalty.

[0055] In the online translation scenario, this application provides a translation tool evaluation method, including:

[0056] The method comprises obtaining a plurality of original texts to be evaluated translated by a translation tool; generating each pair of texts to be evaluated and identification information thereof according to each of the original texts to be evaluated and reference information thereof, wherein the reference information comprises at least one of the original text and the reference translation of the original text to be evaluated, and the identification information is used to indicate the type of the reference information; for each pair of texts to be evaluated, determining a score of the translation to be evaluated in the pair of texts to be evaluated according to a translation evaluation model, the pair of texts to be evaluated and identification information thereof; and generating an evaluation result of the translation tool according to the scores of the plurality of translations to be evaluated.

[0057] After the evaluation result is generated, the evaluation result may be sent to the translation tool or a maintenance terminal of the translation tool to display the evaluation result, so as to facilitate maintenance of the translation tool based on the evaluation result.

[0058] In the translation education scenario, in order to improve the evaluation efficiency of translation assignments submitted by students, teachers can input the translation assignments (translations) submitted by students, as well as at least one of the original text and reference answers (reference translations) into the translation evaluation model, so that the translation evaluation model can output the scores of each translation assignment.

[0059] In a translation education scenario or a translation teaching scenario, this application provides a translation evaluation method, including:

[0060] A text pair to be evaluated input by a user (such as a teacher, a teaching assistant, etc.) is obtained, and identification information of the text pair to be evaluated is obtained, wherein the text pair to be evaluated includes a translation to be evaluated and at least one type of reference information, the reference information includes at least one of an original text and a reference translation of the translation to be evaluated, and the identification information is used to indicate the type of the reference information; a score of the translation to be evaluated is determined according to the text pair to be evaluated, the identification information, and a translation evaluation model.

[0061] After obtaining the score of the translation to be evaluated, the score can be added to the corresponding translation to be evaluated, thereby realizing automatic scoring of the translation job. The translations to be evaluated with scores added can also be packaged and sent to the user terminal, so that the user can send the translations to the corresponding student's terminal. The user can also modify the score added to the translation to be evaluated, and feedback the modified score and the translation to be evaluated.

[0062] The embodiments of the present application can be used to implement translation quality evaluation in various input scenarios. Figure 1 A schematic diagram of an application scenario of an embodiment of the present application is shown in FIG. Figure 1 As shown, the machine translation terminal 102 translates the input original text based on a translation operator, such as a Transformer model, a recurrent neural network model, etc., such as real-time, online translation or offline translation, and outputs a translation of the original text. Figure 1 In the example, a user sends the original text to the machine translation terminal 102 through a user terminal, such as the user terminal 11 and the user terminal 12. The user can also input the original text into the robot translation terminal 102 through the input device of the robot translation terminal 102, such as through the keyboard, serial port, microphone, camera and other input devices of the robot translation terminal 102. The original text can be in the initial form of voice, image or text. If it is voice, the original text in the form of text can be obtained through voice recognition; if it is an image, the original text in the form of text can be obtained by extracting the characters in the image. In order to evaluate the translation quality, the translation output by the machine translation terminal 102, the original text or the reference translation of the translation, can be input into the translation evaluation terminal 104, and the translation evaluation operator set in the translation evaluation terminal 104, such as the BLEU operator, TER (Translation Edit Rate) and other statistical-based translation evaluation operators or model-based translation evaluation operators, etc., is used to evaluate the translation, and the evaluation result is output or displayed, so as to screen or improve the automatic translation evaluation operator in the robot translation terminal 102 based on the evaluation result to improve the translation quality.

[0063] The translation and at least one of the original text and the reference translation can be sent to the translation evaluation terminal 104 through the machine translation terminal 102 or the user terminal, such as the user terminal 12, so that the translation evaluation terminal 104 evaluates the translation based on the input text (including the translation and at least one of the original text and the reference translation) and the translation evaluation operator to obtain an evaluation result.

[0064] The translation evaluation operator can be divided into three types according to the input: reference translation type, original text type and reference translation combined with original text type. The input of the reference translation type includes translation and reference translation, the input of the original text type includes original text and translation, and the input of the reference translation combined with original text type includes reference translation, original text and translation. The translation evaluation operator provided in this application, i.e., the translation evaluation model provided in the subsequent embodiments, supports translation evaluation under the above three input types, and is a unified translation evaluation framework.

[0065] In the related art, the model-based translation evaluation operator (i.e., the translation evaluation model) only supports one of the input types, such as the reference translation type, the original text type, or the reference translation combined with the original text type, and is unable to perform translation evaluation in other scenarios, resulting in a limited scope of application of the model. In other input scenarios, users need to manually improve the input to adapt to the translation evaluation model, and the model is less convenient to use.

[0066] In order to improve the application scope of the translation evaluation model and the convenience of using the translation evaluation model, an embodiment of the present application provides a translation evaluation method suitable for multiple input scenarios. The main concept of the translation evaluation method is: identifying the type of reference information in the text pair to be evaluated or the current input scenario through identification information, and based on the input type or scenario, inputting the text pair to be evaluated into a pre-trained translation evaluation model to obtain the translation in the text pair to be evaluated or the score of the translation to be evaluated. The translation evaluation model provided by the present application can be competent for translation evaluation in a variety of input scenarios, improves the convenience of translation quality evaluation, and increases the application scope of the translation evaluation model.

[0067] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0068] Figure 2A flowchart of a translation evaluation method provided in an embodiment of the present application is provided. The method can be executed by any device that is provided with the translation evaluation model provided in an embodiment of the present application, such as the above-mentioned translation evaluation terminal 104. The specific form of the translation evaluation terminal 104 can be a computer, a server, a handheld device, etc.

[0069] like Figure 2 As shown, the translation evaluation method includes the following steps:

[0070] Step S201, obtaining a text pair to be evaluated and identification information.

[0071] The to-be-evaluated text pair includes a to-be-evaluated translation and at least one type of reference information, the reference information includes at least one of an original text and a reference translation of the to-be-evaluated translation, and the identification information is used to indicate the type of the reference information.

[0072] Specifically, the text pair to be evaluated may consist of at least two texts, one of which is the translation to be evaluated, and the other text may be the original text or reference translation of the text to be evaluated.

[0073] Exemplarily, HYP represents the translation to be evaluated, SRC represents the original text of the translation to be evaluated, and REF represents the reference translation of the translation to be evaluated. Then, the text pair to be evaluated including the translation to be evaluated and the corresponding original text may be [HYP, SRC], the text pair to be evaluated including the translation to be evaluated and the corresponding reference translation may be [HYP, REF], and the text pair to be evaluated including the translation to be evaluated, the corresponding original text and the corresponding reference translation may be [HYP, SRC, REF].

[0074] In one embodiment, different texts in the to-be-evaluated text pair may be separated by a separator, such as SEP.

[0075] Specifically, the identification information may include identifications of each reference information in the text pair to be evaluated, such as the above-mentioned SRC and REF.

[0076] Specifically, the text pair to be evaluated can be formed by splicing the translation to be evaluated and its reference information, that is, the translation to be evaluated and at least one of the original text and the reference translation of the translation to be evaluated can be spliced ​​to obtain the text pair to be evaluated including the translation to be evaluated.

[0077] Specifically, the identification information may include three types, such as first identification information, second identification information and third identification information. The first identification information is used to indicate that the reference information only includes the original text, the second identification information is used to indicate that the reference information only includes the reference translation, and the third identification information is used to indicate that the reference information includes the original text and the reference translation.

[0078] Specifically, after obtaining the texts such as the translation to be evaluated, the original text, and the reference translation, the texts may be transformed to obtain the translation to be evaluated, the original text, and the reference translation represented by vectors.

[0079] Specifically, the types of reference information can include three types in total, such as single reference type and multiple reference type, among which the single reference type means that the reference information includes only one of the original text and the reference translation, and the multiple reference type means that the reference information includes the original text and the reference translation. The single reference type can also be divided into the original text type and the reference translation type. The reference information of the original text type is the original text, and the reference information of the reference translation type is the reference translation.

[0080] Exemplarily, the original text type may be represented by SRC, the reference translation type may be represented by REF, and the multi-reference type may be represented by SRC+REF.

[0081] In one embodiment, the identification information can be specifically used to identify the type of each text in the pair of texts to be evaluated, each text including the above-mentioned translation to be evaluated and each reference information. The translation to be evaluated can be identified by "HYP" or "H", the reference translation can be identified by "REF" or "R", and the original text can be identified by "SRC" or "S".

[0082] In one embodiment, there may be multiple reference translations for a translation to be evaluated.

[0083] Exemplarily, a user may submit or upload each text in a text pair to be evaluated, such as a translation to be evaluated, an original text, a reference translation, etc., through an interactive interface, and when submitting each text, select a text label for the text, thereby determining the type of reference information of the translation to be evaluated, i.e., identification information of the text pair to be evaluated, based on the label of each text.

[0084] In one embodiment, the upload order of the translation to be evaluated, the original text and the reference translation may be set by default, so that the type of reference information of the translation to be evaluated, ie, identification information of the text pair to be evaluated, is determined based on the upload order.

[0085] Step S202: determining the score of the translation to be evaluated according to the text pair to be evaluated, the identification information and the translation evaluation model.

[0086] The translation evaluation model is a pre-trained model for translation evaluation, that is, a trained translation evaluation model.

[0087] Specifically, the text pair to be evaluated and the identification information may be input into a trained translation evaluation model, and the trained translation evaluation model may output a score of the translation to be evaluated in the text pair to be evaluated, such as 87 points.

[0088] Optionally, the text to be evaluated can be preprocessed based on the identification information to obtain an input sequence of the translation evaluation model. The preprocessed text pair to be evaluated, that is, the input sequence, is input into the translation evaluation model, and the translation evaluation model outputs the score of the translation to be evaluated in the text pair to be evaluated.

[0089] Specifically, based on the identification information, the individual texts in the text pair to be evaluated may be concatenated to obtain the preprocessed text pair to be evaluated, or to obtain an input sequence of the translation evaluation model.

[0090] Among them, the text pair to be evaluated can be composed of two or three texts, one of which is the translation to be evaluated, and the remaining texts can be at least one of the original text and the reference translation of the translation to be evaluated, that is, the "text" referred to in the embodiments of the present application covers the translation to be evaluated, the original text and the reference translation.

[0091] For example, taking the case where the text pair to be evaluated includes the translation to be evaluated, the original text and the reference translation, the preprocessed text pair to be evaluated or the input sequence is x SRC+REF The expression of x is: SRC+REF =Concat(h, s, r), where Concat(· represents a concatenation operation, vector h represents the translation to be evaluated, vector s represents the original text, and vector r represents the reference translation. SRC+REF Input the translation evaluation model to obtain the score of the translation to be evaluated predicted by the translation evaluation model.

[0092] Specifically, after concatenating each component of the text pair to be evaluated according to the identification information, embedding processing may be performed on the concatenated text pair to be evaluated to convert each word segment in the text pair to be evaluated into a vector, thereby obtaining an input sequence of the translation evaluation model.

[0093] By preprocessing the text pairs to be evaluated based on identification information, the text pairs to be evaluated in various input scenarios are converted into input sequences with a unified format, making the translation evaluation model compatible with various input scenarios and improving the application scope of the model.

[0094] Optionally, the preprocessed text pair to be evaluated is input into the translation evaluation model to obtain a score of the translation to be evaluated, including:

[0095] The preprocessed text pair to be evaluated is input into the translation evaluation model; an attention matrix of the text pair to be evaluated is generated through the translation evaluation model based on the attention mechanism and the identification information; and a score of the translation to be evaluated is determined according to the attention matrix.

[0096] In one embodiment, the above preprocessing step can be omitted, that is, step S202 can be replaced by the following steps:

[0097] An attention matrix of the to-be-evaluated text pair is generated via the translation evaluation model based on an attention mechanism and the identification information; and a score of the to-be-evaluated translation is determined according to the attention matrix.

[0098] Specifically, the translation evaluation model may include a language model layer or an encoder, which is used to encode the text pair to be evaluated, so as to determine the score of the translation to be evaluated in the text pair to be evaluated based on the encoding of the text pair to be evaluated.

[0099] For example, Figure 3 For this application Figure 2 The structural diagram of the translation evaluation model in the embodiment shown is as follows: Figure 3 As shown in Figure 1, the translation evaluation model includes an input concatenation layer, an encoder, a pooling layer, and a feedforward neural network layer. The input concatenation layer is used to receive the texts and text labels input by the user and output the text pairs to be evaluated. Figure 3 In the figure, a text pair to be evaluated includes a translation to be evaluated, an original text and a reference translation. The encoder is used to encode the text pair to be evaluated to obtain an encoding vector. The pooling layer is used to map the encoding vector to a weight vector, and the feedforward neural network layer is used to obtain a score of the translation to be evaluated according to the weight vector.

[0100] In one embodiment, the encoder is a recurrent neural network.

[0101] In one embodiment, the encoder is a language model, such as an attention-based language model.

[0102] Exemplarily, the encoder may be composed of one or more Transformer encoders.

[0103] Specifically, the Query vector and Key vector of the text pair to be evaluated can be generated based on the encoder, and the attention matrix can be generated based on the Query vector and the Key vector based on the attention layer. The attention matrix is ​​then mapped into a weight vector through the pooling layer, and the score of the translation to be evaluated is output based on the weight vector through the output layer or the feedforward network layer.

[0104] The translation evaluation method provided in this embodiment is aimed at translation evaluation in multiple input scenarios, such as the input scenarios being the original text scenario, the reference translation scenario, and the original text and reference translation scenario. The current input scenario is identified through identification information, and the score of the translation in the text pair is determined according to the text pair input in the current input scenario through the designed translation evaluation model, thereby realizing translation evaluation in multiple input scenarios through the same translation evaluation model, improving the application scope and flexibility of the translation evaluation model, and improving the convenience of translation evaluation.

[0105] Figure 4 A flowchart of another translation evaluation method provided in an embodiment of the present application. Figure 2 Based on the embodiment shown, step S202 is further refined, such as Figure 4 As shown, the translation evaluation method may include the following steps:

[0106] Step S401, obtaining a text pair to be evaluated and identification information.

[0107] Step S402: determining the position of each text in the pair of texts to be evaluated according to the identification information.

[0108] Specifically, the position of each text in the input sequence of the translation evaluation model can be preset, such as the translation to be evaluated or the translation, the original text and the reference translation from left to right, and the type of each text in the text pair to be evaluated is identified based on the identification information, and then the position of each text is determined.

[0109] Specifically, the user can upload each text in the text pair to be evaluated, such as the translation to be evaluated, the original text of the translation to be evaluated, and the reference translation through the user interaction interface. When uploading the original text or the reference translation, the user can add identification information to the original text or the reference translation, or before uploading the text, select the type of each text to be uploaded or the identification information of the reference information, or after uploading the text, select the type of each text to be uploaded or the identification information of the reference information to be uploaded.

[0110] Exemplarily, in the user interaction interface, the user can first select an input scenario or directly adopt a default input scenario. The input scenario is used to limit the composition or reference information of the text pair to be evaluated that is input into the translation evaluation model. The input scenario can be divided into a reference translation scenario (the text pair to be evaluated includes the translation to be evaluated and the reference translation), an original text scenario (the text pair to be evaluated includes the translation to be evaluated and the original text), and a three-input scenario (the text pair to be evaluated includes the translation to be evaluated, the reference translation, and the original text). Furthermore, based on the input scenario selected by the user and the order of the subsequent uploaded texts, the identification information of each text is determined, such as the default user's first uploaded text is the translation to be evaluated. If the input scenario is a reference translation scenario or a three-input scenario, the second uploaded text is the reference translation. If the input scenario is an original text scenario, the second uploaded text is the original text. If the input scenario is a three-input scenario, the third uploaded text is the original text.

[0111] Exemplarily, the user interaction interface may include three input boxes, where input boxes 501 to 503 are used to input the translation to be evaluated, the reference translation and the original text, respectively. Each input box is used to input a type of text. The user can enter the corresponding text in the input box and click the upload completion or evaluation button. Based on the identification of each input box, the type of text entered into the input box is determined, that is, the above-mentioned identification information is determined, and based on the identification information, the input data in each input box is spliced ​​into an input sequence of the translation evaluation model. Figure 5 A schematic diagram of a user interaction interface provided by an embodiment of the present application, such as Figure 5 As shown, the user inputs content in input box 501 and input box 502 respectively, and clicks button 504 to trigger translation evaluation. Then, in the background, based on the identifier of input box 501, such as input_HYP, the identifier information of the content input in input box 501 is determined to be HYP, that is, the content input in input box 501 is determined to be the translation to be evaluated. Based on the identifier of input box 502, such as input_REF, the identifier information of the content input in input box 502 is determined to be REF, that is, the content input in input box 502 is determined to be the reference translation. HYP and reference translation x REF Generate an input sequence corresponding to the text pair to be evaluated, such as: {[C], x HYP , [S], x REF , [S]}, the input sequence is input into the translation evaluation model, and the score of the translation to be evaluated can be obtained, and the score can be displayed in the preset area of ​​the user interaction interface, Figure 5 In the example, "Score: 93.1" is displayed to the user. Other user interaction interfaces may also be used to upload the text pair to be evaluated and add the identification information. The specific form of the user interaction interface is not limited in this application.

[0112] Step S403 , splicing the texts in the to-be-evaluated text pair according to the positions of the texts to obtain a preprocessed to-be-evaluated text pair.

[0113] Specifically, the position of the text can be represented by numbers or letters. Taking the text pair to be evaluated including the translation to be evaluated, the reference translation and the original text as an example, the position of the text can be 1, 2, 3, or L, M, R, etc. "1" or "L" represents the text located at the leftmost side of the text pair to be evaluated, such as the translation to be evaluated, "2" or "M" represents the text located in the middle of the text pair to be evaluated, such as the reference translation, and "3" or "R" represents the text located at the rightmost side of the text pair to be evaluated, such as the original text.

[0114] Specifically, the texts in the to-be-evaluated text pair may be concatenated based on the positions of the texts to obtain an input sequence of the translation evaluation model, that is, the preprocessed to-be-evaluated text pair.

[0115] Furthermore, the concatenated texts to be processed may be embedded to obtain digitally represented text pairs to be evaluated, so that the translation evaluation model can predict the score based on the digitally represented text pairs to be evaluated.

[0116] Step S404: input the preprocessed text pair to be evaluated into the translation evaluation model.

[0117] Specifically, the preprocessed text pair to be evaluated may be input into an input embedding layer, an input layer or a language model layer of a trained translation evaluation model to obtain a score of the translation to be evaluated in the text pair to be evaluated based on the translation evaluation model.

[0118] Step S405: Generate an attention matrix of the text pair to be evaluated via the translation evaluation model based on the attention mechanism and the identification information.

[0119] Specifically, an attention matrix of the text pair to be evaluated can be generated through an attention layer in the translation evaluation model based on the attention mechanism and the identification information. The translation evaluation model can first perform feature extraction or semantic encoding on the input sequence of the text pair to be evaluated, such as nonlinear transformation, to obtain an encoding vector of the text pair to be evaluated; and then generate an attention matrix of the text pair to be evaluated based on the encoding vector and the identification information based on the attention layer.

[0120] Specifically, based on the identification information, the interaction mode of the encoding vectors corresponding to each text can be determined, and then based on the determined interaction mode and attention mechanism, each encoding vector is processed to obtain the attention matrix of the text pair to be processed.

[0121] In one embodiment, the encoding vector of the text to be evaluated may include the Query vector and the Key vector of each text in the pair of texts to be evaluated.

[0122] Optionally, based on the attention mechanism and the identification information, an attention matrix of the text pair to be evaluated is generated, including:

[0123] Based on the attention mechanism, an encoding vector of the text pair to be evaluated is generated; when the identification information indicates that the reference information is the original text or the reference translation, an attention matrix of the text pair to be evaluated is generated according to the encoding vector, wherein the encoding vector is composed of multiple attention values; when the identification information indicates that the reference information is the original text and the reference translation, an attention matrix of the text pair to be evaluated is generated according to a correction matrix and the encoding vector; wherein the correction matrix is ​​used to limit the interaction mode of the attention values ​​corresponding to the translation to be evaluated, the original text and the reference translation in the encoding vector, so that the values ​​corresponding to the original text and the reference translation in the attention matrix are independent of the values ​​corresponding to the translation to be evaluated (such as the Key vector and the Query vector).

[0124] The attention matrix includes the attention value or attention score of each input in the input sequence corresponding to the text pair to be evaluated.

[0125] Specifically, the correction matrix can limit the interaction between the attention value output in each attention layer and the input value, so that the attention value corresponding to the partial input (such as the reference translation or the original text) in each attention layer is only related to the set input value, such as only related to the value of the input itself, or the attention value corresponding to the input is limited to be independent of the translation to be evaluated, such as limiting the attention corresponding to the reference translation to be related only to the value of the reference translation (such as the Key vector, the Query vector), and the attention value corresponding to the original text is related to the values ​​of the original text and the reference translation (such as the Key vector, the Query vector).

[0126] Specifically, based on the input sequence and the weight matrix of the attention layer of the translation evaluation model, the Query vector and Key vector of each input in the input sequence corresponding to the text pair to be processed are calculated; based on the identification information, the Query vector of each input of the input sequence and the Key vector of each input, the attention matrix of the text pair to be evaluated is generated. When the identification information indicates that the reference information is the original text or the reference translation, that is, the input of the input sequence is two sentences, the attention matrix of the text pair to be evaluated is generated according to each encoding vector. When the identification information indicates that the reference information is the original text and the reference translation, that is, the input of the input sequence is three sentences, the attention matrix of the text pair to be evaluated is generated according to the correction matrix and each encoding vector.

[0127] For example, the expression of the attention matrix A is:

[0128]

[0129] Where Q is the Query vector of each input in the input sequence, K is the Key vector of each input in the input sequence, and M is the above correction matrix. The element m in the correction matrix is ij The expression is:

[0130]

[0131] Among them, area U is the area in the attention matrix where the interaction method needs to be restricted. The attention value of the input corresponding to area U is independent of the Key vector corresponding to area U, that is, the correlation between the input Query vector in area U and the Key vector corresponding to area U, i.e., the attention value, will not be calculated, thereby achieving the suppression of the input interaction method.

[0132] Specifically, region U may include: a region where the Query vector of the original text interacts with the Key vector of the translation to be evaluated, a region where the Query vector of the reference translation interacts with the Key vector of the translation to be evaluated, and a region where the Query vector of the reference translation interacts with the Key vector of the original text, such as Figure 7 The area corresponding to the “×” in the figure.

[0133] By introducing the correction matrix, a regional attention mechanism is realized, thereby limiting the correlation between the attention value of the original text or reference translation and the corresponding value of the translation to be evaluated, and realizing input interaction suppression, so that the calculated attention value of the original text or reference translation is independent of the translation to be evaluated. Based on the determined attention value, the translation to be evaluated is evaluated, thereby improving the accuracy of the evaluation of the translation to be evaluated.

[0134] In one embodiment, the language model layer may include multiple attention layers, each of which may adopt the above-mentioned regional attention mechanism. The above-mentioned correction matrix may be used to limit the values ​​of the reference translation or original text output between each attention layer to be independent of the input value of the translation to be evaluated, thereby achieving interaction inhibition and prohibiting the calculation of the value of the reference translation or original text between each attention layer.

[0135] In one embodiment, the above step S405 can be replaced by generating an attention matrix for the text pair to be evaluated based on a regional attention mechanism or based on an attention mechanism and a correction matrix via a translation evaluation model. Compared with the traditional attention mechanism, the regional attention mechanism adds a correction matrix, which is used to limit the attention values ​​corresponding to the original text and the reference translation when the reference information indicated by the identification information is the original text and the reference translation, and the interaction mode with the input value of the translation to be evaluated (the value in the edit vector) when calculating the attention matrix, so as to realize the regional attention mechanism.

[0136] Step S406: Determine the score of the translation to be evaluated according to the attention matrix via the translation evaluation model.

[0137] Specifically, the attention matrix can be input into the pooling layer of the translation evaluation model, and the output of the pooling layer can be input into the feed-forward neural network layer to obtain the score of the translation to be evaluated in the text pair to be evaluated.

[0138] In this embodiment, for translation evaluation in a multi-input scenario, the position of each text in the text pair to be evaluated is determined by identification information, and then the texts in the text pair to be evaluated are spliced ​​based on the position to obtain an input sequence for inputting the translation evaluation model, thereby reducing the dimension of the model input; the input sequence is input into the trained translation evaluation model, the input sequence is encoded by the translation evaluation model, and the attention matrix of the input sequence is determined based on the attention mechanism, and the score of the translation to be evaluated is obtained based on the attention matrix, thereby realizing a strategy of performing translation evaluation in a multi-input scenario based on a unified translation evaluation model, thereby improving the convenience of translation evaluation and the application scope of the translation evaluation model, and at the same time, the accuracy of model evaluation is improved through the introduction of the attention mechanism.

[0139] Figure 6 A flowchart of a method for training a translation evaluation model provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the training method of the translation evaluation model includes the following steps:

[0140] Step S601: Obtain a training sample set and identification information of each training text pair in the training sample set.

[0141] The training sample set includes a plurality of training text pairs, at least some of which are composed of translations, rating labels, original texts and reference translations, and the other training texts are composed of translations and rating labels, and original texts or reference translations. The original text is translated by a translation tool, a translation plug-in or a translation model to obtain a translation. The reference translation can be a translation of the original text provided by an authoritative expert or manually. The reference translation of the same original text has the same language as the translation.

[0142] The translation evaluation model is trained by training text pairs corresponding to various input scenarios in the training sample set, so that the trained translation evaluation model can evaluate the translation quality in various input scenarios, thereby improving the application scope of the model.

[0143] For example, taking the translation of a Chinese sentence into English as an example, the original text may be "I like cake", the reference translation may be "I like cake", and the translation to be evaluated may be "I like cakes".

[0144] Specifically, the training text pair can be obtained by concatenating a translation, a score label, and the original text and / or a reference translation of the translation.

[0145] Specifically, each object constituting the training text pair (such as the translation, the rating label, the original text or the reference translation) may be separated by a separator, and the separator may be "SEP", "S" or other symbols.

[0146] Exemplarily, the translations in the training sample set may be translations output by a translation model, a translation tool, a translation plug-in, or the above-mentioned translation terminal 102, such as translations output by a transaction platform during cross-border e-commerce trade, translations historically translated by online translation tools, etc.; they may also be translations uploaded by users, such as translations or translation assignments uploaded by teachers in a translation education scenario.

[0147] For example, each original text and its translation translated by the translation terminal 102 at a historical time may be collected and used as the original text and the translation in the training text pair, respectively.

[0148] For example, public data provided by machine translation evaluation competitions over the years can be collected and integrated to obtain a training sample set for model training. The public data includes multiple original texts, reference translations of the original texts, and translations of each original text output by the participating translation terminal 102. The scoring labels can be determined based on the scores of the translations by various experts in the competition results.

[0149] Optionally, the rating labels in the training sample set can be manually annotated, such as by experts.

[0150] In order to improve the annotation efficiency of the scoring labels in the training sample set, the scoring labels of a small part of the translations in the training sample set can be annotated manually, and the scoring labels of most of the translations are automatically annotated by the translation evaluation operator. The ratio of manual annotation to automatic annotation can be 1:9, 2:8 or other ratios. The annotation method of the training sample set based on the scoring labels can be divided into an artificial training sample set and a pseudo training sample set. The scoring labels in each training text pair in the artificial training sample set are annotated manually, and the training text pairs in the artificial training sample set can be called artificial training text pairs; the scoring labels in each training text pair in the pseudo training sample set are annotated based on the translation evaluation operator, the training text pairs in the pseudo training sample set can be called pseudo training text pairs, and the scoring labels in the pseudo training sample set can be called pseudo scoring labels.

[0151] The translation evaluation operator may be any operator other than the above-mentioned translation evaluation model, such as BLEU, TER, etc.

[0152] In one embodiment, after obtaining the complete translation, the complete translation may be divided to obtain various translation segments, wherein the translation segment includes at least one sentence, such as a paragraph, and a translation segment is regarded as a translation. Taking the training text pair including the translation, the reference translation and the original text as an example, the original text segment and the reference translation segment corresponding to the translation segment are respectively used as the original text and the reference translation of the translation, and the translation, the translation label, the reference translation and the original text are spliced ​​to obtain the training text pair corresponding to the translation.

[0153] In one embodiment, the training text pairs in the training sample set can be divided into a first training text pair, a second training text pair and a third training text pair according to their composition, wherein the first training text pair consists of a translation, a score label of the translation and the original text of the translation, the second training text pair consists of a translation, a score label of the translation and a reference translation of the translation, and the third training text pair consists of a translation, a score label of the translation, the original text of the translation and the reference translation of the translation. The ratio of the number of the first training text pair, the second training text pair and the third training text pair in the training sample set should be approximately 1:1:1, so that the evaluation accuracy of the translation evaluation model trained based on the training sample set is high in various input scenarios.

[0154] Step S602: training a translation evaluation model according to each training text pair and its identification information in the training sample set to obtain a trained translation evaluation model.

[0155] Specifically, each training text pair and its identification information are input into the translation evaluation model in turn, and the evaluation error is determined based on the predicted score output by the translation evaluation model and the corresponding score label. The parameters of the translation evaluation model are fine-tuned through back propagation of the evaluation error.

[0156] Specifically, each training text pair can be preprocessed according to the identification information, and the translation evaluation model can be trained based on each preprocessed training text pair to obtain a trained translation evaluation model. The preprocessing method of the training text pair is similar to the preprocessing method of the text pair to be evaluated in the aforementioned embodiment, and only "the text pair to be evaluated" is replaced by "the training text pair", which will not be repeated here.

[0157] Optionally, the translation evaluation model is trained according to each training text pair and its identification information in the training sample set to obtain a trained translation evaluation model, including:

[0158] According to the identification information, the position of each text in the training text pair is determined; according to the position of each text, each text in the training text pair is spliced ​​to obtain a preprocessed text pair to be trained; according to each preprocessed training text pair, the translation evaluation model is trained to obtain a trained translation evaluation model; wherein each text in the training text pair includes a translation, and includes at least one of an original text and a reference translation of the translation.

[0159] By preprocessing the training text pairs based on identification information, the training text pairs in various input scenarios are converted into input sequences with a unified format, making the translation evaluation model compatible with various input scenarios and improving the application scope of the model.

[0160] Optionally, training a translation evaluation model according to the training sample set and identification information of each training text pair therein to obtain a trained translation evaluation model includes:

[0161] According to the training sample pairs in the training sample set, the translation evaluation model is trained iteratively, and after the training is completed, the trained translation evaluation model is output; wherein the following steps are performed in one iterative training process:

[0162] The training text pairs corresponding to the current iteration are sequentially input into the translation evaluation model; for each training text pair corresponding to the current iteration, the attention matrix of the training text pair is generated through the translation evaluation model based on the attention mechanism and the identification information of the training text pair, and the prediction score corresponding to the training text pair is determined according to the attention matrix of the training text pair; the prediction error corresponding to the current iteration is calculated according to each of the prediction scores and the score label corresponding to each prediction score; and the parameters of the translation evaluation model are adjusted based on the prediction error corresponding to the current iteration.

[0163] In one embodiment, the number of training text pairs corresponding to each iteration may be the same or as much the same as possible, and the number may be 100, 50 or other values.

[0164] By introducing the attention mechanism into the translation evaluation model and performing translation evaluation based on the attention matrix, the evaluation accuracy of the translation evaluation model is improved.

[0165] Optionally, generating an attention matrix of the training text pair based on the attention mechanism and the identification information of the training text pair includes:

[0166] Based on the attention mechanism, an encoding vector of the training text pair is generated; when the identification information indicates that the reference information is the original text or the reference translation, an attention matrix of the training text pair is generated according to the encoding vector; when the identification information indicates that the reference information is the original text and the reference translation, an attention matrix of the training text pair is generated according to a correction matrix and the encoding vector; wherein the correction matrix is ​​used to control the attention values ​​corresponding to the original text and the reference translation in the attention matrix to be independent of the values ​​corresponding to the translation in the encoding vector.

[0167] The method of generating the attention matrix of the training text pair can refer to the method of generating the attention matrix of the text pair to be evaluated in the aforementioned embodiment, that is, it can refer to the relevant description of step S405, and only replace "text pair to be evaluated" with "training text pair", which will not be repeated here.

[0168] By introducing the correction matrix, a regional attention mechanism is realized, which limits the correlation between the attention value of the original text or reference translation and the corresponding value of the translation, realizes input interaction suppression, and makes the calculated attention value of the original text or reference translation independent of the translation. Based on the determined attention value, the predicted score of the translation is determined, which improves the accuracy of translation evaluation.

[0169] Exemplarily, the translation evaluation model may include an encoder, such as a Transformer encoder, through which the input training text pair is encoded, and a prediction score of the translation in the training text pair is generated based on the encoded training text pair through a conversion layer or an output layer.

[0170] Exemplarily, the translation evaluation model includes a recurrent neural network layer, through which an attention matrix of a training text pair is generated, and a predicted score of a translation in the training text pair is output based on the attention matrix through a conversion layer or an output layer.

[0171] In one embodiment, the translation evaluation model includes an input concatenation layer, a language model layer, a pooling layer, and an output layer. The language model layer includes a language model, which can be a pre-trained language model, such as a BERT model, a Transformer model, etc. The input concatenation layer is used to concatenate input training text pairs to obtain an input sequence; the language model layer is used to generate an attention matrix according to the input sequence; the pooling layer is used to convert the attention matrix into a weight vector; and the output layer is used to generate a predicted score for the translation in the input training text pair according to the weight vector.

[0172] Specifically, the language model layer can calculate the attention matrix of the input sequence based on the attention mechanism, where the attention matrix is ​​composed of the attention value of each input in the input sequence, and then determine the attention matrix based on the attention matrix.

[0173] For example, Figure 7 For this application Figure 6 The schematic diagram of the structure of the translation evaluation model provided in the embodiment shown is as follows: Figure 7 As shown in FIG. 1 , the translation evaluation model includes: a language model layer, a pooling layer, and a feedforward neural network layer. The training text pairs can be converted into input sequences x by preprocessing, where x = [x1, x2, …, x L ]; input the input sequence into the language model layer of the translation evaluation model. When the input sequence only includes the information of the translation and the original text (corresponding to the SRC branch), or only includes the information of the translation and the reference translation (corresponding to the REF branch), the attention matrix of the input sequence is calculated based on the full attention mechanism (Full Attention) through the language model layer; when the input sequence includes the information of the translation, the original text and the reference translation (corresponding to the SRC+REF branch), the attention matrix of the input sequence is calculated based on the regional attention mechanism (Regional Attention) through the language model layer; the attention matrix is ​​mapped into a weight vector through the pooling layer, and the predicted score of the translation in the training text pair is output based on the weight vector through the feedforward neural network layer.

[0174] The full attention mechanism means that the attention value corresponding to each input in the input sequence of the language model layer is related to each input. The regional attention mechanism limits the attention value corresponding to some inputs in the input sequence to only be related to the value of one or more specified inputs, thus achieving input suppression. Figure 7 In , HYP represents the translation to be evaluated, SRC represents the original text, and REF represents the reference translation. Figure 7 As shown in the figure, the attention value at the “×” (the correlation between the Query vector and the Key vector) is independent of the value of the input sequence at the corresponding position during calculation, thus realizing the regional attention mechanism.

[0175] For example, taking the input sequence or training text pair as an example, the input sequence or training text pair only includes the information of the translation and the reference translation. Indicates that, among them, l h is the length of the original text in the input sequence, l r is the length of the reference translation in the input sequence; then the attention matrix output by the language model layer The corresponding relationship is:

[0176]

[0177] Where d is the dimension of the output representation of the language model layer PLM, which can be understood as the number of attention layers or encoder layers included in the language model layer. The attention matrix is ​​pooled through the pooling layer to obtain the weight vector H REF:

[0178]

[0179] Finally, the feedforward neural network layer outputs the predicted score p according to the weight vector REF :

[0180] p REF =Feedforward(H REF )∈R 1

[0181] The training method of the translation evaluation model provided in the present embodiment includes a collection of training sample sets including training text pairs under various input scenarios, such as training text pairs corresponding to a translation and an original text, training text pairs corresponding to a translation and a reference translation, and training text pairs corresponding to a translation, an original text, and a reference translation. The translation evaluation model is trained by using training text pairs of rich types, so that the trained translation evaluation model has the ability to evaluate translations under various input scenarios, thereby improving the application scope of the translation evaluation model and the convenience of using the model. At the same time, the model is trained based on samples under various input scenarios, i.e., multi-task model training, which improves the learning ability of the model and further improves the accuracy of the model.

[0182] Optional, Figure 8 For this application Figure 6 A flowchart of a method for determining a pseudo-scoring label in the embodiment shown is shown in FIG. Figure 8 As shown, the method for determining a pseudo scoring label or the process for determining a scoring label in a pseudo training sample set specifically includes the following steps:

[0183] Step S801: for each translation evaluation operator, a translation score of each translation in a pseudo training sample set is generated based on the translation evaluation operator.

[0184] Specifically, for each translation in the pseudo training sample set, the translation score of the translation may be evaluated by using multiple translation evaluation operators, thereby obtaining multiple translation scores corresponding to the translation.

[0185] For example, taking the training samples in the training sample set including the translation, the original text and the reference translation as an example, for one of the translation evaluation operators, the training sample pair D can be obtained. i : Among them, h i is the training sample pair D i The translation in r i is the training sample pair D i Reference translation in s i is the training sample pair D i The original text in The translation h output by the translation evaluation operator i The translation score of .

[0186] Step S802: for each translation evaluation operator, based on the translation score corresponding to the translation evaluation operator, sort the translations in the pseudo training sample set to obtain a sorting result corresponding to the translation evaluation operator.

[0187] For each translation evaluation operator, the translations in the pseudo training sample set may be sorted in order from low to high translation scores to obtain the ranking of each translation in the sorting results corresponding to each translation evaluation operator.

[0188] For example, the translation score of one of the translation evaluation operators corresponding to the translation The ranking in the sorting result Q (in descending order) corresponding to the translation evaluation operator The expression can be:

[0189] Step S803: for each of the translations in the pseudo training sample set, determine a score label of the translation according to the ranking of the translation in the sorting results corresponding to each of the translation evaluation operators.

[0190] Specifically, the score label of the translation may be determined based on the average or weighted average of the rankings of the translation in the sorting results corresponding to various translation evaluation operators.

[0191] Specifically, the score label of the translation may be determined based on the average value and variance of the ranking of the translation in the sorting results corresponding to each translation evaluation operator.

[0192] By annotating the score labels of each translation in the pseudo training sample set through a score ranking method based on multiple translation evaluation operators, the score labels are automatically annotated while improving the accuracy of score label annotation, thereby improving the quality of translation evaluation model training.

[0193] Accordingly, the training process of the translation evaluation model can be specifically as follows:

[0194] Based on each training text pair in the pseudo training sample set, the translation evaluation model is pre-trained; based on each training text pair in the labeled training sample set, the pre-trained translation evaluation model is fine-tuned to obtain a trained translation evaluation model.

[0195] Fig. 9A flowchart of a translation evaluation method provided in another embodiment of the present application is provided. The method can be executed by any device that is provided with the translation evaluation model provided in the embodiment of the present application, such as the above-mentioned translation evaluation terminal 104. The specific form of the translation evaluation terminal 104 can be a computer, a server, a handheld device, etc.

[0196] like Fig. 9 As shown, the translation tool evaluation method includes the following steps:

[0197] Step S901: obtaining a plurality of original texts translated by a translation tool and requiring evaluation.

[0198] The translation tool may be an online translation tool or an offline translation tool, may be an independent translation software or a translation application, or may be a plug-in set in other translation software, such as a mini-program.

[0199] Specifically, various original texts translated by the translation tool in the most recent period, such as 10 days, 30 days, etc., and the translations to be evaluated of various original texts may be obtained.

[0200] In an educational scenario, the user will provide a reference translation of the original text, that is, the original text also has a corresponding reference translation, or in a translation tool testing scenario, when the translation tool is translating the original text, the user will also provide a reference translation of the original text, and then the original text translated by the translation tool, the reference translation corresponding to the original text, and the translation output of the original text to be evaluated can be obtained.

[0201] Step S902: Generate each pair of texts to be evaluated and their identification information according to each of the translations to be evaluated and their reference information.

[0202] The reference information includes at least one of the original text and the reference translation of the translation to be evaluated, and the identification information is used to indicate the type of the reference information.

[0203] Step S903: for each text pair to be evaluated, according to the translation evaluation model, the text pair to be evaluated and its identification information, determine the score of the translation to be evaluated in the text pair to be evaluated.

[0204] The translation evaluation model is a pre-trained model used to evaluate the quality of translation.

[0205] Step S904: generating evaluation results of the translation tool according to the scores of the translations to be evaluated.

[0206] Among them, the evaluation results can be used to indicate whether the translation tool is qualified.

[0207] Exemplarily, the translation tool may be set on the above-mentioned translation terminal 102, and the translation terminal 102 may send multiple original texts, the translations to be evaluated of each original text, and the reference translations corresponding to some original texts to the translation evaluation terminal 104, so that the translation evaluation terminal 104 executes the translation tool evaluation method provided in this embodiment, outputs the evaluation result of the translation tool, and sends the evaluation result to the translation terminal 102.

[0208] Each step provided in this embodiment, such as step S903, can refer to Figure 2 and Figure 4 The corresponding steps are explained and will not be repeated here.

[0209] In one embodiment, the evaluation result may also include identifications of the translations to be evaluated whose scores are lower than a preset score.

[0210] Specifically, an evaluation result of the translation tool may be generated based on one or more of the evaluation parameters such as the average value, minimum value, variance, etc. of each translation to be evaluated, so as to determine whether the translation tool is qualified. If the translation tool is unqualified, targeted maintenance may be performed on the translation tool based on the generated evaluation result.

[0211] Figure 2 , Figure 4 and Fig. 9 The translation evaluation model in the illustrated embodiment may be trained using the training method for the translation evaluation model provided in any of the above embodiments.

[0212] An embodiment of the present application provides a training device for a translation evaluation model, the device comprising: a training set acquisition module and a model training module.

[0213] Among them, the training set acquisition module is used to obtain the identification information of the training sample set and each training text pair therein, wherein the training text pair includes a translation and at least one type of reference information, the reference information includes at least one of an original text and a reference translation, and the identification information is used to indicate the type of reference information in the training text pair; the model training module is used to train the translation evaluation model according to the identification information of each training text pair in the training sample set to obtain a trained translation evaluation model.

[0214] Optionally, the training text pairs in the training sample set are divided into pseudo training text pairs and artificial training text pairs, the scoring labels in the artificial training text pairs are manually annotated, each artificial training text pair constitutes an artificial training sample set, and each pseudo training text pair constitutes a pseudo training sample set. The training device of the translation evaluation model also includes a pseudo label annotation module, which is used to: for each translation evaluation operator, based on the translation evaluation operator, generate a translation score for each translation in the pseudo training sample set; for each translation evaluation operator, based on the translation score corresponding to the translation evaluation operator, sort the translations in the pseudo training sample set to obtain the sorting result corresponding to the translation evaluation operator; for each translation in the pseudo training sample set, determine the scoring label of the translation according to the ranking of the translation in the sorting result corresponding to each translation evaluation operator.

[0215] Optionally, the model training module includes: a preprocessing unit, used to determine the position of each text in the training text pair according to the identification information, and splice each text in the training text pair according to the position of each text to obtain a preprocessed text pair to be trained; a model training unit, used to train the translation evaluation model according to each preprocessed training text pair to obtain a trained translation evaluation model; wherein each text in the training text pair includes a translation, and includes at least one of the original text and a reference translation of the translation.

[0216] Optionally, the model training module is specifically used to: perform cyclic iterative training on the translation evaluation model according to the training sample pairs in the training sample set, and output the trained translation evaluation model after the training is completed; wherein the following steps are performed during one iterative training process:

[0217] The training text pairs corresponding to the current iteration are sequentially input into the translation evaluation model; for each training text pair corresponding to the current iteration, the attention matrix of the training text pair is generated through the translation evaluation model based on the attention mechanism and the identification information of the training text pair, and the prediction score corresponding to the training text pair is determined according to the attention matrix of the training text pair; the prediction error corresponding to the current iteration is calculated according to each of the prediction scores and the score label corresponding to each prediction score; and the parameters of the translation evaluation model are adjusted based on the prediction error corresponding to the current iteration.

[0218] Optionally, based on the attention mechanism and the identification information of the training text pair, an attention matrix for the training text pair is generated, including: based on the attention mechanism, generating an encoding vector for the training text pair; when the identification information indicates that the reference information is the original text or the reference translation, generating the attention matrix for the training text pair according to the encoding vector; when the identification information indicates that the reference information is the original text and the reference translation, generating the attention matrix for the training text pair according to a correction matrix and the encoding vector; wherein the correction matrix is ​​used to control the attention values ​​corresponding to the original text and the reference translation in the attention matrix to be independent of the values ​​corresponding to the translation in the encoding vector.

[0219] The training device of the translation evaluation model provided in the embodiment of the present application can be used to perform the above Figure 6 and Figure 8 The technical solutions provided in any corresponding embodiment have similar implementation principles and technical effects, which will not be described in detail in this embodiment.

[0220] The embodiment of the present application provides a translation evaluation device, which includes: a text pair acquisition module and a score determination module.

[0221] Among them, the text pair acquisition module is used to acquire the text pair to be evaluated and identification information, wherein the text pair to be evaluated includes the translation to be evaluated and at least one type of reference information, the reference information includes at least one of the original text and the reference translation of the translation to be evaluated, and the identification information is used to indicate the type of the reference information; the score determination module is used to determine the score of the translation to be evaluated based on the text pair to be evaluated, the identification information and the translation evaluation model.

[0222] Optionally, the device also includes: a model training module, used to train the translation evaluation model based on a training sample set to obtain a trained translation evaluation model, wherein the training sample set includes multiple training text pairs, some of the training text pairs consist of a translation, a scoring label, an original text and a reference translation, and another part of the training text consists of a translation and a scoring label, and an original text or a reference translation.

[0223] Optionally, the training text pairs in the training sample set are divided into pseudo training text pairs and artificial training text pairs, the scoring labels in the artificial training text pairs are manually annotated, and each of the pseudo training text pairs constitutes a pseudo training sample set. The device also includes: a pseudo label annotation module, which is used to generate, for each translation evaluation operator, a translation score of each translation in the pseudo training sample set based on the translation evaluation operator; for each translation evaluation operator, based on the translation score corresponding to the translation evaluation operator, sorting each translation in the pseudo training sample set to obtain a sorting result corresponding to the translation evaluation operator; for each translation in the pseudo training sample set, determining the scoring label of the translation according to the ranking of the translation in the sorting results corresponding to each translation evaluation operator.

[0224] Optionally, the model training module is specifically used to: pre-train the translation evaluation model based on each training text pair in the pseudo training sample set; fine-tune the pre-trained translation evaluation model based on each training text pair in the labeled training sample set to obtain a trained translation evaluation model.

[0225] Optionally, the score determination module includes: a preprocessing unit, used to preprocess the text pair to be evaluated according to the identification information; and a score determination unit, used to input the preprocessed text pair to be evaluated into the translation evaluation model to obtain the score of the translation to be evaluated.

[0226] Optionally, the preprocessing unit is specifically used to: determine the position of each text in the text pair to be evaluated according to the identification information; splice each text in the text pair to be evaluated according to the position of each text to obtain a preprocessed text pair to be evaluated; wherein each text in the text pair to be evaluated includes the translation to be evaluated, and at least one of the original text and the reference translation of the translation to be evaluated.

[0227] Optionally, the score determination unit is specifically used to: input the preprocessed text pair to be evaluated into the translation evaluation model; generate an attention matrix of the text pair to be evaluated based on the attention mechanism via the translation evaluation model; and determine the score of the translation to be evaluated according to the attention matrix.

[0228] Optionally, the score determination module includes: an attention matrix generation unit, which is used to generate an attention matrix of the text pair to be evaluated based on the attention mechanism and the identification information via the translation evaluation model; and a translation scoring unit, which is used to determine the score of the translation to be evaluated based on the attention matrix.

[0229] Optionally, the score determination module further includes: a preprocessing unit, configured to preprocess the text pair to be evaluated according to the identification information.

[0230] Correspondingly, the attention matrix generating unit is specifically used to: receive the preprocessed text pair to be evaluated via the translation evaluation model, and generate an attention matrix of the text pair to be evaluated based on the attention mechanism and the identification information.

[0231] Optionally, the attention matrix generating unit is specifically used to: generate the encoding vector of the text pair to be evaluated based on the attention mechanism; when the identification information indicates that the reference information is the original text or the reference translation, generate the attention matrix of the text pair to be evaluated according to the encoding vector; when the identification information indicates that the reference information is the original text and the reference translation, generate the attention matrix of the text pair to be evaluated according to the correction matrix and the encoding vector; wherein the correction matrix is ​​used to control the attention values ​​corresponding to the original text and the reference translation in the attention matrix to be independent of the values ​​corresponding to the translation to be evaluated in the encoding vector.

[0232] The training device of the translation evaluation model provided in the embodiment of the present application can be used to perform the above Figure 2 and Figure 4 The technical solutions provided in any corresponding embodiment have similar implementation principles and technical effects, which will not be described in detail in this embodiment.

[0233] The embodiment of the present application provides another translation evaluation device, including: a text acquisition module and a text processing module.

[0234] A translation acquisition module is used to acquire translations to be evaluated of multiple original texts translated by a translation tool; a text pair generation module is used to generate each text pair to be evaluated and its identification information according to each translation to be evaluated and its reference information, wherein the reference information includes at least one of the original text and the reference translation of the translation to be evaluated, and the identification information is used to indicate the type of the reference information; a score determination module is used to determine the score of the translation to be evaluated in the text pair to be evaluated for each text pair to be evaluated according to a translation evaluation model, the text pair to be evaluated and its identification information; an evaluation result generation module is used to generate an evaluation result of the translation tool according to the score of each translation to be evaluated.

[0235] The translation evaluation device provided in the embodiment of the present application can be used to perform the above Fig. 9 The technical solutions provided in any corresponding embodiment have similar implementation principles and technical effects, which will not be described in detail in this embodiment.

[0236] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the electronic device provided in this embodiment includes:

[0237] At least one processor 1010; and a memory 1020 communicatively connected to the at least one processor; wherein the memory 1020 stores computer-executable instructions; and the at least one processor 1010 executes the computer-executable instructions stored in the memory so that the electronic device executes a method as provided in any of the foregoing embodiments.

[0238] Optionally, the memory 1020 may be independent or integrated with the processor 1010 .

[0239] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0240] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the method provided in any of the aforementioned embodiments can be implemented.

[0241] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method provided in any of the above embodiments when executed by a processor.

[0242] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0243] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0244] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0245] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0246] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0247] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0248] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0249] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods provided in each embodiment of the present application.

[0250] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0251] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for training a translation evaluation model, characterized in that: The method comprises: Acquire a training sample set and identification information of each training text pair in the training sample set, wherein the training text pair includes a translation and at least one type of reference information, the reference information includes at least one of an original text and a reference translation, and the identification information is used to indicate the type of reference information in the training text pair; Preprocessing each training text pair according to the identification information to convert each training text pair into an input sequence with a unified format; The translation evaluation model is trained according to each preprocessed training text pair and its identification information to obtain a trained translation evaluation model.

2. The method according to claim 1, characterized in that: The training text pairs in the training sample set are divided into pseudo training text pairs and artificial training text pairs, the scoring labels in the artificial training text pairs are manually annotated, each artificial training text pair constitutes an artificial training sample set, and each pseudo training text pair constitutes a pseudo training sample set; The method further comprises: For each translation evaluation operator, generating a translation score for each translation in the pseudo training sample set based on the translation evaluation operator; For each translation evaluation operator, sorting the translations in the pseudo training sample set based on the translation score corresponding to the translation evaluation operator to obtain a sorting result corresponding to the translation evaluation operator; For each of the translations in the pseudo training sample set, a score label of the translation is determined according to a ranking of the translation in the sorting results corresponding to each of the translation evaluation operators.

3. The method according to claim 1 or 2, characterized in that: According to the preprocessed training text pairs and their identification information, the translation evaluation model is trained to obtain a trained translation evaluation model, including: Determining the position of each text in the training text pair according to the identification information; splicing the texts in the training text pair according to the positions of the texts to obtain the preprocessed text pair to be trained; Training the translation evaluation model according to each preprocessed training text pair to obtain a trained translation evaluation model; Each text in the training text pair includes a translation and at least one of an original text and a reference translation of the translation.

4. The method according to claim 1 or 2, characterized in that: The translation evaluation model is trained according to the identification information of each training text pair after preprocessing to obtain a trained translation evaluation model, including: According to the training sample pairs in the training sample set, the translation evaluation model is trained iteratively, and after the training is completed, the trained translation evaluation model is output; wherein the following steps are performed in one iterative training process: Inputting the training text pairs corresponding to the current iteration into the translation evaluation model in sequence; For each training text pair corresponding to the current iteration, generating an attention matrix of the training text pair based on the attention mechanism and identification information of the training text pair via the translation evaluation model, and determining a prediction score corresponding to the training text pair according to the attention matrix of the training text pair; Calculate the prediction error corresponding to the current iteration according to each of the prediction scores and the score label corresponding to each of the prediction scores; Based on the prediction error corresponding to the current iteration, parameters of the translation evaluation model are adjusted.

5. The method according to claim 4, characterized in that Based on the attention mechanism and the identification information of the training text pair, an attention matrix of the training text pair is generated, including: Based on the attention mechanism, generating an encoding vector of the training text pair; When the identification information indicates that the reference information is an original text or a reference translation, generating an attention matrix for the training text pair according to the encoding vector; When the identification information indicates that the reference information is an original text and a reference translation, generating an attention matrix for the training text pair according to a correction matrix and the encoding vector; Among them, the correction matrix is ​​used to control the attention values ​​corresponding to the original text and the reference translation in the attention matrix, which are independent of the values ​​corresponding to the translation in the encoding vector.

6. A translation evaluation method, characterized in that: The method comprises: Acquire a text pair to be evaluated and identification information, wherein the text pair to be evaluated includes a translation to be evaluated and at least one type of reference information, the reference information includes at least one of an original text and a reference translation of the translation to be evaluated, and the identification information is used to indicate the type of the reference information; Determining a score of the translation to be evaluated based on the text pair to be evaluated, the identification information, and the translation evaluation model; Wherein, the translation evaluation model is trained based on the method provided in any one of claims 1-5.

7. A translation evaluation method, characterized in that: The method comprises: Obtain translations to be evaluated of multiple original texts translated by translation tools; Generating each pair of texts to be evaluated and identification information thereof according to each of the translations to be evaluated and reference information thereof, wherein the reference information includes at least one of the original text and the reference translation of the translation to be evaluated, and the identification information is used to indicate the type of the reference information; For each text pair to be evaluated, determining the score of the translation to be evaluated in the text pair to be evaluated according to the translation evaluation model, the text pair to be evaluated and its identification information; wherein the translation evaluation model is obtained by training based on the method provided in any one of claims 1 to 5; The evaluation result of the translation tool is generated according to the scores of the translations to be evaluated.

8. An electronic device comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images