Translation evaluation method, translation evaluation model training method and electronic device

By performing noise processing and similarity calculation on the original text, combined with the translation evaluation model, the problem of low accuracy of Chinese translation evaluation in the existing technology is solved, and a more accurate translation quality evaluation is achieved.

CN114997188BActive Publication Date: 2024-11-26ALIBABA (CHINA) CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202210624350.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-01
Publication Date
2024-11-26
Estimated Expiration
2042-06-01

AI Technical Summary

Technical Problem

The existing machine translation quality evaluation method is based on translation only and cannot fully consider the original text characteristics, resulting in low evaluation accuracy.

Method used

By performing noise addition processing on the original text of the evaluated translation, multiple noise addition original texts are generated and inputted into the machine translation model to generate the noise addition translation, calculate the similarity of the noise addition translation, and input the pre-trained translation evaluation based on the original text and similarity. Model, comprehensive evaluation.

Benefits of technology

Improve the accuracy of translation quality evaluation, and provide more accurate evaluation results by taking into account the robustness and internal characteristics of the machine translation model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114997188B_ABST
    Figure CN114997188B_ABST
Patent Text Reader

Abstract

The present application provides a translation evaluation method, a translation evaluation model training method and an electronic device, the translation evaluation method comprising: performing noise processing on an original text of a translation to be evaluated to obtain a plurality of noisy original texts; inputting the plurality of noisy original texts into a machine translation model to obtain a plurality of first noisy translations, wherein the machine translation model is a model for outputting the translation to be evaluated; calculating a first similarity of the plurality of first noisy translations; inputting the original text, the translation to be evaluated and the first similarity into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated, combining the internal features or state of the machine translation model to evaluate its output translation, thereby improving the accuracy of the translation evaluation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a translation evaluation method, a translation evaluation model training method and an electronic device. Background Art

[0002] Machine translation, also known as automatic translation, is a process that uses computers to convert text in a source language (original text) into text in a target language (translation). With the increase in the application scenarios of machine translation, the evaluation of translation quality has also received widespread attention.

[0003] Machine Translation Metric is mainly measured by measuring the consistency between the translation result and the language expression of a given part. Common translation quality evaluation methods are mainly based on BLEU (Bilingual Evaluation Understudy, a bilingual translation quality evaluation auxiliary tool) and BERT (Bidirectional Encoder Representations from Transformers, Tansformer's bidirectional encoder representation) models. The above models are only based on the translation results of the machine translation model, that is, the translation, and the evaluation accuracy is low and cannot meet the needs. Summary of the invention

[0004] The present application provides a translation evaluation method, a translation evaluation model training method and an electronic device, which comprehensively analyzes the translation output by a machine translation model and its internal features, such as a first similarity, to perform machine translation quality evaluation, thereby improving the accuracy of translation evaluation.

[0005] In a first aspect, the present application provides a translation evaluation method, comprising:

[0006] The original text of the translation to be evaluated is subjected to noise processing to obtain multiple noisy original texts;

[0007] Inputting the multiple noisy original texts into a machine translation model to obtain multiple first noisy translations, wherein the machine translation model is a model that outputs the translation to be evaluated;

[0008] Calculating a first similarity between the translation to be evaluated and a plurality of first noisy translations;

[0009] The original text, the translation to be evaluated and the first similarity are input into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0010] In a second aspect, the present application provides a translation evaluation model training method, comprising:

[0011] Acquire a training sample set, wherein the training sample set includes a plurality of sample pairs, each sample pair includes an original sample, a translation sample, and an evaluation label of the translation sample;

[0012] Performing noise processing on each of the original text samples to obtain a plurality of noisy original texts corresponding to each of the original text samples;

[0013] For each original text sample, multiple noisy original texts corresponding to the original text sample are input into the corresponding machine translation model to obtain multiple noisy translations corresponding to the original text sample, and a first similarity between the translation sample corresponding to the original text sample and the multiple noisy translations corresponding to the original text sample is calculated; wherein the machine translation model corresponding to the noisy original text is a model for translating the original text sample corresponding to the noisy original text;

[0014] The translation evaluation model is trained according to the training sample set and the first similarity to output a trained translation evaluation model.

[0015] In a third aspect, the present application provides a translation evaluation device, comprising:

[0016] The original text noise adding module is used to add noise to the original text of the translation to be evaluated, and obtain multiple noisy original texts;

[0017] A first noisy translation acquisition module, used for inputting the multiple noisy original texts into a machine translation model to obtain multiple first noisy translations, wherein the machine translation model is a model that outputs the translation to be evaluated;

[0018] A first similarity calculation module, used for calculating the first similarities of the plurality of first noisy translations;

[0019] The translation evaluation module is used to input the original text, the translation to be evaluated and the first similarity into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0020] In a fourth aspect, the present application provides a translation evaluation model training device, comprising:

[0021] A training set acquisition module, used to acquire a training sample set, wherein the training sample set includes a plurality of sample pairs, each sample pair includes an original sample, a translation sample and an evaluation label of the translation sample;

[0022] A sample noise adding module, used for performing noise adding processing on each of the original text samples to obtain a plurality of noisy original texts corresponding to each of the original text samples;

[0023] A sample similarity calculation module is used to input multiple noisy original texts corresponding to each original text sample into a corresponding machine translation model to obtain multiple noisy translations corresponding to the original text sample, and calculate a first similarity between the translation sample corresponding to the original text sample and the multiple noisy translations corresponding to the original text sample; wherein the machine translation model corresponding to the noisy original text is a model for translating the original text sample corresponding to the noisy original text;

[0024] The model training module is used to train the translation evaluation model according to the training sample set and the first similarity to output a trained translation evaluation model.

[0025] In a fifth aspect, the present application provides an electronic device, including:

[0026] A processor, and a memory communicatively connected to the processor;

[0027] The memory stores computer-executable instructions;

[0028] The processor executes the computer-executable instructions stored in the memory to implement the method provided in the first aspect or the second aspect of the present application.

[0029] In a sixth aspect, the present application provides a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method provided in the first aspect or the second aspect of the present application.

[0030] In a seventh aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method provided in the first aspect or the second aspect of the present application.

[0031] The translation evaluation method, translation evaluation model training method and electronic device provided by the present application are aimed at the scenario of evaluating the translation output by the machine translation model, and the original text of the translation to be evaluated is denoised to obtain multiple noisy original texts, and the multiple noisy original texts are input into the machine translation model to obtain multiple corresponding noisy translations, and the similarity of the multiple noisy translations, that is, the first similarity, is calculated, and then the first similarity, the translation to be evaluated and its original text are input into a pre-trained translation evaluation model, so as to output the evaluation result of the translation to be evaluated based on the translation evaluation model, so as to achieve comprehensive consideration of the translation output by the machine translation model and the internal features of the machine translation model, that is, the robust characteristics of the structure of the machine translation model itself reflected by the noise adding process and the first similarity, so as to improve the accuracy of the translation quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0033] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application;

[0034] Figure 2 A flowchart of a translation evaluation method provided in an embodiment of the present application;

[0035] Figure 3 A flowchart of another translation evaluation method provided in an embodiment of the present application;

[0036] Figure 4 For this application Figure 3 A schematic diagram of the structure of the translation evaluation model in the illustrated embodiment;

[0037] Figure 5 A flowchart of another translation evaluation method provided in an embodiment of the present application;

[0038] Figure 6 For this application Figure 5 A schematic diagram of the structure of the translation evaluation model in the illustrated embodiment;

[0039] Figure 7 A flowchart of a method for training a translation evaluation model provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0041] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0043] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application is shown in FIG. Figure 1As shown, the machine translation terminal 102 translates the input original text based on a machine translation model, such as a Transformer model, a recurrent neural network model, etc., such as real-time, online translation or offline translation, and outputs a translation of the original text.

[0044] In order to evaluate the translation quality, the translation output by the machine translation terminal 102, the original text or the reference translation of the translation, can be input into the translation evaluation terminal 104, and the translation can be evaluated based on the translation evaluation model set in the translation evaluation terminal 104, such as the BLEU model, TER (Translation Edit Rate), etc., and the evaluation result can be output or displayed to improve the machine translation model in the robot translation terminal 102 based on the evaluation result to improve the translation quality.

[0045] Translation evaluation models can be divided into three types according to input: reference translation type, original text type, and reference translation combined with original text type. The input of the reference translation type includes the translation and the reference translation, the input of the original text type includes the original text and the translation, and the input of the reference translation combined with original text type includes the reference translation, the original text, and the translation.

[0046] Since reference translations usually need to be provided by professional translators, a huge number of training samples are needed to train the translation evaluation model. The training of the model whose input includes the reference translation is labor-intensive and costly. Therefore, the original text type, that is, the translation evaluation type that uses the translation and its original text as input, has been widely used.

[0047] In related technologies, translation evaluation models, such as the BERT model, usually only use the output of the machine translation model, that is, the translation, as a basis for translation quality evaluation. Since the translation is only the result of the machine translation model output for a certain original text, the evaluation result is easily restricted by the original text, such as the simple structure of the original text, the single type of the original text, etc., resulting in a certain one-sidedness of the translation evaluation result based only on the translation, resulting in a low accuracy of translation quality evaluation.

[0048] In order to improve the accuracy of the translation evaluation model, an embodiment of the present application provides a translation evaluation method based on a model using uncertainty quantification. Specifically, the original text is subjected to noise processing, and a translation of the noisy original text is obtained based on a machine translation model. The uncertainty quantification of the similarity between the translation of the noisy original text and the translation to be evaluated, the translation and the original text are input into a pre-trained translation evaluation model to obtain an evaluation result. The translation quality is evaluated through the translation output by the machine translation model and the uncertainty quantification feature of the first similarity that characterizes the internal state of the model, thereby improving the accuracy of the evaluation.

[0049] Uncertainty measurement features are used to estimate the uncertainty of the model on the output results. In this application, the output results are translations, such as the translation to be evaluated. Model uncertainty mainly includes accidental uncertainty and cognitive uncertainty. Accidental uncertainty is also called data uncertainty, which is caused by the noise of the data itself, such as the original text, training set, etc. Cognitive uncertainty is also called model uncertainty, which is caused by the uncertainty of model parameters and model structure. By evaluating the uncertainty of the model, the accuracy of the evaluation of the model output, that is, the translation, can be improved.

[0050] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0051] Figure 2 A flowchart of a translation evaluation method provided in an embodiment of the present application is provided. The method can be executed by any device that is provided with a translation evaluation model provided in an embodiment of the present application, such as the above-mentioned translation evaluation terminal 104 or an electronic device in subsequent embodiments. The specific form of the translation evaluation terminal 104 can be a computer, a server, a handheld device, etc.

[0052] like Figure 2 As shown, the translation evaluation method includes the following steps:

[0053] Step S201 , performing noise processing on the original text of the translation to be evaluated to obtain a plurality of noisy original texts.

[0054] The translation to be evaluated is a translation output by a machine translation model that needs to be evaluated for translation quality, which can be a paragraph or sentence of an article. The original text of the translation to be evaluated can correspond to any language, such as Chinese, English, Japanese, German, etc. The language of the translation to be evaluated can be any language different from the language of its original text, such as English, Japanese, German, Chinese, etc.

[0055] For example, the translation to be evaluated can be the translation output by each machine translation model in the machine translation competition. The translation to be evaluated can also be the translation assignment submitted by a student. The translation to be evaluated can also be the translation output by the translation software in international trade.

[0056] The original text can be input into a machine translation model, and then the machine translation model is used to obtain a translation of the original text, that is, the translation to be evaluated.

[0057] Specifically, after obtaining the translation to be evaluated and the original text, the translation to be evaluated and the original text may be converted to obtain the translation to be evaluated and the original text represented by vectors.

[0058] Specifically, a certain proportion of the participles in the original text may be randomly replaced with other participles to obtain a noisy original text.

[0059] Specifically, a certain proportion of the word segmentations in the original text can be randomly deleted to obtain a noisy original text.

[0060] Specifically, based on the noise adding algorithm, the original text of the translation to be evaluated can be randomly noised N times to obtain N different noisy original texts, where N is a positive integer, such as 5, 10, 15, etc. The noise adding algorithm can be used to randomly change part of the word segmentation in the original text, such as using a mask to replace part of the word segmentation in the original text, and selecting a replacement word or a predicted word from the dictionary library for each mask according to a certain rule, so as to obtain the noisy original text.

[0061] For example, the original text is "Today's weather is pretty good, suitable for going out for a walk", and the original text after adding the mask can be "Today [mask 1] is pretty good, suitable for going out [mask 2]", that is, "weather" in the original text is replaced with "mask 1" and "walk" is replaced with "mask 2", the predicted word corresponding to mask 1 is "sunshine", and the predicted word corresponding to mask 2 is "play", then the noisy original text is "Today's sunshine is pretty good, suitable for going out for play".

[0062] Step S202: input the multiple noisy original texts into a machine translation model to obtain multiple first noisy translations.

[0063] The machine translation model is a model that outputs the translation to be evaluated. The machine translation model can be any model, such as a statistical translation model, a neural network machine translation model (NMT), such as a Tansformer model, an LSTM neural network model, a recurrent neural network model, etc.

[0064] The N noisy original texts corresponding to the original text are respectively input into the machine translation model corresponding to the original text, so that the robot translation model outputs the corresponding N first noisy translations. The machine translation model corresponding to the original text is a machine translation model that translates the original text to obtain the translation to be evaluated.

[0065] Step S203: calculating the first similarities of the plurality of first noisy translations.

[0066] The first similarity is used to characterize the similarity between the first noisy translations, and the similarity between the translations can be represented by the edit distance.

[0067] Specifically, for the original text x, N different noise additions are performed to obtain N noisy original texts The original text x and N noisy original texts Input the machine translation model respectively to obtain the translation to be evaluated and N first noisy translations, then the first similarity can include the N first noisy translations The similarity between any two translations in , such as the average value, sum value, median value, mode value, etc. of the similarity between any two translations.

[0068] Exemplarily, the first similarity S1 may be:

[0069]

[0070] in, is the first noisy translation of the ith is the jth first noisy translation, and the value range of i and j are both 1 to N. represents the similarity between the i-th first noisy translation and the j-th first noisy translation, which can be represented by the minimum edit distance.

[0071] In one embodiment, the first similarity may be the similarity between the translation to be evaluated and the plurality of first noisy translations.

[0072] Step S204: input the original text, the translation to be evaluated and the first similarity into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0073] Among them, the translation evaluation model can be any multilingual model, such as XLM-R (Cross-lingualLanguage Model Rretraining-RoBERTa), M-Bert (Multilingual-Bert) and other pre-trained language models.

[0074] Specifically, the original text and the translation to be evaluated can be segmented, and each segmentation of the original text and each segmentation of the translation to be evaluated can be concatenated with a specific separator to obtain an input feature vector, and the input feature vector and the first similarity can be input into a pre-trained translation evaluation model, and the evaluation result of the translation to be evaluated can be output through the translation evaluation model.

[0075] The translation evaluation method provided by the present application aims at the scenario of evaluating the translation output by a machine translation model, adds noise to the original text of the translation to be evaluated, thereby obtaining multiple noisy original texts, inputs the multiple noisy original texts into the machine translation model, obtains multiple corresponding noisy translations, calculates the similarity of the multiple noisy translations, i.e., the first similarity, and then inputs the first similarity, the translation to be evaluated and its original text into a pre-trained translation evaluation model, thereby outputting an evaluation result of the translation to be evaluated based on the translation evaluation model, realizing comprehensive consideration of the translation output by the machine translation model and the internal features of the machine translation model, i.e., evaluating the translation through the robust characteristics of the structure of the machine translation model itself reflected by the noise adding process and the first similarity, thereby improving the accuracy of translation quality evaluation.

[0076] Figure 3 A flowchart of another translation evaluation method provided in an embodiment of the present application. Figure 2 Based on the embodiment shown, step S201 and step S204 are further refined as follows: Figure 3 As shown, the translation evaluation method may include the following steps:

[0077] Step S301, execute step S302 and step S303 N times to obtain N noisy original texts.

[0078] Wherein, N is a positive integer greater than or equal to 2, such as 3, 5, 10, 20, etc., and N is an adjustable parameter.

[0079] Step S302: randomly deleting word segments in the original text with a first probability, and randomly inserting masks into the original text with a second probability.

[0080] Among them, the first probability p i and the second probability p d It can be between 5% and 30%, such as 10% or 15%. The first probability p i and the second probability p d They can be the same or different. The first probability p i and the second probability p d All of them can be adjustable parameters. Mask is used to mask the word segmentation at the corresponding position.

[0081] For example, taking the original text "haven't seen you in forever" as an example, through the above noise adding step, that is, by randomly inserting the mask with a certain probability or proportion <masked>and randomly delete tokens, the original text can be processed as " <masked>haven’t <masked>You forever". By executing step S302 N times, N masked original texts can be obtained.

[0082] Step S303: input the masked original text into the masked language model to obtain a noisy original text.

[0083] The Masked Language Model (MLM) is used to predict the word segmentation corresponding to the mask of the masked input, which can be a pre-trained language model, such as BERT, RoBERTa and other models.

[0084] Whenever a masked original text is obtained, it is input into the masked language model, so that the mask therein is predicted based on the context by the masked language model to obtain the noisy original text corresponding to the masked original text.

[0085] In one embodiment, after executing step S302 N times, the obtained N masked original texts are respectively input into the masked language model, thereby obtaining N different noisy original texts.

[0086] In the above embodiment, <masked>haven’t <masked>Taking the masked original text "you forever" as an example, the corresponding noisy original text can be "I haven't loved you forever". That is, the masked language model predicts that the two masks are "I" and "loved".

[0087] Step S304: inputting N noisy original texts into a machine translation model to obtain N first noisy translations.

[0088] The machine translation model is a model that outputs the translation to be evaluated.

[0089] The N noisy original texts obtained through the above steps are sequentially input into the machine translation model of the translation original text to obtain N first noisy translations.

[0090] Step S305: Calculate the first similarities of the N first noisy translations.

[0091] The first similarity is one of the uncertainty measurement features. By calculating the similarity between the translation of the noisy original text and the translated noisy original text (i.e., the first noisy translation) by the machine translation model, the robustness of the internal structure of the machine translation model can be tested.

[0092] The higher the first similarity, the more robust the machine translation model is, and the better the quality of the translation produced by the machine translation model is.

[0093] Through the above-mentioned denoising algorithm, multiple different noisy original texts can be quickly obtained, and through the reasonable setting of the first probability and the second probability, the process of translating the noisy original text through the machine translation model, that is, through the similarity of the noisy translations, can well characterize the robustness of the machine translation model, and then combine this robustness to evaluate the subsequent translation quality, so as to improve the accuracy of the translation quality evaluation.

[0094] Step S306: input the original text, the translation to be evaluated, and the first similarity into a pre-trained translation evaluation model.

[0095] Optional, Figure 4 For this application Figure 3 The structural diagram of the translation evaluation model in the embodiment shown is as follows: Figure 4 As shown, the translation evaluation model includes a pre-trained language layer, a feedforward network layer and an output layer. The pre-trained language layer includes a pre-trained language model, such as a cross-language pre-trained language model such as XLM-R, M-Bert, etc. The pre-trained language layer is used to receive the original text and the translation to be evaluated, such as an input vector concatenated from the original text and the translation to be evaluated, and generate a semantic feature vector based on the original text and the translation to be evaluated; the feedforward network layer is used to receive the semantic feature vector, and generate a weight vector based on the semantic feature vector and uncertainty measurement features, such as the first similarity; the output layer is used to receive the weight vector, and generate an evaluation result of the translation to be evaluated based on the weight vector.

[0096] In one embodiment, the uncertainty measurement feature includes at least one of the first similarity, the second similarity, the predicted probability of each mask, the frequency of each word segment in the original text, and the predicted probability of each word segment in the translation to be evaluated. The above uncertainty measurement features are used to evaluate the machine translation model from multiple dimensions, which will be described in detail in subsequent embodiments.

[0097] The feed forward network layer may include a linear regression model to linearly map the semantic feature vector and the uncertainty measurement feature into a weight vector.

[0098] By introducing uncertainty measurement features that characterize the internal structural characteristics of the machine translation model into the translation evaluation model, such as the first similarity and the second similarity that characterize the robustness of the machine translation model, the frequency of word segmentation that characterizes the characteristics of the training data, the mask prediction probability that characterizes the difficulty of translating the original text, and the word segmentation prediction probability that characterizes the translation accuracy, through the construction of the above uncertainty measurement features, the translation evaluation model can combine the internal state of the machine translation model and the input data to evaluate the translation quality, thereby improving the accuracy of the translation evaluation.

[0099] Step S307 , generating a semantic feature vector according to the original text and the translation to be evaluated via the pre-trained language layer of the translation evaluation model.

[0100] Specifically, the original text and the translation to be evaluated can be concatenated with a specific separator, such as sep, and the concatenated original text and translation are input into the pre-trained language layer of the translation evaluation model, thereby outputting the semantic feature vectors corresponding to the original text and the translation to be evaluated through the pre-trained language model in the pre-trained language layer, such as XLM-R.

[0101] Step S308: Generate a weight vector according to the semantic feature vector and the uncertainty measurement feature via the feedforward network layer of the translation evaluation model.

[0102] Specifically, the semantic feature vector and the uncertainty measurement feature, such as the first similarity, the second similarity, etc., may be concatenated and input into the feedforward network layer after concatenation, so as to output a weight vector through the feedforward network layer.

[0103] Step S309: generating an evaluation result of the translation to be evaluated according to the weight vector via the output layer of the translation evaluation model.

[0104] Illustratively, the evaluation result may be a score of the translation to be evaluated, such as 95 points.

[0105] In this embodiment, for the scenario of evaluating the translation output by the machine translation model, the word segmentation of the original text is randomly deleted and randomly masked by the first probability and the second probability respectively, and then the mask is predicted by the mask language model to obtain a plurality of noisy original texts, and the plurality of noisy original texts are input into the machine translation model to obtain a plurality of corresponding noisy translations, and by constructing a first similarity as an uncertainty measurement feature, the first similarity is concatenated with the semantic feature vector input by the pre-trained language layer of the translation evaluation model, and the translation quality is predicted by the feedforward network layer and the output layer of the translation evaluation model, so that the translation output by the machine translation model and the robustness of the machine translation model are comprehensively considered to evaluate the translation, thereby improving the accuracy of the translation quality evaluation.

[0106] In one embodiment, the uncertainty measurement feature of the feed-forward network layer input to the translation evaluation model further includes: the prediction probability of each mask.

[0107] Optionally, the method further includes: obtaining the predicted probability of each mask output by the mask language model. Accordingly, the above step S204 is specifically: inputting the original text, the translation to be evaluated, the first similarity and the predicted probability of each mask into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0108] The prediction probability of each mask is another uncertainty measurement feature mentioned in this application. When the mask language model performs mask prediction, it will output the predicted word segmentation corresponding to the mask and the predicted probability of the predicted word segmentation, such as 0.85. In order to further improve the accuracy of translation evaluation, the predicted probability of the mask, that is, the predicted probability of the predicted word segmentation corresponding to the mask, can be combined. The predicted probability of the mask can characterize the difficulty of translating the original text. If the predicted probability of the mask is high, it indicates that the noise processing of the original text has a low impact on the translation of the translated text, that is, the integrity of the original text itself and the difficulty of translation are low.

[0109] Specifically, the original text and the translation to be evaluated can be input into the pre-trained language layer of the translation evaluation model to obtain a semantic feature vector, and the first similarity and the predicted probability of each mask can be input into the feedforward network layer of the translation evaluation model, such as by concatenating the first similarity, the predicted probability and the semantic feature vector and then inputting them into the feedforward network layer, thereby obtaining the evaluation result of the translation to be evaluated through the feedforward network layer and the output layer.

[0110] By introducing uncertainty measurement features that characterize the difficulty of translating the original text, namely the predicted probability of the mask, into the translation evaluation model, the translation quality is evaluated by using the difficulty of translating the original text to assist the translation evaluation model, thereby improving the accuracy of translation quality evaluation.

[0111] In one embodiment, the uncertainty metric feature of the feedforward network layer input into the translation evaluation model further includes: a second similarity.

[0112] Optionally, the machine translation model is a neural network machine translation model, and the method further includes: randomly deactivating neurons of the machine translation model with a third probability for multiple times to obtain multiple noisy machine translation models; inputting the original text into the multiple noisy machine translation models to obtain multiple second noisy translations; and calculating the second similarities of the multiple second noisy translations. Accordingly, the above step S204 is specifically: inputting the original text, the translation to be evaluated, the first similarity, and the second similarity into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0113] The third probability may be between 5% and 30%, such as 8%, 10%, 15% and the like.

[0114] Specifically, the uncertainty of the model can be measured by adding noise to the machine translation model, that is, by randomly deactivating the neurons of the machine translation model with a certain probability (third probability) for multiple times, thereby obtaining multiple noisy machine translation models, inputting the above original texts into the multiple noisy machine translation models respectively, obtaining multiple second noisy translations, and calculating the similarity between the multiple second noisy translations, that is, the second similarity. The calculation method of the second similarity is similar to the calculation method of the first similarity, and only the first noisy translation is replaced by the second noisy translation, which will not be repeated here.

[0115] Specifically, the original text and the translation to be evaluated can be input into the pre-trained language layer of the translation evaluation model to obtain a semantic feature vector, and the first similarity and the second similarity can be input into the feedforward network layer of the translation evaluation model, such as by concatenating the first similarity, the second similarity and the semantic feature vector and then inputting them into the feedforward network layer, thereby obtaining the evaluation result of the translation to be evaluated through the feedforward network layer and the output layer.

[0116] By introducing uncertainty measurement features that characterize the robustness of the machine translation model, namely the second similarity, into the translation evaluation model, the robustness of the machine translation model is used to assist the translation evaluation model in evaluating the translation quality, thereby improving the accuracy of the translation quality evaluation.

[0117] In one embodiment, the uncertainty measurement feature of the feedforward network layer input into the translation evaluation model further includes: the frequency of each word segment of the original text.

[0118] Optionally, the method further includes: obtaining the frequency of each word segment in the original text in a training set, wherein the training set is used to train the machine translation model. Accordingly, the above step S204 is specifically: inputting the original text, the translation to be evaluated, the first similarity and the frequency of each word segment in the original text into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0119] The frequency of a segmentation in the training set can be represented by the number of times the segmentation appears in the training set. The higher the frequency of a segmentation, the higher the accuracy of the machine translation model in translating the segmentation.

[0120] By introducing uncertainty measurement features related to the original text segmentation input into the machine translation model, namely the frequency of word segmentation, the translation evaluation model is assisted in evaluating the translation quality through the uncertainty of the original text's own word segmentation, thereby improving the accuracy of translation quality evaluation.

[0121] In one embodiment, the uncertainty measurement feature of the feedforward network layer input into the translation evaluation model further includes: the predicted probability of each word segment in the translation to be evaluated.

[0122] Optionally, the machine translation model is a neural network machine translation model, and the method further comprises: obtaining the predicted probability of each word segment in the translation to be evaluated output by the output layer of the machine translation model. Accordingly, the above step S204 is specifically: inputting the original text, the translation to be evaluated, the first similarity, and the predicted probability of each word segment in the translation to be evaluated into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0123] The predicted probability of a word segmentation is also called the decoding probability of the word segmentation. The neural network machine translation model is usually an autoregressive model, that is, the translation word segmentations are generated one by one from left to right, and the decoding probability of the word segmentation of the translation to be evaluated output by the decoding step is obtained. The decoding probability is input into the feedforward network layer of the translation evaluation model to combine the decoding probability, a measurement feature of uncertainty, to evaluate the translation quality.

[0124] A neural network machine translation model, such as the XLM-R model, usually includes a softmax layer, through which the probability distribution of each word segment of the translation to be evaluated can be output. For each word segment in the translation to be evaluated, the neural network machine translation model outputs the word with the maximum probability in the probability distribution corresponding to the word segment, and the maximum probability is the above-mentioned predicted probability.

[0125] Specifically, the expected prediction probability E of each word segment of the generated translation to be evaluated can be calculated, and the expected E is concatenated with the first similarity and semantic features and input into the feedforward network layer of the translation evaluation model. The calculation formula of the expected prediction probability E of each word segment is as follows:

[0126]

[0127]

[0128] in, is the decoding probability or prediction probability of the tth step, θ is the model parameter of the machine translation model, x is the original text, y t is the model output at time t, y <t is the model output before time t.

[0129] The predicted probability of a word segmentation can represent the confidence of the machine translation model in the word segmentation. The higher the predicted probability, the lower the uncertainty of the translation result, that is, the translation to be evaluated, and the higher the translation quality. By introducing the uncertainty measurement feature of the predicted probability of the word segmentation of the translation to be evaluated into the translation evaluation model, the model evaluates the overall translation to be evaluated based on the confidence or reliability of the word segmentation of the translation to be evaluated, thereby improving the accuracy of translation quality evaluation.

[0130] Figure 5 A flowchart of another translation evaluation method provided in an embodiment of the present application. Figure 2 On the basis of the embodiment shown, step S204 is further refined, and before step S204, a step of obtaining the second similarity, the predicted probability of each mask, the frequency of each word segment in the original text, and the predicted probability of each word segment in the translation to be evaluated is added, such as Figure 5 As shown, the translation evaluation method may include the following steps:

[0131] Step S501, obtaining the frequency of each word segment in the original text in the training set.

[0132] Step S502: Obtain the predicted probability of each word segment in the translation to be evaluated output by the output layer of the machine translation model.

[0133] Step S503, executing step S504 N times to obtain N noisy original texts.

[0134] Wherein, N is a positive integer greater than or equal to 2.

[0135] Step S504: randomly delete the word segments in the original text with a first probability, and randomly insert a mask into the original text with a second probability; and input the masked original text into a masked language model to obtain a noisy original text.

[0136] Step S505: Calculate the first similarities of the N first noisy translations.

[0137] Step S506, obtaining the predicted probability of each mask output by the mask language model.

[0138] Step S507 , randomly inactivating neurons of the machine translation model multiple times with a third probability to obtain multiple noisy machine translation models.

[0139] Step S508: input the original text into the multiple noisy machine translation models to obtain multiple second noisy translations.

[0140] Step S509: calculating the second similarities of the plurality of second noisy translations.

[0141] Step S510, inputting the original text, the translation to be evaluated, the first similarity, the second similarity, the frequency of each word segment in the original text, the predicted probability of each mask, and the predicted probability of each word segment in the translation to be evaluated into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0142] Specifically, the original text and the translation to be evaluated can be input into the pre-trained language layer of the translation evaluation model to obtain a semantic feature vector, and the uncertainty measurement features such as the first similarity, the second similarity, the frequency of each word segment in the original text, the predicted probability of each mask, and the predicted probability of each word segment in the translation to be evaluated can be input into the feedforward network layer of the translation evaluation model. For example, after splicing these uncertainty measurement features with the semantic feature vector, they are input into the feedforward network layer, so as to obtain the evaluation result of the translation to be evaluated through the feedforward network layer and the output layer.

[0143] In one embodiment, only one or more of the above uncertainty measurement features can be selected to be input into the translation evaluation model, that is, concatenated with the semantic feature vector and input into the feedforward network layer of the translation evaluation model, so as to obtain the evaluation result of the translation to be evaluated through the feedforward network layer and the output layer.

[0144] Figure 6 For this application Figure 5 The structural diagram of the translation evaluation model in the embodiment shown is as follows: Figure 6 As shown, the translation evaluation model includes an XLM-R layer, a feedforward network layer and an output layer. The translation evaluation model may also include an input embedding layer (Embedding) (not shown in the figure), which is used to generate a word segmentation vector [x1 x2…x T ] and the word segmentation vectors of the translation to be evaluated [y1 y2…y T ], and concatenate the two through a specific separator to obtain the input vector, such as Figure 6 "[CLS][x1][x2][…][x T ][EOS][SEP][y1][y2][…][y T ][EOS]”Wherein, "CLS" indicates the beginning of the input vector, "EOS" is the end of the word segmentation vector of the original text and the translation to be evaluated, and "SEP" is a specific separator. The XLM-R layer includes an XLM-R model, which inputs the input vector into the XLM-R model, and outputs the semantic feature vectors corresponding to the original text and the translation to be evaluated through the XLM-R model. Then, the semantic feature vector is concatenated with the uncertainty measurement features such as the first similarity, the second similarity, the frequency of each word segment in the original text, the predicted probability of each mask, and the predicted probability of each word segment in the translation to be evaluated, and then input into the feedforward network layer to obtain the weight vector, which is then input into the output layer to obtain the evaluation result.

[0145] Figure 7 A flowchart of a method for training a translation evaluation model provided in an embodiment of the present application, which can be used to train Figures 2 to 6 The translation evaluation model provided by any corresponding embodiment, such as Figure 7 As shown, the training method of the translation evaluation model includes the following steps:

[0146] Step S701, obtaining a training sample set.

[0147] The training sample set includes a plurality of sample pairs, and each sample pair includes an original sample, a translation sample, and an evaluation label of the translation sample.

[0148] Step S702: performing noise processing on each of the original text samples to obtain a plurality of noisy original texts corresponding to each of the original text samples.

[0149] Step S703: for each original text sample, multiple noisy original texts corresponding to the original text sample are input into the corresponding machine translation model to obtain multiple noisy translations corresponding to the original text sample, and the first similarity of the multiple noisy translations corresponding to the original text sample is calculated.

[0150] The machine translation model corresponding to the noisy original text is a model for translating the original text sample corresponding to the noisy original text.

[0151] Step S704: training the translation evaluation model according to the training sample set and the first similarity to output a trained translation evaluation model.

[0152] Specifically, the sample pair can be obtained by concatenating the original sample, the translation sample and the evaluation label.

[0153] Specifically, each object constituting a sample pair (such as a translation sample, an evaluation label, and an original sample) may be separated by a separator, and the separator may be "SEP", "S" or other symbols.

[0154] Exemplarily, the translation samples in the training sample set may be translations output by a machine translation model, a translation tool, a translation plug-in, or the above-mentioned translation terminal 102, such as translations output by a trading platform during cross-border e-commerce trade, translations historically translated by online translation tools, etc.; they may also be translations uploaded by users, such as translations or translation assignments uploaded by teachers in a translation education scenario.

[0155] For example, each original text and its translation translated by the translation terminal 102 at a historical time may be collected and used as the original text sample and the translation text sample in the training text pair, respectively.

[0156] For example, public data provided by machine translation evaluation competitions over the years can be collected and integrated to obtain a training sample set for model training. The public data includes multiple original texts and translations of each original text output by the participating translation terminals 102. Evaluation labels can be determined based on the scores of the translations given by various experts in the competition results.

[0157] Optionally, the evaluation labels in the training sample set can be manually annotated, such as by experts.

[0158] In order to improve the annotation efficiency of evaluation labels in the training sample set, the evaluation labels of a small part of the translations in the training sample set can be annotated manually, and the evaluation labels of most of the translations are automatically annotated by the translation evaluation operator. The ratio of manual annotation to automatic annotation can be 1:9, 2:8 or other ratios. The annotation method of the training sample set based on the evaluation label can be divided into an artificial training sample set and a pseudo training sample set. The evaluation labels in each training text pair in the artificial training sample set are annotated manually, and the training text pairs in the artificial training sample set can be called artificial training text pairs; the evaluation labels in each training text pair in the pseudo training sample set are annotated based on the translation evaluation operator, the training text pairs in the pseudo training sample set can be called pseudo training text pairs, and the evaluation labels in the pseudo training sample set can be called pseudo evaluation labels.

[0159] The translation evaluation operator may be any operator other than the above-mentioned translation evaluation model, such as BLEU, TER, etc.

[0160] In one embodiment, after obtaining the complete translation, the complete translation may be divided to obtain various translation segments, where the translation segment includes at least one sentence, such as a paragraph, and a translation segment is regarded as a translation sample.

[0161] Specifically, each training text pair and the corresponding uncertainty measurement feature are input into the translation evaluation model in turn, and the evaluation error is determined based on the predicted score output by the translation evaluation model and the corresponding evaluation label. The parameters of the translation evaluation model are fine-tuned through back propagation of the evaluation error.

[0162] In one embodiment, in addition to the uncertainty measurement feature of the first similarity, the second similarity mentioned in the above embodiment, the frequency of each word segment in the original text, the predicted probability of each mask, and the predicted probability of each word segment in the translation to be evaluated, these uncertainty measurement features can also be introduced into the translation evaluation model, then the above step S704 can be replaced by: according to the training sample set, and at least one of the second similarity, the frequency of each word segment in the original text, the predicted probability of each mask, and the predicted probability of each word segment in the translation to be evaluated, the translation evaluation model is trained to output a trained translation evaluation model. The calculation or acquisition method of each uncertainty measurement feature is as shown in the above embodiment, and it is only necessary to replace the original text with the original sample and the translation to be evaluated with the translation sample, which will not be repeated here.

[0163] The translation evaluation method provided in the embodiment of the present application can be applied to the evaluation of translation quality in a variety of scenarios, such as cross-border e-commerce scenarios, online translation scenarios, translation education scenarios, corporate office scenarios, subtitle translation scenarios, etc.

[0164] In the cross-border e-commerce scenario, the two or more parties to the transaction communicate through chat software, audio and video software, trading platforms, etc., and can translate texts (original texts) in other languages ​​into texts (translations) in a specified language based on the translation plug-ins set in the software or platform. After obtaining the translation, the translation output by the translation plug-in (translation to be evaluated), the original text of the translation, and uncertainty measurement features such as the first similarity can be input into the translation evaluation model in an online or offline manner, and the score of the translation is output based on the translation evaluation model, so that the user can judge whether to accept the translation, and the supplier of the translation plug-in can maintain the translation plug-in based on the score to improve the translation quality.

[0165] In an enterprise office scenario, such as an online meeting in a multilingual scenario, the text data (original text) of the recognized conference audio is translated into a translation in a set language (translation to be evaluated) through a translation plug-in set in the conference software. The translation, the original text of the translation, and uncertainty measurement features such as the first similarity are input into a translation evaluation model to obtain a score for the translation, so that the user can judge whether it is necessary to consult the original text or re-translate it with the help of other translation tools based on the score, so as to improve the smoothness of conference communication and enable users to better understand the content of the meeting.

[0166] In cross-border e-commerce scenarios or enterprise office scenarios, this application provides a translation evaluation method for a translation plug-in or a translation plug-in evaluation method, including:

[0167] The method comprises the steps of: obtaining a translation to be evaluated of an original text translated by a translation plug-in; performing noise processing on the original text of the translation to be evaluated to obtain a plurality of noisy original texts; inputting the plurality of noisy original texts into a machine translation model to obtain a plurality of first noisy translations, wherein the machine translation model is a model for outputting the translation to be evaluated; calculating a first similarity of the plurality of first noisy translations; and inputting the original text, the translation to be evaluated and the first similarity into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0168] Among them, the translation plug-in translates the original text into a translation based on a machine translation model, such as the translation to be evaluated.

[0169] In the online translation scenario, the provider of the translation tool needs to evaluate the translation quality of the translation tool regularly or irregularly in order to improve the translation quality. The translation tool, such as a robot, a handheld device, etc., the translation text, the original text of the translation text, and uncertainty measurement features such as the first similarity are input into the translation evaluation model, and the translation quality of the translation tool is evaluated based on the translation evaluation model. When the translation quality of the translation tool is poor, the translation tool can be maintained in time to improve the translation accuracy of the translation tool, thereby improving the user experience and loyalty.

[0170] In the online translation scenario, this application provides a translation tool evaluation method, including:

[0171] A plurality of translations to be evaluated translated by a translation tool is obtained; for each translation to be evaluated, an original text of the translation to be evaluated is subjected to noise processing to obtain a plurality of noisy original texts, and the plurality of noisy original texts are input into a machine translation model to obtain a plurality of first noisy translations, wherein the machine translation model is a model for outputting the translation to be evaluated; a first similarity of the plurality of first noisy translations is calculated; the original text, the translation to be evaluated and the first similarity are input into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated; and an evaluation result of the translation tool is generated based on the evaluation results of a plurality of the translations to be evaluated.

[0172] The translation tool may be an online translation tool or an offline translation tool. It may be an independent translation software or a translation application, or a plug-in set in other translation software, such as a mini-program. The translation tool translates the original text into a translation, such as the translation to be evaluated, based on a machine translation model.

[0173] Specifically, various original texts translated by the translation tool in the most recent period, such as 10 days, 30 days, etc., and the translations to be evaluated of various original texts may be obtained.

[0174] Among them, the translation tool can be used to indicate whether the translation tool is qualified.

[0175] Exemplarily, the translation tool may be set on the above-mentioned translation terminal 102, and the translation terminal 102 may send multiple original texts and the translations to be evaluated of each original text to the translation evaluation terminal 104, so that the translation evaluation terminal 104 executes the translation tool evaluation method provided in this embodiment, outputs the evaluation result of the translation tool, and sends the evaluation result to the translation terminal 102.

[0176] In one embodiment, the evaluation result may also include identifications of the translations to be evaluated whose scores are lower than a preset score.

[0177] Specifically, an evaluation result of the translation tool may be generated based on one or more of the evaluation parameters such as the average value, minimum value, variance, etc. of each translation to be evaluated, so as to determine whether the translation tool is qualified. If the translation tool is unqualified, targeted maintenance may be performed on the translation tool based on the generated evaluation result.

[0178] In one embodiment, the above-mentioned translation tools and translation plug-ins may be collectively referred to as a translation platform, and the translation platform is configured to translate the original text based on a machine translation model to obtain a corresponding translation, such as a translation to be evaluated.

[0179] This application provides a translation evaluation method, comprising:

[0180] The translation platform outputs a translation text to be evaluated and its original text, and the translation platform translates the corresponding original text into the translation text to be evaluated based on a machine translation model; the original text of the translation text to be evaluated is subjected to noise processing to obtain a plurality of noisy original texts; the plurality of noisy original texts are input into the machine translation model to obtain a plurality of first noisy translations; a first similarity of the plurality of first noisy translations is calculated; the original text, the translation text to be evaluated and the first similarity are input into a pre-trained translation evaluation model to obtain an evaluation result of the translation text to be evaluated.

[0181] The various steps provided in this embodiment can refer to Figure 2 , Figure 3 and Figure 5 The corresponding steps are explained and will not be repeated here.

[0182] In the translation education scenario, in order to improve the evaluation efficiency of translation assignments submitted by students, teachers can input the translation assignments (translations), original texts, and uncertainty measurement features such as the first similarity submitted by students into the translation evaluation model, so that the translation evaluation model can output the scores of each translation assignment.

[0183] In a translation education scenario or a translation teaching scenario, this application provides a translation evaluation method, including:

[0184] The method comprises obtaining a translation to be evaluated and its original text input or uploaded by a user (such as a teacher, teaching assistant, etc.), performing noise processing on the original text of the translation to be evaluated to obtain a plurality of noisy original texts; inputting the plurality of noisy original texts into a machine translation model to obtain a plurality of first noisy translations, wherein the machine translation model is a model for outputting the translation to be evaluated; calculating a first similarity of the plurality of first noisy translations; and inputting the original text, the translation to be evaluated and the first similarity into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0185] An embodiment of the present application provides a translation evaluation device, which includes: an original text noise adding module, a first noise added translation acquisition module, a first similarity calculation module and a translation evaluation module.

[0186] Among them, the original text noisy module is used to perform noise processing on the original text of the translation to be evaluated to obtain multiple noisy original texts; the first noisy translation acquisition module is used to input the multiple noisy original texts into the machine translation model to obtain multiple first noisy translations, wherein the machine translation model is a model that outputs the translation to be evaluated; the first similarity calculation module is used to calculate the first similarity of the multiple first noisy translations; the translation evaluation module is used to input the original text, the translation to be evaluated and the first similarity into a pre-trained translation evaluation model to obtain the evaluation result of the translation to be evaluated.

[0187] Optional, original text noise module, specifically used for:

[0188] Execute the following steps N times to obtain N noisy original texts, where N is a positive integer greater than or equal to 2: randomly delete word segments in the original text with a first probability, and randomly insert masks into the original text with a second probability; input the masked original text into the masked language model to obtain a noisy original text.

[0189] Optionally, the device further comprises:

[0190] The mask probability acquisition module is used to obtain the predicted probability of each mask output by the mask language model.

[0191] Accordingly, the translation evaluation module is specifically used to:

[0192] The original text, the translation to be evaluated, the first similarity and the predicted probability of each mask are input into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0193] Optionally, the machine translation model is a neural network machine translation model, and the device further includes:

[0194] The second similarity calculation module is used to randomly inactivate the neurons of the machine translation model with a third probability for multiple times to obtain multiple noisy machine translation models; input the original text into the multiple noisy machine translation models to obtain multiple second noisy translations; and calculate the second similarity of the multiple second noisy translations.

[0195] Accordingly, the translation evaluation module is specifically used to:

[0196] The original text, the translation to be evaluated, the first similarity, and the second similarity are input into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0197] Optionally, the device further comprises:

[0198] The word segmentation frequency acquisition module is used to obtain the frequency of each word segment in the original text in a training set, wherein the training set is used to train the machine translation model.

[0199] Accordingly, the translation evaluation module is specifically used to:

[0200] The original text, the translation to be evaluated, the first similarity and the frequency of each word segment in the original text are input into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0201] Optionally, the machine translation model is a neural network machine translation model, and the device further includes:

[0202] Obtaining the predicted probability of each word segment in the translation to be evaluated output by the output layer of the machine translation model;

[0203] Accordingly, the translation evaluation module is specifically used to:

[0204] The original text, the translation to be evaluated, the first similarity and the predicted probability of each word segment in the translation to be evaluated are input into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

[0205] The training device of the translation evaluation model provided in the embodiment of the present application can be used to perform the above Figure 2 , Figure 3 and Figure 5 The technical solutions provided in any corresponding embodiment have similar implementation principles and technical effects, which will not be described in detail in this embodiment.

[0206] The present application provides a translation tool evaluation device, including:

[0207] A training set acquisition module is used to acquire a training sample set, wherein the training sample set includes multiple sample pairs, each sample pair includes an original sample, a translation sample and an evaluation label of the translation sample; a sample denoising module is used to perform denoising processing on each of the original samples to obtain multiple noisy original texts corresponding to each original sample; a sample similarity calculation module is used to input multiple noisy original texts corresponding to the original sample into a corresponding machine translation model for each original sample to obtain multiple noisy translations corresponding to the original sample, and calculate a first similarity between the translation sample corresponding to the original sample and the multiple noisy translations corresponding to the original sample; wherein the machine translation model corresponding to the noisy original text is a model for translating the original sample corresponding to the noisy original text; a model training module is used to train the translation evaluation model according to the training sample set and the first similarity to output a trained translation evaluation model.

[0208] The training device of the translation evaluation model provided in the embodiment of the present application can be used to perform the above Figure 7 The technical solutions provided in any corresponding embodiment have similar implementation principles and technical effects, which will not be described in detail in this embodiment.

[0209] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the electronic device provided in this embodiment includes:

[0210] At least one processor 810; and a memory 820 communicatively connected to the at least one processor; wherein the memory 820 stores computer-executable instructions; and the at least one processor 810 executes the computer-executable instructions stored in the memory so that the electronic device performs a method as provided in any of the foregoing embodiments.

[0211] Optionally, the memory 820 may be independent or integrated with the processor 810 .

[0212] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0213] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the method provided in any of the aforementioned embodiments can be implemented.

[0214] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method provided in any of the above embodiments when executed by a processor.

[0215] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0216] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0217] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0218] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0219] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0220] It should be noted that, in this article, the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0221] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0222] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods provided in each embodiment of the present application.

[0223] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0224] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.< / masked> < / masked> < / masked> < / masked> < / masked>

Claims

1. A translation evaluation method, characterized in that: The method comprises: The original text of the translation to be evaluated is subjected to noise processing to obtain multiple noisy original texts; Inputting the multiple noisy original texts into a machine translation model to obtain multiple first noisy translations, wherein the machine translation model is a model that outputs the translation to be evaluated; the machine translation model is a neural network machine translation model; Calculating first similarities of the plurality of first noisy translations; Randomly deactivating neurons of the machine translation model multiple times with a third probability to obtain multiple noisy machine translation models; Inputting the original text into the multiple noisy machine translation models to obtain multiple second noisy translations; Calculating the second similarities of the plurality of second noisy translations; inputting the original text, the translation to be evaluated, the first similarity and the second similarity into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

2. The method according to claim 1, characterized in that The original text is processed with noise to obtain multiple noisy original texts, including: Perform the following steps N times to obtain N noisy original texts, where N is a positive integer greater than or equal to 2: Randomly deleting word segments in the original text with a first probability, and randomly inserting masks into the original text with a second probability; The masked original text is input into the masked language model to obtain a noisy original text.

3. The method according to claim 2, characterized in that The method further comprises: Obtaining the predicted probability of each mask output by the mask language model; Inputting the original text, the translation to be evaluated and the first similarity into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated, including: The original text, the translation to be evaluated, the first similarity and the predicted probability of each mask are input into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

4. The method according to claim 1, characterized in that: The method further comprises: Obtaining the frequency of each word segment in the original text in a training set, wherein the training set is used to train the machine translation model; Inputting the original text, the translation to be evaluated and the first similarity into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated, including: The original text, the translation to be evaluated, the first similarity and the frequency of each word segment in the original text are input into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

5. The method according to claim 1, characterized in that The machine translation model is a neural network machine translation model, and the method further includes: Obtaining the predicted probability of each word segment in the translation to be evaluated output by the output layer of the machine translation model; Inputting the original text, the translation to be evaluated and the first similarity into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated, including: The original text, the translation to be evaluated, the first similarity and the predicted probability of each word segment in the translation to be evaluated are input into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

6. The method according to any one of claims 2-3, characterized in that: The translation evaluation model includes a pre-trained language layer, a feed-forward network layer and an output layer; The pre-trained language layer is used to generate a semantic feature vector according to the original text and the translation to be evaluated; The feedforward network layer is used to generate a weight vector according to the semantic feature vector and the uncertainty measurement feature; The output layer is used to generate an evaluation result of the translation to be evaluated according to the weight vector; The uncertainty measurement feature includes the first similarity and at least one of the second similarity, the predicted probability of each mask, the frequency of each word segment in the original text, and the predicted probability of each word segment in the translation to be evaluated.

7. A translation evaluation method, characterized in that: The method comprises: Obtaining a translation to be evaluated and its original text output by a translation platform, wherein the translation platform translates the corresponding original text into the translation to be evaluated based on a machine translation model; the machine translation model is a neural network machine translation model; The original text of the translation to be evaluated is subjected to noise processing to obtain multiple noisy original texts; Inputting the multiple noisy original texts into the machine translation model to obtain multiple first noisy translations; Calculating first similarities of the plurality of first noisy translations; Randomly deactivating neurons of the machine translation model multiple times with a third probability to obtain multiple noisy machine translation models; Inputting the original text into the multiple noisy machine translation models to obtain multiple second noisy translations; Calculating second similarities of the plurality of second noisy translations; The original text, the translation to be evaluated, the first similarity, and the second similarity are input into a pre-trained translation evaluation model to obtain an evaluation result of the translation to be evaluated.

8. A translation evaluation model training method, characterized in that: The method comprises: Acquire a training sample set, wherein the training sample set includes a plurality of sample pairs, each sample pair includes an original sample, a translation sample, and an evaluation label of the translation sample; Performing noise processing on each of the original text samples to obtain a plurality of noisy original texts corresponding to each of the original text samples; For each original text sample, multiple noisy original texts corresponding to the original text sample are input into the corresponding machine translation model to obtain multiple noisy translations corresponding to the original text sample, and a first similarity of the multiple noisy translations corresponding to the original text sample is calculated; wherein the machine translation model corresponding to the noisy original text is a model for translating the original text sample corresponding to the noisy original text; The translation evaluation model is trained according to the training sample set and the first similarity to output a trained translation evaluation model, wherein the translation evaluation model is used to perform an operation of evaluating the translation to be evaluated as claimed in any one of claims 1 to 6 or claim 7.

9. An electronic device, comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

Citation Information

Patent Citations

  • Machine translation method and device, electronic equipment and storage medium

    CN111814493A

  • Translation quality evaluation method and device

    CN111985251A

  • Translation evaluation method, device and system and computer storage medium

    CN112446221A

  • Machine translation model performance detection method and related equipment

    CN113705253A

  • A scoring model training method and device, a terminal and a storage medium

    CN113836946A