Multilingual translation data quality assessment system, method, device and storage medium

Abstract

The present invention discloses a multilingual translation data quality assessment system, method, device and storage medium. The system includes: a data enhancement module based on constraint decoding, which can obtain multilingual sentence pair data from a multilingual high-quality text sentence pair library, and perform local constraint translation to obtain forged sentence pair data; an uncertainty feature construction module based on a translation vocabulary, which can obtain sentence pair data from a multilingual high-quality text sentence pair library, and use Monte Carlo sampling to perform multiple forward propagations to obtain the confidence in the translation process; a data selection module for quality assessment confidence, which can obtain sentence pair data from a multilingual quality assessment corpus, and use Monte Carlo sampling to perform multiple forward propagations to calculate the comprehensive confidence, and select the manual re-examination data from the latest multilingual quality assessment model prediction score data according to the comprehensive confidence. It can improve the assessment quality and the efficiency and effect of quality inspectors.

Description

Technical Field

[0001] The present invention relates to the field of machine translation, and in particular to a multilingual translation data quality assessment system and method. Background Art

[0002] In recent years, the phased breakthroughs in technologies such as deep learning and transfer learning, as well as the large amount of data generated by the Internet, have brought a lot of opportunities for the development of natural language processing. A large number of natural language processing tasks have also made breakthrough progress. For example, the perplexity of language models has dropped from hundreds to dozens, or even more than ten; the BLEU score of machine translation has increased from 10 to 20 to 40 to 50. The reliability of using machines to automatically process texts is getting higher and higher, allowing more work to be done by machines. Deep learning usually relies on a large amount of labeled data. The amount and quality of the data directly determine the quality of the deep learning model. The popularity of deep learning and big data has also driven the development of the data annotation industry, from the initial pure manual data annotation to the use of certain machine capabilities to assist data annotation. The results of the machine translation system will be used to assist manual annotation, and the data will be screened through the quality assessment system. With the accumulation of large-scale parallel corpora, the continuous improvement of computing power, and the widespread application of Transformer models, the translation level of machine translation in general fields has even exceeded that of humans, making machine translation a technology with relatively high technical maturity in natural language processing, and it is also gradually applied in people's daily lives.

[0003] With the accumulation of machine translation data, data quality has become a factor that restricts the effectiveness of machine translation. Although thanks to cross-language pre-training technology, quality assessment models applied to machine translation tasks have made initial progress, especially in Chinese and English scenarios with rich corpus, wide dissemination and application scope and large number of people; however, multilingual translation scenarios still face the lack of high-quality bilingual sentence pairs and quality scoring data for cross-language language model and quality assessment training. Therefore, multilingual data quality assessment models have not yet reached the level of being practical and easy to use.

[0004] The existing multilingual translation data quality assessment system mainly faces the following two major problems: First, the quality assessment training lacks high-quality multilingual parallel data and assessment annotation data, and the model assessment effect needs to be improved; second, the coordination efficiency between quality inspectors and quality assessment models is not high, and the labor cost is high. Therefore, how to improve the multilingual quality assessment effect and improve the efficiency and effectiveness of manual quality inspection are issues that need to be solved.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a multilingual translation data quality assessment system, method, device and storage medium, which can improve the multilingual quality assessment effect and improve the efficiency and effect of manual quality inspection, thereby solving the above-mentioned technical problems existing in the prior art.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] The embodiment of the present invention provides a multilingual translation data quality assessment system, including: a multilingual quality assessment model, and also including:

[0009] A data enhancement module based on constraint decoding, an uncertainty feature construction module based on translation vocabulary, and a data selection module for quality assessment confidence;

[0010] The data enhancement module based on constraint decoding can obtain multilingual sentence pair data from a multilingual high-quality text sentence pair library, and perform local constraint translation on the obtained sentence pair data based on a multilingual machine translation model to obtain forged sentence pair data. The obtained forged sentence pair data is used in the training of a multilingual quality assessment model in a multi-task training framework during the training phase;

[0011] The uncertainty feature construction module based on the translation vocabulary can obtain sentence pair data from a multilingual high-quality text sentence pair library, and perform multiple forward propagations on the obtained sentence pair data using Monte Carlo sampling based on a multilingual machine translation model to obtain sentence-level model uncertainty features and sentence-level data uncertainty features of the translation in the sentence pair data during the forward propagation process, and the sentence-level model uncertainty features and sentence-level data uncertainty features are used in the quality assessment training of the multilingual quality assessment model in the training phase; and as confidence in the translation process in the reasoning phase;

[0012] The data selection module for the quality assessment confidence can receive the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary, and perform multiple forward propagations on the acquired sentence pair data using Monte Carlo sampling based on the multilingual quality assessment model to obtain a multidimensional score list, calculate the standard deviation of the multidimensional score list as the confidence in the quality assessment process of the latest multilingual quality assessment model, obtain a comprehensive confidence based on the confidence in the quality assessment process and the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary, and screen out manual review data from the multilingual quality assessment model prediction score data based on the comprehensive confidence.

[0013] The embodiment of the present invention further provides a multilingual translation data quality assessment method using the multilingual translation data quality assessment system of the present invention, comprising the following steps:

[0014] Training phase:

[0015] Acquire multilingual sentence pair data from a multilingual high-quality text sentence pair library through the data enhancement module based on constraint decoding of the system, perform local constraint translation on the acquired sentence pair data based on a multilingual machine translation model to obtain forged sentence pair data, and use the obtained forged sentence pair data in a multi-task training framework to train a multilingual quality assessment model during a training phase;

[0016] Acquire sentence pair data from a multilingual high-quality text sentence pair library through the uncertainty feature construction module based on the translation vocabulary of the system, perform multiple forward propagations on the acquired sentence pair data using Monte Carlo sampling based on a multilingual machine translation model, and obtain sentence-level model uncertainty features and sentence-level data uncertainty features of the translation in the sentence pair data during the forward propagation process. The sentence-level model uncertainty features and sentence-level data uncertainty features are used in quality assessment training of a multilingual quality assessment model during the training phase;

[0017] Reasoning stage:

[0018] The uncertainty feature construction module based on the translation vocabulary of the system is used to calculate the sentence-level model uncertainty feature and sentence-level data uncertainty feature of the translation in the sentence pair data in the forward propagation process as the confidence in the translation process;

[0019] The confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary is received by the data selection module of the quality assessment confidence of the system, and the sentence pair data obtained is forward propagated multiple times by Monte Carlo sampling based on the multilingual quality assessment model to obtain a multidimensional score list, and the standard deviation of the multidimensional score list is calculated as the confidence in the quality assessment process of the latest multilingual quality assessment model, and the comprehensive confidence is obtained according to the confidence in the quality assessment process and the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary, and the manual review data is screened out from the predicted score data of the multilingual quality assessment model according to the comprehensive confidence.

[0020] The embodiment of the present invention further provides a processing device, including:

[0021] at least one memory for storing one or more programs;

[0022] At least one processor can execute one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor can implement the method described in the present invention.

[0023] An embodiment of the present invention provides a readable storage medium storing a computer program, which can implement the method described in the present invention when the computer program is executed by a processor.

[0024] Compared with the prior art, the multilingual translation data quality assessment system, method, device and storage medium provided by the present invention have the following beneficial effects:

[0025] By utilizing local constrained decoding of the translation model for data enhancement, a large amount of forged data before manual post-editing can be constructed unsupervisedly, and joint training can be performed based on a multi-task learning framework and a quality assessment scoring task to alleviate the problem of scarcity of annotated data for quality assessment. In addition, in view of the scarcity of annotated data, especially the quality assessment problem for some zero-sample languages, the present invention constructs uncertainty features based on the vocabulary probability distribution of the translation model, and provides quality inspectors with the confidence level of the current model evaluation while estimating the scores of multilingual sentence pairs. The confidence level also comprehensively considers the uncertainty score estimates of the current sentence pair by the translation model and the quality assessment model, and provides quality inspectors with a more reasonable initial screening of data, thereby improving the efficiency and effectiveness of quality inspection by quality inspectors, and effectively solving the problems of low collaborative efficiency and high labor costs between quality inspectors and quality assessment models. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0027] Figure 1 A schematic diagram of the structure of a multilingual translation data quality assessment system provided by an embodiment of the present invention.

[0028] Figure 2 A schematic diagram of the structure of a data enhancement module based on constraint decoding of a multilingual translation data quality assessment system provided by an embodiment of the present invention.

[0029] Figure 3 A schematic diagram of the composition of an uncertainty feature construction module based on a translation vocabulary of a multilingual translation data quality assessment system provided by an embodiment of the present invention.

[0030] Figure 4 A schematic diagram of the structure of a data selection module based on translation and quality assessment confidence of a multilingual translation data quality assessment system provided by an embodiment of the present invention.

[0031] Figure 5 A flowchart of a method for evaluating the quality of multilingual translation data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the specific content of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0033] First, the terms that may be used in this article are explained as follows:

[0034] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y means both “X” or “Y” and “X and Y”.

[0035] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0036] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0037] Unless otherwise specified or limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this article can be understood according to specific circumstances.

[0038] When concentration, temperature, pressure, size or other parameters are expressed in the form of a numerical range, the numerical range should be understood to specifically disclose all ranges formed by the pairing of any upper limit, lower limit, and preferred value in the numerical range, regardless of whether the range is explicitly stated; for example, if a numerical range of "2 to 8" is stated, the numerical range should be interpreted as including ranges such as "2 to 7", "2 to 6", "5 to 7", "3 to 4 and 6 to 7", "3 to 5 and 7", "2 and 5 to 7", etc. Unless otherwise specified, the numerical ranges stated herein include both their end values ​​and all integers and fractions within the numerical range.

[0039] The orientation or position relationship indicated by terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. are based on the orientation or position relationship shown in the drawings and are only for the convenience and simplification of description, and do not explicitly or implicitly indicate that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation of this document.

[0040] The multilingual translation data quality assessment system and method provided by the present invention are described in detail below. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professionals in the field. If the specific conditions are not specified in the embodiments of the present invention, the conditions are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. If the manufacturer is not specified in the reagents or instruments used in the embodiments of the present invention, they are all conventional products that can be purchased commercially.

[0041] like Figure 1 As shown, an embodiment of the present invention provides a multilingual translation data quality assessment system, including: a multilingual quality assessment model, and also including:

[0042] A data enhancement module based on constraint decoding, an uncertainty feature construction module based on translation vocabulary, and a data selection module for quality assessment confidence;

[0043] The data enhancement module based on constraint decoding can obtain sentence pair data from a multilingual high-quality text sentence pair library, and perform local constraint translation on the obtained sentence pair data based on a multilingual machine translation model to obtain forged sentence pair data. The obtained forged sentence pair data is used in the training of a multilingual quality assessment model in a multi-task training framework during the training phase;

[0044] The uncertainty feature construction module based on the translation vocabulary can obtain sentence pair data from a multilingual high-quality text sentence pair library, and perform multiple forward propagations on the obtained sentence pair data using Monte Carlo sampling based on a multilingual machine translation model to obtain sentence-level model uncertainty features and sentence-level data uncertainty features of the translation in the multilingual sentence pair data during the forward propagation process, and the sentence-level model uncertainty features and sentence-level data uncertainty features are used in the quality assessment training of the multilingual quality assessment model in the training phase; and as confidence in the translation process in the reasoning phase;

[0045] The data selection module for the quality assessment confidence can receive the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary, and perform multiple forward propagations on the acquired sentence pair data using Monte Carlo sampling based on the multilingual quality assessment model to obtain a multidimensional score list, calculate the standard deviation of the multidimensional score list as the confidence in the quality assessment process of the multilingual quality assessment model, obtain a comprehensive confidence based on the confidence in the quality assessment process and the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary, and screen out manual review data from the multilingual quality assessment model prediction score data based on the comprehensive confidence.

[0046] Specifically, in the training of the multilingual quality assessment model through the multi-task training framework mentioned above, multi-task learning mainly includes two aspects: one is to use classification task learning based on (source text, real translation data) and (source text, local forged translation data) data to identify forged translations that may have omissions and mistranslations; the other is the regression task based on (source text, annotated translation) data training, that is, the scoring model. The neural network parameters used for the two tasks are shared and trained simultaneously.

[0047] like Figure 2 As shown, in the above system, the data enhancement module based on constraint decoding includes:

[0048] A first multilingual machine translation model and a local constraint decoding submodule; wherein;

[0049] The first multilingual machine translation model has an input end connected to a multilingual high-quality text sentence pair library, and an output end connected to an input end of a local constraint decoding submodule, and can obtain sentence pair data from the multilingual high-quality text sentence pair library during a training phase;

[0050] The local constraint decoding submodule is respectively provided with a constraint decoding output terminal and a forged data output terminal. The constraint decoding output terminal can randomly select a text within a preset percentage range from the translation of the sentence pair data as a constraint part, and perform constraint decoding together with the source text of the sentence pair data through the first multilingual machine translation model to obtain first forged data; and randomly select a text within a preset percentage range from the source text of the sentence pair data as a constraint part, and perform reverse translation together with the translation of the sentence pair data through the first multilingual translation model to obtain second forged data, and output the obtained first forged data and the second forged data together as forged data through the forged data output terminal.

[0051] In the above system, the local constraint decoding submodule randomly selects the first 30% to 70% of the text from the translation of the sentence pair data output by the first multilingual machine translation model as the constraint part;

[0052] The local constraint decoding submodule randomly selects the first 30% to 70% of the text from the source text of the clause pair data as the constraint part.

[0053] like Figure 3 As shown, in the above system, the uncertainty feature construction module based on the translation vocabulary includes:

[0054] The second multilingual machine translation model, the Monte Carlo sampling submodule and the vocabulary probability distribution calculation submodule; wherein,

[0055] The second multilingual machine translation model has an input end connected to a multilingual high-quality text sentence pair library, and an output end connected to an input end of a Monte Carlo sampling submodule, and can obtain sentence pair data from the multilingual high-quality text sentence pair library during a training phase;

[0056] The Monte Carlo sampling submodule can perform multiple forward propagations on the sentence pair data obtained by the second multilingual machine translation model using Monte Carlo sampling to obtain a multi-dimensional word list probability distribution matrix for each position of the translation;

[0057] The vocabulary probability distribution calculation submodule is connected to the output end of the Monte Carlo sampling submodule, and can calculate the sentence-level model uncertainty characteristics and sentence-level data uncertainty characteristics of the translation in the sentence pair data during the forward propagation process according to the multi-dimensional word vocabulary probability distribution matrix as the confidence in the translation process.

[0058] In the above system, the vocabulary probability distribution calculation submodule calculates the sentence-level model uncertainty features and sentence-level data uncertainty features of the translation in the sentence pair data during the forward propagation process according to the multi-dimensional word vocabulary probability distribution matrix in the following manner:

[0059] The model uncertainty characteristic for each position of the translation during decoding is calculated as follows: and sentence-level model uncertainty characteristics They are:

[0060]

[0061]

[0062] Where i represents the i-th position in the translation; j represents the j-th forward propagation process; represents the predicted probability center of the ith position in the N forward propagation processes; Euc_dist represents the Euclidean distance between the word list probability distribution and the predicted probability center in each forward propagation process, which serves as the prediction confidence of the model for the current decoding sequence under the constraints of the translation; represents the probability distribution of the i-th position in the translation during the j-th forward propagation process in the word list probability distribution matrix; T represents the length of the translation end sentence; t represents each word in the translation end sentence; represents the average of the model uncertainty characteristics at all positions of each word in the translation during decoding; Represents the average value of the model uncertainty characteristics of all positions of all words in the translation during decoding, that is, the sentence-level model uncertainty characteristics;

[0063] The data uncertainty characteristic of each position of the translation during decoding is calculated as follows and sentence-level data uncertainty characteristics They are:

[0064]

[0065]

[0066] The meanings of the symbols are as follows: i represents the i-th position in the translation; j represents the j-th forward propagation process; N represents the number of forward propagations; Entropy() represents the information entropy function; represents the probability distribution of the i-th position in the translation during the j-th forward propagation process in the word list probability distribution matrix; T represents the length of the translation end sentence; t represents each word in the translation end sentence; represents the average value of the uncertainty characteristics of the data at all positions of each word in the translation during decoding; It represents the average value of the data uncertainty characteristics of all positions of all words in the translation during the decoding process, that is, the sentence-level data uncertainty characteristics.

[0067] like Figure 4As shown, in the above system, the data selection module based on translation and quality assessment uncertainty includes:

[0068] Multilingual quality assessment model, confidence calculation submodule and data screening submodule; among them,

[0069] The multilingual quality assessment model can obtain sentence pair data from a multilingual quality assessment corpus, and use Monte Carlo sampling to perform multiple forward propagations on the obtained sentence pair data to obtain a multi-dimensional score list;

[0070] The confidence calculation submodule is connected to the output end of the multilingual quality assessment model, and can calculate the standard deviation of the multidimensional score list as the confidence in the quality assessment process of the latest multilingual quality assessment model, and obtain the comprehensive confidence according to the confidence in the quality assessment process and the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary;

[0071] The data screening submodule is connected to the output end of the confidence calculation submodule, and can screen out manual review data from the latest multilingual quality assessment model prediction score data according to the comprehensive confidence.

[0072] like Figure 5 As shown, the embodiment of the present invention also provides a multilingual translation data quality assessment method using the multilingual translation data quality assessment system, comprising the following steps:

[0073] Training phase:

[0074] Acquire multilingual sentence pair data from a multilingual high-quality text sentence pair library through the data enhancement module based on constraint decoding of the system, perform local constraint translation on the acquired sentence pair data based on a multilingual machine translation model to obtain forged sentence pair data, and use the obtained forged sentence pair data in a multi-task training framework to train a multilingual quality assessment model during a training phase;

[0075] Acquire sentence pair data from a multilingual high-quality text sentence pair library through the uncertainty feature construction module based on the translation vocabulary of the system, perform multiple forward propagations on the acquired sentence pair data using Monte Carlo sampling based on a multilingual machine translation model, and obtain sentence-level model uncertainty features and sentence-level data uncertainty features of the translation in the sentence pair data during the forward propagation process. The sentence-level model uncertainty features and sentence-level data uncertainty features are used in quality assessment training of a multilingual quality assessment model during the training phase;

[0076] Reasoning stage:

[0077] The uncertainty feature construction module based on the translation vocabulary of the system is used to calculate the sentence-level model uncertainty feature and sentence-level data uncertainty feature of the translation in the sentence pair data in the forward propagation process as the confidence in the translation process;

[0078] The confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary is received by the data selection module of the quality assessment confidence of the system, and the sentence pair data obtained is forward propagated multiple times by Monte Carlo sampling based on the multilingual quality assessment model to obtain a multidimensional score list, and the standard deviation of the multidimensional score list is calculated as the confidence in the quality assessment process of the latest multilingual quality assessment model, and the comprehensive confidence is obtained according to the confidence in the quality assessment process and the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary, and the manual review data is screened out from the predicted score data of the multilingual quality assessment model according to the comprehensive confidence.

[0079] In the above method, the acquired sentence pair data is locally constrained translated based on the multilingual machine translation model in the following manner to obtain forged sentence pair data, including: randomly selecting a preset percentage range of text from the translation of the sentence pair data output by the first multilingual machine translation model based on the multilingual machine translation model as a constraint part, and performing constraint decoding together with the source text of the sentence pair data through the first multilingual machine translation model to obtain first forged data; and randomly selecting a preset percentage range of text from the source text of the sentence pair data as a constraint part, and performing reverse translation together with the translation of the sentence pair data through the first multilingual translation model to obtain second forged data, and outputting the obtained first forged data and the second forged data together as forged data.

[0080] In the above method, the sentence pair data obtained by the multilingual machine translation model is forward propagated multiple times using Monte Carlo sampling to obtain the sentence-level model uncertainty features and sentence-level data uncertainty features of the translation in the sentence pair data during the forward propagation process, including:

[0081] The sentence pair data obtained by the second multilingual machine translation model is forward propagated multiple times using Monte Carlo sampling, and a multidimensional word list probability distribution matrix is ​​obtained for each decoded position. The sentence-level model uncertainty features and sentence-level data uncertainty features of the translation in the sentence pair data in the forward propagation process are calculated according to the multidimensional word list probability distribution matrix, including:

[0082] The model uncertainty characteristic for each position of the translation during decoding is calculated as follows: and sentence-level model uncertainty characteristics They are:

[0083]

[0084]

[0085] Where i represents the i-th position in the translation; j represents the j-th forward propagation process; represents the predicted probability center of the ith position in the N forward propagation processes; Euc_dist represents the Euclidean distance between the word list probability distribution and the predicted probability center in each forward propagation process, which serves as the prediction confidence of the model for the current decoding sequence under the constraints of the translation; represents the probability distribution of the i-th position in the translation during the j-th forward propagation process in the word list probability distribution matrix; T represents the length of the translation end sentence; t represents each word in the translation end sentence; represents the average of the model uncertainty characteristics at all positions of each word in the translation during decoding; Represents the average value of the model uncertainty characteristics of all positions of all words in the translation during decoding, that is, the sentence-level model uncertainty characteristics;

[0086] The data uncertainty characteristic of each position of the translation during decoding is calculated as follows and sentence-level data uncertainty characteristics They are:

[0087]

[0088]

[0089] The meanings of the symbols are as follows: i represents the i-th position in the translation; j represents the j-th forward propagation process; N represents the number of forward propagations; Entropy() represents the information entropy function; W j i represents the probability distribution of the i-th position in the translation during the j-th forward propagation process in the word list probability distribution matrix; T represents the length of the translation end sentence; t represents each word in the translation end sentence; represents the average value of the uncertainty characteristics of the data at all positions of each word in the translation during decoding; It represents the average value of the data uncertainty characteristics of all positions of all words in the translation during the decoding process, that is, the sentence-level data uncertainty characteristics.

[0090] In the above method, the data selection module for quality assessment confidence obtains the comprehensive confidence U according to the confidence in the quality assessment process and the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary in the following manner, including:

[0091]

[0092] Among them, the meaning of each parameter is as follows: W1 represents the proportion of the model uncertainty feature at the sentence level, and its initial value is 1; Represents the sentence-level model uncertainty feature; W2 represents the proportion of sentence-level data uncertainty features, and its initial value is 1; represents the data uncertainty feature at the sentence level; W3 represents the proportion of confidence in the quality assessment process of the latest multilingual quality assessment model, and its initial value is 1; σ represents the standard deviation of the multidimensional score list, which is used as the confidence in the quality assessment process of the latest multilingual quality assessment model.

[0093] The embodiment of the present invention further provides a processing device, including:

[0094] at least one memory for storing one or more programs;

[0095] At least one processor can execute one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor can implement the above method.

[0096] The embodiment of the present invention further provides a readable storage medium storing a computer program, which can implement the above method when the computer program is executed by a processor.

[0097] In summary, it can be seen that the multilingual translation data quality assessment system and method of the embodiment of the present invention, on the one hand, uses local constraint decoding of the translation model for data enhancement, so that a large amount of forged data before manual post-editing can be constructed unsupervisedly, and joint training is performed based on a multi-task learning framework and a quality assessment scoring task to alleviate the problem of scarcity of quality assessment annotation data; on the other hand, uncertainty features are constructed based on the vocabulary probability distribution of the translation model to alleviate the scarcity of annotation data, especially for the quality assessment problem of some zero-sample languages; while estimating the scores of multilingual sentence pairs, the confidence level of the current model evaluation is provided to the quality inspectors, and the confidence level comprehensively considers the uncertainty score estimation of the current sentence pair by the translation model and the quality assessment model, so as to provide the quality inspectors with a more reasonable data initial screening, thereby improving the efficiency and effect of the quality inspection by the quality inspectors, and well solving the problem of low collaborative efficiency and high labor cost between the quality inspectors and the quality assessment model.

[0098] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the multilingual translation data quality assessment system and method provided by the embodiments of the present invention are described in detail below with reference to specific embodiments.

[0099] Example 1

[0100] The embodiment of the present invention provides a multilingual translation data quality assessment system, which is based on a multi-task learning framework and an uncertainty estimation method. In the case of scarcity of multilingual high-quality parallel sentence pairs and scoring annotation data, the system can perform multilingual sentence pair effect assessment, which can solve the problem of lack of annotation data required for quality scoring in the current multilingual quality assessment system. Figure 1 As shown, the system includes: a multilingual quality assessment model, a data enhancement module based on constraint decoding, an uncertainty feature construction module based on a translation vocabulary, and a data selection module based on translation and quality assessment confidence; wherein,

[0101] (I) The composition of the data enhancement module based on local constraint decoding is as follows Figure 2 As shown, it includes: a first multilingual machine translation model and a local constraint decoding submodule; preferably, the first multilingual machine translation model adopts the latest multilingual machine translation model;

[0102] The specific processing method of the data enhancement module is as follows:

[0103] (11) Assume that there is a high-quality multilingual text sentence pair library, denoted by<x,y> ;

[0104] (12) The first multilingual machine translation model from<x,y> Randomly select sentence pairs <x l ,y l >, the local constraint decoding submodule targets high-quality sentence pairs <x l ,y l >, from the translation y l The first 30% to 70% of the text is randomly selected as the constraint part, which is consistent with the source text x l The data are sent to the first multilingual machine translation model for constraint decoding to obtain forged data. <x l ,y d > Similarly, from the source text x l The first 30% to 70% of the text is randomly selected as the constraint part, and reverse translated using the multilingual translation model to obtain fake data <x d ,y l >

[0105] (12) Repeat the above process until the high-quality multilingual corpus<x,y> After traversing completely, all the fake data are obtained.

[0106] The above-mentioned data enhancement module based on local constrained decoding realizes data enhancement of local constrained decoding by jointly training the task of classifying high-quality translations and translations with unstable machine translation quality, such as multiple translation omissions. Since the two tasks are strongly correlated, the bottom features of the model can be effectively shared, and at the same time, the overfitting problem caused by data scarcity can be alleviated, and the generalization ability of the trained model is improved. It avoids the problem that the correlation between the classification task goal and the multilingual quality assessment goal is reduced due to the uncontrollable style of the translation (the same source text can be expressed in multiple translations), thereby weakening the data enhancement effect. It solves the problems of scarce multilingual quality assessment scoring data, few data sources, and expensive labeling.

[0107] (II) The composition of the module based on the uncertainty feature of the probability distribution of the translation vocabulary is as follows Figure 3 As shown, it includes: a second multilingual machine translation model, a Monte Carlo sampling submodule and a vocabulary probability distribution calculation submodule; preferably, the second multilingual machine translation model adopts the latest multilingual machine translation model;

[0108] The specific processing method of the uncertainty feature construction module is as follows:

[0109] (21) Assume that there is a high-quality multilingual text sentence pair library, denoted by<x,y> , the second multilingual machine translation model θ, the vocabulary size is V;

[0110] (22) The second multilingual machine translation model sequentially extracts sentence pairs from the corpus <x i ,y i >, assuming the translation y i The length of is T. Based on the second multilingual machine translation model θ, Monte Carlo sampling is used to perform forward propagation N times. For each decoding position, an N×V-dimensional word list probability distribution matrix can be obtained, which is recorded as {W} t＝1..T ;

[0111] (23) The vocabulary probability distribution calculation submodule defines the uncertainty characteristics of the model for each position in the translation. and data uncertainty characteristics The formula is calculated to and The formula is as follows:

[0112]

[0113]

[0114] Where i represents the i-th position in the translation; j represents the j-th forward propagation process; It represents the predicted probability center of the ith position in the N forward propagation processes; Euc_dist represents the Euclidean distance between the word list probability distribution and the probability center in each forward process. The smaller the distance, the lower the model uncertainty, indicating that the model has greater confidence in predicting the current decoding sequence under the constraints of the translation; is the sentence-level model uncertainty feature;

[0115] Data uncertainty characteristics The formula is as follows:

[0116]

[0117]

[0118] in, Represents the data uncertainty feature at the sentence level; data uncertainty measures the uncertainty caused by data noise in the prediction process. The present invention uses average entropy to model the data uncertainty in the prediction probability distribution set. For each position prediction, if the distribution probability of each word in the vocabulary is more discrete, the entropy value is smaller, indicating that the data uncertainty is smaller;

[0119] (24) Introducing the model uncertainty characteristics and data uncertainty characteristics of each position mentioned above into the subsequent quality assessment training can effectively alleviate the quality assessment effect under few samples.

[0120] The uncertainty feature construction method of the translation model based on the vocabulary probability distribution of the present invention uses the latest multilingual translation model to perform score modeling on the current multilingual quality inspection data under different data and model disturbances, thereby improving the quality assessment effect; it avoids the existing scheme that often calculates the probability expectation, variance or text similarity features of each translated word in the prediction sequence based on Monte Carlo dropout, and ignores the vocabulary probability distribution information of each frame in the translation process; for example, when translating the yth i When a word is selected, assuming that the word is in the first place in the vocabulary and the word with the highest vocabulary probability is in the second place, then the vocabulary probability may exist in the following two situations: A: [0.4, 0.3, ...] and B: [0.4, 0.01, ...]. If only the probability of the first word is considered, the model will think that A and B are equally good, but from the perspective of vocabulary probability distribution, the model is more confident in choosing B than A. This solves the problem of how to introduce some glass-box artificially constructed translation features to assist training due to the scarcity of multilingual quality assessment annotation data, especially for quality assessment of some zero-shot languages.

[0121] (III) The composition of the data selection module based on translation and quality assessment uncertainty Figure 4As shown, it includes: a multilingual quality assessment model, a confidence calculation submodule and a data screening submodule; preferably, the multilingual quality assessment model adopts the latest multilingual quality assessment model;

[0122] The data selection module processes as follows:

[0123] (31) Assume that the corpus to be evaluated for multilingual quality is<x,y> , multilingual quality assessment model θ;

[0124] (32) The multilingual quality assessment model sequentially extracts sentence pairs from the corpus <x i ,y i >, based on the multilingual quality assessment model θ, Monte Carlo sampling is used to perform forward propagation N times, and an N-dimensional score list can be obtained, denoted as {S} t＝1..N ;

[0125] (33) Confidence calculation submodule calculates the score list {S} t＝1..N The standard deviation σ of is used as the confidence in the quality assessment process. The smaller the standard deviation score, the higher the confidence of the quality assessment model in the prediction.

[0126] (34) This embodiment comprehensively utilizes the confidence level in the translation process and The confidence σ in the quality assessment process is used to screen the data for manual re-inspection, which can avoid the problem of being too one-sided due to simply using the confidence of the quality assessment or translation model to measure whether the current data needs manual quality inspection. The confidence calculation submodule calculates the comprehensive confidence U according to the following comprehensive confidence calculation formula:

[0127]

[0128] (35) The data screening submodule sorts the model prediction score data according to the comprehensive confidence U, and selects a predetermined proportion P (the proportion can be freely set according to needs) for manual quality inspection; the above parameters W1, W2, and W3 can be initialized to 1 in the initial stage, and can be trained based on the data after manual quality inspection. The efficiency and effectiveness of quality inspectors can be improved through active learning. For example, the data after manual quality inspection can be used as labels, and the above parameters can be used as variables to simply train a fully connected neural network to determine the above parameters. In the inference stage, a random dropout mechanism is used to create disturbances to test the confidence of the multilingual quality assessment model in the current prediction.

[0129] The data selection module of the present invention performs preliminary screening of manual quality inspection data based on the confidence of the translation and quality assessment model, estimates the confidence of the quality assessment model, selects the data that the model cannot determine in combination with the translation confidence and the quality assessment confidence, and submits it to the quality inspector for manual evaluation, thereby improving the efficiency and effect of the entire system, avoiding the problem that the existing quality assessment model can only give the model a specific score for the current score estimate, but does not provide the quality inspector with the confidence of the current model evaluation. For example, assuming that the score range is 0-5 points, and the quality assessment model gives a score of 5 for a multilingual sentence pair, the quality inspector cannot know the degree of confidence of the model when scoring. Therefore, random sampling is often used during quality inspection, or different random sampling is performed on low-scoring data and high-scoring data. However, this method has a very low recall rate and efficiency for data with incorrect machine scoring.

[0130] In summary, the system and method of the embodiment of the present invention is based on the latest machine translation system, and improves the operating efficiency and reliability of the entire system by constraining translation data enhancement based on a multi-task learning framework and introducing uncertainty features in translation and quality assessment. Compared with the prior art, the advantages are as follows:

[0131] Advantage 1: By using the local constrained decoding technology of the translation model for data enhancement, a large amount of forged data before manual post-editing can be constructed unsupervisedly, and joint training is performed based on the multi-task learning framework and the quality assessment scoring task. Due to the correlation between the two tasks, this method can alleviate the overfitting problem caused by data scarcity and improve the generalization ability of the model.

[0132] Advantage 2: Uncertainty estimation is introduced simultaneously in the multilingual translation and quality assessment process. On the one hand, the multilingual translation model is used to construct uncertainty features based on the probability distribution of the vocabulary, thereby improving the effect of the multilingual quality assessment model under scarce data; on the other hand, the confidence scores of the translation model and the quality assessment model for the current sentence pair are comprehensively considered, so that the data that the model cannot determine is selected based on the confidence and handed over to the quality inspectors for manual evaluation, thereby improving the efficiency and effectiveness of the entire system.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0134] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A multilingual translation data quality assessment system, comprising: The multilingual quality assessment model is characterized by further comprising: A data enhancement module based on constraint decoding, an uncertainty feature construction module based on translation vocabulary, and a data selection module for quality assessment confidence; The data enhancement module based on constraint decoding can obtain sentence pair data from a multilingual high-quality text sentence pair library, and perform local constraint translation on the obtained sentence pair data based on a multilingual machine translation model to obtain forged sentence pair data. The obtained forged sentence pair data is used in the training of a multilingual quality assessment model in a multi-task training framework during the training phase; The uncertainty feature construction module based on the translation vocabulary can obtain sentence pair data from a multilingual high-quality text sentence pair library, and perform multiple forward propagations on the obtained sentence pair data using Monte Carlo sampling based on a multilingual machine translation model to obtain sentence-level model uncertainty features and sentence-level data uncertainty features of the translation in the sentence pair data during the forward propagation process, and the sentence-level model uncertainty features and sentence-level data uncertainty features are used in the quality assessment training of the multilingual quality assessment model in the training phase; and as confidence in the translation process in the reasoning phase; The data selection module for the quality assessment confidence can receive the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary, and perform multiple forward propagations on the acquired sentence pair data using Monte Carlo sampling based on the multilingual quality assessment model to obtain a multidimensional score list, calculate the standard deviation of the multidimensional score list as the confidence in the quality assessment process of the latest multilingual quality assessment model, obtain a comprehensive confidence based on the confidence in the quality assessment process and the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary, and screen out manual review data from the multilingual quality assessment model prediction score data based on the comprehensive confidence.

2. The multilingual translation data quality assessment system according to claim 1, characterized in that: The data enhancement module based on constraint decoding includes: A first multilingual machine translation model and a local constraint decoding submodule; wherein; The first multilingual machine translation model has an input end connected to a multilingual high-quality text sentence pair library, and an output end connected to an input end of a local constraint decoding submodule, and can obtain sentence pair data from the multilingual high-quality text sentence pair library; The local constraint decoding submodule is respectively provided with a constraint decoding output terminal and a forged data output terminal. The constraint decoding output terminal can randomly select a text within a preset percentage range from the translation of the sentence pair data as a constraint part, and perform constraint decoding together with the source text of the sentence pair data through the first multilingual machine translation model to obtain first forged data; and randomly select a text within a preset percentage range from the source text of the sentence pair data as a constraint part, and perform reverse translation together with the translation of the sentence pair data through the first multilingual translation model to obtain second forged data, and output the obtained first forged data and the second forged data together as forged data.

3. The multilingual translation data quality assessment system according to claim 2, characterized in that: The local constraint decoding submodule randomly selects the first 30% to 70% of the text from the translation of the sentence pair data output by the first multilingual machine translation model as the constraint part; The local constraint decoding submodule randomly selects the first 30% to 70% of the text from the source text of the clause pair data as the constraint part.

4. The multilingual translation data quality assessment system according to any one of claims 1 to 3, characterized in that: The uncertainty feature construction module based on the translation vocabulary includes: The second multilingual machine translation model, the Monte Carlo sampling submodule and the vocabulary probability distribution calculation submodule; wherein, The second multilingual machine translation model has an input end connected to a multilingual high-quality text sentence pair library, and an output end connected to an input end of a Monte Carlo sampling submodule, and can obtain sentence pair data from the multilingual high-quality text sentence pair library during a training phase; The Monte Carlo sampling submodule can perform multiple forward propagations on the sentence pair data obtained by the second multilingual machine translation model using Monte Carlo sampling to obtain a multi-dimensional word list probability distribution matrix for each position of the translation; The vocabulary probability distribution calculation submodule is connected to the output end of the Monte Carlo sampling submodule, and can calculate the sentence-level model uncertainty characteristics and sentence-level data uncertainty characteristics of the translation in the sentence pair data during the forward propagation process according to the multi-dimensional word vocabulary probability distribution matrix as the confidence in the translation process.

5. The multilingual translation data quality assessment system according to claim 1 or 2, characterized in that: The vocabulary probability distribution calculation submodule calculates the sentence-level model uncertainty features and sentence-level data uncertainty features of the translation in the sentence pair data during the forward propagation process according to the multi-dimensional word vocabulary probability distribution matrix in the following manner: The model uncertainty characteristic of each position of the translation is calculated as follows: and sentence-level model uncertainty characteristics They are: Where i represents the i-th position in the translation; j represents the j-th forward propagation process; represents the predicted probability center of the ith position in the N forward propagation processes; Euc_dist represents the Euclidean distance between the word list probability distribution and the predicted probability center in each forward propagation process, which is the prediction confidence of the model for the current decoding sequence under the constraints of the translation; W j i represents the element at the ith position in the translation during the jth forward propagation in the probability distribution matrix of the word list; T represents the length of the sentence on the translation side; t represents each word in the sentence on the translation side; represents the average of the model uncertainty characteristics at all positions of each word in the translation during decoding; Represents the average value of the model uncertainty characteristics of all positions of all words in the translation during decoding, that is, the sentence-level model uncertainty characteristics; The data uncertainty characteristic of each position in the translation is calculated as follows and sentence-level data uncertainty characteristics They are: The meanings of the symbols are as follows: i represents the i-th position in the translation; j represents the j-th forward propagation process; N represents the number of forward propagations; Entropy() represents the information entropy function; W j i represents the probability distribution of the i-th position in the translation during the j-th forward propagation process in the word list probability distribution matrix; T represents the length of the translation end sentence; t represents each word in the translation end sentence; represents the average value of the uncertainty characteristics of the data at all positions of each word in the translation during decoding; It represents the average value of the data uncertainty characteristics of all positions of all words in the translation during the decoding process, that is, the sentence-level data uncertainty characteristics.

6. The multilingual translation data quality assessment system according to claim 1 or 2, characterized in that: The data selection module based on translation and quality assessment uncertainty includes: Multilingual quality assessment model, confidence calculation submodule and data screening submodule; among them, The multilingual quality assessment model can obtain sentence pair data from a multilingual quality assessment corpus, and use Monte Carlo sampling to perform multiple forward propagations on the obtained sentence pair data to obtain a multi-dimensional score list; The confidence calculation submodule is connected to the output end of the multilingual quality assessment model, and can calculate the standard deviation of the multidimensional score list as the confidence in the quality assessment process of the latest multilingual quality assessment model, and obtain the comprehensive confidence according to the confidence in the quality assessment process and the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary; The data screening submodule is connected to the output end of the confidence calculation submodule, and can screen out manual review data from the latest multilingual quality assessment model prediction score data according to the comprehensive confidence.

7. A method for evaluating the quality of multilingual translation data using the multilingual translation data quality evaluation system according to any one of claims 1 to 6, characterized in that: The steps include: Training phase: Acquire multilingual sentence pair data from a multilingual high-quality text sentence pair library through the data enhancement module based on constraint decoding of the system, perform local constraint translation on the acquired sentence pair data based on a multilingual machine translation model to obtain forged sentence pair data, and use the obtained forged sentence pair data in a multi-task training framework to train a multilingual quality assessment model during a training phase; Acquire sentence pair data from a multilingual high-quality text sentence pair library through the uncertainty feature construction module based on the translation vocabulary of the system, perform multiple forward propagations on the acquired sentence pair data using Monte Carlo sampling based on a multilingual machine translation model, and obtain sentence-level model uncertainty features and sentence-level data uncertainty features of the translation in the sentence pair data during the forward propagation process. The sentence-level model uncertainty features and sentence-level data uncertainty features are used in quality assessment training of a multilingual quality assessment model during the training phase; Reasoning stage: The uncertainty feature construction module based on the translation vocabulary of the system is used to calculate the sentence-level model uncertainty feature and sentence-level data uncertainty feature of the translation in the sentence pair data in the forward propagation process as the confidence in the translation process; The confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary is received by the data selection module of the quality assessment confidence of the system, and the sentence pair data obtained is forward propagated multiple times by Monte Carlo sampling based on the multilingual quality assessment model to obtain a multidimensional score list, and the standard deviation of the multidimensional score list is calculated as the confidence in the quality assessment process of the latest multilingual quality assessment model, and the comprehensive confidence is obtained according to the confidence in the quality assessment process and the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary, and the manual review data is screened out from the predicted score data of the multilingual quality assessment model according to the comprehensive confidence.

8. The multilingual translation data quality assessment method according to claim 7, characterized in that: In the method, the data enhancement module based on constraint decoding performs local constraint translation on the acquired sentence pair data based on a multilingual machine translation model to obtain forged sentence pair data in the following manner, including: randomly selecting a preset percentage range of text from the translated text of the sentence pair data output by the first multilingual machine translation model of the data enhancement module based on constraint decoding as a constraint part, and performing constraint decoding together with the source text of the sentence pair data through the first multilingual machine translation model to obtain first forged data; and randomly selecting a preset percentage range of text from the source text of the sentence pair data as a constraint part, and performing reverse translation together with the translated text of the sentence pair data through the first multilingual translation model to obtain second forged data, and outputting the obtained first forged data and the second forged data together as forged data; In the method, the uncertainty feature construction module based on the translation vocabulary performs multiple forward propagations on the acquired sentence pair data using Monte Carlo sampling based on the multilingual machine translation model in the following manner, and obtains the sentence-level model uncertainty features and sentence-level data uncertainty features of the translation in the sentence pair data during the forward propagation process, including: The sentence pair data obtained by the second multilingual machine translation model based on the translation vocabulary uncertainty feature construction module is forward propagated multiple times using Monte Carlo sampling, and a multidimensional word vocabulary probability distribution matrix is ​​obtained for each decoded position. The sentence-level model uncertainty features and sentence-level data uncertainty features of the translation in the sentence pair data in the forward propagation process are calculated according to the multidimensional word vocabulary probability distribution matrix, including: The model uncertainty characteristic of each position of the translation is calculated as follows: and sentence-level model uncertainty characteristics They are: Where i represents the i-th position in the translation; j represents the j-th forward propagation process; represents the predicted probability center of the ith position in the N forward propagation processes; Euc_dist represents the Euclidean distance between the word list probability distribution and the predicted probability center in each forward propagation process, which is the prediction confidence of the model for the current decoding sequence under the constraints of the translation; W j i represents the element at the ith position in the translation during the jth forward propagation in the probability distribution matrix of the word list; T represents the length of the sentence on the translation side; t represents each word in the sentence on the translation side; represents the average of the model uncertainty characteristics at all positions of each word in the translation during decoding; Represents the average value of the model uncertainty characteristics of all positions of all words in the translation during decoding, that is, the sentence-level model uncertainty characteristics; The data uncertainty characteristic of each position in the translation is calculated as follows and sentence-level data uncertainty characteristics They are: The meanings of the symbols are as follows: i represents the i-th position in the translation; j represents the j-th forward propagation process; N represents the number of forward propagations; Entropy() represents the information entropy function; W j i represents the probability distribution of the i-th position in the translation during the j-th forward propagation process in the word list probability distribution matrix; T represents the length of the translation end sentence; t represents each word in the translation end sentence; represents the average value of the uncertainty characteristics of the data at all positions of each word in the translation during decoding; Represents the average value of the data uncertainty characteristics of all positions of all words in the translation during decoding, that is, the sentence-level data uncertainty characteristics In the method, the data selection module for quality assessment confidence obtains the comprehensive confidence U according to the confidence in the quality assessment process and the confidence in the translation process output by the uncertainty feature construction module based on the translation vocabulary in the following manner, including: Among them, the meaning of each parameter is as follows: W1 represents the proportion of the model uncertainty feature at the sentence level, and its initial value is 1; Represents the sentence-level model uncertainty feature; W2 represents the proportion of sentence-level data uncertainty features, and its initial value is 1; represents the uncertainty feature of sentence-level data; W3 represents the proportion of confidence in the quality assessment process of the latest multilingual quality assessment model, and its initial value is 1; σ represents the standard deviation of the multidimensional score list, which is used as the confidence in the quality assessment process of the latest multilingual quality assessment model.

9. A processing device, characterized in that: include: at least one memory for storing one or more programs; At least one processor can execute one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor can implement the method described in any one of claims 7-8.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method described in any one of claims 7 to 8 can be implemented.

Images