A method for unsupervised multilingual translation quality assessment without reference translations

Abstract

The application discloses a kind of unsupervised multilingual translation quality evaluation methods not depending on reference translation, it is related to translation quality evaluation technical field, comprising the following steps: S1: calculating accuracy score;S2: calculating fluency score;S3: calculating translation quality score, this unsupervised multilingual translation quality evaluation method not depending on reference translation, unlike prior art, the translation accuracy score of original text and translation is calculated using multilingual pre-training model, the generation probability of translation is calculated using language model, and is mapped as translation fluency score, compared with the two methods of depending on artificial labeling and depending on reference translation, the present method does not depend on artificial labeling data and reference translation, is suitable for multilingual translation direction scene, maintenance cost is low, and index is strong in interpretability.

Description

Technical Field

[0001] This invention relates to the field of translation quality assessment technology, specifically to an unsupervised multilingual translation quality assessment method that does not rely on reference translations. Background Technology

[0002] Existing translation quality assessment methods mainly fall into two categories: The first relies on manual annotation to score the translation quality, using annotated data to train a scoring model. The input is the source text and the translation, and the output is a quality assessment score. This method is costly, and the more languages ​​translated, the higher the requirements for data scale and quality. The computational and maintenance costs are also high, making it difficult to meet the needs of multilingual translation scenarios. The second relies on reference translations, using metrics such as BLEU and METEOR to calculate the difference between the translation and the reference translation. This method is not applicable to real-world scenarios, as most translation scenarios lack reference translations, making it impossible to calculate metrics like BLEU. Therefore, we propose an unsupervised multilingual translation quality assessment method that does not rely on reference translations. Summary of the Invention

[0003] The purpose of this invention is to provide an unsupervised multilingual translation quality assessment method that does not rely on reference translations, in order to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: an unsupervised multilingual translation quality assessment method that does not rely on reference translations, comprising the following steps:

[0005] S1: Calculate the accuracy score;

[0006] S2: Calculate the fluency score;

[0007] S3: Calculate the translation quality score;

[0008] In step S1, the operation steps are as follows:

[0009] S11: Enter the original text and translation;

[0010] S12: Segment the original text and translation;

[0011] S13: Input the word segmentation results into the multilingual pre-training model and output the original word vectors and the translated word vectors;

[0012] S14: Perform SVD decomposition on the original text word vectors and the translated text word vectors to obtain the common components of all word vectors in the original text and the translated text;

[0013] S15: Perform two operations: remove common components from all word vectors in the original text or translation, or do not remove them.

[0014] S16: Sentence vectors without removing principal components are generated through mean pooling, while sentence vectors with removed principal components are generated by repeating the above pooling operation.

[0015] S17: Calculate the cosine similarity between the original sentence vector and the translated sentence vector, and then convert the cosine similarity into an accuracy score out of 100.

[0016] S18: Calculate the accuracy score;

[0017] In step S2, the operation steps are as follows:

[0018] S21: Enter the translation;

[0019] S22: Segment the translation into words and obtain m translation phrases;

[0020] S23: Input the first m-1 translated word groups into the autoregressive language model in sequence to obtain the conditional probability p(w) of the m-th translated word group. m |w1,w2,...,w m-1 );

[0021] S24: Multiply the conditional probabilities of the m translation phrases to obtain the translation generation probability p(tgt);

[0022] S25: Map p(tgt) to 0-1 using the sigmoid function, which is a 100-point scale, where α is a hyperparameter with a value of 10.

[0023] S26: Calculate the fluency score.

[0024] In step S3, the operation steps are as follows:

[0025] S31: Calculate the weighted sum of the translation accuracy score and the fluency score;

[0026] S32: Calculate the translation quality score.

[0027] Preferably, the decomposition formula in step S14 and the elimination formula in step S15 are expressed as follows:

[0028]

[0029]

[0030] Among them, A m*n It is the vector set of all phrases in the original or translated text, where m is the number of phrases in the original or translated text, determined by the number of phrases after word segmentation, and n is the dimension of the word vector, typically 768. m*k It is an m x k ​​left singular matrix, ∑ k*k V is a k x k singular value matrix.k*n It is a k x n right singular matrix, representing the common components of the first k n-dimensional word vectors, where k is the number of components, and in this project, it is set to 1. The calculation result is A m*n The common component matrix in the matrix, which is the same as matrix A. m*n Both are m rows and n columns, therefore matrix A m*n Subtracting the common component matrix from the common component matrix yields a new matrix A'. m*n A' m*n It is the updated set of word vectors of the original text or translation after removing k principal components.

[0031] Preferably, the calculation formula for step S16 is expressed as follows:

[0032]

[0033]

[0034] Where S represents the sentence vector without removing principal components, and S' represents the sentence vector with removing principal components.

[0035] Preferably, the calculation formula in step S17 is expressed as follows:

[0036]

[0037]

[0038] Where accuracy represents the accuracy score of the original sentence vector and the translated sentence vector without removing principal components, S src S represents the vector of the original sentence without removing principal components. tgt ||S represents the vector of the translated sentence without removing principal components. src || represents the magnitude of the original sentence vector without removing the principal components, ||S tgt || represents the modulus of the translated sentence vector without removing principal components. accuracy' represents the accuracy score of the original sentence vector and the translated sentence vector after removing principal components, S' src S' represents the vector of the original sentence after removing the principal components. tgt ||S' represents the vector of the translated sentence after removing the principal components. src || represents the magnitude of the original sentence vector after removing the principal components, ||S' tgt || represents the modulus of the translated sentence vector after removing the principal components.

[0039] Preferably, the formula for calculating the translation generation probability p(tgt) in step S23 is as follows:

[0040] p(tgt)=p(w1)p(w2|w1)p(w3|w1,w2)...p(w m |w1,w2,…,w m-1 ).

[0041] Preferably, the formula for calculating the fluency score in step S26 is as follows:

[0042] fluency=sigmoid(-m / log(p(tgt))*α)*100.

[0043] Preferably, the formula for calculating the translation quality score in step S32 is expressed as follows:

[0044] score=0.1*accuracy+0.2*accuracy'+0.7*fluency

[0045] Here, score refers to the translation quality score, accuracy is the accuracy score without removing principal components, with a weight of 0.1, accuracy' refers to the accuracy score after removing principal components, with a weight of 0.2, and fluency is the fluency score, with a weight of 0.7.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] This method utilizes a multilingual pre-trained model to calculate the translation accuracy scores of the source and target texts, and uses a language model to calculate the generation probability of the target text, which is then mapped to the fluency score of the target text. Compared with the two methods that rely on manual annotation and reference translations, this method does not rely on manually annotated data or reference translations, making it suitable for multilingual translation scenarios. It also has low maintenance costs and strong interpretability of the metrics. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the calculation of the translation quality score in this invention;

[0049] Figure 2 This is a flowchart of the accuracy score calculation for this invention;

[0050] Figure 3 This is a flowchart of the fluency score calculation for this invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Please see Figure 1-3 This invention provides a technical solution: an unsupervised multilingual translation quality assessment method that does not rely on reference translations, comprising the following steps:

[0053] S1: Calculate the accuracy score;

[0054] S2: Calculate the fluency score;

[0055] S3: Calculate the translation quality score;

[0056] In step S1, the operation steps are as follows:

[0057] S11: Enter the original text and translation;

[0058] S12: Segment the original text and translation;

[0059] S13: Input the word segmentation results into the multilingual pre-training model and output the original word vectors and the translated word vectors;

[0060] S14: Perform SVD decomposition on the original text word vectors and the translated text word vectors to obtain the common components of all word vectors in the original text and the translated text;

[0061] S15: Perform two operations: remove common components from all word vectors in the original text or translation, or do not remove them.

[0062] S16: Sentence vectors without removing principal components are generated through mean pooling, while sentence vectors with removed principal components are generated by repeating the above pooling operation.

[0063] S17: Calculate the cosine similarity between the original sentence vector and the translated sentence vector, and then convert the cosine similarity into an accuracy score out of 100.

[0064] S18: Calculate the accuracy score;

[0065] In step S2, the operation steps are as follows:

[0066] S21: Enter the translation;

[0067] S22: Segment the translation into words and obtain m translation phrases;

[0068] S23: Input the first m-1 translated word groups into the autoregressive language model in sequence to obtain the conditional probability p(w) of the m-th translated word group. m |w1,w2,...,w m-1 );

[0069] S24: Multiply the conditional probabilities of the m translation phrases to obtain the translation generation probability p(tgt);

[0070] S25: Map p(tgt) to 0-1 using the sigmoid function, which is a 100-point scale, where α is a hyperparameter with a value of 10.

[0071] S26: Calculate the fluency score.

[0072] In step S3, the operation steps are as follows:

[0073] S31: Calculate the weighted sum of the translation accuracy score and the fluency score;

[0074] S32: Calculate the translation quality score.

[0075] Furthermore, the decomposition formula in step S14 and the elimination formula in step S15 are expressed as follows:

[0076]

[0077]

[0078] Among them, A m*n It is the vector set of all phrases in the original or translated text, where m is the number of phrases in the original or translated text, determined by the number of phrases after word segmentation, and n is the dimension of the word vector, typically 768. m*k It is an m x k ​​left singular matrix, ∑ k*k V is a k x k singular value matrix. k*n It is a k x n right singular matrix, representing the common components of the first k n-dimensional word vectors, where k is the number of components, and in this project, it is set to 1. The calculation result is A m*n The common component matrix in the matrix, which is the same as matrix A. m*n Both are m rows and n columns, therefore matrix A m*n Subtracting the common component matrix from the common component matrix yields a new matrix A'. m*n A' m*n It is the updated set of word vectors of the original text or translation after removing k principal components.

[0079] Furthermore, the calculation formula for step S16 is expressed as follows:

[0080]

[0081]

[0082] Where S represents the sentence vector without removing principal components, and S' represents the sentence vector with removing principal components.

[0083] Furthermore, the calculation formula in step S17 is expressed as follows:

[0084]

[0085]

[0086] Where accuracy represents the accuracy score of the original sentence vector and the translated sentence vector without removing principal components, S src S represents the vector of the original sentence without removing principal components. tgt ||S represents the vector of the translated sentence without removing principal components. src || represents the magnitude of the original sentence vector without removing the principal components, ||S tgt || represents the modulus of the translated sentence vector without removing principal components. accuracy' represents the accuracy score of the original sentence vector and the translated sentence vector after removing principal components, S' src S' represents the vector of the original sentence after removing the principal components. tgt ||S' represents the vector of the translated sentence after removing the principal components. src || represents the magnitude of the original sentence vector after removing the principal components, ||S' tgt || represents the modulus of the translated sentence vector after removing the principal components.

[0087] Furthermore, the formula for calculating the translation generation probability p(tgt) in step S23 is as follows:

[0088] p(tgt)=p(w1)p(w2|w1)p(w3|w1,w2)...p(w m |w1,w2,…,w m-1 ).

[0089] Furthermore, the formula for calculating the fluency score in step S26 is as follows:

[0090] fluency=sigmoid(-m / log(p(tgt))*α)*100.

[0091] Furthermore, the formula for calculating the translation quality score in step S32 is expressed as follows:

[0092] score=0.1*accuracy+0.2*accuracy'+0.7*fluency

[0093] Here, score refers to the translation quality score, accuracy is the accuracy score without removing principal components, with a weight of 0.1, accuracy' refers to the accuracy score after removing principal components, with a weight of 0.2, and fluency is the fluency score, with a weight of 0.7.

[0094] The final translation quality assessment results are presented in the following examples and Table 1.

[0095] original:

[0096] Economic construction departments under the People's Governments at various levels should include the prevention of air pollution in their production and construction plans in accordance with the requirements for atmospheric environment protection as set forth by the respective people'sGovernments, and have it put into effect.

[0097] Translation A:

[0098] The economic construction departments of people's governments at all levels shall, in accordance with the requirements for atmospheric environmental protection put forward by the people's governments, incorporate the prevention and control of air pollution into their production and construction plans and implement them.

[0099] Translation B:

[0100] The economic construction departments at all levels of the people's government shall, in accordance with the atmospheric environmental protection requirements stipulated by their respective people's governments, incorporate the prevention of air pollution into their production and construction plans and implement them.

[0101] Translation C:

[0102] Air pollution prevention and control work should be incorporated into production and construction plans and implemented accordingly.

[0103] Table 1. Translation quality assessment scores for translations A, B, and C.

[0104] accuracy accuracy flu score Translation A 81.62 52.34 87.39 82.73 Translation B 82.88 41.51 87.34 81.87 Translation C 68.87 25.52 87.56 77.61

[0105] Taking Chinese-English translation as an example, the original text is an English sentence, and the Chinese translations A and B are from two different machine translation engines. In the human evaluation (12 people), translation A (12 votes) is significantly better than translation B (0 votes). The Chinese translation C is the machine translation after manual deletion (simulating a scenario of omission). After using the evaluation index system of this method, the results are shown in Table 1. Translation A has the highest total score. Translation B is more inclined to "literal translation", so its fluency score is lower than that of translation A. Translation C has omissions. Although it has the highest fluency score, its two translation accuracy index scores are far lower than those of translations A and B.

[0106] In summary, the accuracy score measures the degree of matching between the source and translation texts. The higher the semantic similarity between the source and translation sentence vectors, the better the match. The translation accuracy score is calculated by inputting the source and translation texts into a multilingual pre-trained model to obtain source and translation word vectors. Then, source and translation sentence vectors are calculated using SVD decomposition and mean pooling operations. The cosine similarity between the source and translation sentence vectors is then calculated and converted into a 100-point accuracy score. The fluency score refers to the smoothness of the translation's word choice and sentence structure. Some literal translations may sound machine-translated and not conform to native speaker expression habits; therefore, fluency is one of the indicators of translation quality. The fluency score is calculated by inputting the translation phrases into an autoregressive language model to calculate the generation probability of the translated text, mapping it to a 100-point score. Finally, the translation accuracy score and fluency score are weighted and summed to obtain the translation quality score.

[0107] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0108] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for unsupervised multilingual translation quality estimation without reference translation, characterized in that, The method comprises the following steps: S1: calculating an accuracy score; S2: calculating a fluency score; S3: calculating a translation quality score; In step S1, the operation steps are as follows: S11: inputting the original text and the translation; S12: performing word segmentation on the original text and the translation; S13: inputting the word segmentation result into a multilingual pre-training model to output original text word vectors and translation word vectors; S14: performing SVD decomposition on the original text word vector set and the translation word vector set to obtain common components of all original text word vectors and all translation word vectors; S15: performing two operations of eliminating and not eliminating the common components of all original text word vectors or all translation word vectors; S16: generating a sentence vector without eliminating the principal component through mean pooling operation, and generating a sentence vector by repeating the above pooling operation; S17: calculating the cosine similarity between the original text sentence vector and the translation sentence vector, and then converting the cosine similarity into an accuracy score of 100 points; S18: calculating the accuracy score; In step S2, the operation steps are as follows: S21: inputting the translation; S22: performing word segmentation on the translation to obtain m translation word groups; S23: inputting the first m-1 translation word groups into the autoregressive language model in sequence to obtain a conditional probability p(w m |w1,w2,...,w m-1 ) of the mth translation word group w S24: multiplying the conditional probability of the m translation word groups to obtain a translation generation probability p(tgt); S25: mapping p(tgt) to 0-1 through a sigmoid function, and mapping it into a score of 100 points, where α is a hyperparameter and takes a value of 10; S26: calculating the fluency score; In step S3, the operation steps are as follows: S31: performing weighted summation on the translation accuracy score and the fluency score; S32: calculating the translation quality score.

2. The method of claim 1, wherein, The decomposition formula of step S14 and the elimination formula of step S15 are as follows: where A m*n is the vector set of all phrases in the original or translation, m is the number of phrases in the original or translation, which is determined by the number of phrases after segmentation, n is the dimension of the word vector, generally taking the value of 768, U m*k is an m-row k-column left singular matrix, ∑ k*k is a k-row k-column singular value matrix, V k*n is a k-row n-column right singular matrix, representing the common components of all n-dimensional word vectors in the first k, and k is the number of components, which is 1 in this project, The calculation result of A m*n is the common component matrix in A m*n , which is m-row n-column. Therefore, matrix A m*n is subtracted from the common component matrix to finally obtain a new matrix A′ m*n , and A′ m*n is the updated vector set of the original or translation after removing the k principal components.

3. The method of claim 1, wherein, The calculation formula of step S16 is as follows: Wherein, S represents a sentence vector without eliminating the principal component, and S' represents a sentence vector eliminating the principal component.

4. The method of claim 1, wherein, The calculation formula in step S17 is as follows: wherein accuracy represents the accuracy score of the original sentence vector and the translated sentence vector without removing the principal component, S src represents the original sentence vector without removing the principal component, S tgt represents the translated sentence vector without removing the principal component, ||S src || represents the norm of the original sentence vector without removing the principal component, ||S tgt || represents the norm of the translated sentence vector without removing the principal component; accuracy' represents the accuracy score of the original sentence vector and the translated sentence vector with removing the principal component, S' src represents the original sentence vector with removing the principal component, S' tgt represents the translated sentence vector with removing the principal component, ||S' src || represents the norm of the original sentence vector with removing the principal component, ||S' tgt || represents the norm of the translated sentence vector with removing the principal component.

5. The method of claim 1, wherein, The calculation formula of the translation generation probability p(tgt) in step S23 is as follows: p(tgt) = p(w1)p(w2|w1)p(w3|w1,w2)...p(w m |w1,w2,...,w m-1 ).

6. The method of claim 1, wherein, The calculation formula of the fluency score in step S26 is as follows: fluency=sigmoid(-m / log(p(tgt))*α)*100.

7. The method of claim 1, wherein, The calculation formula of the translation quality score in step S32 is as follows: score=0.1*accuracy+0.2*accuracy′+0.7*fluency Wherein, score refers to the translation quality score, accuracy is the accuracy score without eliminating the principal component, the weight is 0.1, accuracy' refers to the accuracy score eliminating the principal component, the weight is 0.2, and fluency is the fluency score, the weight is 0.7.

Images