Sentence Granularity Transformation Testing Method for Neural Machine Translation System

Abstract

The present invention is aimed at the translation robustness evaluation problem existing in the neural machine translation system, and discloses a sentence granularity metamorphosis test method for the neural machine translation system, which belongs to the field of metamorphosis test. The method first performs data preprocessing operations on two groups of texts that need to be similarity calculated to generate two groups of word lists; then, the two groups of word lists are constructed with a word bag model based on TF-IDF, and the angle cosine value of the two groups of text vectors is calculated; at the same time, the Jaccard similarity coefficient and edit distance of the two groups of word lists are calculated; finally, the similarity between sentences and the metamorphosis relationship satisfaction rate are calculated according to the defined similarity calculation formula and metamorphosis relationship. The present invention improves the difficulty of a single cosine similarity method in reflecting the semantic changes brought about by different word orders of sentences, strengthens the ability to distinguish the semantic changes brought about by word order changes in sentences, and improves the accuracy of sentence similarity calculation and translation quality evaluation.

Description

Technical Field

[0001] The invention belongs to the field of metamorphic testing, and specifically relates to the problem of calculating sentence granularity similarity during a metamorphic testing process of a neural machine translation system. A sentence granularity metamorphic testing method for a neural machine translation system is proposed. Background Art

[0002] With the development of deep learning, machine translation based on neural networks has made great progress. However, the neural network model used in the neural machine translation system lacks interpretability and comprehensibility, which leads to the problem of test prediction in the testing process of the neural machine translation system. Metamorphosis testing is a test method proposed to solve the problem of test prediction. Therefore, metamorphosis testing is also one of the main methods for testing neural machine translation systems. In the metamorphosis test of the neural machine translation system, the metamorphosis relationship of sentence granularity depends on the accuracy of the sentence similarity calculation results. Its accuracy affects the metamorphosis relationship satisfaction rate, which in turn leads to deviations in the evaluation results of the neural machine translation system. Therefore, it is of great practical and research significance to improve the accuracy of sentence granularity similarity calculation and evaluate the translation robustness of the neural machine translation system.

[0003] Although the similarity calculation method based on cosine distance is well constructed, it ignores the original word order, grammar and semantics of the text. The present invention uses TF-IDF text vectorization, adds a weight to each inter-class information, and then introduces a measurement factor that can reflect the commonality and difference of symbol strings to transform the single cosine similarity calculation method, thereby improving the accuracy of sentence similarity calculation and strengthening the ability to distinguish the semantic changes caused by word order changes in sentences, thereby improving the accuracy of the metamorphosis test in evaluating the translation robustness of the neural machine translation system. Summary of the invention

[0004] The sentence granularity transformation test method for neural machine translation system is, firstly, based on the TF-IDF-based bag-of-words model text vectorization, by improving the word frequency and adding a weight to reduce the role of secondary feature items and enhance the role of primary feature items. Secondly, the Jaccard coefficient can reflect the commonality between sentences, and the edit distance can reflect the difference between sentences. The two are combined to make up for the shortcoming that the cosine distance is difficult to capture the semantic changes of sentences due to the change of word order.

[0005] The present invention is a sentence granularity degradation test method for a neural machine translation system, which is characterized by comprising the following steps:

[0006] Step 1: Obtain bilingual text data in Chinese and English from the corpus and generate test cases based on the metamorphosis relationship. Specifically, input the English data in each field into the neural network-based translation system for three round-trip translations: First, convert the source English sentence S en Input into the translation system to generate Chinese result S zh , and then S zh Input to the translation system to generate English sentence S' en , and finally S' en Enter the translation system again to generate the Chinese translation result S' zh ;

[0007] Step 2: Perform data preprocessing operations on the source Chinese and English data of the dataset and the generated test cases: word segmentation, deletion of stop words and punctuation marks. Because the structures of Chinese and English are different, English sentences also need to be converted to lowercase letters and word form restoration to obtain the word list W;

[0008] Step 3: The two sentences to be similarity calculated are recorded as S1 and S2, and the sentences after data preprocessing form a word list W S1 and W S2 , through the bag-of-words model based on TF-IDF, W S1 and W S2 Expressed as a vector V S1 and V S2 , the traditional calculation is to regard the weights of all feature words as the same, but for data sets with different domain data, it is necessary to reduce the role of secondary feature items, enhance the role of primary feature items, improve the word frequency, and add a weight. The improved calculation formula is as follows:

[0009] tf-idf(t,d)=tf t,d ·idf t ·c i

[0010]

[0011] where tf t,d idf is the traditional TF-IDF algorithm calculation process, maxλ n,i Indicates that the i-th feature word appears most frequently in the n-type text data, ∑λ n Indicates the total number of words in n types of text;

[0012] Step 4: Calculate V S1 and V S2 The cosine value of the angle cos(V S1 ,V S2 ), calculate the Jaccard coefficient J(W S1 ,W S2), the calculation formulas for cosine similarity and Jaccard coefficient are as follows:

[0013]

[0014]

[0015] in, Represents two vectors V S1 and V S2 The sum of the products of the corresponding components, Represents vector V S1 Length, Represents vector V S2 The length, W S1 ∩W S2 Indicates the words that coexist in sentences S1 and S2, W S1 ∪W S2 Represents all words in sentences S1 and S2 except for the words that are the same;

[0016] Step 5: Perform data preprocessing on the source Chinese and English data of the dataset and the generated test cases: segment words and remove punctuation marks, generate a set of common subsequences (W1, W2, ..., W n ), S1 and S2 are segmented and punctuation marks are removed to generate a word list that retains the word order of the original sentence. Then the edit distance E(S1, S2) between S1 and S2 is calculated to measure the difference in word order between two sentences with common words. The calculation formula of the edit distance is as follows:

[0017]

[0018] Where k and l are the subscripts of string S1 and string S2 respectively. When min(k,l)=0, it means that one of string S1 and S2 is an empty string. Then, the conversion from x to y only requires max(k,l) single-character editing operations. Therefore, the editing distance between them is max(k,l), which is the maximum of k and l. When min(k,l)≠0, Indicates deletion of S 1k , Indicates inserting S 2l , Indicates replacement of S 2l , is the indicator function, when S 1k =S 2l When , the function value is 0; otherwise, the function value is 1;

[0019] Step 6: Calculate the similarity of sentences S1 and S2, Smilarity(S1,S2). Smilarity(S1,S2) is the product of the suppression factor consisting of the product of the Jaccard coefficient and the edit distance and the cosine similarity. The calculation formula is as follows:

[0020]

[0021]

[0022] Among them, the Jaccard coefficient and the edit distance product K S1,S2 It has an inhibitory effect on cosine similarity, which can make up for the fact that cosine similarity cannot capture the semantic changes caused by word order changes. S1,S2 It is constructed in the form of a logarithmic decay function;

[0023] Step 7: Calculate the similarity of the two sentences according to the similarity calculation formula, and replace the source English sentence S in step 1 with en And the generated test case S zh , S' en and S' zh , according to Figure 1 Repeat steps 3 to 5 until all sentence pairs S are calculated. en and S' en , S zh and S' zh ;

[0024] Step 8: Obtain the transformation relationship satisfaction rate based on the transformation relationship. The definition of the transformation relationship is as follows:

[0025] Similarity(S zh ,S' zh ) / Similarity(S en ,S' en )≥1

[0026] Among them, Similarity(S zh ,S' zh ) represents the similarity result of the target language Chinese sentence pair, Similarity(S en ,S' en ) represents the similarity result of the source language English sentence pair. Only the similarity of the sentence pair in a single language is used as the evaluation result. The quality of the translation results of the back translation or the original translation may be high or low. Therefore, the ratio of the sentence similarity results of the two languages ​​is used to evaluate the translation quality.

[0027] Step 9: Calculate the translation quality evaluation score of each translation engine based on the sentence transformation relationship satisfaction rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The figure is a flow chart of the sentence similarity calculation method of the present invention.

[0029] Figure 2 It is a schematic diagram of the specific implementation process of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present invention clearer, taking the Google translation engine as an example, the specific implementation method of the sentence granularity transformation test method for the neural machine translation system proposed in the present invention is described.

[0031] Step 1: Select Google Translate as the translation model T, and select 5,000 pairs of Chinese and English sentences in the five fields of education, Weibo, news, spoken language, and subtitles from the UM-Corpus dataset as the original datasets D1 and D2;

[0032] Step 2: Input D1 into the translation model T to obtain the Chinese test case set D 1t , D 1t Input it into the translation model T to get the English test case set D1', and then input D1' into the translation model T to get the Chinese test case set D 1t ';

[0033] Step 3: Compare the original data set D1 and the generated test case set D 1t , D1', D 1t 'Perform data preprocessing operations, first D1, D 1t , D1', D 1t 'Word segmentation, D1 and D1' are English data sets, using Python's NLTK toolkit to implement word segmentation, D 1t and D 1t The Chinese dataset is implemented using Python's jieba word segmentation technology; then stop words and punctuation marks are deleted from the Chinese and English datasets, and lowercase letter conversion and word form restoration are performed on the English datasets D1 and D1' to generate word lists W1 and W 1t 、W1'、W 1t ';

[0034] Step 4: Calculate the Jaccard coefficient J(W1,W1') of W1 and W1', W 1t and W 1t 'Jaccard coefficient J(W 1t ,W 1t ');

[0035] Step 5: Compare the original data set D1 and the generated test case set D 1t , D1', D1t 'Perform word segmentation and remove punctuation to generate a word list D that only retains the original word order in the sentence 1c , D 1tc , D 1c '、D 1tc ', the operation is the same as step 1, and then the edit distance E(D1,D1') of the sentence pairs in the English dataset D1 and D1' is calculated. 1t and D 1t The edit distance E(D 1t ,D 1t ');

[0036] Step 6: Set W1, W 1t 、W1'、W 1t 'Represented as vector data sets V1, V through the improved TF-IDF bag of words model 1t 、V1'、V 1t ', then construct the cosine value of the angle between the vector form datasets V1 and V1' of the English datasets D1 and D1', cos(V1,V1'), the Chinese dataset D 1t and D 1t 'Vector form data set V 1t and V 1t 'The cosine value of the angle cos(V 1t ,V 1t ');

[0037] Step 7: Calculate the similarity of sentence pairs in the English datasets D1 and D1', and first construct the suppression factor K D1，D1’ , K D1 , D1' is composed of J(W 1t ,W 1t ')·E(D 1c ,D 1c ') The product is formed in the form of a logarithmic decay function, and then K is calculated D1，D1’ The product of V1 and cos(V1,V1') is the similarity set Smilarity D1 , and similarly calculate the Chinese data set D 1t and D 1t 'Similarity set Smilarity D1t ;

[0038] Step 8: According to the transformation relationship, the similarity set Smilarity D1t and Smilarity D1 The corresponding Chinese sentence similarity and English sentence similarity are compared to Smilarity D1t / Smilarity D1, and then calculate the transformation relationship satisfaction rate and translation quality evaluation score of data set D1 on Google translation engine by similarity ratio;

[0039] The above process can realize the sentence granularity transformation test method for neural machine translation system. The above steps are as follows: Figure 2 As shown in the figure, it is divided into three parts: test case generation, text preprocessing, text vectorization representation and similarity calculation. Among them, the text vectorization representation mainly adds a weight on the basis of the TF-IDF bag-of-words model to achieve the effect of distinguishing data from different fields; the similarity calculation process adds the Jaccard coefficient that can reflect the commonality between two sentences and the edit distance that reflects the difference between two sentences as inhibition factors, improving the single cosine similarity method that is difficult to reflect the semantic changes brought about by different word orders in sentences.

[0040] This method is experimented under the framework of the metamorphic test for neural machine translation systems. Tables 1 and 2 are the experimental results of the PC value and SRC value comparison of the similarity calculation method based on cosine distance and the similarity calculation method based on edit distance on the dataset UM-Corpus. Among them, BLEU and METEOR represent the automatic evaluation indicators of machine translation. BLEU focuses on accuracy, METEOR focuses on accuracy and recall, and the correlation coefficient and p value are Pearson correlation coefficients. This method uses the Chinese data in the 5000 pairs of data sets selected as reference translations, and translates 5000 source English sentences by different translation engines according to the field. The Pearson correlation coefficient and Spearman rank correlation coefficient are compared between the BLEU and METEOR results of the reference translation. The higher the correlation coefficient, the closer the translation result evaluation is to the reference translation result, and the more accurate the sentence similarity calculation method is. The p value reflects the significance of the correlation. The smaller the p value, the more significant the correlation coefficient. From Tables 1 and 2, we can see that the correlation values ​​based on the cosine distance method are improved on each translation engine compared with the edit distance method. The PC value is increased by an average of 20.5%, the SRC value is increased by an average of 12%, and the p value is low, indicating that the PC value is more reliable.

[0041] Table 1 PC values ​​of different similarity calculation methods and reference translation experimental results

[0042]

[0043] Table 2 SRC values ​​of different similarity calculation methods and reference translation experimental results

[0044]

Claims

1. A sentence granularity transformation test method for a neural machine translation system, characterized in that The following steps are involved: Step 1: Obtain bilingual text data in Chinese and English from the corpus and generate test cases based on the metamorphosis relationship. Specifically, input the English data in each field into the neural network-based translation system for three round-trip translations: First, convert the source English sentence S en Input into the translation system to generate Chinese result S zh , and then S zh Input to the translation system to generate English sentence S' en , and finally S' en Enter the translation system again to generate the Chinese translation result S' zh ; Step 2: Perform data preprocessing operations on the source Chinese and English data of the dataset and the generated test cases: word segmentation, deletion of stop words and punctuation marks. Because the structures of Chinese and English are different, English sentences also need to be converted to lowercase letters and word form restoration to obtain the word list W; Step 3: The two sentences to be similarity calculated are recorded as S1 and S2, and the sentences after data preprocessing form a word list W S1 and W S2 , through the bag-of-words model based on TF-IDF, W S1 and W S2 Expressed as a vector V S1 and V S2 , the traditional calculation is to regard the weights of all feature words as the same, but for data sets with different domain data, it is necessary to reduce the role of secondary feature items, enhance the role of primary feature items, improve the word frequency, and add a weight. The improved calculation formula is as follows: tf-idf(t,d)=tf t,d ·idf t ·c i where tf t,d idf is the traditional TF-IDF algorithm calculation process, maxλ n,i Indicates that the i-th feature word appears most frequently in the n-type text data, ∑λ n Indicates the total number of words in n types of text; Step 4: Calculate V S1 and V S2 The cosine value of the angle cos(V S1 ,V S2 ), calculate the Jaccard coefficient J(W S1 ,W S2 ), the cosine similarity and Jaccard coefficient calculation formulas are as follows: in, Represents two vectors V S1 and V S2 The sum of the products of the corresponding components, Represents vector V S1 Length, Represents vector V S2 The length, W S1 ∩W S2 Indicates the words that coexist in sentences S1 and S2, W S1 ∪W S2 Represents all words in sentences S1 and S2 except for the words that are the same; Step 5: Perform data preprocessing on the source Chinese and English data of the dataset and the generated test cases: segment words and remove punctuation marks, generate a set of common subsequences (W1, W2, ..., W n ), S1 and S2 are segmented and punctuation marks are removed to generate a word list that retains the word order of the original sentence. Then the edit distance E(S1, S2) between S1 and S2 is calculated to measure the difference in word order between two sentences with common words. The calculation formula of the edit distance is as follows: Where k and l represent the subscripts of string S1 and string S2 respectively. When min(k,l)=0, it means that one of string S1 and S2 is an empty string. Then, only max(k,l) single-character edit operations are required to convert from x to y. Therefore, the edit distance between them is max(k,l), which is the maximum of k and l. When min(k,l)≠0, Indicates deletion of S 1k , Indicates inserting S 2l , Indicates replacement of S 2l , is the indicator function, when S 1k =S 2l When , the function value is 0; otherwise, the function value is 1; Step 6: Calculate the similarity of sentences S1 and S2, Smilarity(S1,S2). Smilarity(S1,S2) is the product of the suppression factor consisting of the product of the Jaccard coefficient and the edit distance and the cosine similarity. The calculation formula is as follows: Among them, the Jaccard coefficient and the edit distance product K S1,S2 It has an inhibitory effect on cosine similarity, which can make up for the fact that cosine similarity cannot capture the semantic changes caused by word order changes. S1,S2 It is constructed in the form of a logarithmic decay function; Step 7: Calculate the similarity of the two sentences according to the similarity calculation formula, and replace the source English sentence S in step 1 with en And the generated test case S zh , S' en and S' zh , repeat steps 3 to 6 until all sentence pairs S are calculated en and S' en , S zh and S' zh ; Step 8: Obtain the transformation relationship satisfaction rate based on the transformation relationship. The definition of the transformation relationship is as follows: Similarity(S zh ,S' zh ) / Similarity(S en ,S' en )≥1 Among them, Similarity(S zh ,S' zh ) represents the similarity result of the target language Chinese sentence pair, Similarity(S en ,S' en ) represents the similarity result of the source language English sentence pair. Only the similarity of the sentence pair in a single language is used as the evaluation result. The quality of the translation results of the back translation or the original translation may be high or low. Therefore, the ratio of the sentence similarity results of the two languages ​​is used to evaluate the translation quality. Step 9: Calculate the translation quality evaluation score of each translation engine based on the sentence transformation relationship satisfaction rate.

2. The sentence granularity transformation test method for a neural machine translation system according to claim 1, characterized in that: This method can be applied to the robustness evaluation of translation quality of neural machine translation systems.

Images