A machine translation software defect detection method based on combined semantics

Abstract

The application discloses a machine translation software defect detection method based on combined semantics, comprising the following steps: S1, obtaining different Chinese translation sentences of the same English source sentence from different translation software; S2, using a word alignment model to correspond English words in the source sentence with segmented Chinese words in the Chinese translation sentences; S3, using a sentence compression model and a syntactic structure analysis method to obtain a main part and each additional part of the English source sentence respectively and form a sub-sentence set; S4, aligning each part of the English source sentence obtained in the step S3 with the corresponding translation part to obtain aligned translation of the main part and the additional part of the source sentence; and S5, detecting errors, including semantic similarity calculation and synonym checking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of testing machine translation. In particular, a method for detecting defects of machine translation software based on combined semantics. BACKGROUND

[0002] Machine translation software, such as Google Translate [1] and Bing Translator [2] , is widely used in international communication in various fields, including academic, tourism and business activities. It is reported that as of 2016, the use of Google Translate has exceeded 500 million people [3] times. Currently, commercial machine translation software such as Google Translate can provide translation results close to the quality of human translation [4] . The widespread use of machine translation has greatly improved the communication efficiency between people from different parts of the world. However, like traditional software, machine translation software can also have translation errors, which can have serious consequences, including economic losses [5] , personal safety risks [6] , and even diplomatic incidents [7] , becoming the source of international tension and conflict [8] .

[0003] In view of the widespread use of machine translation software and the losses caused by its translation errors, it is very important to ensure the quality of its translation results. Currently, there are many works related to the testing methods of machine translation software [9-13] , and many errors have been detected. These methods mainly mutate a single word of a given type to obtain a variant sentence [9,11-13] , and use certain metamorphic relationships, such as structural invariance

[10] , pathological invariance

[11] and reference transparent input

[12] to construct test predictions. The generation of variant sentences in such methods is limited by the replaceable word types of the language model, and some sentences in which the words do not exist in the language model vocabulary cannot generate variant sentences for translation error detection. At the same time, due to the use of language models to replace words in sentences, the variant sentences themselves may have semantic and logical errors, thereby increasing the false positives of test results. In addition, these methods rely on text-level comparison indicators, such as BLEU

[14] , ROUGE

[30] and edit distance

[15] , or syntactic structure to measure the similarity of translated sentences. However, these indicators can only calculate the similarity of two sentences from the text or syntactic structure, and cannot reflect the semantic similarity of the sentences. While BERT

[31] and their derived models tend to encode all sentences into a small region of the space, which makes most sentence pairs have a high similarity score

[32] Therefore, new indicators are needed to measure semantic similarity when evaluating

[16] CIT

[10] Syntactic structure invariant relations are used for test case generation and test prediction construction. Unlike methods that replace individual words, CIT generates test cases with the same main part and different additional parts. Compared with word variation-based methods, CIT has a higher test accuracy, but different test sentences generated by this method may still refer to the same problem in the original sentence (as shown in Figure 2 Thus, these word replacement-based methods have the following four problems: (1) limited source sentences. Limited to the types of words that can be replaced by different variation methods. (2) limited errors that can be detected. A single detection system cannot identify errors present in the unchanged part of the source sentence. (3) introduce errors when generating test cases. Word replacement based on variation methods introduces errors. (4) inaccurate measurement indicators. Text similarity-based (such as edit distance) or syntactic structure-based similarity measurement indicators cannot accurately identify semantic similarity. (5) high redundancy. The variation sentence generation method based on multiple replacements of individual words will result in high redundancy when detecting errors. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art, based on the problems of limited test source sentences, limited detectable errors, test case generation introducing errors, inaccurate measurement indicators, and high redundancy in existing work. A machine translation software defect detection method based on combined semantics is provided, which needs to have the following characteristics: (1) can handle any test source sentence; (2) can find more potential translation errors; (3) has higher accuracy and lower redundancy when detecting errors.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] A machine translation software defect detection method based on combined semantics, comprising:

[0007] S1. Obtain different Chinese translation sentences for the same English source sentence from different translation software;

[0008] S2. Use a word alignment model to correspond the English words in the source sentence to the segmented Chinese words in the Chinese translation sentence;

[0009] S3. Obtain the main part and each additional part of the English source sentence respectively using the way of sentence compression model and syntactic structure parsing, and jointly constitute the sub-sentence set;

[0010] S4. Align each part of the English source sentence obtained in step S3 with its corresponding translation part to obtain the aligned translation of the main part and the additional part of the source sentence;

[0011] S5. Error detection: including semantic similarity calculation and synonym checking;

[0012] In the semantic similarity calculation, the SimCSE model is used as the representation model of the sentence vector, and then the cosine similarity of the sentence vector is calculated to obtain the similarity score, and the sentence vector similarity threshold is set to 0.75; for the sentence vector below the similarity threshold, synonym checking is performed;

[0013] In the synonym checking, the synonym dictionary tool is used to complete the synonym checking; when the translation of all sub-sentences is semantically similar, it is considered that the translation of the corresponding English source sentence is correct; if the translation of a sub-sentence is not semantically similar, it is considered that the translation of the corresponding English source sentence is not similar, and it is considered that there is an error in the translation result of the translation software.

[0014] Further, in step S1, the translation software includes Google Cloud Translation, Azure Microsoft Translator API and Baidu Translate API, for each source sentence, the above three translation software are called to obtain three Chinese translation sentences, and then the Chinese translation sentences are paired to obtain three translation sentence pairs, namely <Google, Bing>, <Bing, Baidu> and <Google, Baidu>.

[0015] Further, in step S2, the word alignment model AWESOME is used for word alignment, the input of the word alignment model AWESOME is the token sequence of the English source sentence and its corresponding Chinese translation sentence, and the output is a list of numbers i-j, indicating that the i-th word of the English source sentence is aligned with the j-th word of the Chinese translation sentence.

[0016] Further, in step S3, the sentence compression model SLAHAN is used, which can remove redundant information in the sentence to generate a simple sentence, for the remaining part of the English source sentence except the main part, each additional part is obtained by searching in the syntactic structure tree, and each additional part is combined to constitute the remaining part.

[0017] Further, in step S4, a clause part of the English source sentence is given, and according to the alignment result of the words, the corresponding Chinese words in the Chinese translation sentence are found, and the Chinese words are sorted according to the position of the Chinese words in the Chinese translation sentence. The part of the Chinese translation sentence that aligns with the same clause part in the English source sentence forms a translation clause pair.

[0018] Further, the synonym checking in step S5 includes the following steps:

[0019] (501) Given a pair of sentences, use jieba and LTP to perform word segmentation for each sentence to obtain token sequence wordList;

[0020] Query the synonym dictionary to obtain synoListDict composed of the synonym list of each token in wordList;

[0021] (502) Use the function checkSynonym to input the synoListDict of the two sentences respectively, and check the synonyms at the word level, that is, when a word in the sentence finds its synonym in the given synoListDict, it will be deleted from the sentence; Stop words will also be deleted from the given sentence;

[0022] (503) When there is no word in the given sentence, return True, indicating that each word in the sentence can find its synonym in the synonym dictionary of the other sentence; Synonym checking is performed on both sentences, and only when both synonym checking results return True, True is reported, indicating semantic similarity;

[0023] (504) Further, word-level synonym analysis is performed using the bag-of-words model defined in formulas (1) (2); Sim(B1,B2) calculates the minimum value between Sim s (B1,B2) and Sim s (B1,B2); B1 and B2 represent the two bags of words after deleting the same words and stop words, respectively; represents the corresponding word vector obtained using the Chinese-Word-Vector model on w Calculate the cosine similarity of and

[0024] Sim s ​In the definition of formula (2), first, the maximum cosine similarity of each word in B1 and all words in B2 is calculated, and then the minimum value is found therein; for each word in B1, a word in B2 is found which has the maximum cosine similarity, that is, the highest semantic similarity, and then the minimum cosine value is selected from all the cosine values calculated for the word in B1; if the minimum cosine value is still greater than the threshold value, it means that all the words in B1 can find semantically similar words in B2; Sim s There is a direction, so two directions are calculated and the smaller direction is selected; the bag-of-words model similarity threshold is set to 0.45.

[0025] Sim(B1,B2)=min(Sim s (B1,B2),Sim s (B1,B2)) (1)

[0026]

[0027] The application further provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the machine translation software defect detection method based on combined semantics when executing the program.

[0028] The application further provides a computer readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the steps of the machine translation software defect detection method based on combined semantics.

[0029] Compared with the prior art, the technical scheme of the application has the beneficial effects that:

[0030] 1. The application selects the method of difference test. As long as the test system (i.e. translation software) supports, it can be applied to any sentence in the source language (such as English). By giving an input sentence in English, two Chinese translation sentences are obtained from two different machine translation software (such as Google and Baidu).

[0031] 2. The application calculates the semantic similarity based on the combination principle, uses the sentence compression model syntactic structure analysis

[18] to obtain the part-whole structure relationship between a sentence and one of its clauses. Then, the language model is used for sentence representation, the semantic similarity of each clause part is calculated, and the translation error is detected according to the similarity score. Based on the new semantic similarity calculation method, the method of the application combines the grammatical structure and semantic information together, and has a higher error detection accuracy compared with the existing evaluation indexes using only text similarity and only semantic similarity.

[0032] 3. The application provides a machine translation software testing method based on difference testing, which can find errors that cannot be found by single software under test, has no limitation on testing input sentences and has lower redundancy when detecting errors.

[0033] 4. The method is tested on three widely used translation software and compared with two most advanced methods CIT and CAT. The experimental results show that DCS is superior to the other two methods in terms of accuracy and redundancy. The average values of precision, recall and F value of DCS on the three translation software are the highest, reaching 0.81, 0.83 and 0.82 respectively, and the redundancy is the lowest, reaching 1.06. In terms of testing efficiency, DCS reaches the same testing efficiency as CIT. The average preprocessing time of DCS and CIT on sentences is the same, the variation time consumption is 0.1 seconds lower than CIT, and the average defect detection time of sentence pairs is 0.25 seconds, slightly higher than the 0.19 seconds of CIT. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The flowchart of the machine translation software defect detection method in the embodiment.

[0035] Figure 2 The schematic diagram of redundant information.

[0036] Figure 3 The schematic diagram of splitting the source sentence into the main part and the modification part. DETAILED DESCRIPTION

[0037] The application will be further described below in conjunction with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0038] In order to better distinguish the errors generated by machine translation software, the embodiment adopts the error types defined in [21 ,22] The types of translation errors and corresponding examples of translation errors are shown in Table 1. The first column shows the error type, and the last three columns show the source sentence, the corresponding Chinese translation sentence and the English meaning of the Chinese translation sentence. The error is highlighted in bold.

[0039] The error types are described as follows:

[0040] (1) Wrong Choice of Words: There is a corresponding translation in the original text, but the translation is wrong or inaccurate

[0041] (2) Under Translation: Omit or repeat the translation content.

[0042] (3) Unclear Logic: The order of word translation is wrong.

[0043] (4) Opposite Meaning: The semantic of the translated sentence is opposite to the source sentence.

[0044] (5) Named Entity Problem: The named entity in the source sentence is wrongly translated or not translated.

[0045] (6) Quantifier / Time Problem: The translation of quantifier or time in the source sentence is incorrect or not translated.

[0046] (7) Others: Errors that are not of the above types.

[0047] Table 1 Translation Error Types and Example Descriptions

[0048]

[0049]

[0050]

[0051] For the sake of simplicity, the method of the present application is named DCS in the following sections, i.e., the difference test based on combined semantics [17,34] . The core idea of the method of the present application is that the translations from different translation software should be consistent in semantics. According to the principle of combination

[35] (which stipulates that the meaning of a complex expression is determined by the meaning of each component and the rules for combining them), the semantic similarity of each pair of translations corresponding to each clause is calculated. In particular, a pre-trained machine learning model SimCSE

[23] is used to measure semantic similarity, supplemented by synonym checking. The method of the present application can determine the phrases with translation errors based on word alignment.

[0052] The specific process of the experiment is shown in Figure 1 . Taking English-Chinese translation as an example, Baidu translation and Google translation are used as the translation software to be tested, and the test method of the present application mainly includes five steps:

[0053] (1) Obtain different translation results from different translation software for the same English source sentence;

[0054] (2) Use a word alignment model to correspond the English words in the source sentence and the segmented Chinese words;

[0055] (3) Use a sentence compression model and a syntactic structure analysis to obtain the main part and the additional part of the English source sentence to form a clause set;

[0056] (4) Based on the word alignment information and the split clauses, the alignment translation of the main part and the additional part of the source sentence can be obtained;

[0057] (5) The last part is error detection based on the calculated semantic similarity. When the translation of all clauses is similar in semantics, it is considered that the two translations of the source sentence are probably correct; if the translation of a clause is not similar in semantics, it is considered that the translation pair is not similar, and one of the translation results of different translation software may have an error.

[0058] S1. Obtain translation results

[0059] Call the official API provided by the test target, namely Google Cloud Translation

[24] , Azure Microsoft Translator API

[25] and Baidu Translate API

[26] to obtain the translated sentences (Chinese) of the source sentences (English). For each source sentence, call three APIs to obtain three Chinese translation sentences, and then pair the Chinese translation sentences to obtain three translation sentence pairs, namely <Google, Bing>, <Bing, Baidu> and <Google, Baidu>, The order in the sentence pair does not need to be considered. Figure 1 The running example of the Google and Baidu Chinese translation sentence pair is shown.

[0060] S2. Word alignment

[0061] Word alignment is a natural language processing technique that establishes a connection between two words if there is a translation relationship between them. This embodiment uses the most advanced word alignment model AWESOME

[27] , which has achieved the best performance in word alignment of five languages including English-Chinese alignment. AWESOME uses a pre-trained language model, which is fine-tuned on parallel text to improve alignment quality. The input of AWESOME is the token sequence of the source sentence and its corresponding Chinese translation sentence, and the output is a list of numerical pairs i-j, indicating that the i-th word of the source sentence is aligned with the j-th word of the Chinese translation sentence.

[0062] S3. Split the source sentence

[0063] Use sentence compression and syntactic structure parsing

[28] to obtain the structural composition of the source sentence. This embodiment uses the most advanced sentence compression model SLAHAN

[29] , it can remove the redundant information in the sentence to generate simple sentences, preserving the syntactic structure and important content. SLAHAN is a Seq2Seq model, and the output constitutes the main part of the source sentence. For the remaining part of the source sentence except the main part, search for each additional part (i.e. the clause whose root node is the "Clause" attribute, such as SBAR) in the syntactic structure tree, as shown in Figure 3 .

[0064] S4. Clause alignment

[0065] After obtaining the clause components of the source sentence and the word alignment pair, the alignment of the clause part is performed, that is, each part of the source sentence obtained in the split source sentence is aligned with its corresponding translation part. Given a clause part of the source sentence, according to the word alignment result, find the corresponding Chinese words in the Chinese translation sentence, and sort them according to their positions in the Chinese translation sentence. The part of the Chinese translation sentence that aligns with the same clause part in the source sentence will form a translation clause pair.

[0066] Figure 1 This is clearly illustrated by the example in

[0067] S5. Error detection

[0068] (501) Semantic similarity calculation

[0069] The present embodiment adopts the SimCSE framework

[19] , which enhances pre-trained language models such as BERT

[31] and RoBERTa

[36] with a contrastive objective function for sentence embeddings. This model improves the representation of sentences by using a self-supervised training method, which is superior to previous methods on the semantic text similarity task. Therefore, the SimCSE model fine-tuned using Chinese semantic similarity training data is selected as the representation model for obtaining sentence vectors. Then, the cosine similarity on the sentence vectors is calculated to obtain the similarity score.

[0070] Like existing methods that manually set thresholds [9,11], a trade-off between false positives and false negatives needs to be made. In the method of the present invention, a high similarity score means semantic consistency between the two parts. However, setting a very high score can result in a large number of false positives, i.e. semantically similar sentence pairs are reported as incorrect sentence pairs, while setting a low similarity score can result in a large number of false negatives, i.e. semantically inconsistent sentence pairs are not detected. Through the analysis of false positives, since most of the false positives are due to synonyms, which are not correctly embedded by the language model. Therefore, the similarity threshold is set to 0.75. For the false positives that can be introduced below the threshold, further synonym checking is carried out.

[0071] (502) Synonym checking

[0072] The present embodiment uses the synonym dictionary of Harbin Institute of Technology

[37] for synonym checking. The detailed checking process is shown in Algorithm 1, where the input is a pair of sentences to be checked and a stop word list. Given a pair of sentences, first use jieba

[38] and LTP

[39] to perform word segmentation for each sentence to obtain the token sequence wordList (lines 1-2). Then, query the synonym dictionary to obtain synoListDict consisting of a list of synonyms for each token in wordList (lines 3-4).

[0073] The function checkSynonym takes as input the synoListDict of the two sentences respectively, and checks for synonyms at the word level, i.e. when a word in a sentence finds its synonym in the given synoListDict, it will be removed from the sentence (lines 9-16). Stop words will also be removed from the given sentence (lines 17-20). When there are no words left in the given sentence, return True, indicating that every word in the sentence has its synonym in the synonym dictionary of the other sentence. By performing synonym checking on both sentences (lines 5-6), only when both synonym checking results return True (line 7), it is reported that the semantics are similar.

[0074] Since the number of synonyms in the synonym dictionary of Harbin Institute of Technology is small (77343 words), the number of false positives is also limited. Further word-level synonym analysis is performed using the bag-of-words model defined in equations (1) (2). Sim(B1,B2) calculates the minimum value between Sim s (B1,B2) and Sim s (B1,B2). B1 and B2 represent the two bags of words after removing the same words and stop words, respectively. denotes the corresponding word vector obtained using the Chinese-Word-Vector model on w, Compute and the cosine similarity of Sim s , which is defined in Equation 2, first computes the maximum cosine similarity of each word in B1 to all words in B2, and then finds the minimum among them. For each word in B1, find a word in B2 that has the maximum cosine similarity (highest semantic similarity), and then select the minimum cosine value among all the cosine values computed for the word in B1. If the minimum cosine value is still greater than a certain threshold, it means that all the words in B1 can find semantically similar words in B2. Sim s has direction, so we need to compute both directions and select the smaller one. According to experience, the similarity threshold of the bag-of-words model is set to 0.45.

[0075] Sim(B1,B2)=min(Sim s (B1,B2),Sim s (B1,B2)) (1)

[0076]

[0077]

[0078]

[0079] Specifically, based on the above technical solution, the embodiment selects three most widely used translation software, namely Google translation, Bing translation and Baidu translation, and takes English-Chinese translation as the test target system. The three translation software are popular commercial translation software, with a large number of international and domestic user groups. They all provide Web interface and API, so that users and developers can obtain instant translation results. In the experiment, the API provided by each translation software is directly called, that is, Google Cloud Translation API

[24] , Azure Microsoft Translator API

[25] and Baidu Translate API

[26] , to obtain the translated sentence.

[0080] First, the experimental setup is performed, and the embodiment performs the experiment on a server with 512GB and Nvidia 2080RTX GPU, and uses Ubuntu 18.04 system in software aspect. In the model aspect, an open source sentence representation model SimCSE

[23] and a word alignment model awesome-align

[27] The dataset is collected from CNN

[41] , BBC

[42] and China Daily

[43] articles. Eight different categories of articles, namely business, culture, entertainment, health, politics, sports, technology and travel, are specially searched. Then, grammatically and semantically correct sentences are selected from each category to form a list of sentences for each category. Then, 100 sentences are randomly selected from the list of sentences for each category to form a dataset containing 800 sentences. Eight categories are initially selected to evaluate whether the performance of DCS is consistent in different semantic contexts. The dataset collected in this embodiment has no intersection with the dataset used in the prior art [9 , 10, 13 ,20] , which ensures a fair comparison with existing methods from a data perspective. The statistics of the sentences in the dataset are shown in Table 2.

[0081] Table 2 Statistics of the test dataset

[0082]

[0083] Secondly, regarding the evaluation indicators, this embodiment uses precision, recall and F value to evaluate the test effect. The formula for calculating precision is:

[0084]

[0085] where R represents the test output of the test method, and p is a sentence pair in R. When there is a translation error in either of the two sentences in the p sentence pair, error(p) is set to 1, otherwise it is set to 0.

[0086] The formula for calculating recall is:

[0087]

[0088] where All represents all sentence pairs, and the denominator represents the number of sentence pairs with translation errors. In the numerator, R represents the test output of the test method, and the numerator represents the number of all sentence pairs with translation errors in the test output.

[0089] Taking into account the influence of precision and recall, F1-Score is used as the harmonic mean of precision and recall.

[0090]

[0091] To quantitatively measure how the detected problems help to produce unique errors, and thus help to locate and fix the errors, the redundancy of each method is calculated. A formal definition of redundancy is given, which is defined as the average number of detected problems that point to a unique error in the source sentence. |R| is the number of suspect sentence pairs reported by the testing method, and w represents a word or phrase in the source sentence. suspect(w) is set to 1 if the given word or phrase in the source sentence is a suspected mistranslation, i.e., the inconsistent word / phrase reported in the Chinese translation sentence aligns with w in the source sentence. For example, in the example shown in Figure 2 There are three sentences corresponding to the seed sentence generation, and three sentence pairs, i.e., the seed sentence paired with each generated sentence, will report inconsistent problems. However, the three reported sentences all provide the same information, i.e., the word "hitting" in the source sentence is likely a mistranslation. |R| is 3, and the number of suspected error words / phrases in the source sentence is 1. Therefore, the redundancy is 3.

[0092]

[0093] Then, regarding the selection of the comparison method, the present embodiment compares DCS with the two latest and best-performing research results, CIT

[10] and CAT

[20] . CIT adopts syntactic structure invariance as the metamorphic relationship, and forms a sentence pair with the original sentence by adding an adjunct to the basic sentence. It is also compared with CAT, which is a contemporary work of CIT, and its method generates a variant sentence by replacing a single word to form a sentence pair with the original sentence. According to the link provided in their paper, the published open-source model is adopted. The hyperparameters of the comparison methods are consistent with the parameters in the original paper. CIT

[10] and CAT

[20] have been shown to outperform existing methods, including SIT [9] , Purity

[12] and TransRepair

[13] , respectively. Therefore, the present embodiment only compares with CIT and CAT.

[0094] Finally, regarding the experimental results, Table 3 lists the precision, recall and F-value of DCS, CIT and CAT on Baidu translation, Google translation and Bing translation, respectively. As can be seen from the table, DCS has the best effect on the three machine translation software, with an average precision of 81%, which is better than CIT (69%) and CAT (51%). The average recall of DCS is 83%, which is also much higher than CIT (43%) and CAT (74%).

[0095] The DCS performed stably in different categories, but in all the tested translation categories, the sports category performed best, and the health category performed worst. In the sports category, the number of DCS false positive samples was 16, accounting for 10% of the total percentage, and synonym omission (Synonym) was the main cause of false positives, accounting for 5% of the total. The number of false negative samples accounted for 13% of the total, and same error (Same Error) accounted for 6% of the total. WOW accounted for 59% of its error types, which was much higher than the average of 44%. This may be mainly because there are many technical terms in a specific field, such as Derby, which means “a sports competition between teams from the same region or town” in sports, and is not correctly translated in the sports category. DCS performs better on this error detection because the results of two different translation engines are compared, which is more likely to reveal differences in word selection.

[0096] Table 3 Comparison of effectiveness of DCS, CIT and CAT

[0097]

[0098]

[0099] Table 4 lists the number of each type of translation error found by DCS using the data set of eight categories on three translation software respectively, totaling 509. It can be observed that WOW causes the most translation errors, followed by UT and UL. Among the three translation software as test systems, Bing translation has the least number of translation errors, while Baidu translation has the most number of errors. At the same time, DCS can detect all types of translation errors. Tables 2 and 3 show that DCS can find various types of errors in the process of Chinese translation sentences by translation software, and its output results have high accuracy for developers to refer to.

[0100] Table 4 Number of errors reported by DCS in each category

[0101]

[0102] In order to determine whether DCS is efficient, three types of average time are recorded, namely preprocessing time, variation time and defect detection time. The preprocessing time refers to the average time consumed by those processes that do not require machine translation or do not depend on Chinese translation sentences. For DCS and CIT, the preprocessing time is composed of sentence compression and syntax structure parsing time. The variation time refers to the average time spent on generating variant sentences; for CIT, the variation time refers to using Bert

[31] Model replacement time: the time for the model to replace the words in the sentence to generate the variant sentence; for CAT, the variant time refers to the time for the Bert model to replace the words in the sentence to generate the variant sentence. Defect detection time refers to the average time of the process related to calling the test target or depending on the translation result; for DCS, the defect detection time includes the average consumption time of word alignment, clause alignment and semantic similarity analysis; CIT uses the average consumption time of syntax structure comparison to check the correctness of the translation; CAT uses the average consumption time of LCS to calculate the similarity of the sentence pair. There is no time to compare the calling of the test target, because it is the same for the three methods. In Table 5, CAT is the most efficient, because it does not call the complex models such as sentence compression and syntax structure analysis used by CIT and DCS. The defect detection time of the method of the present application is relatively high, because the word alignment and sentence representation model need to be loaded.

[0103] Table 5 average execution time (in seconds)

[0104]

[0105] In Table 6, redundancy provides a quantitative method to measure the information that each problem translation pair output by the detection method can provide, and it can also be used to measure the workload of the manual checking and analysis work required by the tester. As shown in Table 6, the redundancy of CIT is about 4, which means that the average of about 4 sentence pair problems reported by CIT involve the same error mapping, so the tester manually checks 4 redundant reports and obtains one unique information. For DCS and CAT, the redundancies are 1.08 and 1.17 respectively, which are much smaller than CIT. Considering the unique information provided by each reported sentence pair, the execution time advantage of CIT in Table 4 is offset. The faster average execution time and lower redundancy reduce the workload of the tester and the development personnel.

[0106] Table 6 redundancy of comparison methods

[0107]

[0108] Table 7 lists the DCS and DCS-comp that eliminates the step of combining semantics (i.e. DCS-comp, which eliminates the step of combining semantics in DCS, and the method of the present application, which eliminates the step of combining semantics in DCS) Figure 1 In steps 2, 3 and 4, the Chinese translation sentence obtained in step 1 is directly subjected to step 5 defect detection, and in the table it can be seen that the number of false negatives of the method of directly subjecting the translation sentence pair to defect detection is very large, which shows that even if the pre-training model is enhanced using the latest contrastive learning framework SimCSE, the problem of "collapse" will still occur in the sentence representation process

[33] That is, the semantically irrelevant sentences are encoded into a smaller space region so that the similarity is higher. After adding steps 2, 3, and 4, it can be seen from Table 6 that the number of false negatives has decreased significantly, although it has led to an increase in the number of false positives. However, the precision (P), recall (R), and F value in the last three columns are all higher than those of DCS-comp, indicating that the combined semantic method is more effective for shortening the sentence pair and then performing defect detection.

[0109] Table 7 Comparison of DCS and DCS-comp

[0110]

[0111] To prove that the method of the present application can complement the two methods CIT and CAT, two sets of complementary experiments were performed. For the data sets of the two experiments, CIT_Gen is the sentence pair data set generated by CIT variation, and CAT_Gen is the sentence pair data set generated by CAT variation. As can be seen from Table 8, by using the sentences generated by CIT and CAT as test inputs, and still obtaining higher precision and recall than CIT and CAT on this input, the effectiveness of the experimental method of the present embodiment is proved. To prove the complementary effect of DCS on the CIT and CAT experimental methods, the two sets of experimental results were used to verify the correction of false positives of DCS on the other method in this experiment. In the complementary experiment of DCS and CIT, the true positive cases of DCS were used to correct 53%, 64%, and 43% of the false negatives of CIT in the test results of Baidu, Google, and Bing translation software, respectively. In the complementary experiment of DCS and CAT, the true positives of DCS were used to correct 38%, 46%, and 65% of the false negatives of CAT in Baidu, Google, and Bing translation, respectively.

[0112] Table 8 DCS complementary experiment with CIT and CAT

[0113]

[0114]

[0115] To better explain the method of the present application, all DCS reported results were analyzed, and the reasons for the false positive and false negative samples were given.

[0116] (1) Reasons for false positive samples:

[0117] DCS reported 152 false positives, of which 105 were related to Bing translation, 91 were related to Google translation, and 108 were related to Baidu translation. The main reasons for false positives are summarized as follows.

[0118] ① Synonym: The semantics of synonyms are not correctly represented and not reflected in the dictionary. For example, in the first example study shown in Table 9, the phrase "Begrudgingly" is translated as "reluctantly" in Baidu Translate and "unwillingly" in Google Translate. However, the semantics of this synonym are not correctly captured, resulting in false positives. For this reason, 67 false positives were detected, accounting for 41% of all false positives in DCS.

[0119] ② Word Alignment Error: Incorrect word alignment between English and Chinese can cause false positive results. For example, in Table 9, "indescribable" is not correctly aligned with "无法形容" in Baidu Translate. Therefore, the Chinese clauses of "Is something indescribable" are "是一件的事情" (Baidu Translate) and "是难以形容的事情" (Google Translate) respectively. The semantic similarity between these two Chinese sub-words is lower than the given threshold, resulting in a false positive. For this reason, 48 false positives were detected.

[0120] ③ Structure Difference: The translated sentences have different syntactic structures, which leads to different aligned parts. Therefore, the semantic similarity is measured by the metrics of this embodiment. Taking the example study in Table 8 as an example, the sentence "which is gaining increasing global influence in recent years" is translated as an attributive clause of "第十二届北京国际电影节" by Bing Translate, but as an additional sentence of "第十二届北京国际电影节" by Baidu Translate. Both are correct syntactic structures in Chinese, but there is no word directly aligned with "which" in the sentence translated by Bing Translate, while there is a word "该电影节" directly aligned with "哪" in the sentence translated by Baidu Translate, resulting in false positives. 20 false positive samples were detected due to this reason.

[0121] ④ Polysemy: The semantics of polysemous words are reasonable and correct without more context. For example, in the second example study shown in Table 9, the word "lifting" is translated as "好起来" by Google Translate and "提升" by Bing Translate, both of which are correct in the given context. For this reason, 17 false positives were detected.

[0122] Table 9 False Positive Example Studies

[0123]

[0124]

[0125] (2) Causes of false negative samples:

[0126] DCS reported 152 false negatives, of which 89 were Bing translation related, 85 were Google translation related, and 88 were Baidu translation related. The main causes of false negatives are summarized as follows.

[0127] ① Inaccurate Semantic Representation: The semantic representation of the language model is inaccurate, so the cosine similarity cannot measure semantic similarity. For example, in the first example study in Table 10, Bing translation has opposite meaning on the sentence "scenes of people everywhere", which is an opposite meaning error. There are 76 false negative samples of this type.

[0128] ② Same Error: Both translated sentences have the same error, for example, due to the omission of translation content, both sentences have high semantic similarity. In the second example study in Table 10, Baidu translation and Google translation both did not translate the phrase "which she quoted", and the method based on difference test cannot detect this. There are 55 false negative samples of this type.

[0129] Table 10 False Negative Sample Study

[0130]

[0131]

[0132] In summary, the machine translation software difference test method based on combined semantics DCS of the embodiment does not directly compare the semantics of the Chinese translation sentences obtained from two machine translation software, but utilizes the combined principle to calculate the semantic similarity of the corresponding components of the Chinese translation sentences. Word alignment and syntactic structure analysis are used to obtain the corresponding components. The method of the present application is compared with the two methods of the current latest CIT and CAT. The experimental results show that the method DCS of the present application has higher precision, and the average values of the precision, recall rate and F value of DCS on the three translation software are the highest, reaching 0.81, 0.83 and 0.82 respectively; the generated redundant information is the least, and the redundancy is the lowest, reaching 1.06; the time consumption is equivalent to CIT, and the average preprocessing time of the two methods is the same, which is 0.49 seconds, the variation time is lower than that of CIT by 0.1 seconds, and the average defect detection time of the sentence pair is 0.25 seconds, which is slightly higher than that of CIT by 0.19 seconds.

[0133] Preferably, the embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all the steps of the machine translation software defect detection method based on combined semantics in the above embodiments, and the electronic device specifically includes the following contents:

[0134] a processor, a memory, a communications interface and a bus;

[0135] The processor, the memory and the communications interface complete the communication with each other through the bus; the communications interface is used to realize the information transmission between the server-side device, the metering device and the user-side device and other related devices.

[0136] The processor is used to call the computer program in the memory, and the processor realizes all the steps of the machine translation software defect detection method based on combined semantics in the above embodiments when executing the computer program.

[0137] The embodiments of the present application also provide a computer readable storage medium capable of implementing all the steps of the machine translation software defect detection method based on combined semantics in the above embodiments, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize all the steps of the machine translation software defect detection method based on combined semantics in the above embodiments.

[0138] Each embodiment in the specification is described in a progressive manner, and the same and similar parts between each embodiment can be referred to each other, and each embodiment mainly describes the differences from other embodiments. Especially, for the hardware+program type embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0139] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of the application as expressed by the claims that follow, some further examples of aspects of the application are discussed below.

[0140] Although the present application provides method operations steps as in an embodiment or flow chart, more or fewer operations steps can be included based on the implementation. The order in which the steps are listed in the embodiments is merely one way of executing the steps, and is not meant to be the only way of executing the steps. The steps can be executed in the order listed in the embodiments or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0141] Those skilled in the art will appreciate that embodiments of the application can be practiced in a method, system, or computer program product. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, magnetic disks, CD-ROMs, optical storage media such as DVD s, etc.) embodying computer readable program code.

[0142] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0144] The present application is not limited to the above-described embodiments. The above description of the specific embodiments is intended to describe and illustrate the technical solutions of the present application, and the specific embodiments described above are merely illustrative and are not restrictive. Without departing from the purpose of the present application and the scope protected by the claims, those of ordinary skill in the art can make many forms of specific changes under the inspiration of the present application, and these all belong to the protection scope of the present application.

[0145] References:

[0146] [1]“The google translate engine,”2022,https: / / translate.google.com.

[0147] [2]“The bing microsoft translator,”2022,https: / / cn.bing.com / translator.

[0148] [3]B.Turovsky,“Ten years of google translate,”2022,https: / / blog.google /

[0149] products / translate / ten-years-of-google-translate / .

[0150] [4]H.Hassan,A.Aue,C.Chen,V.Chowdhary,J.Clark,C.Federmann,X.Huang,M.

[0151] Junczys-Dowmunt,W.Lewis,M.Li et al.,“Achieving human parity onautomatic chinese to english news translation,”arXiv preprint arXiv:1803.05567,2018.

[0152] [5]“Lost in translation:13international marketing fails,”2022,

[0153] https: / / www.businessnewsdaily.com / 5241-international-marketing-fails.html.

[0154] [6]“Palestinian man is arrested by police after posting‘good morning’in arabic on facebook which was wrongly translated as‘attack them’,”2017,

[0155] https: / / www.dailymail.co.uk / news / article-5005489 /

[0156] Good-morning-Facebook-post-leads-arrest-Palestinian.html.

[0157] [7]“After google translate’s latest update,bbc culture finds history’s biggest language mistakes–including a us president stating‘i desire thepoles carnally’.”2015,https: / / www.bbc.com / culture / article / 20150202-the-greatest-mistranslations-ever.

[0158] [8]M.C.Mason,“Strategic insights:Lost in translation,”2017,https: / /

[0159] press.armywarcollege.edu / articles\editorials / 401 / .

[0160] [9] P. He, C. Meister, and Z. Su, “Structure-invariant testing for machine translation,” in Proceedings of the ACM / IEEE 42nd International Conference on Software Engineering, ser. ICSE’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 961–973.

[0161]

[10] P. Ji, Y. Feng, J. Liu, Z. Zhao, and B. Xu, “Automated testing for machine translation via constituency invariance,” in 2021 36th IEEE / ACM International Conference on Automated Software Engineering (ASE), 2021, pp. 468–479.

[0162]

[11] S. Gupta, P. He, C. Meister, and Z. Su, Machine Translation Testing via Pathological Invariance. New York, NY, USA: Association for Computing Machinery, 2020, p. 863–875.

[0163]

[12] P. He, C. Meister, and Z. Su, “Testing machine translation via referential transparency,”

[0164] 2021 IEEE / ACM 43rd International Conference on Software Engineering (ICSE), pp. 410–422, 2021.

[0165]

[13] Z. Sun, J. M. Zhang, M. Harman, M. Papadakis, and L. Zhang, “Automatic testing and improvement of machine translation,” in Proceedings of the ACM / IEEE 42nd International Conference on Software Engineering, ser. ICSE’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 974-985.

[0166]

[14] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: A method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ser. ACL’02. USA: Association for Computational Linguistics, 2002, p. 311-318.

[0167]

[15] E. Ristad and P. Yianilos, “Learning string-edit distance,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 5, pp. 522-532, 1998.

[0168]

[16] Y. Wu, M. Schuster, Z. Chen, Q. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, u. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, and J. Dean, “Google’s neural machine translation system: Bridging the gap between human and machine translation,” 09 2016.

[0169]

[17] J. Hintikka, “A hundred years later: The rise and fall of frege’s influence in language theory,” Synthese, vol. 59, no. 1, pp. 27-49, 1984.

[0170]

[18] R. A. Hudson, “Constituency and dependency,” 1980.

[0171]

[19] T. Gao, X. Yao, and D. Chen, “Simcse: Simple contrastive learning of sentence embeddings,” arXiv preprint arXiv:2104.08821, 2021.

[0172]

[20] Z. Sun, M. J. Zhang, Y. Xiong, M. Harman, M. Papadakis, and L. Zhang, “Improving machine translation systems via isotopic replacement,” in Proceedings of the ACM / IEEE 44nd International Conference on Software Engineering, ser. ICSE’22 (To appear). New York, NY, USA: Association for Computing Machinery, 2022.

[0173]

[21] H.Zhao and Q.Liu,“Common error analysis of machine translationoutput,”11 2014.

[0174]

[22] D.Vilar,J.Xu,L.D’Haro,and H.Ney,“Error analysis of machinetranslation output,”05

[0175] 2006,pp.697–702.

[0176]

[23] “The chinese sentence representation model,”2022.[Online].Available:

[0177] https: / / huggingface.co / cyclone / simcse-chinese-roberta-wwm-ext

[0178]

[24] “Google cloud translation,”2022,https: / / cloud.google.com / translate.

[0179]

[25] “Azure microsoft translator,”2022,https: / / azure.microsoft.com / en-us /

[0180] services / cognitive-services / translator / .

[0181]

[26] “Baidu translator api,”2022,https: / / api.fanyi.baidu.com / .

[0182]

[27] Z.-Y.Dou and G.Neubig,“Word alignment by fine-tuning embeddingson parallel corpora,”01 2021,pp.2112–2128.

[0183]

[28] “Stanford parser,” 2022, https: / / nlp.stanford.edu / software / lexparser.shtml.

[0184]

[29] H. Kamigaito and M. Okumura, “Syntactically look-ahead attention network for sentence compression,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, 2020, pp. 8050- 8057.

[0185]

[30] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out, 2004, pp. 74-81.

[0186]

[31] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.

[0187]

[32] S. Liu, R. Guo, B. Zhao, T. Chen, and M. Zhang, “Appcorp: A corpus for android privacy policy document structure analysis,” CoRR, vol. abs / 2005.06945, 2020.

[0188]

[33] Y. Yan, R. Li, S. Wang, F. Zhang, W. Wu, and W. Xu, “ConSERT: A contrastive framework for self-supervised sentence representation transfer,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1 : Long Papers). Online : Association for Computational Linguistics, Aug. 2021, pp. 5065-5075.

[0189]

[34] Z. Liu, Y. Lin, and M. Sun, Compositional Semantics. Singapore: Springer Singapore, 2020, pp. 43-57, https: / / doi.org / 10.1007 / 978-981-15-5573-2\3.

[0190]

[35] F. J. Pelletier, “The principle of semantic compositionality,” Topoi, vol. 13, no. 1, pp. 11-24, 1994.

[0191]

[36] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.

[0192]

[37] B. Turovsky, “Hit ir-lab tongyici cilin,” 2022, http: / / ir.hit.edu.cn / demo / ltp / Sharing\Plan.html.

[0193]

[38] “jieba”chinese text segmentation,”2022,https: / / github.com / fxsjy / jieba.

[39] “Language technology platform,”2022,https: / / github.com / HIT-SCIR / ltp.

[40] S.Li,Z.Zhao,R.Hu,W.Li,T.Liu,and X.Du,“Analogical reasoning on chinese morphological and semantic relations,”arXiv preprint arXiv:1805.06504,2018.

[41] “Cable news network,”2022,https: / / edition.cnn.com / .

[0194]

[42] “British Broadcasting Corporation”2022,https: / / www.bbc.co.uk /

[0195]

[43] “China Daily Website”2022,https: / / www.chinadaily.com.cn /

Claims

1. A method for detecting software defects in machine translation based on combined semantics, characterized in that, Comprise: S1. Obtain different Chinese translation sentences from different translation software for the same English source sentence; S2. Use a word alignment model to align English words in the source sentence with segmented Chinese words in the Chinese translation sentence; S3. Use a sentence compression model and a syntactic structure analysis method to obtain the main part and each additional part of the English source sentence respectively and jointly constitute a sub-sentence set; S4. Align each part of the English source sentence obtained in step S3 with its corresponding translation part to obtain the aligned translation of the main part and the additional part of the source sentence; S5. Error detection: including semantic similarity calculation and synonym checking; In semantic similarity calculation, SimCSE model is used as the representation model of sentence vector, then the cosine similarity on the sentence vector is calculated to obtain the similarity score, and the sentence vector similarity threshold is set to 0.75; for the sentence vector below the similarity threshold, synonym checking is performed; In synonym checking, synonym dictionary tool is used to complete synonym checking; when the translation of all sub-sentences is semantically similar, it is considered that the translation of the corresponding several sentences of the English source sentence is correct; if the translation of a sub-sentence is not semantically similar, it is considered that the translation of the corresponding English source sentence is not similar, and it is considered that there is an error in the translation result of a certain translation software.

2. The method of claim 1, wherein the method is based on a combination of semantics. In step S1, the translation software includes Google Cloud Translation, Azure Microsoft Translator API and Baidu Translate API, for each source sentence, the above three translation software are called to obtain three Chinese translation sentences, and then the Chinese translation sentences are paired to obtain three translation sentence pairs, namely <Google, Bing>, <Bing, Baidu> and <Google, Baidu>.

3. The method of claim 1, wherein the method further comprises: In step S2, the word alignment model AWESOME is used for word alignment, the input of the alignment model AWESOME is the token sequence of the English source sentence and its corresponding Chinese translation sentence, and the output is a list of digital pairs i-j, indicating that the i-th word of the English source sentence is aligned with the j-th word of the Chinese translation sentence.

4. The method of claim 1, wherein the method further comprises: In step S3, the sentence compression model SLAHAN is used, which can remove redundant information in the sentence to generate a simple sentence, for the remaining part of the English source sentence except the main part, each additional part is obtained by searching in the syntactic structure tree, and each additional part is combined to constitute the remaining part.

5. The method of claim 1, wherein the method further comprises: In step S4, a sub-sentence part of the English source sentence is given, according to the alignment result of the words, the corresponding Chinese words in the Chinese translation sentence are found, and the Chinese words are sorted according to the position of the Chinese words in the Chinese translation sentence. The part aligned with the same sub-sentence part in the English source sentence in the Chinese translation sentence forms a translation sub-sentence pair.

6. The method of claim 1, wherein the method further comprises: In step S5, synonym checking includes the following steps: (501) Given a pair of sentences, use jieba and LTP to perform tokenization for each sentence to obtain token sequence wordList; Query the synonym dictionary to obtain synoListDict composed of synonym list of each token in wordList; (502) The function checkSynonym takes as input the synoListDict of both sentences respectively and checks for synonyms at word level, i.e. when a word in a sentence finds its synonym in the given synoListDict, it is removed from the sentence; stop words are also removed from the given sentence; (503) When there are no words in the given sentence, True is returned, indicating that every word in the sentence finds its synonym in the synonym dictionary of the other sentence; synonym checking is done for both sentences and True is reported only when both synonym checks return True, indicating semantic similarity; (504)Further word-level synonym analysis using the bag-of-words model defined in (1) (2); Sim(B1, B2) computes the minimum of Sim s (B1, B2) and Sim s (B1, B2) between the minimum of Sim denotes the corresponding word vector obtained using the Chinese-Word-Vector model on w, computes the cosine similarity of and and Sim s In the definition of formula (2), first calculate the maximum cosine similarity of each word in B1 with all words in B2, then find the minimum value among them; for each word in B1, find a word in B2 that has the maximum cosine similarity, that is, the highest semantic similarity, then select the minimum cosine value among all cosine values calculated for the word in B1; if the minimum cosine value is still greater than the threshold value, it means that all words in B1 can find semantically similar words in B2; Sim s There is a direction, so calculate two directions and select the smaller one; the bag-of-words similarity threshold is set to 0.45; Sim(B1, B2) = min(Sim s (B1, B2), Sim s (B1, B2)) (1) 7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the method for detecting defects in machine translation software based on combined semantics according to any one of claims 1 to 6 when executing the program.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the method for detecting defects in machine translation software based on combined semantics according to any one of claims 1 to 6 when executed by the processor.

Images