A testing method for a translator based on syntax tree pruning

Abstract

The present invention relates to a machine translation testing method. The method constructs a dependency syntax tree for a sentence, prunes the syntax tree according to specific rules, destroys the validity of the sentence based on a set of dependency syntax tree-level deletion operators, deletes words or phrases from the original sentence to generate a new grammatically and semantically valid sentence, then inputs the original text and the newly generated sentence into the machine translation system being tested, calculates the bag-of-words distance, sorts and expands the sentences according to the bag-of-words distance, selects the five sentences with the largest distance, manually labels the original sentence and the translated sentence results, marks the erroneous sentences, and completes the test of the machine translation system. The purpose of the present invention is to solve the problem that current machine translation testing mainly generates test cases by replacing some words in the sentence, and the test performance is mainly limited by the maturity of the language model used. While amplifying the data, the invariance of the basic structure of the sentence is guaranteed, thereby discovering more errors, many of which could not be discovered by previous machine translation testing technologies.

Description

Technical Field

[0001] The present invention belongs to the field of machine translation in information technology, and is particularly suitable for machine translation testing in machine translation. Its purpose is to generate test sentences in machine translation testing, and is a translation machine testing method that can generate a large number of test sentences. Background Art

[0002] Machine translation, the task of converting one natural language into another using computers, is a hot topic in artificial intelligence research. In recent years, with the advancement of deep learning, neural machine translation models based on sequence-to-sequence structures have surpassed statistical machine translation models in translation tasks across multiple language pairs and are widely used in commercial translation systems. While the actual application results of commercial translation systems clearly demonstrate significant performance improvements for neural machine translation models, they also suffer from more significant robustness issues compared to statistical machine translation models.

[0003] In recent years, machine translation software has experienced rapid growth, driven by the development of deep neural networks. People frequently use machine translation software in daily life, such as ordering food at foreign restaurants, receiving medical diagnoses and treatment from foreign doctors, and reading international political news online. However, due to the complexity and intractability of the underlying neural networks, modern machine translation software remains far from robust. Similar to traditional software (such as web servers), the reliability of machine translation software is crucial. However, due to the complexity of the neural networks that drive these systems, modern translation software can return incorrect translations, leading to semantic misunderstandings, financial losses, threats to personal safety and health, and political conflict. Existing machine translation testing techniques mostly generate test cases by replacing a single word in a sentence using a language model. While these methods can detect some translation errors in specific sentences, their performance is limited by the maturity of existing language models. Therefore, a translation testing method based on syntax tree pruning has significant research significance and practical value.

[0004] In traditional systems, decision logic is embedded in the source code. In contrast, the output of a DNN system depends primarily on millions of parameters, which are optimized after training. The neural network models used in neural machine translation systems lack interpretability and comprehensibility. During neural machine translation, the sentence to be translated is converted into a multidimensional vector within the neural network structure. This conversion involves complex steps and numerous parameters, making it difficult to understand the actual meaning of each step. Furthermore, neural network models are highly dependent on training data. The same network structure can produce significantly different parameter values when trained with different training datasets, resulting in less stable outputs. Furthermore, recent testing methods for DNN systems have primarily focused on models with a small number of possible outputs (such as image classifiers). Enumerating all possible outputs is a thorny issue in machine translation, making it difficult to test. Furthermore, existing machine translation techniques primarily generate test cases by replacing a single word in a sentence.

[0005] The difficulty of testing neural network models means that white-box testing methods are less feasible, while machine translation primarily uses black-box testing methods. Currently, common methods for solving deep neural network system testing problems can be divided into two categories. The first category is based on difference testing, which determines whether an error has occurred by testing whether the same input has the same output when implemented based on the same protocol. The other category is based on metamorphic testing, which describes the relationship between changes in the system's input and output by defining a metamorphic relationship. In previous work on the quality assessment of neural machine translation systems, methods based on metamorphic testing were more common, and the key to this method lies in the definition of the metamorphic relationship. In response to this, the present invention proposes a translation machine testing method based on syntax tree pruning, which tests the translation machine based on the assumption that the new sentence generated by eliminating irrelevant information should retain the core semantic information of the original sentence. Summary of the Invention

[0006] The present invention solves the current problem of difficulty in translation machine testing by proposing a translation machine testing method based on syntax tree pruning, thereby helping machine translation systems improve translation quality and reduce the possibility of translation errors, thereby providing better translation quality for the public.

[0007] To achieve the above objectives, the present invention proposes a translation machine testing method based on syntax tree pruning. First, pruned sentences are generated. For each unlabeled sentence, the present invention generates a list of new sentences by dependency syntax tree pruning; secondly, filtering by grammatical and semantic information. We filter out new sentences whose similarity (with their source sentences) is less than a given threshold, which indicates the degree of retention of relevant content; then collect target sentences. We feed the source sentences and the newly generated sentences back to the machine translation system being tested, and collect their target sentences. Finally, translation errors are detected, and the structure of the translated sentences is compared with the structure of the translated original sentences. If there is a large difference between the structures, a potential error is reported. Specifically, the method includes the following steps:

[0008] 1) Read the source sentence for machine translation from the file, perform grammatical analysis on the sentence, build a grammar tree through a neural network-based parser, output basic dependency grammar analysis, and build a grammar tree based on the dependency grammar analysis;

[0009] Here are the steps:

[0010] 1a) Use Stanford CoreNLP from Stanford University to parse the text sentences and convert the input sentences into dependency triples. Stanford dependencies represent the grammatical relationships between words in a sentence. They are designed to be easy to understand and effective for those who want to extract textual relationships. Universal dependencies (UDs) are triples consisting of: relation name, regulator, and dependency.

[0011] 2b) Based on the common dependencies, we use the tree library from Python's treelib library to construct a dependency syntax tree based on the common dependencies. Each node in the tree consists of the current word, its relationship with the parent node, and a leaf node.

[0012] 2) Scan the sentence to determine which sentence type it belongs to, and whether it is a simple sentence, a complex sentence, or a compound sentence; split the compound sentence into simple sentences, and retain the non-operational dependencies;

[0013] There are five basic sentence types: SV, SVC, SVO, SVOO, and SVOC. These components are set as inseparable content.

[0014] SV stands for Subject + Verb. A subject-verb structure refers to a sentence consisting of one or more subjects and one or more predicates. A subject-verb sentence consists of two components that represent the relationship between the statement and the statement being made. A sentence in which the statement being made is the subject and the subject-verb phrase serves as the predicate is called a subject-verb-predicate sentence.

[0015] SVC stands for Subject + Link.V (copula) + predicate. The Subject-Link-V structure refers to an English sentence consisting primarily of a subject, a linking verb, and a predicate. The subject is the central element of the sentence, while the linking verb has a meaning and cannot stand alone. The predicate is used to modify the subject.

[0016] SVO, Subject (subject) + Verb (predicate) + Object (object). The grammatical order is subject-predicate-object structure.

[0017] SVOO, Subject + Verb + Indirect object + Direct object. When there are two objects in a sentence, the one that refers to a thing or event is the direct object. The one that refers to a person (or animal) is the indirect object. The indirect object is the thing or person affected, while the direct object is the recipient of the action. For example, in "He passed me the ball," "me" is the indirect object, and "the ball" is the direct object.

[0018] SVOC stands for Subject + Verb + Object + Complement. Object complements refer to transitive verbs in a sentence whose meaning is incomplete even with an object. They require another sentence component to further explain the object's meaning, status, etc. This is often referred to as an object complement. The object and its complement form a compound object. The first part of a compound object is usually a noun or pronoun, while the second part expresses the action, identity, or characteristics of the first noun or pronoun, known as the object complement.

[0019] Here are the steps:

[0020] 1a) Using Stanford CoreNLP from Stanford University, we perform component analysis on sentences in a text. A component parse tree breaks down a text into phrases. Non-leaf nodes in the tree represent phrase types, while leaf nodes represent words in a sentence. Edges are unlabeled. By identifying non-leaf nodes, we determine whether the sentence type is simple, complex, or compound.

[0021] 2b) Based on the component analysis results, split the complex sentence into simple or complex sentences to avoid multiple occurrences of S, V, O, and C. Based on common dependencies, set S, V, O, and C-type words as non-deletable components.

[0022] 3) Prune the dependency syntax tree from bottom to top, and remove the content on the right side of the dependency relationship. At the same time, STP needs to refer to the mapping table and determine whether a cascade pruning should be performed until only the basic dependency syntax tree structure remains;

[0023] 1a) If a sentence with a parallel relationship is encountered, it is split into multiple independent sentences, placed in the expansion result list, and a new dependency syntax tree is generated. It enters the new Prune function for pruning.

[0024] 2b) Enter the Prune function, prune the dependency syntax tree of the statement, and determine whether there should be a cascade pruning according to the reference mapping table, and whether there are sentences with parallel relationships.

[0025] 3c) If pruning is possible, prune the branches and put the results into the augmentation result list.

[0026] 4) Translate the original sentence and the augmented sentence, filter out potential translation errors based on the bag-of-words distance, and manually annotate them. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flowchart of the translation machine testing method based on syntax tree pruning

[0028] Figure 2 for Figure 1 Constructing a syntax tree flowchart in

[0029] Figure 3 for Figure 1 Flowchart of statement type analysis and preprocessing in annotations

[0030] Figure 4 for Figure 1 Flowchart of statement pruning in annotations DETAILED DESCRIPTION

[0031] In order to better understand the technical content of the present invention, specific examples are given and described below with reference to the accompanying drawings.

[0032] Figure 1 The present invention is a flowchart of a translation machine testing method based on syntax tree pruning.

[0033] Construct a syntax tree: Use the relevant syntax analysis library to obtain dependency syntax structure information, extract the semantic trunk and related semantic components, and construct a tree storage structure from the triples.

[0034] Sentence pattern detection: Based on the dependency syntax tree extracted from the source sentence, sentence pattern detection is performed on the dependency syntax tree. In linguistics, sentence pattern refers to the basic types of sentences, from which other types are derived. There are five basic sentence types: SV, SVC, SVO, SVOO, and SVOC.

[0035] Syntax tree pruning: Based on the dependency syntax tree generated from the source sentence, nodes in the tree are deleted according to specific rules to find the nodes that can be deleted. The deletion method is to delete the nodes with the highest level first. This method can obtain more augmented sentences.

[0036] Sentence screening: Potentially incorrect sentences are screened based on the bag-of-words distance, with sentences with the largest distance selected as potentially incorrect. Bag-of-words distance refers to a simple representation method based on the bag-of-words model. It ignores grammar and even word order, while preserving polysemy. The bag-of-words distance compares the number of distinct elements in the bag-of-words vector, resulting in the bag-of-words distance.

[0037] Manual inspection: After screening, it is still not certain whether there are errors in the translated sentences. Professionals (people majoring in relevant languages) are needed to screen the sentences to determine whether the sentences are legal and correct.

[0038] Figure 2 The flowchart for constructing the syntax tree mainly involves converting the basic universal dependency triples into a tree data structure. The specific steps are as follows:

[0039] Step 1: Use Python's basic IO method, the read function, to read the file content to be read and determine whether there are any illegal characters. If any illegal characters are encountered, an exception is thrown and waits for manual processing. Otherwise, all read statements are put into a list and wait for syntax parsing.

[0040] Step 2: Use Stanford CoreNLP, a natural language processing tool from Stanford University, to import Stanford CoreNLP and perform syntax analysis on the sentence to obtain the universal dependency (UD) triples: relation name, regulator, and dependency.

[0041] Step 3: Use Python's treelib library to build tree structure data based on common dependency triples and save it in the dictionary of the corresponding statement.

[0042] Figure 3 This is a flow chart for sentence pattern detection. The main detection process is to determine which sentence pattern it belongs to and whether it is a simple sentence, a complex sentence, or a compound sentence. The specific steps are as follows:

[0043] Step 1: Sentence component analysis: Use the Parse method in StanfordCoreNLP, a natural language processing tool from Stanford University, to convert the source sentence into a string of type String. Then, use the fromstring method in the tree package in the Python Nltk library to convert the results of the sentence component analysis into a tree storage structure.

[0044] Step 2: Determine simple sentences: Traverse the tree storage structure of sentence component analysis and follow the following rules to convert complex sentences into simple sentences while retaining inseparable components. The specific rules are as follows:

[0045] A1: Identify nodes labeled SBAR. If the number of leaf nodes of the node is greater than 1, it is considered that there is a clause. Split the sentence into two sentences: a sentence without SBAR nodes and a sentence with only SBAR nodes, and store them in the corresponding data lists.

[0046] A2: Identify the node with the label HYPH. If the number of leaf nodes of the node is greater than 1, it is considered to be an inseparable word and is placed in the inseparable list.

[0047] Step 3: Determine the sentence structure: Use the Depd_Travesal function to set the corresponding main clause components as unmodifiable parts according to the dependency syntax tree of the sentence. The rules are as follows:

[0048] A: Tags are the following names that are considered to be the main components of the sentence: ROOT, cop, obj, dobj, iobj, nsubj, xsubj

[0049] Figure 4 This is a flowchart for pruning a dependency syntax tree. The dependency syntax tree is pruned from bottom to top, and the content on the right side of the dependency relationship is deleted. The specific steps are as follows:

[0050] Step 1: Parallel sentence determination: traverse the dependency syntax tree to determine whether there are parallel components and split the parallel sentence into non-parallel sentences. The specific rules are as follows:

[0051] A1: Read the node dependencies, which are tags: begin, end. Identify nodes with tags conj and cc. When conj appears, consider the node to be a parallel component and add it to a dictionary with begin as the key and end as the content. When cc appears, consider the begin of cc's begin as the key in the dictionary and set the content as the key.

[0052] For example: (cc, N1, N2), (conj, N3, N1), the dictionary corresponding to cc is [n3] = N2.

[0053] A2: After reading the parallel components, determine the parallel components and generate non-parallel statements. The specific rules are as follows:

[0054] The parallel components should be cop, obj, dobj, iobj, nsubj, and xsubj. If they are basic components, modify the syntax tree, set the parallel components as new essential components, and delete the other parallel components.

[0055] When the parallel component should be root, that is, a verb, adjust the syntax tree to modify root into a parallel component and delete other parallel components.

[0056] The parallel components are other components. Modify the syntax tree and delete the other parallel components.

[0057] A3: Read the statement to cancel the parallelization and enter the pruning function. The node deletion rules are as follows:

[0058] The tags are root, cop, obj, dobj, iobj, nsubj, and xsubj. They are basic components and cannot be deleted.

[0059] The tags are dep, aux, auxpass, ccomp, xcomp, expl, det, mwe, mark, nummod, prt, goeswith, and case, which are cascade components and need to be deleted in conjunction with the parent node.

[0060] If the label is of other types, the node can be directly deleted, the subtree with the node as the root is directly deleted, and the newly generated statement is added to the expansion list.

[0061] Even if the label is of other types, if the child node contains non-deletable nodes, it cannot be deleted.

[0062] A4: Store the data, translate the original sentence and the augmented sentence, calculate the bag-of-words distance, and output the candidate errors according to the set number of candidate errors to Excel.

[0063] Finally, seek out relevant professionals for review.

[0064] The following examples illustrate the effects of the present invention:

[0065] We tested machine translation using CNN news data, including 100 sentences from political news and 100 sentences from economic news. Our testing revealed 214 errors in Google Translate and 242 in Bing Translate. Examples of the errors we found are as follows:

[0066] Source sentence: Young people of color became a majority of K12 public schoolstudents in 2014.

[0067] In 2014, people of color became the majority of K-12 public school students.

[0068] Purpose sentence:

[0069] Young people became a majority in 2014.

[0070] In 2014, young people became the majority.

[0071] We found that the word Young was not translated in the original sentence, which means that the translation machine missed the word Young, which is an incorrect translation result.

[0072] The following are examples of splitting parallel sentences into non-parallel sentences:

[0073] Source sentence: The old rule started to seem dated and out of place.

[0074] The old rules are starting to look outdated and outdated.

[0075] Purpose sentence:

[0076] The old rule started to seem dated.

[0077] The old rules seem outdated.

[0078] The old rule started to seem out of place.

[0079] The old rules are starting to seem wrong.

[0080] The source sentence was split into two parallel sentences. While ensuring legality, a new sentence was generated. The incorrect translation of the source sentence in the machine translation was discovered, and the outdated translation results were repeated twice.

Claims

1. A defect detection method for a translation machine, characterized in that First, a dependency syntax tree is constructed for the sentence. Specific rules are then designed to prune the syntax tree, removing words or phrases from the original sentence to generate new grammatically and semantically valid sentences. The original text and the newly generated sentences are then fed into the machine translation system being tested. Finally, the bag-of-words distance is calculated, and the sentences are expanded by sorting them according to their bag-of-words distance. The five sentences with the largest distances are selected, and the original and translated sentences are manually labeled, marking any incorrect sentences. The specific steps of this method are as follows: 1) Read the source sentence for machine translation from the file, perform syntactic analysis on the sentence, and build a syntax tree using the neural network-based parser provided by StanfordCoreNLP; 2) Perform sentence pattern detection to identify simple, complex, and compound sentences, and segment compound sentences into simple sentences. Finally, identify and retain basic sentence structures, including subject-verb-complement, subject-verb, subject-verb-object, subject-verb-object-object, and subject-verb-object-complement structures, and set them as inoperable dependencies; 3) Prune the dependency syntax tree from bottom to top and delete the content on the right side of the dependency relationship; at the same time, STP needs to refer to the mapping table and determine whether there should be a cascade pruning process until only the basic dependency syntax tree structure remains; 4) Filter potential erroneous sentences according to the bag-of-words distance and manually label them.

2. The defect detection method for a translation machine according to claim 1, characterized in that: In step three, the dependency syntax tree is pruned from bottom to top, and the content on the right side of the dependency relationship is deleted. This method designs a relationship mapping table and, based on the corresponding relationship type, designs related basic operations, including whether to directly delete or cascade delete, split operations, etc. The statements are pruned according to the mapping table, and the pruned code is stored in the expansion list.

Images