Automatic speech translation test method based on semantic analysis

Abstract

A semantic analysis-based automatic test method for a speech translation system is characterized by comprising the following steps of: acquiring a syntactic structure of a sentence by using syntactic analysis, and constructing a metamorphic relationship; according to the method, the performance of the speech translation model can be evaluated without reference texts or additional annotation information, the problems that a current speech translation test method depends on the reference texts, and test oracles are difficult to construct are mainly solved, and a new concept and syntactic structure invariance are provided. A dependency structure and a component structure are often used for describing a syntactic structure of a sentence, and according to definitions in linguistics, a modifier is an optional modification part and cannot strongly influence the syntactic structure of the sentence. The method comprises the following steps of: generating an audio amplification template by using a text test case generation tool and a phoneme alignment tool; a test audio is generated by using noise injection and audio splicing, a sentence structure is expressed by using a syntactic structure analytic tree, and translation errors are detected based on syntactic structure invariance.

Description

Technical Field

[0001] The present invention belongs to the field of software testing, and in particular to speech translation testing and automated testing methods. It is a new attempt to perform automated testing on a speech translation system based on a semantic analysis method. Different from the previous speech translation system testing methods, this method uses the semantic analysis method in the field of natural language processing, the fuzzy testing concept and the metamorphosis testing method in the field of software testing to solve the problems of speech translation testing relying on annotation content, low accuracy, incomprehensible internal parameters, diverse output results, and difficulty in constructing test predictions. Background Art

[0002] Speech translation technology can process speech signals in one language into text in another language. With the maturity of machine learning methods (such as deep learning), speech translation based on neural networks has become mainstream. The core component of the current mainstream speech translation system is its internal deep neural network model. The traditional speech translation system is mainly a cascade model composed of a speech recognition model and a machine translation model. In recent years, with the successful application of the Transformer framework in many fields, researchers have proposed a method to directly build an end-to-end speech translation system. Currently, many suppliers have launched speech translation services, such as Google Translate, Youdao Translation, iFlytek, etc. For many people, translation functions are indispensable in daily life.

[0003] Therefore, the robustness of speech translation systems has received attention. The core component of a deep neural network-driven speech translation system is its internal deep neural network model, the performance of which is highly dependent on the size and quality of the training dataset. Speech translation systems perform poorly under low-resource conditions, but perform better under high-resource conditions. In order to improve the performance of speech translation systems, developers need more comprehensive test data as well as translation error information to measure the impact of scarce training resources on the model. By analyzing these errors, developers can obtain information about the strengths and weaknesses of the current model within the speech translation system to propose improvement measures. Collecting erroneous translations and their corresponding correct translations and supplementing the training set for fine-tuning or retraining is an effective way to improve the performance of speech translation systems. Therefore, many translation service providers have introduced feedback mechanisms that require users to submit incorrect translations or corresponding correct translations that they have found. Identifying these incorrect translation results is the collection of

[0004] By analyzing these results, developers can better identify system defects and propose improvement measures. Youdao Translation and Bing Translation have added feedback interfaces to their translation service pages to facilitate users to provide direct feedback on incorrect translation results. Google has even established a translation community and rewarded contributors to encourage them to provide more valuable feedback.

[0005] In order to analyze and improve the performance of speech translation systems, many researchers have focused on how to design better speech translation testing methods to more efficiently detect speech translation system errors. However, testing speech translation systems remains a challenging task. Unlike traditional software, speech translation systems build their complex internal structures in a data-driven manner, often involving millions of parameter settings, which makes many traditional testing methods unsuitable. In addition, the physical deployment environment of speech translation systems often contains factors such as environmental noise and reverberation. These factors can cause the actual input to deviate from the distribution of the training dataset, leading to unpredictable behavior in application scenarios. Therefore, when designing test methods, it is also necessary to consider the impact of the application environment on speech input in order to evaluate the true performance of the speech translation system. In addition, since the output space of speech translation is huge (for example, the same speech may have multiple correct translation results), it is very challenging to build a test oracle. Previously, some methods for testing speech translation systems have been proposed. Common speech translation systems can be divided into cascade systems and end-to-end systems. Cascade systems first use a speech recognition model to convert speech into text, and then use a machine translation model to translate the recognized text into the target language. End-to-end systems can directly process speech input to generate translation results. The overall performance of the cascade system can be evaluated by testing the speech recognition module and the translation module respectively through existing speech recognition test methods (such as Asdf and AequeVox) and machine translation test methods (such as Purity and SemMT). However, this test strategy only targets specific modules in the cascade system, ignoring other modules that may cause translation errors, such as speech preprocessing modules, recognition text correction modules, etc. In addition, this test strategy is not suitable for end-to-end systems without intermediate steps, and using end-to-end systems instead of cascade systems is the current trend in the development of speech translation.

[0006] Based on the above background and analysis of related work, this patent proposes an automated testing method for speech translation systems based on semantic analysis to solve the problems of current speech translation testing, such as reliance on annotation content, low accuracy, incomprehensible internal parameters, diversity of output results, and difficulty in constructing test predictions. This patent proposes a new concept, namely, syntactic structure invariance. The syntactic structure here includes component structure and dependency structure. In linguistics, a sentence consists of basic components and adjuncts. The basic components are the core of the sentence, and the adjuncts are optional modifiers that cannot have a strong impact on the syntactic structure of the sentence. This feature indicates that the syntactic structure of a given sentence is usually retained in the syntactic structure of the sentence after the modifier is added, and can be used to reflect the syntactic structure inclusion relationship between the two sentences. This invariance relationship can be mapped to the speech pair containing such a sentence pair. We define this feature as syntactic structure invariance and use it to construct metamorphosis relations to solve the test prediction problem in automated testing of speech translation systems. Summary of the invention

[0007] The problem to be solved by the present invention is that, unlike traditional software systems, speech translation systems based on neural networks do not rely on manually designed rules or business logic, but are automatically learned through a large amount of training data, and the internal logic of the model is constructed with millions of model parameters. This feature makes the behavior of such speech translation systems difficult to explain, and also makes it difficult to effectively apply traditional testing methods. In addition, building test predictions for speech translation systems is also a difficult problem. Since the output space of speech translation is usually very large, it becomes extremely complicated to accurately define test predictions. Some existing evaluation indicators usually rely on these test predictions without considering the impact of the diversity of speech translation system outputs. Therefore, the present invention aims to solve the problems faced by current speech translation systems in testing, such as dependence on annotation content, low accuracy, incomprehensible internal parameters, diversity of output results, and difficulty in building test predictions.

[0008] The technical solution of the present invention is: an automated testing method for a speech translation system based on semantic analysis, characterized in that semantic analysis is used to obtain the syntactic structure of the sentence corresponding to the translation result of the audio, and a metamorphosis relationship is constructed based on the invariance of the syntactic structure, so that the performance of the speech translation system can be evaluated without reference to text. The input of this method is a set of unlabeled single-language audios, each of which contains a sentence. The output is a list of suspected error reports. Each error report contains three parts: 1) a sentence s contained in the audio and its translation T; 2) a sentence s′ contained in a variant of the audio and its translation T′. 3) a syntactic structure analysis tree path that causes the syntactic structure invariance to be destroyed.

[0009] The method consists of the following three modules / steps:

[0010] 1) Generate a template for audio amplification: Figure 1 As shown in step (1) in FIG. 1 , for a given audio S, the method first extracts the basic structural part (subject, predicate, object or subject, predicate complement) and the attached words of the corresponding sentence s from S through the speech decomposition method, and determines the start and end time of each word in the sentence, thereby decomposing the speech block from the speech and generating a template. Each template consists of a basic speech (the audio containing the basic structural part) and corresponding slot positions, which record the start and end time of the speech block. Finally, the generated template contains the basic speech S0 and multiple slots l1, l2, ..., l n , these slots are used to splice corresponding speech blocks.

[0011] 2) Test audio assembly: This method generates a set of test audio based on the template generated in the previous stage for testing the speech translation system, such as Figure 1 As shown in step (2) in the figure. For convenience, we use a j Represents the corresponding slot l j First, based on the basic voice and slot positions in the template, this method assembles the test voice by splicing the voice blocks into the basic voice. In order to enhance diversity and connect with actual application scenarios, the voice transformation operation is used to generate additional voices in the process. For the basic voice S0, we can get voice S after the voice transformation operation. 1.0 , S 2.0 ,...,S n.0 Then, the method sorts the speech blocks in the template according to the start and end time sequence to form an ordered sequence of speech blocks. i.0 , we can get from S i.0 To S i.n A set of audio. Among them, S i.j Indicates S i.0 The jth generation. Whenever a new speech S is generated i.j When , this method will get the first speech block a in the current sequence j , and obtain its i.j Find the corresponding slot l j Then, this method will a j and S i.j Synthesize into the next generation of speech S i.j+1 .

[0012] 3) Obtain translation results and detect translation errors: Figure 1As shown in step (3), this method uses the speech translation service API provided by various translation platforms to obtain translations, such as Google Translate, Youdao Translation, iFLYTEK Translation, etc. In this method, the source language is English and the template language is Chinese. By calling the API and giving the language setting and the audio file path, a return value containing the translation result can be obtained. This method uses syntactic structure invariance to detect translation errors. In linguistics, dependency structure and component structure are often used to describe the syntactic structure of a sentence, where the component structure reflects the whole-part relationship between the sentence and one of its direct components, while the dependency relationship represents the part-part relationship between pairs of words. Figure 2 A specific example is given to illustrate the difference between constituent structure and dependency structure. Dependency focuses on word grammar (e.g., "fish" is the object of "eats"), while constituent structure focuses on phrase structure grammar (e.g., the verb phrase "eatsfish" consists of the verb "eats" and the noun "fish"). In linguistics, adjuncts are optional modifying parts that cannot have a strong impact on the syntactic structure of a sentence. This feature indicates that the syntactic structure of a given sentence is usually retained in the syntactic structure of the sentence after the modifier is added, and can be used to reflect the syntactic structure inclusion relationship between two sentences. This method defines this feature as syntactic structure invariance. And translation errors are detected based on syntactic structure invariance. This method implements a parse tree path matching algorithm, which sequentially matches a pair of parent-child audio (S i ,S i+1 )'s translation results (T i ,T i+1 ) performs path matching of dependency structure parse tree and component structure parse tree, and combines the matching results of the two parse trees to determine whether there is a translation error. The specific design of the matching function is as follows Figure 3 as shown.

[0013] The present invention is characterized in that:

[0014] 1. Propose a new concept, syntactic structure invariance, which can be used to detect speech translation errors.

[0015] 2. Based on the sentence compression model and speech transformation method, a new audio amplification method is proposed.

[0016] 3. Using syntactic structure invariance and audio amplification, we implement the automatic testing method SPlice for speech translation system based on syntactic structure analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The overall flow chart for the implementation of the present invention.

[0018] Figure 2 This is an example diagram of the selection parse tree and dependency parse tree in key step 3.

[0019] Figure 3 Design diagram for the matching function in key step 3. DETAILED DESCRIPTION

[0020] This method realizes an automatic testing method of speech translation system based on syntactic structure analysis, which mainly adopts text test case generation technology, phoneme alignment technology, speech transformation technology, syntactic structure analysis technology, etc.

[0021] 1. Generate template

[0022] In the present invention, firstly, a template for guiding the generation of test audio needs to be generated. The template includes a basic voice and multiple slots. The basic voice includes the basic structural part of the sentence in the input audio, that is, the subject-verb-object or subject-verb-complement. The slot is used to splice the voice block containing the attached words. In order to obtain the template from the input audio, the method first extracts the basic structural part and attached words of the sentence in the seed voice through NLPLego. NLPLego is a text test case generation tool designed based on the sentence parsing process in classical linguistics, which can decompose the sentence into a template containing the basic structural part and additional components. Next, the method uses the phoneme alignment technology to determine the start and end time of each word in the sentence, so as to align the seed voice and text in time. This method chooses to use Montreal Forced Aligner (MFA) performs phoneme alignment. MFA is a Kaldi-based tool for accurately aligning audio and corresponding text. It is widely used in speech data processing and automatic speech recognition tasks. Finally, through the results of NLPLego and MFA, this method constructs a template containing basic speech and slot positions in this step, records the start and end time of the speech block, and thus generates a complete speech template. Below is a display of the generated template, where each sentence or phrase represents the corresponding speech block.

[0023]

[0024] 2. Generate test audio

[0025] This section mainly introduces the specific design and implementation method of test audio generation. In order to make the test audio conform to the actual application scenario of the speech translation system and ensure the validity of the phoneme alignment results, this method chooses to inject environmental noise into the generated audio as the implementation of the speech transformation operation. Environmental noise refers to the sounds generated by industrial production, transportation and social life. In the test speech assembly process, this method chooses to use the noise library NOISEX-92, a noise library containing a variety of environmental sounds, to inject noise into the generated speech. In addition, this method chooses to use the pydub library for audio processing as the technical implementation of the test audio generation method. pydub is a Python library designed for audio file operations and provides a convenient interface for performing various speech operations including format conversion, segmentation, volume adjustment, merging, etc. In order to maintain the quality of the generated speech, we perform additional processing on each speech during the assembly stage. According to the position of punctuation marks in the corresponding text, we adjust the rhythm of the speech and add appropriate pauses; at the same time, during the splicing process, the audio at the splicing point is smoothed to make it more fluent and natural.

[0026] At the same time, in order to ensure the intelligibility and high quality of the generated speech, we selected two evaluation indicators: short-time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ). STOI is an important measure of speech intelligibility, with a value range of 0 to 1, where 1 means that the speech is completely understandable and 0 means that it is incomprehensible. PESQ is one of the most commonly used speech quality evaluation indicators, with a value range of -0.5 to 4.5. The higher the index, the better the speech quality. In order to generate as many test cases as possible while maintaining semantic integrity and high speech quality, we selected STOI$>0.98$ and PESQ$>2$ as thresholds after testing different values. This is because we hope that the generated test audio maintains high intelligibility and is as close to the real application scenario as possible. In reality, the input audio is often accompanied by some environmental noise, which makes the speech translation system should be able to accept some input audio with not so high quality and give accurate translation. Finally, in this step, the method can generate multiple groups of test audio according to the given input audio through noise injection and splicing of attached word audio blocks.

[0027] 3. Translation error detection

[0028] This section describes how to apply syntactic structure invariance to speech translation error detection, as well as the specific implementation of the speech translation error detection method. For the translation results of the test speech, this method chooses to use Stanford CoreNLP to extract its component analysis and dependency analysis results, and converts these results into the format of a syntactic tree. Stanford CoreNLP is a natural language processing toolkit for linguistic analysis of text data, supporting component analysis and dependency analysis of multiple languages. In the component analysis results, we use the root node to represent the entire sentence, the leaf node to represent the word in the sentence, and the internal node to represent the phrase, thereby generating a component analysis tree of the translation results. In the dependency analysis results, the nodes in the syntactic tree represent the words in the sentence, and the edges connecting the nodes represent the dependency relationship between the two words. For each pair of translation results (T i ,T i+1 ), we compare the component syntax trees and dependency syntax trees generated from the translation results to determine whether the syntactic structure invariance is preserved.

[0029] In the component syntax tree, this method first obtains all component paths from the root node to the leaf node. Each path shows the syntactic decomposition process from the root to the leaf, providing information about different levels of the component structure. In the dependency syntax tree, each dependency path only contains the dependency relationship stored on the edge, and omits the word information stored on the node. This method inputs the component path and dependency path list into the designed matching function to determine whether there is an error path. An error path refers to a path that violates the transformation relationship. In the path comparison process, we check whether there is a path in the target path in which each node appears in the original path in sequence.

[0030] This method uses a matching algorithm to compare the component syntactic parse trees and dependency syntactic parse trees of the parent-child speech pair respectively. Error paths may appear in one or both comparison processes, and in this method, these error paths are considered to represent potential errors in the speech translation results. However, during the implementation process, errors in syntactic tree construction may also introduce error paths, even if the translation is correct. For example, components marked as FW in component analysis results or dependencies marked as dep in dependency analysis results may indicate problems in the Stanford CoreNLP analysis process. In this case, when one type of syntactic tree has an error, we will give priority to the comparison results of another type of syntactic tree to improve the accuracy of the output results.

Claims

1. An automated testing method for a speech translation system based on semantic analysis, characterized in that Use syntactic analysis to obtain the syntactic structure of the sentence and construct the metamorphosis relationship, so as to evaluate the performance of the speech translation system without the need for reference text; this method takes a set of unlabeled single-language audios as input, each of which contains a complete sentence, and outputs a list of suspected translation errors detected in the translation results. The specific steps are as follows: 1) Generate templates: Use text test case generation technology and phoneme alignment technology to extract the basic structure parts (subject, predicate, object or subject, predicate complement) and dependent words of the corresponding sentence s from each input audio S, and determine the start and end time of each word in the sentence, so as to decompose the speech block from the speech and generate templates; each template consists of basic speech (including the audio of the basic structure part) and corresponding slot positions, which record the start and end time of the speech block corresponding to the dependent word; 2) Generate test audio: This method generates a set of test audio based on the template generated in the previous stage for testing the speech translation system, and assembles the test speech by splicing the attached word speech block into the basic speech; the present invention uses the speech transformation operator to generate multiple variants of the template; for each variant, the attached word speech block is inserted in sequence to generate multiple sets of test audio; 3) Obtaining translation results and detecting translation errors: Using the API provided by the translation platform to obtain translations, where the source language is set to English adapted by this method, and the target language is set to Chinese adapted by this method; When calling the API, complete the language setting and the path of the audio file to be translated, and you can get a return value containing the translation result; this method uses syntactic structure invariance to detect translation errors. Based on the syntactic structure invariance detection, this method implements a parse tree path matching algorithm, which performs syntactic structure parse tree path matching on the translation results (Ti, Ti+1) of the parent and child audio (Si, Si+1) in turn, and determines whether there is a translation error based on the matching results.

2. The method for automatic testing of a speech translation system based on semantic analysis according to claim 1 is characterized in that It uses text test case generation technology, phoneme alignment technology, voice transformation technology, syntactic structure analysis technology, etc.

3. The method for automatic testing of a speech translation system based on semantic analysis according to claim 1 is characterized in that Syntactic structurality is used to detect translation errors. In linguistics, the syntactic structure of a given sentence is usually retained in the syntactic structure of the sentence after the addition of modifiers, which can be used to reflect the syntactic structure inclusion relationship between two sentences. The present invention defines this feature as syntactic structure invariance.

4. The method for automatic testing of a speech translation system based on semantic analysis according to claim 1 is characterized in that It is compatible with the voice translation services provided by Google Translate, Youdao Translation and iFLYTEK Translation.

Images