An automated speech translation testing method based on semantic analysis

Abstract

A kind of voice translation system automatic test method based on semantic analysis, characterized by using syntax analysis to obtain the syntax structure of sentence, and build metamorphic relationship, without reference text or additional annotation information can evaluate the performance of voice translation model.The method mainly solves the problems existing in current voice translation test method, such as dependence on reference text and difficulty in constructing test prediction.This invention proposes a new concept, the invariance of syntax structure.In linguistics, dependency structure and constituent structure are often used to describe the syntax structure of a sentence.According to the definition in linguistics, modifier is an optional modification part, which cannot have a strong impact on the syntax structure of the sentence.The steps of the method include using text test case generation tool and phoneme alignment tool to generate audio augmentation template, using noise injection and audio splicing to generate test audio, using syntax structure parse tree to express sentence structure, and detecting translation errors based on the invariance of syntax structure.

Description

Technical Field

[0001] This invention belongs to the field of software testing, specifically relating to speech translation testing and automated testing methods. Automated testing of speech translation systems based on semantic analysis is a novel approach. Unlike previous speech translation system testing methods, this method utilizes semantic analysis from natural language processing, fuzzy testing concepts from software testing, and metamorphic testing methods to address the problems of speech translation testing, such as reliance on labeled content, low accuracy, incomprehensible internal parameters, diverse output results, and difficulty in constructing test predictions. Background Technology

[0002] Speech translation technology can process speech signals from one language into text in another language. With the maturity of machine learning methods (such as deep learning), neural network-based speech translation has become mainstream. The core component of current mainstream speech translation systems is their internal deep neural network model. Traditional speech translation systems mainly consist of a cascaded model combining a speech recognition model and a machine translation model. In recent years, with the successful application of the Transformer framework in many fields, researchers have proposed a method to directly build end-to-end speech translation systems. Currently, many vendors have launched speech translation services, such as Google Translate, Youdao Translate, and iFlytek. For many people, translation functionality is indispensable in daily life.

[0003] Therefore, the robustness of speech translation systems has received considerable attention. The core component of a deep neural network-driven speech translation system is its internal deep neural network model, the performance of which is highly dependent on the size and quality of the training dataset. Speech translation systems perform poorly under low-resource conditions but better under high-resource conditions. To improve the performance of speech translation systems, developers need more comprehensive test data and translation error information to measure the impact of scarce training resources on the model. By analyzing these errors, developers can obtain information about the strengths and weaknesses of the current speech translation system's internal model and propose improvement measures. Collecting erroneous translations and their corresponding correct translations, and supplementing the training set for fine-tuning or retraining, is an effective method to improve the performance of speech translation systems. Therefore, many translation service providers have introduced feedback mechanisms, requiring users to submit incorrect translations or their corresponding correct translations. Identifying these incorrect translations is a process of data collection.

[0004] This is an important way to obtain effective data. By analyzing these results, developers can better identify system flaws and propose improvements. Youdao Translate and Bing Translate have both added feedback interfaces to their translation service pages, allowing users to directly provide feedback on incorrect translation results. Google has even established a translation community and rewards contributors to incentivize them to provide more valuable feedback.

[0005] To analyze and improve the performance of speech translation systems, many researchers have focused on designing better speech translation testing methods to more efficiently detect errors. However, testing speech translation systems remains a challenging task. Unlike traditional software, speech translation systems are built with complex internal structures in a data-driven manner, often involving millions of parameter settings, making many traditional testing methods unsuitable. Furthermore, the physical deployment environment of speech translation systems often includes factors such as environmental noise and reverberation. These factors can cause actual inputs to deviate from the training dataset distribution, leading to unpredictable behavior in application scenarios. Therefore, when designing testing methods, it is also necessary to consider the impact of the application environment on speech input to evaluate the true performance of the speech translation system. In addition, due to the vast output space of speech translation (e.g., the same speech may have multiple correct translations), constructing test predictions is extremely challenging. Previously, several methods for testing speech translation systems have been proposed. Common speech translation systems can be divided into cascaded systems and end-to-end systems. Cascaded systems first use a speech recognition model to convert speech into text, and then use a machine translation model to translate the recognized text into the target language. End-to-end systems can directly process speech input to generate translation results. The overall performance of a cascaded system can be evaluated by testing the speech recognition module and the translation module separately using existing speech recognition testing methods (such as Asdf and AequeVox) and machine translation testing methods (such as Purity and SemMT). However, this testing strategy only targets specific modules in a cascaded system, ignoring other modules that may cause translation errors, such as speech preprocessing modules and recognized text correction modules. Furthermore, this testing strategy is not suitable for end-to-end systems without intermediate steps, and using end-to-end systems instead of cascaded systems is the current trend in speech translation development.

[0006] Based on the above background and analysis of related work, this patent proposes an automated testing method for speech translation systems based on semantic analysis to address the problems of current speech translation testing, such as reliance on labeled content, low accuracy, incomprehensible internal parameters, diverse output results, and difficulty in constructing test predictions. This patent introduces a new concept: syntactic structure invariance. Here, syntactic structure includes constituent structure and dependency structure. In linguistics, a sentence consists of basic constituents and auxiliary words. Basic constituents are the core of the sentence, while auxiliary words are optional modifiers that do not strongly influence the sentence's syntactic structure. This feature indicates that the syntactic structure of a given sentence is usually preserved in the syntactic structure of the sentence after its modifiers are added, which can be used to reflect the syntactic structure inclusion relationship between two sentences. This invariant relationship can be mapped to speech pairs containing such sentence pairs. We define this feature as syntactic structure invariance and use it to construct metamorphic relations to solve the test prediction problem in automated testing of speech translation systems. Summary of the Invention

[0007] The problem this invention aims to solve is that, unlike traditional software systems, neural network-based speech translation systems do not rely on manually designed rules or business logic. Instead, they learn automatically through massive amounts of training data and construct their internal logic using millions of model parameters. This characteristic makes the behavior of such speech translation systems difficult to interpret and also makes traditional testing methods ineffective. Furthermore, constructing test predictions for speech translation systems is also a challenge. Because the output space of speech translation is typically very large, accurately defining test predictions becomes extremely complex. Existing evaluation metrics often rely on these test predictions without considering the impact of the diversity of speech translation system outputs. Therefore, this invention aims to address the problems currently faced by speech translation systems in testing, including reliance on labeled content, low accuracy, incomprehensible internal parameters, diverse output results, and the difficulty in constructing test predictions.

[0008] The technical solution of this invention is: an automated testing method for a speech translation system based on semantic analysis. Its characteristic is that it uses semantic analysis to obtain the syntactic structure of the translated sentence corresponding to the audio, constructs metamorphic relations based on syntactic structure invariance, and can evaluate the performance of the speech translation system without reference text. The input of this method is a set of unlabeled monolingual audios, each containing a sentence. The output is a list of suspected error reports. Each error report contains three parts: 1) the sentence s contained in the audio and its translation T; 2) the sentence s′ contained in the variant of the audio and its translation T′; 3) the syntactic structure analysis tree path that caused the violation of syntactic structure invariance.

[0009] This method comprises the following three modules / steps:

[0010] 1) Generate templates for audio amplification: such as Figure 1 As shown in step (1), for a given audio S, this method first extracts the basic structural parts (subject-verb-object or subject-verb-complement) and auxiliary words of the corresponding sentence s from S through speech decomposition, and determines the start and end times of each word in the sentence, thereby decomposing the speech blocks from the speech and generating templates. Each template consists of basic speech (containing the audio of the basic structural parts) and corresponding slot positions, which record the start and end times of the speech blocks. Finally, the generated template contains the basic speech S0 and multiple slots l1, l2, ..., l n These slots are used to splice together corresponding voice blocks.

[0011] 2) Test Audio Assembly: This method generates a set of test audio based on the template generated in the previous stage to test the speech translation system, such as... Figure 1 As shown in step (2). For ease of representation, we use a. j Represents the corresponding slot l j The method assembles test speech by concatenating speech blocks into the basic speech based on the basic speech and slot positions in the template. To enhance diversity and align with real-world applications, speech transformation operations are used to generate additional speech during the process. For the basic speech S0, we can obtain speech S after the speech transformation operation. 1.0 S 2.0 ,...,S n.0 Then, this method sorts these speech blocks sequentially according to the start and end times recorded in the template, forming an ordered sequence of speech blocks. For each speech S... i.0 We can obtain from S i.0 To S i.n A set of audio. Among them, S i.j S represents i.0 The jth generation. Whenever a new speech S is generated. i.j At that time, this method will obtain the first speech block 'a' in the current sequence. j and obtain its S i.j Find the corresponding slot l in the middle. j Then, this method will a j and S i.j Synthesized into next-generation speech S i.j+1 .

[0012] 3) Obtain translation results and detect translation errors: such as Figure 1As shown in step (3), this method uses the speech translation service APIs provided by various translation platforms to obtain translations, such as Google Translate, Youdao Translate, and iFlytek Translate. In this method, the source language is English and the template language is Chinese. By calling the API and providing the language settings and audio file path, a return value containing the translation results can be obtained. This method uses syntactic invariance to detect translation errors. In linguistics, dependency structure and constituent structure are often used to describe the syntactic structure of a sentence. Constitutive structure reflects the whole-part relationship between a sentence and one of its direct constituents, while dependency relationship represents the part-part relationship between pairs of words. Figure 2 A concrete example is given to illustrate the difference between constituent structure and dependency structure. Dependency relations focus on word-level grammar (e.g., "fish" is the object of "eats"), while constituent structure focuses on phrase-level grammar (e.g., the phrasal phrase "eatsfish" consists of the verb "eats" and the noun "fish"). In linguistics, adjuncts are optional modifiers that do not strongly influence the syntactic structure of a sentence. This feature indicates that the syntactic structure of a given sentence is usually preserved in the syntactic structure of the sentence after its modifiers are added, and can be used to reflect the syntactic inclusion relationship between two sentences. This method defines this feature as syntactic structure invariance and detects translation errors based on syntactic structure invariance. This method implements a parse tree path matching algorithm, sequentially matching a pair of parent-child audio sequences (S... i ,S i+1 Translation results (T) i ,T i+1 The dependency structure parse tree and the constituent structure parse tree are used for path matching, and the matching results of the two parse trees are combined to determine whether translation errors exist. The specific design of the matching function is as follows: Figure 3 As shown.

[0013] The features of this invention are:

[0014] 1. A new concept, syntactic structure invariance, is proposed, which can be used to detect errors in speech translation.

[0015] 2. Based on sentence compression models and speech transformation methods, a new audio augmentation method is proposed.

[0016] 3. SPlice, an automated testing method for speech translation systems based on syntactic structure analysis, is implemented by utilizing syntactic structure invariance and audio amplification. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the overall implementation of the present invention.

[0018] Figure 2 This is an example diagram of the selection parse tree and dependency parse tree in key step 3.

[0019] Figure 3 This is the flowchart for the matching function in key step 3. Detailed Implementation

[0020] This method implements an automated testing approach for speech translation systems based on syntactic structure analysis. It mainly employs text test case generation technology, phoneme alignment technology, speech transformation technology, and syntactic structure parsing technology.

[0021] 1. Generate template

[0022] In this invention, a template is first generated to guide the generation of test audio. The template contains basic speech and multiple slots. The basic speech includes the basic structural parts of sentences in the input audio, i.e., subject-verb-object or subject-verb-complement. The slots are used to concatenate speech blocks containing auxiliary words. To obtain the template from the input audio, this method first extracts the basic structural parts and auxiliary words of sentences from the seed speech using NLPLego, a text test case generation tool designed based on the sentence parsing process in classical linguistics, capable of decomposing sentences into templates containing basic structural parts and auxiliary components. Next, this method uses phoneme alignment technology to determine the start and end times of each word in the sentence, so as to align the seed speech with the text in time. This method uses Montreal Forced... Aligner (MFA) performs phoneme alignment. MFA is a Kaldi-based tool used for precise phoneme alignment between audio and corresponding text, and is widely used in speech data processing and automatic speech recognition tasks. Finally, using the results of NLPLego and MFA, this method constructs a template containing basic speech and slot positions in this step, recording the start and end times of speech blocks, thereby generating a complete speech template. Below is a demonstration of the generated template, where each sentence or phrase represents the corresponding speech block.

[0023]

[0024] 2. Generate test audio

[0025] This section mainly introduces the specific design and implementation method of test audio generation. To ensure the test audio conforms to the actual application scenario of the speech translation system and to guarantee the effectiveness of phoneme alignment results, this method chooses to inject environmental noise into the generated audio as the implementation of speech transformation operations. Environmental noise refers to sounds generated in industrial production, transportation, and social life. In the test speech assembly process, this method uses the NOISEX-92 noise library, which contains a variety of environmental sounds, to inject noise into the generated speech. In addition, this method uses the pydub library for audio processing as the technical implementation of the test audio generation method. pydub is a Python library designed specifically for audio file operations, providing convenient interfaces for performing various speech operations, including format conversion, segmentation, volume adjustment, and merging. To maintain the quality of the generated speech, we performed additional processing on each speech during the assembly stage. Based on the position of punctuation marks in the corresponding text, we adjusted the rhythm of the speech and added appropriate pauses; at the same time, we smoothed the audio at the splicing points during the splicing process to make it more fluent and natural.

[0026] To ensure the intelligibility and high quality of the generated speech, we selected two evaluation metrics: Short-Time Objective Intelligibility (STOI) and Perceived Speech Quality (PESQ). STOI is an important measure of speech intelligibility, with values ​​ranging from 0 to 1, where 1 indicates fully intelligible speech and 0 indicates intelligible speech. PESQ is one of the most commonly used speech quality evaluation metrics, with values ​​ranging from -0.5 to 4.5, where a higher value indicates better speech quality. To generate as many test cases as possible while maintaining semantic integrity and high speech quality, we selected STOI > 0.98 and PESQ > 2 as thresholds after testing with different values. This is because we want the generated test audio to maintain high intelligibility while being as close as possible to real-world application scenarios. In reality, input audio often includes some environmental noise, which means that the speech translation system should be able to accept some lower-quality input audio and provide accurate translations. Finally, in this step, our method can generate multiple sets of test audio based on the given input audio through noise injection and concatenation of auxiliary word audio blocks.

[0027] 3. Translation error detection

[0028] This section describes how to apply syntactic structure invariance to speech translation error detection, and the specific implementation of the speech translation error detection method. For the translation results of the test speech, this method chooses to use Stanford CoreNLP to extract its constituent analysis and dependency analysis results, and transforms these results into a syntactic tree format. Stanford CoreNLP is a natural language processing toolkit for linguistic analysis of text data, supporting constituent analysis and dependency analysis for multiple languages. In the constituent analysis results, we use the root node to represent the entire sentence, the leaf nodes to represent the words in the sentence, and the internal nodes to represent phrases, thus generating a constituent analysis tree of the translation results. In the dependency analysis results, the nodes in the syntactic tree represent the words in the sentence, and the edges connecting the nodes represent the dependency relationships between two words. For each pair of translation results (T... i ,T i+1 We compare the constituent syntax tree and dependency syntax tree generated from the translation results to determine whether syntactic structure invariance is preserved.

[0029] In the constituent syntax tree, this method first obtains all constituent paths from the root node to the leaf node. Each path illustrates the syntactic decomposition process from root to leaf, providing information about different levels of constituent structure. In the dependency syntax tree, each dependency path only contains the dependency relations stored on the edges, omitting the word information stored on the nodes. This method inputs the lists of constituent paths and dependency paths into a designed matching function to determine if there are any erroneous paths. Erroneous paths are those that violate transformation relations. During path comparison, we check if there exists a path in the target path where each node appears sequentially in the original path.

[0030] This method uses a matching algorithm to compare the constituent parse trees and dependency parse trees of parent-child speech pairs separately. Error paths may appear in one or both comparison processes; in this method, these error paths are considered to represent potential errors in the speech translation results. However, in practice, errors in parse tree construction can also introduce error paths, even if the translation is correct. For example, constituents marked as FW in the constituent analysis results or dependencies marked as dep in the dependency analysis results may indicate a problem in the Stanford CoreNLP analysis process. In such cases, when an error occurs in one type of parse tree, we prioritize comparing results from the other type of parse tree to improve the accuracy of the output.

Claims

1. An automated testing method for a speech translation system based on semantic analysis, characterized in that, The input is unlabeled monolingual audio, each audio containing a complete sentence, and the output is a list of detected suspected translation errors. The method evaluates the performance of the speech translation system without referencing a translation by constructing syntactic structure invariant metamorphic relations. The method includes the following steps: (1) Speech template construction steps: In order to convert unstructured speech input into structured test templates, the method first uses phoneme alignment technology to process the input audio, determines the start and end times of each word in the sentence, thereby decomposing the continuous speech signal into discrete speech blocks corresponding to words; at the same time, the syntactic analysis is performed on the text sentence corresponding to the audio to extract the text and auxiliary words corresponding to its basic structural parts. Based on the results of the phoneme alignment and syntactic analysis, a speech template is constructed. The template consists of basic speech blocks representing the basic structural parts of a sentence, and one or more slots that record the start and end time information of the speech blocks of auxiliary words. (2) Test audio generation and enhancement steps: In order to generate diverse test cases to comprehensively test the speech translation system, the method is based on the above speech template and generates seed test audio by sequentially splicing auxiliary word speech blocks into basic speech blocks; To simulate speech changes in a real-world environment, the method further employs a speech transformation operator to process the seed test audio or the basic speech block, and performs smoothing processing on the spliced ​​audio to ensure that its speech intelligibility and quality indicators meet preset thresholds; by changing the inserted auxiliary word speech block or adjusting the parameters of the speech transformation operator, a series of variant test audios with a derivation relationship to the seed test audio are generated. (3) Translation error detection step: In order to automatically identify translation errors without the need for a reference translation, the method inputs the seed test audio and its variant test audio into the speech translation system to obtain the corresponding translated text results; based on the principle of syntactic structure invariance, the syntactic structure of the translated text results of the parent and child test audio is parsed and compared, wherein the syntactic structure invariance refers to the characteristic that the deep syntactic structure relationship of a sentence remains unchanged after adding modifiers; When the comparison result indicates that the syntactic structure consistency is violated, the speech translation system is determined to have a suspected translation error and an error report is output.

Images