Intelligent text dialogue model fuzzy testing method based on uncertainty

Abstract

The invention relates to an intelligent text dialogue model fuzzy testing method based on uncertainty, and belongs to the field of neural networks. According to the method, a series of metamorphic relation operators of text data are constructed, and the operators comprise synonym replacement, back translation and word insertion and are suitable for text dialogue tasks. In terms of semantic retention strategies, these operators can be used to generate new test text sentences without changing the semantics of the original sentences. When the test case is generated, the uncertainty of the tested dialogue model to the test data can be calculated. Under the guidance of uncertainty, the fuzzy test technology can effectively select test data most likely to trigger potential defects of the model. And finally, a large number of test cases used for testing the dialogue model are automatically generated on the basis of limited seed test data through the fuzzy test technology, test resources are saved, and efficient fault revealing capacity is achieved.

Description

Technical Field

[0001] The invention belongs to the field of neural networks, and in particular relates to a fuzzy testing method for an intelligent text dialogue model based on uncertainty. Background Art

[0002] With the significant progress of recurrent neural network (RNN) models in processing serialized data, intelligent dialogue systems have developed rapidly. Practice has shown that the intelligent text dialogue model driven by the RNN structure has some potential defects that are difficult to be discovered manually, which makes the testing of this model particularly important. Fuzz testing, as an effective software testing method, can effectively discover potential vulnerabilities and defects in the system by generating a large amount of unexpected, abnormal or randomly mutated input data. Traditional fuzz testing usually guides the automatic generation process through indicators such as code coverage. However, for intelligent models, code coverage cannot reflect the logical structure relationship processed inside the model, so it cannot guide the generation of more effective test data. Based on the idea of ​​traditional code coverage, intelligent model structure coverage is proposed as an indicator to measure the internal activation of the model, which can guide the generation of test data. In neural networks, the activation state of the intelligent model structure reflects how the model processes input data, especially different inputs may trigger different model structure activations, thereby affecting the output of the model. Intelligent model structure coverage calculates the activation of each model structure to ensure that different areas of the model are fully activated and tested, thereby helping to identify potential defects.

[0003] The current intelligent neural network model mainly uses intelligent model structure coverage indicators such as neuron coverage as guiding indicators for fuzz testing to guide the automatic generation process of test data. However, most of the existing intelligent model structure coverage indicators are difficult to adapt to the intelligent text dialogue model driven by the RNN structure. A small number of intelligent model structure coverage indicators that adapt to the RNN structure are invalid when guiding the generation of fuzz test data, and may even be misleading. When there is a lack of effective guiding indicators, the data generated by the fuzz test will lack clear direction guidance, and the model will be tested under invalid test targets or low-quality test scenarios, which will not only be unhelpful to the security improvement of the intelligent text dialogue model, but will also waste a lot of computing resources. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The technical problem to be solved by the present invention is how to provide a fuzzy testing method for an intelligent text dialogue model based on uncertainty, so as to solve the problem that the existing RNN dialogue model fuzzy testing methods generally lack an effective test guidance mechanism, the amplified test cases deviate from the original semantics, and the existing test predictions cannot be used to verify the results.

[0006] (II) Technical solution

[0007] In order to solve the above technical problems, the present invention proposes a fuzzy testing method for an intelligent text dialogue model based on uncertainty, which comprises the following steps:

[0008] Step 1: Conversational Text Transformation Relationship Construction

[0009] A series of transformation relation operators for text data are constructed for text transformation, including synonym replacement, back translation and word insertion, which are suitable for text dialogue tasks.

[0010] Step 2: Semantic preservation of text data

[0011] In order to ensure that the semantics of the generated text is preserved, the ROUGE-1 indicator is used to evaluate the quality of the text generated by the conversion strategy;

[0012] Step 3: Uncertainty-guided Text Model Fuzz Testing Method

[0013] When generating test cases, the uncertainty of the tested dialogue model to the test data is calculated. Under the guidance of uncertainty, the fuzzy testing technology selects the test data that is most likely to trigger the potential defects of the model;

[0014] Step 4: Model testing and tuning

[0015] The constructed fuzzy test set is used to test the text dialogue model to detect its performance in performing intent detection and slot filling tasks. The model is retrained using fine-tuning. Through a continuous testing-retraining iterative process, the model's fault tolerance and stability are continuously enhanced.

[0016] (III) Beneficial effects

[0017] The present invention proposes a fuzzy testing method for an intelligent text dialogue model based on uncertainty. First, the present invention constructs a series of transformation relation operators for text data, which include synonym replacement, back translation and word insertion, and are suitable for text dialogue tasks. In terms of semantic retention strategy, these operators can be used to generate new test text sentences without changing the semantics of the original sentences. When generating test cases, the present invention calculates the uncertainty of the tested dialogue model to the test data. Under the guidance of uncertainty, fuzzy testing technology can effectively select test data that is most likely to trigger potential defects of the model. Finally, the fuzzy testing technology of the present invention automatically generates a large number of test cases for testing dialogue models based on limited seed test data, saving test resources and having efficient fault revealing capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1This is the technical roadmap of fuzzy testing of the uncertainty-based intelligent text dialogue model of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0020] The purpose of this invention is to propose a fuzzy testing technology for intelligent text dialogue models based on uncertainty. The current fuzzy testing of intelligent text dialogue models faces the following challenges:

[0021] On the one hand, existing RNN dialogue model fuzz testing methods generally lack an effective test guidance mechanism. Traditional indicators such as code coverage are difficult to directly apply to neural network structures, and test cases generated or converted by model structure coverage are often inefficient and difficult to trigger potential defects of the model in a targeted manner.

[0022] On the other hand, when performing test data augmentation, it is necessary to ensure that the generated samples are consistent with the original input at the semantic level. This is because the testing of intelligent dialogue systems usually relies on test oracles built based on the original data to determine whether the output is correct. If the augmented test cases deviate from the original semantics, the existing test oracles will not be able to verify the results.

[0023] The overall technical route of the present invention is as follows Figure 1 As shown, it includes: construction of dialogue text metamorphosis relationship based on domain knowledge, semantic preservation of text data and uncertainty-guided text fuzz testing method.

[0024] The present invention comprises the following steps: firstly, aiming at the characteristics of text modality in a dialogue task, a transformation relation operator based on domain knowledge is constructed to ensure that for the text before and after the operator conversion, there is a certain known and automatically determined relationship between the output of the model to be tested; secondly, a guided conversion generation algorithm is carried out to guide the construction of a test case set that can reveal the problems of the dialogue model as much as possible based on the uncertainty of the dialogue model for the test data; then, semantics is retained in the generated test cases so that the test data can still maintain the original semantics after conversion, thereby ensuring the validity and authenticity of the test; finally, the test case is input into the intelligent text dialogue model, and whether the output of the dialogue model causes problems and errors is judged according to the semantic consistency, the test case set that reveals the problems of the dialogue model is recorded, and the dialogue model is retrained and tuned.

[0025] Step 1: Conversational Text Transformation Relationship Construction

[0026] The dialogue text metamorphosis relationship of the present invention can generate a test sentence with a conversion operator from seed test data.

[0027] The method for constructing the relationship between dialogue text transformation aims to realize the seed conversion method of the RNN-driven dialogue model. Specifically, the RNN-driven dialogue model is represented as The dialogue model can continuously detect the intent of the input sentence and fill the slots. The metamorphosis relationship can be defined as: given a seed sentence set The transformation operator t can be transformed from each sentence s without affecting the intent and slot label R of s:

[0028]

[0029] The present invention constructs three types of sentence conversion operation methods, namely synonym replacement, back translation and word insertion. For each type of conversion operation, the present invention implements several conversion operators, which are defined as follows:

[0030] 1) Synonym replacement: This series of operators transforms sentences by replacing single words with synonyms, but does not change the meaning of the sentence. The present invention uses the large-scale English vocabulary WordNet to find and replace synonyms. The library groups words according to their meanings, and each phrase with the same meaning is a synonym set. The synonym operators used in the present invention include noun conversion, verb conversion, and adjective conversion. For a given sentence, the present invention randomly replaces synonyms with a slot label of "O". Since the replaced words originally have no entity labels, the slot labels and intention labels remain consistent with the original data.

[0031] 2) Back translation: This series of operators translates the target seed into an intermediate language and then translates it back to the original language. Since machine translation usually restructures the sentence structure, the converted data can not only keep the meaning of the sentence unchanged, but also improve the richness of the sentence structure. Regarding the intermediate language for back translation, the present invention selects Chinese, French and Spanish, which have completely different alphabets and grammars.

[0032] 3) Word insertion: This series of operators transforms sentences by inserting words. When the dialogue model performs intent detection and slot filling, it is actually trying to understand the input sentence. Therefore, if a few words irrelevant to the intent are added to the sentence while maintaining grammatical correctness, it may cause potential interference to the model. The present invention uses the BERT architecture, a masked language model (MLM) for automatically generating natural language. This type of model task predicts the missing words in a sentence based on the semantics of the context. The present invention uses three pre-trained MLM models (BERT, POBERTA, and ALBERT) and uses different parameter configurations to insert new words into the original sentence. At the same time, the present invention controls the number of words between 1 and 3. Therefore, the MLM model can insert both words and phrases when inserting vacancies.

[0033] Step 2: Semantic preservation of text data

[0034] The focus of the semantic preservation strategy for text data is to ensure that the text is realistically converted so as to test the intelligent text dialogue model. In order to ensure that the semantics of the generated text is preserved, the present invention adopts the ROUGE-1 indicator to evaluate the quality of the text generated by the conversion strategy. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of metrics for evaluating automatic summarization and machine translation software in natural language processing. The ROUGE-1 score is calculated by calculating the difference between the tag set of each word in the candidate sentence and the reference sentence. Specifically, for a given original sentence q and the corresponding converted sentence q′, the ROUGE-1 score can be calculated as follows:

[0035]

[0036] Where tok is the set of words in the text sentence.

[0037] Step 3: Uncertainty-guided Text Model Fuzz Testing Method

[0038] The present invention guides the fuzzy test direction based on the uncertainty of the test sample by the dialogue model. The present invention uses the Gini coefficient as a measure of uncertainty. The Gini coefficient measures uncertainty based on the probability of calculating the label by the last layer of the softmax function in the evaluation dialogue model, so the Gini coefficient can only be applied to a one-dimensional softmax matrix. The slot filling task in the text dialogue model cannot directly calculate the Gini coefficient because it will output a two-dimensional softmax matrix. Therefore, the present invention calculates the Gini coefficient of the model processing intent detection task, which is defined as follows:

[0039]

[0040] Among them, P t,i represents the probability that the test text t belongs to the intent category i.

[0041] For the softmax layer of the intent classification model under test in the text dialogue model, the test cases with higher Gini coefficients indicate that the model has uncertainty in the prediction results, so it is easier to find potential defects in the model. Similar to the traditional software testing method based on code coverage, the goal of this fuzzy testing method is to generate a test set that can improve the Gini coefficient. Since the test goal of the present invention is to generate more data that may cause incorrect behavior in the prediction results of the text system, the present invention uses the uncertainty of the Gini coefficient to guide the sentence conversion.

[0042] Since a single transformation of a sentence can increase the richness of data samples, this fuzzy testing method will apply multiple transformation combinations to detect more erroneous behaviors of the model when generating a test set. However, if all combinations of transformation methods are applied to the original test set, a large number of test samples will be generated, resulting in low test efficiency and high storage space costs. Moreover, if the number of combinations is not limited, the combinations are theoretically endless. In response to the above problems, the present invention proposes an uncertainty-guided fuzzy algorithm to guide the generation of transformation data. Algorithm 1 introduces a method for increasing uncertainty in the process of combining transformation operators. When generating transformation data, the fuzzy testing method applies the Gini function as a guide to ensure that the generated test data is more likely to discover potential errors in the system than the original test data. The main implementation process of the algorithm is as follows: first, the seed text is randomly transformed, and then the Gini coefficient of the model under test is calculated. If the Gini coefficient of the transformed text rises to a specified threshold τ compared to the Gini coefficient of the original text, and the ROUGE-1 quality score of the transformed text is higher than the specified threshold t, the generated sentence will be added to the seed sentence list. On the contrary, the generated sentence will be directly discarded because the ROUGE-1 quality score or Gini coefficient is small, which indicates that the seed transformation quality is low, or The confidence level of the input sentence is high, and it is not easy to make mistakes.

[0043]

[0044]

[0045] Step 4: Model testing and tuning

[0046] The present invention uses the constructed fuzzy test set to test the text dialogue model to detect the performance of the text dialogue model in performing intent detection and slot filling tasks. In terms of intent detection, whether the model can accurately identify the various intents contained in the user input. For the slot filling task, the accuracy of the model in extracting key information is evaluated. The robustness and accuracy of the text dialogue model in intent understanding and information extraction are fully verified through test evaluation. In addition, the present invention constructs a high-quality training data set based on the constructed fuzzy test set, and retrains the model in a fine-tuning manner. During the retraining process, the model parameters are adjusted so that the model can correctly handle these error scenarios and improve the robustness of the model to various abnormal inputs. Through a continuous test-retraining iterative process, the fault tolerance and stability of the model are continuously enhanced.

[0047] The present invention proposes a fuzzy testing method for an intelligent text dialogue model based on uncertainty. First, the present invention constructs a series of transformation relation operators for text data, which include synonym replacement, back translation and word insertion, and are suitable for text dialogue tasks. In terms of semantic retention strategy, these operators can be used to generate new test text sentences without changing the semantics of the original sentences. When generating test cases, the present invention calculates the uncertainty of the tested dialogue model to the test data. Under the guidance of uncertainty, fuzzy testing technology can effectively select test data that is most likely to trigger potential defects of the model. Finally, the fuzzy testing technology of the present invention automatically generates a large number of test cases for testing dialogue models based on limited seed test data, saving test resources and having efficient fault revealing capabilities.

[0048] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A fuzzy testing method for an intelligent text dialogue model based on uncertainty, characterized in that: The method comprises the following steps: Step 1: Conversational Text Transformation Relationship Construction A series of transformation relation operators for text data are constructed for text transformation, including synonym replacement, back translation and word insertion, which are suitable for text dialogue tasks. Step 2: Semantic preservation of text data In order to ensure that the semantics of the generated text is preserved, the ROUGE-1 indicator is used to evaluate the quality of the text generated by the conversion strategy; Step 3: Uncertainty-guided Text Model Fuzz Testing Method When generating test cases, the uncertainty of the tested dialogue model to the test data is calculated. Under the guidance of uncertainty, the fuzzy testing technology selects the test data that is most likely to trigger the potential defects of the model; Step 4: Model testing and tuning The constructed fuzzy test set is used to test the text dialogue model to detect its performance in performing intent detection and slot filling tasks. The model is retrained using fine-tuning. Through a continuous testing-retraining iterative process, the model's fault tolerance and stability are continuously enhanced.

2. The uncertainty-based intelligent text dialogue model fuzzy testing method according to claim 1, characterized in that: The step one comprises: The RNN-driven dialogue model is represented as The dialogue model can continuously detect the intent of the input sentence and fill the slot; the metamorphosis relationship is defined as: given a seed sentence set The transformation operator t is derived from each sentence s without affecting the intent and slot label R of s:

3. The uncertainty-based intelligent text dialogue model fuzzy testing method according to claim 2, characterized in that: The synonym replacement operator transforms a sentence by replacing a single word with a synonym, but does not change the meaning of the sentence. The large-scale English word library WordNet is used to find and replace synonyms. The library groups words according to their meaning, and each group of words with the same meaning is a synonym set. The synonym operators used include noun conversion, verb conversion, and adjective conversion. For a given sentence, a synonym with a slot label of "O" is randomly replaced. Since the replaced words originally have no entity labels, the slot labels and intent labels remain consistent with the original data.

4. The uncertainty-based intelligent text dialogue model fuzzy testing method according to claim 2, characterized in that: The back-translation operator translates the target seed into an intermediate language and then translates it back to the original language. Since machine translation reorganizes the sentence structure, the converted data not only keeps the meaning of the sentence unchanged, but also improves the richness of the sentence structure. Regarding the intermediate language for back-translation, Chinese, French, and Spanish are selected, which have completely different alphabets and grammars.

5. The uncertainty-based intelligent text dialogue model fuzzy testing method according to claim 2, characterized in that: The word insertion operator transforms sentences by inserting words. Using the BERT architecture, a masked language model (MLM) is used to automatically generate natural language. This type of model task predicts missing words in a sentence based on the semantics of the context. Three pre-trained MLM models are used with different parameter configurations to insert new words into the original sentence. At the same time, the number of words is controlled between 1 and 3.

6. The uncertainty-based intelligent text dialogue model fuzzy testing method according to claim 5, characterized in that: Three pre-trained MLM models include: BERT, POBERTA, and ALBERT.

7. The uncertainty-based intelligent text dialogue model fuzzy testing method according to any one of claims 1 to 6, characterized in that: The second step comprises: The ROUGE-1 score is calculated by calculating the difference between the token set of each word in the candidate sentence and the reference sentence; for a given original sentence q and the corresponding transformed sentence q′, the ROUGE-1 score can be calculated as follows: Where tok is the set of words in the text sentence.

8. The uncertainty-based intelligent text dialogue model fuzzy testing method according to claim 7, characterized in that: The step three includes: using the Gini coefficient as a measure of uncertainty; the Gini coefficient measures uncertainty based on the probability of calculating the label by the last layer of the softmax function in the evaluation dialogue model, so the Gini coefficient can only be applied to a one-dimensional softmax matrix; the slot filling task in the text dialogue model cannot directly calculate the Gini coefficient because it will output a two-dimensional softmax matrix; therefore, the Gini coefficient of the calculation model processing the intent detection task is defined as follows: Among them, P t,i represents the probability that the test text t belongs to the intent category i.

9. The uncertainty-based intelligent text dialogue model fuzzy testing method according to claim 8, characterized in that: When generating transformed data, fuzz testing methods apply the Gini function as a guide to ensure that the generated test data is more likely to find potential errors in the system than the original test data; The implementation process of the algorithm includes: firstly, randomly transforming the seed text, and then calculating the Gini coefficient of the model under test; If the Gini coefficient of the converted text rises to a specified threshold τ compared to the Gini coefficient of the original text, and the ROUGE-1 quality score of the converted text is higher than the specified threshold t, the generated sentence will be added to the list of seed sentences; otherwise, the generated sentence will be directly discarded because of a small ROUGE-1 quality score or Gini coefficient, which indicates that the seed conversion quality is low, or The confidence level of the input sentence is high, and it is not easy to make mistakes.

10. The uncertainty-based intelligent text dialogue model fuzzy testing method according to claim 9, characterized in that: The step four includes: in terms of intent detection, detecting whether the model can accurately identify various intents contained in user input; for slot filling tasks, evaluating the accuracy of the model in extracting key information; comprehensively verifying the robustness and accuracy of the text dialogue model in intent understanding and information extraction through test evaluation; in addition, building a high-quality training data set based on the constructed fuzzy test set, retraining the model in a fine-tuning manner, adjusting the model parameters during the retraining process so that the model can correctly handle these error scenarios, and improving the robustness of the model to various abnormal inputs; through a continuous test-retraining iterative process, continuously enhancing the fault tolerance and stability of the model.

Images