Test method of text sentiment analysis task model based on genetic algorithm

Abstract

The invention discloses a test method of a text sentiment analysis task model based on a genetic algorithm. The test method comprises the following steps: S1, randomly selecting an original seed test case from existing text data; s2, mutation operator design is carried out, wherein a sentiment analysis mutation operator and a general text mutation operator are included; s3, constructing a search framework, calling the search framework based on a genetic algorithm, selecting original seed text test cases according to the evaluation score of the dual-target fitness function to obtain candidate test cases, applying the mutation operator to the candidate test cases in a combined manner to search for an optimal solution of the test cases, and automatically generating a final test case; and S4, testing the sentiment analysis task model to be tested by using the test case to obtain a defect evaluation result based on the metamorphic relationship. According to the method, a test means for comprehensively and efficiently evaluating the defects of the sentiment analysis model can be provided, and the evaluation efficiency and accuracy of the reliability and robustness of the SA system are improved.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing (NLP) and deep learning testing technology, and in particular relates to a method for testing a sentiment analysis task model using automatically generated test cases. Background Art

[0002] 1. Sentiment Analysis Task Model

[0003] Sentiment Analysis (SA) task models use computational methods to identify, extract, quantify, and study the subjective emotions, evaluations, and attitudes contained in text data. This task is usually formalized as a classification problem. Based on machine learning or deep learning models (such as recurrent neural networks, convolutional neural networks, Transformers, etc.), the input text is mapped to predefined sentiment labels. Depending on the granularity of the predefined sentiment label set, sentiment analysis tasks can usually be divided into: binary classification (for example, distinguishing between "positive" and "negative"), tertiary classification (for example, adding a "neutral" label), and fine-grained classification (for example, dividing into five levels of "very positive", "positive", "neutral", "negative", and "very negative") to capture more specific sentiment intensity.

[0004] 2. Transformation Test

[0005] Metamorphic testing is a software testing technique whose core concept is to validate program behavior by leveraging the characteristics of the software under test, known as metamorphic relations (MRs). In sentiment analysis, existing metamorphic testing methods primarily test sentiment analysis systems by constructing equivalence relations (such as synonym substitution, tense shift, and entity substitution, with the expected unchanged sentiment polarity) and inequality relations (such as inserting negation or sentiment-laden adjectives, with the expected change in sentiment polarity).

[0006] 3. Search-based software testing

[0007] Search-Based Software Testing (SBST) is a technique that uses metaheuristic search algorithms (such as genetic algorithms, simulated annealing, and particle swarm optimization) to automatically generate test data. SBST transforms the test case generation problem into an optimization problem, defining one or more fitness functions to evaluate the quality of test cases. Genetic algorithms (GAs) are a commonly used search technique in SBST. They mimic natural selection and genetic mechanisms, iteratively improving the candidate test case population through operations such as selection, crossover, and mutation, thereby finding high-quality test solutions in a vast search space. This approach provides an effective approach for generating complex and targeted test cases.

[0008] The existing testing methods for sentiment analysis systems have the following problems: First, in terms of the diversity and coverage of test case generation, existing methods generally rely on a limited number of mutation rules for single mutations, or generate test cases based on simple templates. This results in limited semantic changes in the generated test cases, making it difficult to touch the deep-seated defects of the model, and the ability to reveal errors needs to be improved. Secondly, in terms of the quality and false alarm rate of test cases, the existing metamorphic testing methods lack explicit constraints on key qualities such as sample naturalness and semantic preservation in the process of generating test cases. Such low-quality test cases are not only difficult to accurately reflect the true capabilities of the model, but also easily lead to false alarms of defects, which seriously weakens the effectiveness of the test. Therefore, this field urgently needs a testing method that can generate test cases with strong error-revealing capabilities and low false alarm rates. Summary of the Invention

[0009] In order to solve several key problems existing in the above-mentioned current sentiment analysis system testing methods, the present invention proposes a testing method for a text sentiment analysis task model based on a genetic algorithm. The test cases of the sentiment analysis task model are automatically generated based on the search framework of the genetic algorithm, thereby realizing comprehensive and efficient testing of the text sentiment analysis task model.

[0010] The present invention provides a testing method for a text sentiment analysis task model based on a genetic algorithm, comprising the following steps:

[0011] S1, randomly select candidate test cases from existing text data;

[0012] S2. Design mutation operators, including sentiment analysis mutation operators and general text mutation operators;

[0013] S3. Constructing a search framework, calling the search framework based on a genetic algorithm, selecting original seed text test cases based on the evaluation scores of the dual-objective fitness function, obtaining candidate test cases, applying the mutation operator to the candidate test case combinations, finding test case optimization solutions, and automatically generating test cases;

[0014] S4. Using the test case to test the sentiment analysis task model to be tested to obtain a defect assessment result based on the metamorphosis relationship.

[0015] In some embodiments, the mutation operator includes a sentiment analysis mutation operator, and the sentiment analysis mutation operator further includes a reinforcement-type sentiment mutation operator and a negation-type sentiment mutation operator.

[0016] In some embodiments, the mutation operator further includes a general text mutation operator, which includes a word replacement operation that does not change the semantics. The operation is based on the screening of candidate nodes in the lexical relationship dependency tree, and selects some nodes that may have the greatest contribution to the overall sentiment as potential mutation positions.

[0017] In some embodiments, the selection of the potential mutation position is based on dependency syntax analysis, SO value calculation based on extension rules, efficient candidate node screening and simulated SO change prediction mechanism to randomly select nodes from safe mutation positions.

[0018] In some embodiments, in S3, candidate test cases are selected based on the dual-objective fitness function evaluation scores, and the mutation operator is applied to the selected candidate test case combinations to find test case optimization solutions. This step further includes:

[0019] The search framework is called to iteratively perform a search based on a genetic algorithm: i) population initialization: replace words in the original seed text test case, randomly generate original seed test cases as samples, and obtain an initial population; ii) fitness evaluation: design a fitness function, and use the fitness function to obtain the evaluation score of the candidate test case; iii) selection, crossover and mutation: select the parent individual of the candidate test case from the current population according to the evaluation score of the candidate test case, exchange some text fragments in the parent individual, generate new child individuals, and then apply the mutation operator to all child individuals with a set probability to obtain child mutant individuals; iv) next generation population generation: merge the child mutant individuals, select the same number of child mutant individuals as the initial population to form the next generation population, and when the next generation population no longer changes or reaches the specified number of iterations, the test case optimization solution to be found is obtained.

[0020] In some embodiments, the combination of applying different types of mutation operators to the selected candidate test cases in S3 further includes determining and applying an enhancement-type sentiment mutation operator or a negation-type sentiment mutation operator; wherein, confirming that the node adapts to the enhancement grammatical phenomenon corresponds to the enhancement-type operation, amplifying the SO value of the node, and the conversion function is defined as:

[0021] τ(σ)=σ×(1+β)

[0022] Among them, τ(σ) is the amplified result of the node's SO value, σ is the original SO value of the node, and β is the weighting factor, which adjusts the SO value according to the strength of the strengthening word;

[0023] Confirm that the node adapts to the negation operation corresponding to the negation grammatical phenomenon and weakens the SO value of the node. The conversion function is:

[0024]

[0025] Among them, τ'(σ) is the result of the negation structure to strengthen or weaken the sentiment, σ is the original SO value of the node, and α is the offset factor, which adjusts the SO value according to the influence of the negation word.

[0026] In some embodiments, the dual-objective fitness function is specifically a dual-objective fitness function vector that evaluates the quality of each candidate test case Ind relative to its original seed text test case Orig, and is expressed as follows:

[0027]

[0028] The first objective fitness function f1 is used to quantify the degree to which the candidate test case causes the change in the prediction probability of the sentiment analysis model to be tested, and the second objective fitness function f2 is used to comprehensively evaluate the similarity or preservation degree of the candidate test case with the original text in terms of semantic content and grammatical structure.

[0029] In some embodiments, the first objective fitness function f1 is specifically:

[0030] f1=|Model(Ind)-Model(Orig)|

[0031] Among them, Model(·) is the predicted output of the sentiment analysis task model, Ind is the candidate test case, and Orig is the original seed text test case.

[0032] In some embodiments, the second fitness objective function f2 is specifically:

[0033] f2=(1-λ)Sim(Infer(Orig),Infer(Ind))+λTed(Ind,Orig)

[0034] Among them, Sim(·) is the semantic similarity, TED(·) is the grammatical structure similarity, and λ is the weight factor.

[0035] In some implementations, the final comprehensive evaluation score of test case x is:

[0036]

[0037] Among them, Perplexity(x) is the perplexity index of the final test case x, and f2 is the second fitness function of the final test case x.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] 1) It can provide a comprehensive and efficient testing method to evaluate the defects of sentiment analysis models, and improve the efficiency and accuracy of evaluating the reliability and robustness of the SA system;

[0040] 2) A candidate node screening strategy was designed to select original seed text test cases based on the dual-objective fitness function evaluation scores to obtain candidate test cases. The mutation operator was then applied to the candidate test case combinations, thereby significantly reducing the complexity of subsequent simulation calculations and improving computational efficiency.

[0041] 3) Through this series of position selections, we can safely identify candidate mutation locations and mutation operator types. This ensures that the mutation operation can effectively explore the model's sensitivity to sentiment changes while maintaining the original sentiment label with a high probability, thereby improving the accuracy and efficiency of the test.

[0042] 4) Calling the search framework based on the genetic algorithm, selecting original seed text test cases according to the evaluation scores of the dual-objective fitness function, obtaining candidate test cases, applying the mutation operator to the candidate test case combinations to find test case optimization solutions, and automatically generating test cases. This algorithmic process uses a multi-objective optimization mechanism to ensure the quality of the generated test cases. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is an overall flow chart of the testing method of the text sentiment analysis task model based on the genetic algorithm of the present invention;

[0044] Figure 2 This is an example diagram of the vocabulary dependency tree structure;

[0045] Figure 3 To strengthen the example graph of class mutation operators;

[0046] Figure 4 This is an example diagram of the negative sentiment mutation operator;

[0047] Figure 5 A roadmap for implementing the testing method of the text sentiment analysis task model based on genetic algorithm of the present invention. DETAILED DESCRIPTION

[0048] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0049] like Figure 1 As shown, the specific steps of the overall process package of a testing method for a text sentiment analysis task model based on a genetic algorithm proposed in the present invention are as follows:

[0050] Step 1: Randomly select original seed text test cases from the existing text data;

[0051] Step 2: Design sentiment analysis mutation operators and general text mutation operators; including:

[0052] Sentiment analysis mutation operators include enhanced sentiment mutation operators and negation sentiment mutation operators designed for sentiment grammatical units of different granularities (words, phrases, clauses);

[0053] The general text mutation operator includes a word replacement operation that does not change the semantics. This operation is based on the screening of candidate nodes in the lexical dependency tree, selecting some nodes that may have the greatest contribution to the overall sentiment as potential mutation locations, obtaining the nodes at the potential mutation locations as candidate nodes, and performing predictions on the enhanced sentiment mutation operator and the negative sentiment mutation operator for the candidate nodes at the potential mutation locations. The specific process is described as follows:

[0054] Step 2.1: Perform dependency syntax analysis on the text to obtain the dependency relationship between words and form a dependency tree structure;

[0055] Step 2.2: The text input is general text, and a general text mutation operator is designed; this includes:

[0056] 2.2.1. Design candidate node screening strategy:

[0057] Preprocessing: Calculate the sentiment orientation (SO) based on the lexical dependency tree and sentiment calculation combination rules. Multiple sentiment calculation rules are combined to form a sentiment calculation combination rule. The definition of a sentiment calculation rule includes the following steps: For a given dependency tree structure T = (V, E), a combination operation is represented as a five-tuple o = (τ, C, δ, π, S). τ is a transformation function used to change the SO value of a node. C is a predicate used to determine whether to trigger the operation. δ is a non-negative integer used for the scope of the combination operation. π is a priority used to describe the order in which the various combination operations are applied. S is a scope calculation function used to determine which dependency tree nodes will be affected. After all nodes are assigned initial SO values using the sentiment lexicon, the dependency tree is calculated from top to bottom using the predefined combination operations described above, ultimately obtaining the SO value for each grammatical structure in the entire text. The sentiment calculation rules under the original definition are based on binary sentiment analysis design, and only consider positive and negative sentiment polarity labels. The present invention adaptively expands the basic sentiment calculation rules, and remaps the conversion function value corresponding to each operation for the fine-grained sentiment score under multi-classification labels, so that after mapping, it can truly reflect the sentiment polarity of the text under fine-grained classification. At the same time, it expands the coverage of the sentiment dictionary so that it can cover sentiment vocabulary including data sets in different fields (movies, business reviews, etc.), and has better calculation results when facing multi-field sentiment analysis tasks, so that it can be simultaneously applicable to binary classification and fine-grained multi-category sentiment analysis tasks in different fields.

[0058] 2.2.2. Design a candidate node screening strategy: First, filter out common stop words in the entire lexical dependency tree. Then, based on the initial SO value of each node, select the nodes that are likely to contribute most to the overall sentiment analysis as candidate nodes. The locations of these nodes are potential mutation locations.

[0059] 2.2.3. Design a simulation SO change prediction mechanism: Apply two simulation operations to the current SO values of the candidate nodes screened out in 1.2.1: 1) a multiplication operation to simulate the maximum SO value enhancement effect brought by the enhancement-type mutation operator; and 2) a subtraction operation to simulate the maximum SO value offset effect brought by the negation-type mutation operator; through the SO values of the candidate nodes after these two simulation operations, re-iterate and calculate the final SO value of the entire sentence; finally, by comparing the direction of the sentence SO value after the simulation operation with the direction of the original sentence SO value, identify safe mutation candidate positions and classify them: if the sentiment polarity direction of the sentence remains unchanged after the simulation multiplication operation, the candidate node is identified as a safe position suitable for the enhancement-type mutation operator; if the polarity direction remains unchanged after the simulation subtraction operation, the candidate node is identified as a safe position suitable for the negation-type mutation operator.

[0060] In step 2, we define and implement a series of general text mutation operators. The core design principle of all operators is to keep the original sentiment polarity label unchanged when performing mutation operations on the original text, thus avoiding the need to manually re-label the test cases after mutation. This module includes the following steps:

[0061] To further enhance the semantic diversity of generated test cases and expand the exploration space for higher-order mutations, a set of general text mutation operators has been integrated. These operators share the ability to modify text without changing the core semantics or sentiment of a sentence. These general text mutation operators primarily perform word replacement, replacing words of the same type with words of different parts of speech. For example, for a person name, they could replace "Christopher Nolan" with "Denis Villeneuve."

[0062] Step 4: Implement high-order mutation by performing multiple mutations on the same sample. Combine and apply the aforementioned different types of mutation operators to automatically generate test cases with complex emotional characteristics that can effectively reveal model defects. Build a search framework based on a genetic algorithm, and iteratively evolve a population of test case candidates. In each iteration, evaluate the quality of candidate test cases based on a preset fitness function, and apply genetic operations such as selection, crossover, and mutation to generate new candidate test cases. Ultimately, select a set of test cases that meet the requirements. Specifically, the high-order mutation strategy refers to using a genetic algorithm to perform multiple mutations on samples.

[0063] The search framework mainly consists of four parts: (1) Population initialization: The goal of this stage is to generate initial candidate mutation samples. This stage randomly generates a set of initial test case mutation samples through word replacement to form the initial population of the genetic algorithm. (2) Fitness evaluation: This stage aims to evaluate the quality of each candidate test case, including its error detection ability and test false positive performance. Each individual is evaluated by a fitness function. (3) Selection, crossover and mutation: This stage is the core step of the search algorithm. Through the fitness function obtained in the fitness evaluation part, individuals with higher fitness and better distribution are selected from the current population as parents. These parents will exchange some text fragments (genes) through single-point structural crossover and other methods to generate new offspring individuals. For all offspring individuals, emotion-related mutation operators or general text mutation operators are applied to them with a certain probability to achieve mutation. (4) Generation of the next generation population: The individuals obtained by the above crossover and mutation are merged, and at the same time, the same number of offspring individuals as the initial population are selected according to the NSGA-II rule to form the next generation population. When the next generation population no longer changes or reaches the specified number of iterations, the final test case is selected using the evaluation score.

[0064] In order to effectively guide the search process, a dual-objective fitness function vector is designed in the fitness evaluation part of the search framework to evaluate the quality of each candidate test case Ind relative to its original seed text test case Orig. The expression is as follows:

[0065]

[0066] The first objective fitness function f1 is used to quantify the degree to which the candidate test case causes the change in the predicted probability (or confidence score) of the sentiment analysis model to be tested;

[0067] The first objective fitness function f1 is specifically:

[0068] f1=|Model(Ind)-Model(Orig)|

[0069] Where Model(·) is the predicted output of the sentiment analysis task model. This objective function aims to quantify the extent to which a candidate test case causes a change in the predicted probability (or confidence score) of the sentiment analysis model under test. Since all mutation operators in this invention are based on the premise of keeping the original sentiment label unchanged, a larger f1 value indicates that the candidate test case is closer to the decision boundary of the sentiment analysis task model under test, and thus has a higher probability of revealing potential defects.

[0070] The second objective fitness function f2 is used to comprehensively evaluate the similarity or preservation degree of the candidate test case with the original text in terms of semantic content and grammatical structure.

[0071] The second fitness objective function f2 is specifically:

[0072] f2=(1-λ)Sim(Infer(Orig),Infer(Ind))+λTed(Ind,Orig)

[0073] Here, Sim(·) is the calculated semantic similarity, TED(·) is the grammatical structure similarity obtained by calculating the tree edit distance (Ted) between lexical dependency trees, and λ is a weight factor used to balance the importance of these two aspects in the search guidance process.

[0074] Specifically, calling the search framework to iteratively execute the genetic algorithm search process includes the following steps:

[0075] During the population initialization phase, an initial population of candidate test cases is generated based on the input raw seed text through methods such as word replacement. The subsequent selection operation selects excellent parent individuals based on the aforementioned fitness function. The crossover operation generates new offspring by combining features of parent individuals (such as text fragments). Mutation is a key step in introducing novelty. It applies various defined emotion-related mutation operators or general text mutation operators to offspring individuals with a certain probability, thereby exploring a wider test space. After several generations of evolution, when a preset termination condition (such as the maximum number of iterations) is reached, individuals within the population are finally screened to ensure that the final selected test cases are both high-quality and have high error-detecting potential.

[0076] Step 5. Use the test case generated in step 3 to test the sentiment analysis task model to obtain a defect assessment based on the transformation relationship;

[0077] Each test case generated by the above search framework and its corresponding original seed test case are provided as input to the sentiment analysis task model to be tested, and the model's prediction results (e.g., sentiment category labels) for these two inputs are obtained. Given that the design of all mutation operators in the present invention follows the principle of maintaining the original sentiment polarity unchanged, if the sentiment analysis task model to be tested gives inconsistent sentiment prediction results for the original test case and its corresponding subsequent test case, that is, violates the expected equivalence transformation relationship, then it is determined that the subsequent test case has successfully detected a defect in the sentiment analysis system to be tested.

[0078] The emotion-related mutation operator designed in this paper focuses on modifying the language structure in the text that is directly related to the expression of emotion. Its implementation relies on in-depth linguistic analysis of the input text.

[0079] like Figure 2 The figure shows an example diagram of a vocabulary dependency tree. The present invention uses external tools to perform dependency syntax analysis on the text, obtain the dependency relationship between words, and form a dependency tree structure. Two types of sentiment variation operators are constructed based on the dependency tree. Figure 3 The figure shows the enhanced sentiment mutation operator, which is used to enhance the sentiment of the grammatical structure in the text. Specifically, it includes:

[0080] (1) Strengthening words: For independent adjectives or adverbs, if they are not modified by strengthening words, add appropriate strengthening words before them; if they are modified, they can be combined with the strengthening words into a whole for subsequent operations, or replaced with strengthening words that express a stronger degree.

[0081] (2) Strengthening phrases: For adjective phrases or adverbial phrases, if there is no strengthening word in the phrase, then a strengthening word is added before the core emotional word; if there is already a strengthening word, then it is replaced with a stronger strengthening word according to the intensity level.

[0082] (3) Strengthening clause: Emotional strengthening is achieved by adding a preset trigger phrase (e.g., "in my opinion," "obviously") at the end of the target clause that can enhance the expression of subjective emotions.

[0083] like Figure 4 As shown in Figure 2, the negative sentiment mutation operator mainly achieves non-reversal adjustment of sentiment by removing the negative words in the existing negative structure in the text, aiming to avoid grammatical errors or semantic confusion caused by improper insertion of negative words. It mainly includes two types: (1) Negative words: When dependency analysis shows that there is a clear negative relationship between an adjective or adverb node and the node it directly depends on (such as connected by a negative word), the negative word is removed.

[0084] (2) Negative phrases or clauses: If there is a negative relationship between the core node (usually the node closest to the root node of the dependency tree) and the node it directly depends on in a phrase or clause structure, the corresponding negative word is removed.

[0085] In addition, the present invention also collects five general text mutation operators, including: 1) Synonym replacement: replace specific words in the text with synonyms with similar semantics. 2) Tense change: adjust the tense expression of verbs in the text. 3) Singular and plural change: change the singular and plural forms of nouns. 4) Entity replacement: replace specific entities (such as names of people, places, organization names, etc.) appearing in the text with other entities of the same type. 5) Translation: translate the source text fragment into one or more intermediate languages, and then translate it back to the source language, using the natural changes in the translation process to introduce semantically equivalent but differently expressed texts. Among them, the steps of part-of-speech tagging, word form restoration, etc. involved in the first four mutation operators are implemented by the natural language toolkit NLTK and its integrated WordNet resources to complete the recognition and replacement of named entities using the Tagme tool. The translation operator is implemented through a pre-trained translation model based on MarianMT. The use of these tools ensures the accuracy and efficiency of the mutation operation, providing a high-quality source of mutant individuals for subsequent genetic algorithms.

[0086] The selection of potential mutation positions in the mutation operator is the key to ensuring the effectiveness of the mutation and maintaining the unchanged emotional polarity. It includes dependency syntax analysis, SO value calculation based on extension rules, efficient candidate node screening, and innovative simulation of SO change prediction mechanism. The method of the present invention randomly selects nodes from safe mutation positions based on this mechanism and applies corresponding reinforcement or negation emotion operators. Among them, the reinforcement operation is used to confirm whether the node is suitable for the reinforcement type. This operation is intended to reflect the phenomenon of using reinforcement words or structures (such as "very", "extremely") to strengthen emotions. Its core is to amplify the SO value of the node, and the relevant conversion function is defined as:

[0087] τ(σ)=σ×(1+β)

[0088] Here, τ(σ) represents the amplified SO value of the node, σ represents the original SO value of the node, and β is a weighting factor that adjusts the SO value based on the strength of the intensifier. This function quantifies the impact of intensifiers. By multiplying the SO value by a coefficient greater than 1, we simulate the increase in emotional intensity. This operation is typically applied to adjectives and adverbs, which are the primary carriers of emotional expression and reinforcement.

[0089] The negation operation corresponding to the negation grammatical phenomenon is intended to reflect the phenomenon that negative words (such as "not", "never") or negative structures reverse (or weaken) the emotion. Its conversion function is:

[0090]

[0091] Here, τ'(σ) represents the effect of negation on sentiment, inverting (or weakening) it. σ represents the node's original SO value, and α is an offset factor that adjusts the SO value based on the influence of the negation. This function simulates the process by which negation alters sentiment polarity. If the original sentiment is positive (σ > 0), the SO value decreases after negation, potentially turning it negative. If the original sentiment is negative (σ < 0), the SO value increases after negation, potentially turning it positive or neutral. This reflects the semantic inversion effect of negation.

[0092] Specifically, within the genetic algorithm-based search framework of the present invention, fitness evaluation, selection, crossover, and mutation operations are performed cyclically until a termination condition is reached. The present invention selects NSGA-II as the core optimization algorithm, primarily based on its ability to optimize for multiple objectives. NSGA-II has numerous successful application cases in search-based software engineering (including software testing), and its effectiveness and robustness have been widely verified in academic and practical applications. Since the test case generation of the present invention aims to simultaneously optimize error detection and reduce test false positives—that is, to maximize the degree to which test cases cause changes in model predictions and maximize the similarity and naturalness of test cases with the original text—NSGA-II was selected to achieve efficient multi-objective optimization and achieve a balance between various objectives. NSGA-II also uses a non-dominated sorting method based on Pareto optimality to retain elite individuals in the population, ensuring that excellent solutions are not lost during the evolutionary process. Furthermore, its crowding distance calculation mechanism helps maintain the diversity of solutions on the Pareto front, preventing the algorithm from prematurely converging to a single local optimal solution, thereby obtaining a widely distributed set of high-quality test cases.

[0093] Specifically, during the population initialization phase, for semantic representation and similarity calculation, the present invention adopts sentence-level embedding representation and uses the GloVe (Global Vectors for Word Representation) model to obtain word embeddings. Combined with the Faiss library, this method efficiently performs nearest neighbor searches in the word embedding space, enabling synonym lookup during the initialization phase. This approach allows for differences between individuals in the initial population without destroying the grammatical structure of the original text, making it easier to find optimal test cases during subsequent searches.

[0094] After the genetic algorithm search is completed, the comprehensive evaluation score used to select the final test case is:

[0095]

[0096] Perplexity(x) is a text perplexity metric that automatically determines the naturalness of a text through the model. Combined with a second fitness function, f2, which reflects the degree of semantic and grammatical preservation, a comprehensive score is calculated. Ultimately, the present invention selects the individual with the highest comprehensive score as the final output test case set. This search strategy, based on multi-objective optimization and quality assessment, enables the present method to generate high-quality test cases that effectively challenge model robustness while maintaining a high degree of naturalness and semantic consistency.

[0097] Table 1

[0098]

[0099]

[0100] Table 2

[0101]

[0102] Table 3

[0103]

[0104]

[0105] The verification scheme of the present invention is as follows:

[0106] The data was analyzed using Python language on a server equipped with an Intel(R) Xeon(R) Platinum 8362 central processing unit (CPU) and an NVIDIA RTX3090 graphics processing unit (GPU), and the operating system was Ubuntu 20.04.

[0107] 1) Experimental Setup

[0108] Tested Models: The experiment selected two pre-trained language models based on the Transformer architecture, which are widely used and representative in sentiment analysis tasks, as the test objects: the BERT-base-cased model and the RoBERTa-base model. Both models demonstrated high baseline accuracy on standard sentiment analysis datasets.

[0109] Comparison method: In order to evaluate the relative advantages of the method of the present invention, two existing representative methods that also use the metamorphic testing concept to test sentiment analysis systems were selected as baselines for comparison: (1) CheckList: A comprehensive testing tool for NLP models, which includes testing methods for sentiment analysis tasks and tests by constructing multiple metamorphic relationships (such as inserting sentiment phrases, modifying named entities, etc.). (2) MTE (Metamorphic Testing Effectiveness for Sentiment Analysis): As an evaluation method focusing on the false positive problem of metamorphic testing in the field of sentiment analysis, it defines metamorphic relationships covering multiple grammatical levels. To ensure fairness, the test cases generated by the MTE method were screened based on perplexity to control their generation quality.

[0110] 2) Experimental results

[0111] To verify the defect detection capabilities of the proposed test method on SA systems, we compared it with two representative existing sentiment analysis system testing methods based on metamorphic testing: CheckList and MTE. The experiment selected two widely used open-source pre-trained models (BERT and RoBERTa) as the sentiment analysis systems to be tested and evaluated on six standard datasets (SST-2, SST-5, Yelp-2, Yelp-5, IMDB, and Sentiment140) covering different topics, text lengths, and sentiment label granularity. Each test method was applied to the aforementioned models and datasets, and its defect detection capabilities were measured by counting the number of test cases generated by each method that caused the tested model to violate the preset metamorphic relationship (i.e., detected defects). The experimental data is shown in Table 1, which shows that the proposed test method demonstrated stronger defect detection capabilities in all test scenarios. Specifically, under the same experimental conditions, the total number of defects detected by the proposed method on the BERT and RoBERTa models increased by 16.91% and 7.75%, respectively, compared to the CheckList method. This result proves that the testing method proposed in this invention has obvious advantages in revealing potential defects of sentiment analysis models.

[0112] To verify the performance of the test method proposed in this invention in reducing the false positive rate, experiments were conducted using the same test model, dataset, and comparison method as the aforementioned defect detection capability experiment. Defects reported by each method were manually verified to determine whether they were real defects, thereby calculating the true positive rate (TPR). Specifically, a certain number of samples were randomly sampled from each dataset and the defects reported by each method. Each reported defect was submitted to two researchers with excellent language comprehension skills for independent review and reporting on whether there were false positives, in the form of original text, original labels, generated corresponding test cases, and new labels. The results of the inspection are shown in Table 2. The experimental results show that the test method proposed in this invention performs well in generating test cases and reducing false positives. In all experimental scenarios, this method exhibited a high average true positive rate. On the BERT and RoBERTa models, the average true positive rate of this method was 12.1% and 20.0% higher than that of the MTE method, respectively. In the vast majority (11 / 12) of experimental scenarios, this method achieved the highest true positive rate. Analysis of false positive cases shows that the proposed method effectively ensures the quality (naturalness and semantic coherence) of generated test cases through the semantic and grammatical constraints in its fitness function and the perplexity assessment in the final selection stage. This significantly reduces false positives caused by test case quality issues (such as semantic confusion and grammatical errors) or improper application of metamorphosis relations (such as simply inserting sentiment words or negation words in complex sentences, resulting in expected sentiment judgments being inconsistent with the actual ones). This demonstrates that the proposed method can more accurately identify real defects.

[0113] To evaluate the ability of the testing method proposed in this paper to generate test cases that cover complex linguistic phenomena and analyze the contribution of its core components (particularly sentiment-related mutation operators and high-order mutation strategies) to achieving this coverage, we conducted an in-depth analysis of test cases generated by the method that successfully detected defects. The frequency of use of various mutation operators (including the enhancement and negation classes within the sentiment-related mutation operators, as well as general text mutation operators) during the generation of these defect-exposing test cases was statistically analyzed. The experimental results are shown in Table 3. The results show that the sentiment-related mutation operators designed by the testing method make a significant contribution to defect detection, with both the enhancement class (contributing approximately 67.4%) and the negation class (contributing approximately 21.4%) showing high contributions. The combination of these operators with general operators through high-order mutation enables the generated test cases to more deeply explore the model's ability to understand and process complex linguistic phenomena (such as changes in sentiment intensity, negation logic, and multiple semantic modifications), thereby more effectively testing and exposing potential model flaws in these areas.

[0114] To achieve the aforementioned purpose of the invention, the present invention proposes a search-based sentiment analysis system testing method. The core of the method lies in applying the metamorphic testing principle, combining a set of carefully designed text mutation operators with a genetic algorithm-based optimization search framework to automatically generate high-quality test cases for evaluating sentiment analysis systems.

[0115] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications made without departing from the principles of the present invention should also be considered to fall within the scope of protection of the present invention.

Claims

1. A testing method for a text sentiment analysis task model based on a genetic algorithm, characterized in that: The following steps are involved: S1. Randomly select original seed text test cases from existing text data; S2. Design mutation operators, including sentiment analysis mutation operators and general text mutation operators; S3. Constructing a search framework, calling the search framework based on a genetic algorithm, selecting original seed text test cases based on the evaluation scores of the dual-objective fitness function, obtaining candidate test cases, applying the mutation operator to the candidate test case combinations, finding test case optimization solutions, and automatically generating final test cases; S4. Using the test case to test the sentiment analysis task model to be tested to obtain a defect assessment result based on the metamorphosis relationship.

2. The test method of the text sentiment analysis task model based on genetic algorithm according to claim 1 is characterized in that, The mutation operator includes a sentiment analysis mutation operator, and the sentiment analysis mutation operator further includes a strengthening sentiment mutation operator and a negation sentiment mutation operator.

3. The testing method of the text sentiment analysis task model based on genetic algorithm according to claim 1 is characterized in that, The mutation operator further includes a general text mutation operator, which includes a word replacement operation that does not change the semantics. This operation is based on candidate node screening in the lexical relationship dependency tree, and selects some nodes that may have the greatest contribution to the overall sentiment as potential mutation positions.

4. The testing method of the text sentiment analysis task model based on genetic algorithm according to claim 1 is characterized in that, The selection of the mutation position is based on dependency syntax analysis, SO value calculation based on extension rules, efficient candidate node screening and simulated SO change prediction mechanism to randomly select nodes from safe mutation positions.

5. The testing method of the text sentiment analysis task model based on genetic algorithm according to claim 1 is characterized in that, In S3, candidate test cases are selected based on the evaluation scores of the dual-objective fitness function, and the mutation operator is applied to the selected candidate test case combinations to find the test case optimization solution. This step further includes: The search framework is called to iteratively perform a search based on a genetic algorithm: i) population initialization: replace words in the original seed text test case, randomly generate original seed text test cases as samples, and obtain an initial population; ii) fitness evaluation: design a fitness function, and use the fitness function to obtain the evaluation score of the candidate test case; iii) selection, crossover and mutation: select the parent individual of the candidate test case from the current population according to the evaluation score of the candidate test case, exchange some text fragments in the parent individual, generate new child individuals, and then apply the mutation operator to all child individuals with a set probability to obtain child mutant individuals; iv) next generation population generation: merge the child mutant individuals, select the same number of child mutant individuals as the initial population to form the next generation population, and when the next generation population no longer changes or reaches the specified number of iterations, the test case optimization solution to be found is obtained.

6. The testing method of the text sentiment analysis task model based on genetic algorithm according to claim 1 is characterized in that, S3 further comprises applying different types of mutation operators to the selected candidate test cases, and further comprises judging and applying an enhancement-type sentiment mutation operator or a negation-type sentiment mutation operator respectively; wherein, confirming that the node adapts to the enhancement grammatical phenomenon corresponds to the enhancement-type operation, amplifying the SO value of the node, and defining the conversion function as: τ(σ)=σ×(1+β) Among them, τ(σ) is the amplified result of the node's SO value, σ is the original SO value of the node, and β is the weighting factor, which adjusts the SO value according to the strength of the strengthening word; Confirm that the node adapts to the negation operation corresponding to the negation grammatical phenomenon and weakens the SO value of the node. The conversion function is: Among them, τ'(σ) is the result of the negation structure to strengthen or weaken the sentiment, σ is the original SO value of the node, and α is the offset factor, which adjusts the SO value according to the influence of the negation word.

7. The testing method of the text sentiment analysis task model based on genetic algorithm according to claim 1 is characterized in that, The dual-objective fitness function is expressed as follows: The first objective fitness function f1 is used to quantify the degree to which the candidate test case causes the change in the prediction probability of the sentiment analysis model to be tested, and the second objective fitness function f2 is used to comprehensively evaluate the similarity or preservation degree of the candidate test case with the original text in terms of semantic content and grammatical structure.

8. The testing method of the text sentiment analysis task model based on genetic algorithm according to claim 7 is characterized in that, The first objective fitness function f1 is specifically: f1=|Model(Ind)-Model(Orig)| Among them, Model(·) is the predicted output of the sentiment analysis task model, Ind is the candidate test case, and Orig is the original seed text test case.

9. The testing method of the text sentiment analysis task model based on genetic algorithm according to claim 7 is characterized in that, The second fitness objective function f2 is specifically: f2=(1-λ)Sim(Infer(Orig),Infer(Ind))+λTed(Ind,Orig) Among them, Sim(·) is the semantic similarity, TED(·) is the grammatical structure similarity, and λ is the weight factor.

10. The testing method of the text sentiment analysis task model based on genetic algorithm according to claim 7, characterized in that: The final comprehensive evaluation score of test case x is: Among them, Perplexity(x) is the perplexity index of the final test case x, and f2 is the second fitness function of the final test case x.

Images