A software testing method
By generating an abstract syntax tree and extracting the syntax tree vectors of feature nodes, test cases are automatically generated, solving the problem of time-consuming and labor-intensive manual test case writing in traditional software testing methods, and achieving more efficient and accurate automated testing.
Patent Information
- Application Number
- CN202510584424.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2026-11-10
AI Technical Summary
Traditional software testing methods rely on manually writing test cases, which is costly in terms of manpower, has low testing accuracy, and the automated verification process is not fully optimized, leading to potential software quality issues.
By generating an abstract syntax tree, extracting feature nodes and generating syntax tree vectors, and using natural language processing technology to automatically generate test cases, only the code related to the test cases is vectorized, and similarity comparison is performed to determine whether the source code meets the test requirements.
It improves the accuracy of test results, reduces the impact of irrelevant code, enhances the efficiency and accuracy of automated testing, and reduces the need for manual intervention.
Smart Images

Figure CN123068263A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a software testing method, belonging to the field of software testing. Background Technology
[0002] With the increasing demand for software in society, the frequent iterations and evolutions in software development necessitate extensive testing activities, making software testing an indispensable part of the software development process. Traditional software testing, to better manage and execute testing activities, typically requires testers to spend a significant amount of time mastering the source code, program design, and internal working mechanisms of the software under test. This often relies on the testers' experience to manually develop test plans and design test cases. This traditional testing method, focusing on manually writing a large number of test cases, consumes considerable manpower and time, extends project timelines, and is highly subjective, leading to low testing accuracy, defects, vulnerabilities, and security risks, seriously impacting software quality.
[0003] Based on natural language processing technology, software requirements documents can be automatically processed to generate test cases with semantic logic. This technology has been implemented using mature open-source platforms and toolchains, greatly reducing the manual writing burden on testers. However, although test cases can be automatically generated, subsequent verification still requires manual intervention. As a result, the overall testing process has not yet achieved a complete breakthrough in automation and efficiency optimization, and there is still room for further optimization.
[0004] Chinese patent application publication number CN116756012A discloses a method and apparatus for verifying test cases. This method vectorizes the test cases and source code separately, compares the similarity between these two vectors, and then automatically determines whether the test cases meet user requirements.
[0005] However, its complete vectorization of source code leads to many interfering factors when performing similarity comparisons, resulting in inaccurate comparison results. Summary of the Invention
[0006] The purpose of this invention is to provide a software testing method to solve the problem of inaccurate comparison results when automatically comparing test cases with source code.
[0007] To achieve the above objectives, the present invention includes:
[0008] A software testing method according to the present invention includes the following steps:
[0009] 1) During the process of traversing the abstract syntax tree generated from the source code, extract the corresponding data based on the feature nodes in the abstract syntax tree, and generate the corresponding syntax tree vector based on the extracted data;
[0010] 2) Compare the similarity between the syntax tree vector and the test case vector generated from the corresponding test cases in the source code, and determine whether the source code meets the test requirements based on the comparison results;
[0011] Feature nodes are nodes in the abstract syntax tree that match predefined feature factors; feature factors are determined based on the corresponding words in the text dataset of the test cases.
[0012] Furthermore, the process of determining feature factors includes: 1) segmenting the text dataset of test cases into words; 2) tagging the words in the segmented text dataset with part-of-speech tags; 3) determining the corresponding words from the part-of-speech-tagged text dataset according to the predefined part-of-speech tags, and using the determined words as feature factors.
[0013] Furthermore, when traversing the abstract syntax tree, when a feature node is detected, the feature node and its directly associated child nodes are extracted, and the corresponding syntax tree vector is generated based on the feature node and child nodes.
[0014] Furthermore, when the comparison result is greater than the corresponding set threshold, the corresponding source code is judged to meet the test requirements. The threshold determination process includes: extracting vector segments with the same dimension as the test case vector from the syntax tree vector, calculating the proportion of the feature vector in each vector segment, calculating the average of the proportions based on the proportions and the number of vector segments, and using the average as the threshold; the feature vector is the value of the feature node after vectorization.
[0015] Furthermore, vector segments are extracted sequentially according to the set dimensions.
[0016] Furthermore, we set the dimension to 1.
[0017] Furthermore, test cases are generated based on the requirements document corresponding to the source code.
[0018] Furthermore, natural language processing is used to process the requirements document to obtain test cases.
[0019] Furthermore, the ANTLR tool is used to generate an abstract syntax tree.
[0020] Furthermore, the Word2vec algorithm is used to generate syntax tree vectors or use case vectors.
[0021] The beneficial effects of this invention are as follows: This invention is a pioneering invention. When generating corresponding vectors from source code, it traverses the abstract syntax tree generated from the source code. During the traversal, corresponding nodes are extracted based on determined feature nodes, and then the extracted nodes are transformed into corresponding vectors. The feature nodes are determined based on test cases. Then, the vectors generated from the test cases are compared with the vectors generated from the source code for similarity. Based on the comparison result, it is determined whether the corresponding source code meets the test requirements. This invention only vectorizes the code related to the test cases when vectorizing the source code, which can effectively reduce the influence of irrelevant code and improve the accuracy of the comparison. Attached Figure Description
[0022] Figure 1 This is a schematic diagram of a test process according to the present invention;
[0023] Figure 2 This is a schematic diagram of a test case generation process according to the present invention;
[0024] Figure 3 This is a schematic diagram of a feature factor extraction process according to the present invention;
[0025] Figure 4 This is a schematic diagram of an abstract syntax tree according to the present invention. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0027] The concept of this invention is that, when vectorizing the source code, only the code directly related to the test cases is vectorized, and then the similarity between the vectorized code vector and the vectorized test case vector is compared, and the test requirements are determined based on the comparison results.
[0028] Method Implementation Examples:
[0029] This embodiment provides a software testing method, such as Figure 1 As shown, it includes the following steps:
[0030] S1. Generate test cases based on the software requirements document, and generate corresponding test case vectors based on the test cases;
[0031] S2. Determine feature factors based on test cases;
[0032] S3. Construct an abstract syntax tree based on the source code, which is developed according to the requirements document.
[0033] S4. Traverse the abstract syntax tree and generate the corresponding syntax tree vector based on the feature nodes;
[0034] S5. Compare the similarity between the test case vector and the syntax tree vector, and determine whether the test requirements are met based on the comparison results.
[0035] In step S1, the requirement document is processed using natural language processing technology based on the existing open-source transform model. The self-attention mechanism of the model is used to capture long-distance dependencies in the requirement document and automatically generate corresponding test cases.
[0036] Specifically, such as Figure 2 As shown, for example, if a user's requirement is to query information about oil wells with well number "A-123", the steps to generate test cases based on this requirement include:
[0037] S101. Text Preprocessing: This involves data cleaning, de-colloquializing user requests, which requires natural language processing technology to understand sentence structure and identify core actions. In this example, there is a primary action: "query." This action corresponds to two distinct stages: the implementation of the function and the processing of the function's result.
[0038] S102, Word Segmentation and Semantic Analysis: Using word segmentation tools, sentences are divided into individual word units.
[0039] Original sentence: Query information for oil wells with well number "A-123"; after word segmentation: "query", "well number", "for", "A-123", "of", "oil", "well", "information".
[0040] Then, semantic analysis is performed to filter out stop words in the word segmentation results. Stop words usually refer to words that do not contribute much to the meaning of the sentence but appear frequently in the text, such as "and", "will", "to carry out", etc.
[0041] After filtering out stop words, the sentence becomes "query", "hash code", "for", "A-123", "of", "oil well", and "information".
[0042] S103. Perform context analysis on the words obtained in S102, extract key information and features, conduct context matching, and generate corresponding test cases that fit the user's usage scenario:
[0043] Test Case 1: Normal Query
[0044] T1. Log in to the system successfully using the correct username and password;
[0045] T2. Navigate to the oil well query page;
[0046] T3. Edit the hash number and input the data;
[0047] T4. Click the query button.
[0048] Test Case 2: Query if hash symbol does not exist
[0049] T1. Navigate to the oil well information query page;
[0050] T2. Enter a non-existent hash symbol in the query input box, such as "B-999";
[0051] T3. Click the query button or perform a query operation;
[0052] T4. Check if the system returns a "hash not found" message or a similar message.
[0053] Test Case 3: Input Format Validation
[0054] T1. Navigate to the oil well information query page;
[0055] T2. Try entering a hash symbol that does not meet the format requirements, such as "123" (without prefix), "A123" (containing spaces), "A#123" (containing the illegal character #);
[0056] T3. Click the query button or perform a query operation;
[0057] T4. Check if the system returns a format error message and prevent the query from executing.
[0058] Test Case 4: Boundary Value Testing
[0059] T1. Try to find hash numbers whose length is close to the maximum length allowed by the system (if the system has restrictions);
[0060] T2. Try to find hash numbers whose length is close to the minimum length allowed by the system (if the system has restrictions);
[0061] T3. Check whether the system can correctly handle these boundary value cases and return the correct results or error messages.
[0062] S104. Use the Word2vec algorithm to map the words in each test case to vectors (v i The formula corresponding to this algorithm is:
[0063] v i =E·O i (1)
[0064] Where E is the pre-trained embedding matrix of the publicly available Word2vec model, with a size of d·N, where d is the dimension of the embedding vectors, N is the total number of words in the vocabulary, and i is the index in the vocabulary. Each column of matrix E represents the embedding vector of a category. iIt is a one-hot encoded vector of size N, with all positions except the i-th position being 0, all other positions being 1. Since the Word2vec algorithm is designed for single words and does not directly support converting entire sentences into single vectors, the average word vector method is used to obtain the corresponding test case vector (L...). n Let the dimension of this vector be k.
[0065]
[0066] Among them, v i is the word vector of the i-th word, and N is the number of words in the test case.
[0067] The test case vector L1 corresponding to test case 1 is then obtained as: [0.69954254, 0.70069895, 0.51637494, 0.56012356, 0.52109646, 0.37018579, 0.50469242, 0.567216, 0.28608799, 0.19791621] (K=10);
[0068] The test case vector L2 corresponding to test case 2 is: [-0.00713902, 0.00124103, -0.00717672,
[0069] -0.00224462, -2.4183906, 9.0092728, 5.1033497, 0.00125673, -0.00224462, 3.0773198, 4.0092667, -0.01324475, 7.7789325] (K=13);
[0070] The test case vector L3 corresponding to test case 3 is: [0.6097682, -0.076375, 0.0674386, -0.697552, -1.329485, 8.88214367, 6.0281, -0.6276944, 0.1033497, 3.782367, 6.0862117, -5.738849, 0.05903312, 0.000103, 5.1033497] (K=15);
[0071] The test case vector L4 corresponding to test case 4 is: [1.9766092, -8.362775, -0.00717672, -3.6944627, 4.7518332, 8.88214367, 9.0092728, -3.6944627, 5.1033497, 8.88214367, 3.0773198, -9.462001, -2.05718902, 0.00102103, 5.1033497, -3.6944627, 0.0037193, -2.05718902, 0.0037193, 3.0773198, -0.000064] (K=21).
[0072] Specifically, such as Figure 3 As shown in S201, data filtering is performed on the four test cases mentioned above to generate the corresponding text datasets.
[0073] Test Case 1: {Log in to the system using username and password, enter the query page, enter the correct hash symbol, perform a query, and determine if it is correct}; Test Case 2: {Enter the query page, enter an incorrect hash symbol, perform a query, and determine if it is correct}; Test Case 3: {Enter the query page, enter an incorrect hash symbol format, perform a query, and determine if it is correct}; Test Case 4: {Enter the query page, enter the longest hash symbol length, enter the shortest hash symbol length, and determine if it is correct}.
[0074] S202. Perform word segmentation on the text dataset obtained in S201: Test case 1: {username, password, login, query, hash, edit, query, judge, return}; Test case 2: {login, query, error hash, edit, query, judge, return}; Test case 3: {hash, edit, error format, query, judge, return}; Test case 4: {hash, edit, longest hash, shortest hash, query, judge, return}.
[0075] S203. Perform part-of-speech tagging on the segmented text dataset obtained in S202. Define words in the segmented text dataset that are verbs, gerunds, nouns, or adjectives as feature factors for the corresponding test cases.
[0076] The characteristic factors for test case 1 are: login\username\password\query\oil well\edit\well number\click\query\button\judgment\return; the characteristic factors for test case 2 are: login\query\edit\incorrect well number\query\judgment\return;
[0077] The feature factors for test case 3 are: hash symbol, edit, error format, query, judgment, return; the feature factors for test case 4 are:
[0078] Hash symbol\Edit\Longest\Shortest\Query\Judgment\Return.
[0079] Then, the eigenfactors are converted from Chinese to English:
[0080] The feature factors after transformation for test case 1 are:
[0081] login\username\password\query\oilWell\input\wellNum\onclick\query\button\if\while\return;
[0082] The transformed feature factors for test case 2 are:
[0083] login\query\input\wellNum\wrong\query\button\if\while\return;
[0084] The transformed feature factors for test case 3 are:
[0085] wellNum\input\wrong\format\query\button\if\while\return;
[0086] The transformed feature factors for test case 4 are:
[0087] wellNum\input\format\longest\shortest\query\button\if\while\return.
[0088] Specifically, in step S3, the source code is first obtained, and the corresponding programming language is selected. For the source code of the business layer and the functional implementation layer, the ANTLR tool is used to construct an abstract syntax tree. The syntax of the programming language used in the source code is defined through the ANTLR syntax rule file (.g file). The ANTLR tool automatically generates a lexer and a parser based on these syntax rules. The lexer is responsible for segmenting the input text into a series of tokens, while the parser constructs the abstract syntax tree based on these tokens. Variable names, function names, and class names are defined using typical identifier naming rules by defining lexical units that begin with a letter (uppercase or lowercase) and an underscore (_). When $text (i.e., the recognized text fragment) equals any one of the four strings "func", "parm", "class", or "var", the following operations can be performed. These strings need to be specially marked because they have special meanings or uses in specific contexts or languages. The `fragment` keyword is used to define segments of lexer rules. These segments are not directly recognized as independent tokens by the lexer; instead, they are used as part of other lexer rules. This mechanism allows for the reuse of complex regular expressions. An abstract syntax tree is constructed by specifying variable data types, error handling keywords, handling null characters, and declaring variables, functions, classes, parameter lists, parameters, code blocks, statements, and expressions within the syntax rules.
[0089] Specifically, in step S4, based on the abstract syntax tree constructed in step S3, the ANTLR listener pattern is used to dynamically monitor key nodes in the code (such as function names, variable names, and class names). According to the depth-first traversal algorithm used in the listener pattern, starting from the root node of the abstract syntax tree, the algorithm traverses the branches of the tree as deeply as possible, obtaining leaf nodes containing feature factors until no more adjacent nodes can be visited. Then, the algorithm backtracks to the nearest accessible unvisited adjacent node and continues traversing. During the traversal, only feature nodes and their directly associated child nodes (such as parameters and return values) are extracted, ignoring irrelevant branches. Here, a feature node refers to a node in the abstract syntax tree that matches the feature factors after Chinese-English conversion. Figure 4 The username, password, return, etc. in the text.
[0090] The extracted data is converted into a syntax tree vector using the Word2vec algorithm. Let the dimension of this vector be B. Then:
[0091] The syntax tree vector TR1 transformed based on the feature factors determined by test case 1 is: [0.69954254, 7.817348144, -0.47717672, 0.4973702475, 4.7516232, 8.88214367, 9.0092728, 0.52109646, 5.1033497, 9.80214367, 3.0762198, 0.37018579, -2.05733902, 5.1033497, 0.4971902475, 3.0773198, -0.098064] (B = 17);
[0092] The syntax tree vector TR2 transformed based on the feature factors determined by test case 2 is: [0.4973702475, 4.7516232, 8.88214367, 0.4973702475, 4.7516232, 8.88214367, 9.0092728, 0.52109646, 5.1033497, 0.00124103, -0.00717672, -0.00] 224462, -2.4183906, 9.0092728, 5.1033497, 0.00125673, -0.00224462, 9.80214367, 3.076 2198, 0.37018579, -2.05733902, 5.1033497, 0.4971902475, 3.0773198, -0.098064] (B=25);
[0093] The syntax tree vector TR3, transformed from the feature factors determined by test case 3, is: [7.654321, 9.9144144, 3.47717672, 0.4973702475, 4.7516232, 8.88214367, 9.0092728, 0.52109646, 5.1033497, 9.80214367, -1.329]. 485, 8.88214367, 6.0281, -0.6276944, 0.1033497, 3.782367, 6.0862117, 3.076219 8, 0.37018579, -2.05733902, 6.009742, 4.90245275, 3.0773198, 8.949224] (B=24);
[0094] The syntax tree vector TR4, transformed from the feature factors determined by test case 4, is: [0.69954254, 7.817348144, -0.00717672, -3.6944627, 4.7518332, 8.88214367, 9.0092728, 4.7516232, 8.88214367, 5.1033497, 9.80214367, 3.0762198, -0.0022]. 4462, -2.4183906, 0.37018579, 9.0092728, -2.05733902, 5.1033497, 5.1033497, -3.6944627, 0. 0037193, -2.05718902, 0.0037193, 3.0773198, 0.4971902475, 0.00125673, -0.00224462] (B=27).
[0095] Specifically, in step S5, cosine similarity is calculated based on the generated use case vector and syntax tree vector using the cosine formula. Since these two vectors may have different dimensions, if the dimensions are different, their dimensions are first unified by taking the least common multiple of the two vectors, and then the average value is used to supplement the vector with the lower dimension. Furthermore, to avoid negative correlation interfering with the judgment, an absolute value is introduced into the cosine formula to unify the similarity within the range [0, 1]. The N-dimensional cosine theorem formula is:
[0096]
[0097] Among them, A i For the i-th use case vector after dimension unification, B i Let be the i-th syntax tree vector after dimension unification, and n be the number of vectors.
[0098] When the calculated cosine similarity is greater than or equal to the set threshold (cosθ ≥ M), it proves that the source code fulfills the corresponding requirements, and confirms that the generated test cases can achieve the expected testing results. The set threshold reflects the degree to which the code fulfills the requirements; the higher the coverage, the higher the similarity requirement.
[0099] This embodiment also provides a process for determining the set threshold:
[0100] From the syntax tree vector generated in step S4, extract vector segments with the same dimension as the use case vectors by extending one dimension: the dimension of use case vector L1 is K=10, and the dimension of syntax tree vector TR1 is B=17, so B-K+1=8 vector segments can be extracted from TR1; the dimension of use case vector L2 is K=13, and the dimension of syntax tree vector TR2 is B=25, so B-K+1=13 vector segments can be extracted from TR2; the dimension of use case vector L3 is K=15, and the dimension of syntax tree vector TR3 is B=24, so B-K+1=10 vector segments can be extracted from TR3; the dimension of use case vector L4 is K=21, and the dimension of syntax tree vector TR4 is B=27, so B-K+1=7 vector segments can be extracted from TR4.
[0101] Then, calculate the proportion P of the feature vectors in each vector segment of each syntax tree vector. Here, the feature vector refers to the vectorized value of the feature node in that vector segment:
[0102]
[0103] Where Y is the number of feature vectors contained in a certain vector segment, and K is the dimension of the vector segment.
[0104] Then, based on the proportion P of the feature vectors of each vector segment in each syntax tree vector, calculate the mean proportion of the feature vectors of each syntax tree vector:
[0105]
[0106] The mean As the corresponding set threshold M, for example, if the set threshold M1 is determined by the syntax tree vector TR1, then M1 will be used as the set threshold when comparing the similarity between the use case vector L1 and the syntax tree vector TR1.
[0107] Based on the experimental data, the threshold ratios for similarity comparison between the four use case vectors and the syntax tree vectors were set as follows: M1 = 0.56, M2 = 0.67, M3 = 0.52, and M4 = 0.31. Then, the use case vectors and syntax tree vectors were compared using cosine similarity, with approximate values of cosθ1 = 0.57, cosθ2 = 0.667, cosθ3 = 0.089, and cosθ4 = 0.421.
[0108] Based on the experimental calculation results, since the cosine similarity values of test cases 1 and 2 compared with the syntax tree vector are greater than the set threshold, it proves that the source code can realize the user requirements related to test cases 1 and 2 in the requirements document, and confirms that the generated test cases are positive examples that can realize the test expectations. Test cases 3 and 4 are negative examples that the current source code fails to realize the user requirements.
[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A software testing method, characterized in that, Includes the following steps: 1) During the process of traversing the abstract syntax tree generated from the source code, extract the corresponding data based on the feature nodes in the abstract syntax tree, and generate the corresponding syntax tree vector based on the extracted data; 2) Compare the similarity between the syntax tree vector and the test case vector generated from the corresponding test cases in the source code, and determine whether the source code meets the test requirements based on the comparison results; The feature nodes are nodes in the abstract syntax tree that match pre-determined feature factors; the feature factors are determined based on the corresponding words in the text dataset of the test cases.
2. The software testing method according to claim 1, characterized in that, The process of determining feature factors includes: 1) segmenting the text dataset of test cases into words; 2) tagging the words in the segmented text dataset with part-of-speech tags; 3) determining the corresponding words from the part-of-speech-tagged text dataset according to the predefined part-of-speech tags, and using the determined words as feature factors.
3. The software testing method according to claim 1, characterized in that, When traversing the abstract syntax tree, when a feature node is detected, the feature node and its directly associated child nodes are extracted, and the corresponding syntax tree vector is generated based on the feature node and the child nodes.
4. The software testing method according to claim 1, characterized in that, When the comparison result is greater than the corresponding set threshold, the corresponding source code is judged to meet the test requirements. The process of determining the threshold includes: extracting vector segments with the same dimension as the test case vector from the syntax tree vector, calculating the proportion of the feature vector in each vector segment, calculating the average of the proportions based on the proportions and the number of vector segments, and using the average as the threshold; the feature vector is the value of the feature node after vectorization.
5. The software testing method according to claim 4, characterized in that, Extract vector segments sequentially according to the set dimensions.
6. The software testing method according to claim 5, characterized in that, The specified dimension is 1.
7. The software testing method according to claim 1, characterized in that, The test cases are generated based on the requirements document corresponding to the source code.
8. The software testing method according to claim 7, characterized in that, The requirements document is processed using natural language processing techniques to obtain the test cases.
9. The software testing method according to claim 1, characterized in that, The abstract syntax tree was generated using the ANTLR tool.
10. The software testing method according to claim 1, characterized in that, Use the Word2vec algorithm to generate syntax tree vectors or use case vectors.
Citation Information
Patent Citations
Test case verification method and device
CN116756012A
