Test case prioritization methods, equipment, and media for large-scale continuous integration

Abstract

The present invention discloses a method, device and medium for prioritizing test cases for large-scale continuous integration, relates to the technical field of computer software testing, and solves the problem that existing test case sorting methods cannot sort test cases quickly, efficiently and accurately. The key points of its technical solution are: first sorting, the first sorting operation includes parsing target code, the target code includes test code, and lemmatizing the target code to obtain at least a test case representation; performing a first sorting on the test cases through a retrieval algorithm to obtain a preliminary test set; and second sorting, the second sorting includes using the preliminary test set after screening and sorting as a training set for a model; performing a second sorting on the preliminary test set using a model trained on the training set to obtain a test case priority sorting result, thereby achieving the purpose of quickly and efficiently sorting the test cases and improving the sorting accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of computer software testing, and more particularly to a method, device and medium for prioritizing test cases for large-scale continuous integration. Background Art

[0002] Continuous integration encourages developers to frequently integrate code changes into the mainline codebase and verifies the quality of submitted code through automated builds and tests. Compared to traditional development models, continuous integration facilitates earlier and faster detection of code errors, thereby supporting rapid product iteration. With the widespread adoption of continuous integration, large amounts of code changes are submitted at an extremely frequent rate. Due to the high frequency of code changes, the number of tests required increases at an alarming rate. Optimizing continuous integration to reduce testing costs and shorten the test feedback cycle to improve testing cost-effectiveness is a major challenge in the practical application of continuous integration.

[0003] Currently, the continuous integration test optimization methods proposed by people are as follows: (1) Traditional TCP technology based on coverage information, risk analysis, etc. becomes inapplicable in typical continuous integration environments because program code changes are submitted extremely frequently, and a large amount of information analysis and collection requires a lot of time cost. (2) Lightweight test case ranking models based on heuristic rules or information retrieval technology ignore information such as code semantics and context, resulting in a large amount of room for improvement in test ranking performance. (3) Test case ranking technology based on deep learning (e.g., reinforcement learning, pre-trained language model BERT) can achieve good ranking performance, but complex model training requires a lot of training time and complex parameter settings.

[0004] In summary, the existing test case sorting methods, such as those based on coverage, information retrieval technology, machine learning, or deep learning methods, are unable to meet the testing requirements in large-scale continuous integration. Summary of the Invention

[0005] The purpose of the present invention is to provide a method, device and medium for prioritizing test cases for large-scale continuous integration, which performs preliminary sorting and screening of test cases in sequence through a first sorting to obtain a screened test set, and uses a second sorting to accurately sort the screened preliminary test set, thereby achieving the purpose of quickly and efficiently sorting test cases and improving sorting accuracy.

[0006] The above technical objectives of the present invention are achieved through the following technical solutions:

[0007] A method for prioritizing test cases for large-scale continuous integration, including:

[0008] A first sorting operation includes parsing target code, the target code including test code, performing lemmatization on the target code to obtain at least test case representations; performing a first sorting on the test cases using a retrieval algorithm to obtain a pre-sorted test set; and

[0009] The second sorting includes using the filtered and sorted preliminary test set as a training set for the model; performing a second sorting on the preliminary test set using the model trained on the training set to obtain a test case priority sorting result.

[0010] Furthermore, the target code also includes a modification program code;

[0011] In the first sorting, the modified program code is lemmatized to obtain a modified program code representation;

[0012] In the second order, when training the model, the training content consists of learning the semantic representation of changes in program code and test cases;

[0013] The correspondence between the modified program code and the test case is: 1 modified program code representation Q corresponds to m test case representations, where m is an integer and is greater than or equal to 1.

[0014] Furthermore, obtaining at least a test case representation includes the following steps:

[0015] Use Javalang of Python language to perform syntax and lexical analysis on test case code, and build an abstract syntax tree to structure the code;

[0016] Traverse the abstract syntax tree to extract identifiers;

[0017] Split the long identifier into individual morphemes according to the long identifier segmentation rule;

[0018] Filter meaningless words and stop words to obtain a test case representation consisting of tokens.

[0019] Furthermore, when repeated tokens appear in the program code representation or test case representation;

[0020] Evaluate the uniqueness of the token, optionally performing the following operations:

[0021] 1) Maintaining token uniqueness: De-duplication of tokens in the modified program code representation or test case representation to maintain the uniqueness of each token in the set;

[0022] 2) Maintain the frequency of occurrence of the original word; allow the recurrence of word units, and record the frequency of word units in the modified program code representation and test case representation.

[0023] Furthermore, in the first sorting, the test cases are sorted by the retrieval algorithm to obtain the first sorted test set, which includes the following operations:

[0024] Given a changed program code, the retrieval algorithm uses the BM25 algorithm to calculate the similarity score between the changed program code and each test case, and then sorts the test cases according to the BM25 algorithm score;

[0025] A certain number of top-ranked test cases and bottom-ranked test cases are selected from the first-ranked test set obtained as the training set of the model, and subsequent operations are completed using this filtering coefficient.

[0026] Furthermore, in the second sorting, the following steps are included:

[0027] Organization of training data; construct pairs of program code and test case representations from the training set, and set labels for each test case representation; given a program code representation Q, the i-th test case representation corresponding to Q is t i and label l i , construct triple T i (Q,t i ,l i );

[0028] The model adopts the Siamese-MiniLM model, and the constructed triples are input into the Siamese-MiniLM model. After model training, the semantic embedding vectors u and v of the changed program code and test cases are obtained, and the test set is sorted according to the semantic similarity between u and v.

[0029] Furthermore, when a test case detects a program error in the last version of the real dataset, it is assigned a first label, which indicates that there is a certain similarity between the modified program code and the test case representation in semantics and functionality;

[0030] On the contrary, other test cases that do not detect program errors are assigned a second label that is different from the first label.

[0031] Furthermore, the loss function is designed by comparing the cosine similarity between the changed program code u and the test case v with the true label of the test case;

[0032] Among them, every time the model is trained for a preset number of steps, it will be evaluated once on the validation set, and the model will be updated according to the loss function, retaining the model with better performance.

[0033] An electronic device comprises: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method described.

[0034] A computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The test cases are preliminarily sorted and screened in turn through the first sorting to obtain the screened test set, and the screened preliminarily sorted test set is accurately sorted using the second sorting, so as to achieve the purpose of quickly and efficiently sorting the test cases and improving the sorting accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0038] Figure 1 Schematic diagram of the process of the method for prioritizing test cases for large-scale continuous integration in this embodiment;

[0039] Figure 2 This is a schematic diagram of the structure of the test precision ranking of the Siamese-MiniLM twin network based on machine translation in this embodiment;

[0040] Figure 3 This is a schematic diagram showing the comparison results between the test case prioritization method for large-scale continuous integration in the example provided in this embodiment, three baseline methods, and actual continuous integration testing. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with examples and drawings. The exemplary embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention.

[0042] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0043] Example: Method, device and medium for prioritizing test cases in large-scale continuous integration,

[0044] like Figure 1 As shown in the figure, a method for prioritizing test cases for large-scale continuous integration includes:

[0045] A first sorting operation includes parsing target code, the target code including test code, performing lemmatization on the target code to obtain at least test case representations; performing a first sorting on the test cases using a retrieval algorithm to obtain a pre-sorted test set; and

[0046] The second sorting includes using the filtered and sorted preliminary test set as a training set for the model; performing a second sorting on the preliminary test set using the model trained on the training set to obtain a test case priority sorting result.

[0047] The target code also includes a changed program code; in the first sorting, the changed program code is lemmatized to obtain a changed program code representation; in the second sorting, when training the model, the training content includes learning the semantic representation of the changed program code and the test case; wherein, the correspondence between the changed program code and the test case is: 1 changed program code representation Q corresponds to m test case representations, where m is an integer and m is greater than or equal to 1.

[0048] Obtain at least a test case representation, and the acquisition step includes the following steps: using Javalang of the python language to perform grammatical and lexical analysis on the test case code, and establishing an abstract syntax tree to structure the code; traversing the abstract syntax tree to extract identifiers; dividing the long identifier into single morphemes according to the long identifier segmentation rule; filtering meaningless words and stop words to obtain a test case representation composed of morphemes.

[0049] Specifically, the test case representation is obtained by tokenizing the test case using an abstract syntax tree (AST). This process involves four steps: a) Parsing the test case and building the AST. Using Javalang (a Python-based language), the test case's syntax and morphology are analyzed and parsed to generate the AST. b) Extracting identifiers. Identifiers that follow naming conventions often carry semantic information related to the test case's functionality. Therefore, all identifiers are extracted by traversing the abstract syntax tree. c) Segmenting long identifiers. To more accurately convey intent, identifiers (such as class names, method names, and variable names) are often named using token concatenation or camelCase, resulting in long identifiers (e.g., the function name printEmployPaychecks()). Therefore, long identifiers are segmented into individual morphemes based on features such as capital letters and symbols to obtain richer semantic features. d) Filtering meaningless words and stop words. In English, there are many frequently used words or phrases such as "a," "the," and "or," which are often articles, prepositions, adverbs, or conjunctions. Indexing all of these words would be a tremendous amount of work for search engines. Therefore, during the experiment, all meaningless morphemes of this type, including English stop words and Java keywords, will be filtered out to obtain a test case representation composed of lemmas.

[0050] When repeated words appear in the changed program code representation or test case representation; the uniqueness of the words is evaluated, including the optional following operations: 1) maintaining the uniqueness of the words; deduplicating the words in the changed program code representation or test case representation to maintain the uniqueness of each token in the set; 2) maintaining the frequency of occurrence of the original words; allowing the repeated occurrence of words, and recording the word frequency of the words in the changed program code representation and test case representation.

[0051] Specifically, the modified program code representation is derived from the modified program code between two versions using the same tokenization method as the test case. A modified program file may contain only one or a few lines of code changes, and the entire code in the modified file is used to modify the modified code in the modified program file. Since tokens may recur in both the modified program code representation and the test case representation, maintaining token uniqueness can have different impacts on model performance. Therefore, after analyzing the impact of different strategies on the model and selecting the strategy that achieves better performance for subsequent operations, the following two strategies are provided: 1) Maintain Token Uniqueness (MU)—Duplicate tokens are removed from the modified program code representation or the test case representation to maintain the uniqueness of each token in the set; 2) Maintain Token Frequency (MF)—To allow token recurrence, the frequency of token occurrence in the modified program code representation or the test case representation is recorded. Compared with existing techniques, this example evaluates the impact of different strategies on model performance and selects the appropriate strategy to achieve optimal model performance. For example, STIR adopts the MF strategy, while the baseline method OptIR adopts the MU strategy.

[0052] In the first sorting, the test cases are sorted by the retrieval algorithm to obtain the preliminary test set, which includes the following operations: given a changed program code, the retrieval algorithm uses the BM25 algorithm to calculate the similarity score between the changed program code and each test case, and then sorts the test cases according to the BM25 algorithm score; a certain number of top-ranked test cases and bottom-ranked test cases are selected from the preliminary test set obtained by the first sorting as the training set of the model, and the subsequent operations are completed using this filtering coefficient.

[0053] In the second sorting, the following steps are included: organization of training data; constructing pairs of program code and test case representations from the training set, while setting labels for each test case representation; given a program code representation Q, the i-th test case representation t corresponding to Q i and label l i , construct triple T i (Q,t i ,l i ); The model adopts the Siamese-MiniLM model, and the constructed triples are input into the Siamese-MiniLM model. After model training, the semantic embedding vectors u and v of the changed program code and test cases are obtained, and the test set is sorted according to the semantic similarity between u and v.

[0054] When a test case detects a program error in the last version of the real dataset, it is assigned a first label to indicate that the modified program code and the test case representation have certain similarities in semantics and functionality; conversely, other test cases that do not detect program errors are assigned a second label that is different from the first label.

[0055] The loss function is designed by comparing the cosine similarity between the changed program code u and the test case v with the true label of the test case; wherein, the model will be evaluated once on the validation set every time the preset number of steps are trained, and the model is updated according to the loss function, and the model with better performance is retained

[0056] Specifically: In this embodiment, the BM25 algorithm is used to quickly sort a large-scale test set. In the first sorting, given a changed program code, the BM25 algorithm is used through the retrieval algorithm to calculate the similarity score between the changed program code and each test case, and then the test cases are sorted from high to low according to the BM25 algorithm score.

[0057] BM25 is a relatively efficient model in the field of information retrieval, ranking text by calculating similarity. However, it considers relevance as a matter of probabilistic statistics rather than the actual semantics of words; considering that method / class names in program code and test cases are almost identical, it can be roughly assumed that the BM25 score indicates the potential semantic similarity between the modified program code and the test case representations. Therefore, in the current implementation, given a modified program code, BM25 is used to calculate the similarity score between the modified program code and each test case, and the test cases are then ranked according to the BM25 score.

[0058] How to select the test set after the BM25 preliminary ranking directly affects the ranking performance of the model in the next stage. If only the top k samples in the pre-sorting results are selected as the training set for the precise ranking stage, the precise ranking model may not explore the data set enough. In order to alleviate this problem, a small number of negative samples with lower priority can be selected to be added to the precise ranking training set. In this embodiment, it is observed that only the top k samples ranked by BM25 scores are selected as training samples for the second stage of precise ranking, and the performance of the model is poor. Therefore, a small number of samples with lower BM25 scores are selected and added to the training set as negative samples to improve its performance. In this embodiment, a screening strategy is designed to select a certain number of top-ranked (high-scoring) tests and a small number of bottom-ranked (low-scoring) tests from the preliminary ranking test set obtained by the first sorting as the training set for the second sorting model. The test cases with higher scores are intended to give high priority to the test cases with higher semantic similarity to the code changes so as to detect integration errors as soon as possible; the test cases with lower scores are intended to enhance the exploration ability of the second sorting model. The time overhead and impact of different screening coefficients on model performance are fully analyzed, and the appropriate filtering strategy is selected, and the subsequent operations are completed with this filtering coefficient. Through careful analysis and adjustment of the filter coefficient, the model achieved better performance; finally, 20%-60% of the test cases before the initial sorting and 2%-6% of the test cases after the sorting were selected as the training set for the model in the precise sorting stage, and all subsequent experiments were completed with this filter coefficient.

[0059] Figure 2 A diagram shows the precise ranking structure of the Siamese-MiniLM machine translation-based twin network. In this example, a Transformer-based pre-trained language model (i.e., Siamese-MiniLM) was used to accurately rank test cases. This method combines the advantages of the MiniLM network's fewer parameters for training semantic representations with the Siamese-BERT network's reduced computational and time overhead in calculating semantic text similarity.

[0060] Specifically, (1) first construct a pair of representations of the changed program code and the test case from the training set. Usually, one query representation corresponds to multiple test case representations, but each test case representation corresponds to a label. The experiment of this scheme is based on the continuous integration test history data of the open source project. If the test case detects a program error in the most recent integrated version, its label is defined as the first label and set to 1, indicating that the changed program code and the test case representation have strong similarities in semantics and functions. Conversely, other test cases that did not detect program errors are defined as the second label and set to 0. These negative samples can promote the model to learn the difference between the changed program code and the test case representation. Given the representation of the changed program code as Q, the i-th test case representation corresponding to Q is t i and label l i , construct triple T i (Q,t i ,l i ). (2) The triples are input into the MiniLM model respectively. The model has a deep self-attention distillation module and has been pre-trained on more than 1 billion sentence pairs. It can learn general language expression knowledge, which is conducive to better performance of downstream tasks. After the MiniLM model learns the changed program code and test case representations, semantic embedding vectors u and v are obtained respectively. Then, the test set is re-sorted according to the semantic similarity between u and v to obtain the final sorting result. (3) In the training phase, the loss function is designed by comparing the cosine similarity between the changed program code u and the test case v with the true label of the test case. After each training step, which is set to 1000 steps here, the model will be evaluated once on the validation set, and the model with better performance will be retained after multiple rounds of training. In the evaluation phase, pairs of changed program code and test case representations are extracted for each project, and then their semantic similarity is calculated through the Siamese-MiniLM model to obtain the final ranked test set, and the method provided in this embodiment is evaluated.

[0061] To reduce errors during model implementation, this implementation uses Sentence-Transformers to implement and train the model. Sentence-Transformers is a comprehensive Python framework that leverages state-of-the-art sentence and image embedding methods for semantic text similarity analysis, semantic search, and paraphrase mining. Based on PyTorch and Transformers, this framework provides a large number of pre-trained models for adaptation to various downstream tasks.

[0062] In summary, the test case prioritization method for large-scale continuous integration provided by this embodiment has the following advantages:

[0063] ① Using code representation and MiniLM language model to accurately capture the semantic information of the code, and calculating the code semantic similarity to sort the test cases, is conducive to improving the accuracy of sorting.

[0064] ② Use the Siamese-MiniLM model as the accurate sorting model. This model combines the advantages of the MiniLM model with fewer parameters and the advantages of the Siamese-BERT network with lower computational cost and time overhead. While ensuring sorting performance, it reduces the time overhead of model calculation and the number of training parameters.

[0065] ③ The two-stage test case sorting strategy of preliminary screening and precise sorting combines the advantages of preliminary screening and precise sorting, which not only improves the sorting performance, but also selects only 55% of the tests for model training in the precise sorting stage, greatly reducing the model training time.

[0066] Example: The large-scale continuous integration test case prioritization method (SatTCP) provided in this embodiment was experimented on 2038 continuous integration versions of 119 open source projects in the Github community. The top 50% of the test cases in the preliminary ranking and the bottom 5% of the test cases in the ranking were selected as the training set for the precise ranking stage model. All subsequent experiments were completed with this filter coefficient to obtain experimental results. The results were compared with three baseline methods and actual continuous integration (CI) tests in terms of defect detection performance (APFD, APFDc) and time overhead. The comparison results are shown as follows: Figure 3 and as shown in Table 1;

[0067]

[0068] Table 1

[0069] Depend on Figure 3 As shown in the experimental results in Table 1, the average APFDc (a comprehensive evaluation index of defect detection performance) of our method across all projects exceeded all baselines by 14.8% to 28.8%. In particular, in large projects, APFDc and APFD exceeded the best-performing baseline by 32.9% and 11% respectively. Furthermore, as the project scales, the method's performance shows a clear upward trend. Furthermore, our method has the lowest total time cost (including test sequencing time and test execution time), reducing actual continuous integration testing time by 71.6% and an average reduction of 33.7% to 41.1% compared to baseline methods.

[0070] This embodiment further provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the described method.

[0071] This embodiment further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0072] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for prioritizing test cases for large-scale continuous integration, characterized in that: include: A first sorting operation includes parsing target code, the target code including test case code and modified program code, and performing lemmatization on the target code to obtain a test case representation and a modified program code representation; Calculate the similarity score between the modified program code and each test case through the retrieval algorithm, perform the first sorting on the test cases, and obtain the preliminary sorted test set; as well as A second sorting, wherein the second sorting includes using the filtered and sorted preliminary test set to construct a training set for the model; performing a second sorting on the preliminary test set using the model trained on the training set to obtain a test case priority sorting result; The second sorting includes the following steps: Organization of training data; construct pairs of program code changes and test case representations from the training set, and label each test case representation based on whether the test case detected a program error in the last version of the real dataset; given a program code change represented as Q, the i-th test case corresponding to Q is represented as t i and label l i , construct triple T i (Q,t i ,l i ); The model adopts the Siamese-MiniLM model, and the constructed triples are input into the Siamese-MiniLM model. After model training, the semantic embedding vectors u and v of the changed program code and test cases are obtained, and the initial test set is sorted according to the semantic similarity between u and v.

2. The method according to claim 1, wherein In the first sorting, the modified program code is lemmatized to obtain a modified program code representation; In the second order, when training the model, the training content consists of learning the semantic representation of changes in program code and test cases; The correspondence between the modified program code and the test case is: 1 modified program code representation Q corresponds to m test case representations, where m is an integer and is greater than or equal to 1.

3. The method according to claim 2, wherein The step of obtaining at least a test case representation includes the following steps: Use Javalang of Python language to perform syntax and lexical analysis on test case code, and build an abstract syntax tree to structure the code; Traverse the abstract syntax tree to extract identifiers; Split the long identifier into individual morphemes according to the long identifier segmentation rule; Filter meaningless words and stop words to obtain a test case representation consisting of tokens.

4. The method according to claim 2, wherein When changing the program code representation or test case representation to cause duplicate tokens; Evaluate the uniqueness of the token, optionally performing the following operations: Maintaining token uniqueness; removing duplicate tokens in the modified program code representation or test case representation to maintain the uniqueness of each token in the set; Maintain the frequency of occurrence of the original words; allow the recurrence of word units, and record the word frequency of the word units in the changed program code representation and test case representation.

5. The method according to claim 2, wherein In the first sorting, the test cases are sorted by the retrieval algorithm to obtain the first sorted test set, which includes the following operations: Given a changed program code, the retrieval algorithm uses the BM25 algorithm to calculate the similarity score between the changed program code and each test case, and then sorts the test cases according to the BM25 algorithm score; A certain proportion of the top-ranked test cases and the bottom-ranked test cases are selected from the first-ranked test set obtained as the training set of the model, and the subsequent operations are completed based on this filtering coefficient.

6. The method according to claim 1, wherein When a test case detects a program error in the last version of the real dataset, it is assigned the first label, which indicates that the modified program code and the test case representation have certain similarities in semantics and functionality; On the contrary, other test cases that do not detect program errors are assigned a second label that is different from the first label.

7. The method according to claim 1, wherein Design a loss function by comparing the cosine similarity between the changed program code u and the test case v with the true label of the test case; Among them, every time the model is trained for a preset number of steps, it will be evaluated once on the validation set, and the model will be updated according to the loss function, retaining the model with better performance.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute instructions to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images