A method for automatically co-evolving production code and test cases

Abstract

The application discloses a kind of automatic coevolution production code and test case method, the edited code sequence, original code, change code and test case are used as code change information, code change information is preprocessed, and the input sequence of classification model is obtained;Input sequence is input to classification model, and the classification model multidimensional space vector representation of extended code change information is obtained and sent into classifier, and whether test sample needs to be updated is obtained;Automatic update test sample, through the encoder in generation model, the generation model multidimensional space vector representation of input sequence is obtained and input to the decoder in generation model, and the probability distribution is obtained, and the probability distribution is restored to the test case generated by the dictionary contained in the encoder-decoder.The application realizes a kind of production-test case coevolution method that can identify obsolete test case and can automatically update obsolete test case.

Description

Technical Field

[0001] This technology relates to the field of automated code judgment and generation, and in particular to a method for automatically co-evolving and generating code and test cases. Background Technology

[0002] Software testing is generally considered one of the most critical parts of the software development lifecycle, helping to identify potential faults and ensure the quality of the software system. Typically, source code is constantly modified to meet new requirements or fix potential problems. To ensure software quality, corresponding test cases should evolve along with the changed production code. However, co-evolution of test cases is a challenging task because even small changes in production code can significantly impact test coverage. Furthermore, maintaining test cases is both time-consuming and labor-intensive. Unfortunately, due to a lack of time to maintain test cases and a lack of sufficient knowledge to determine whether test cases need to evolve, co-evolution of production code and test cases is often lacking during software evolution.

[0003] To illustrate the impact of obsolete test cases, let's take the Conekta project as an example. We can see that the API version in the production code was updated to 2.0.0 on February 16, 2017. However, the API version used in the corresponding test cases wasn't set to 2.0.0 until October 18, 2019. Although the production code passed the tests without errors, the new API version was never tested during this period and may not have revealed any faults in the production code. As described in the test case update submission, "Java library JUnit tests need fixing. Many tests failed due to a wrapper error in the SetApiVersion method. Other minor bugs regarding obsolete references in test cases have been fixed." Many bugs may have been introduced from obsolete test cases. Therefore, many researchers have proposed various techniques to mine and analyze the rules and patterns of co-evolution between production and test cases. Some studies mine data from version control systems to study the co-evolution of production and test cases, while others use association rule mining techniques to generate co-evolutionary patterns. Recently, researchers proposed a method called Sitar, which promotes the co-evolution of production and test cases by extracting features and leveraging machine learning techniques. They focused on committing file-level changes at the granular level. According to their findings, change-type emergencies (EEs) (i.e., updating existing test cases with production code) occur far more frequently than other co-evolutionary types (e.g., adding / removing test cases). However, their method performs worse at predicting this type of co-evolution (EE) than other types (e.g., adding new test cases). Inspired by their research, it is possible to reduce or even avoid the introduction of obsolete test cases if existing test cases requiring updates can be automatically identified and co-evolved. This invention focuses on the co-evolution of method-level production code and its corresponding test cases, as automated unit testing of method-level methods has been a key task in recent years. Summary of the Invention

[0004] This invention aims to co-evolve production code and test cases at the method level, proposing a method for automatically co-evolving production code and test cases. It consists of two phases: an obsolete test identification phase and an obsolete test update phase. A typical use case for this invention is providing test recommendations to developers when making changes to production code. This invention proposes a new method to automate the task of co-evolving production code and test cases. It includes two phases: (1) First, determining whether test cases should be updated when production code changes. (2) Second, updating test cases based on code changes. By utilizing deep learning technology to automate the co-evolution process, this provides developers with a highly efficient method for checking code changes, and helps developers discover problems in test cases early, reducing or even avoiding the introduction of errors, thus contributing to improved software quality.

[0005] This invention is achieved through the following technical solution:

[0006] A method for automatically co-evolving production code and test cases, comprising the following steps:

[0007] S1. Obtain code change information c from the commit log, including the original production code x, the changed production code x′, and the original test case code t;

[0008] S2. Preprocess the code change information c, delete duplicate samples, and construct the edit code sequence e using the original production code x and the changed production code x′. Expand the edit code sequence e to the code change information c to obtain the input sequence I of the classification model.

[0009] S3. Input the input sequence I of the classification model into the classification model to obtain the multidimensional space vector representation R of the classification model with expanded code change information;

[0010] S4. Input the obtained multidimensional space vector representation R into the classifier to determine whether the test sample needs to be updated. If no update is needed, end the method; if an update is needed, proceed to the next step.

[0011] S5. Automatically update test cases by inputting I into the encoder of the generative model to obtain the multidimensional space vector representation of I in the generative model.

[0012] S6. Input the multidimensional space vector representation of the generative model of I into the decoder in the generative model to obtain the probability distribution. Then, use the dictionary contained in the encoder-decoder to restore the probability distribution to the generated test cases.

[0013] Furthermore, S2 specifically refers to:

[0014] Edit code sequence representation is as follows Where x i and x′ i These are the morphemes representing the original and modified production methods at position i; a i Indicates x i Convert to x i The editing actions are divided into four types, a i =insert, delete, replace, or equal; To obtain a representation of code changes, the original and modified production codes are first tokenized into sequences of code lexicals, and these sequences are used to construct the edit code sequence; the edit code triple at position i is obtained by calculating word-level alignment, i.e. <x i ,c′ i ,a i> Concatenate the edit code triples to obtain the edit code sequence e;

[0015] The classification model takes as input the original production code, edited code sequences, modified production code, and original test case code. It then segments the original production code, modified production code, and original test case code into a series of morphemes, namely x = {x1, ..., x...}. m}, x′={x′1,...,x′ n}, t={t1,...,t l}, where m, n, and l represent the lengths of the original production code, the modified production code, and the original test case code, respectively; these morphemes are constructed using the Roberta tokenizer of the CodeT5 model; x, x′, e, and t are concatenated into an input I =<x,x′,e,t> .

[0016] Furthermore, S3 specifically refers to:

[0017] The input sequence I is encoded into a numerical representation using a Transformer-based pre-trained CodeT5 encoder. Then, this encoder converts the numerical representation of the input into a multidimensional vector representation R.

[0018] For each input I in the input sequence I i The Transformer generates three embedding representations for it, namely vector q. i Vector k i sum vector v i In the classification model encoder, I is calculated using the dot product. i Attention score α i，j The method is to use the vector q of the i-th input. i and the j-th input vector k j As shown below:

[0019]

[0020] Where d is q i and k j Dimensions; Attention score α i，j This indicates the level of attention paid to the j-th input when encoding the i-th input;

[0021] Then, attention score α i，j The normalized score is obtained using a softmax function:

[0022]

[0023] To learn relevant or irrelevant tokens in the input sequence, the normalized score of the attention score for each input during encoding of the i-th input is compared with the vector v. j Multiply and add:

[0024]

[0025] Finally, the value z of the last hidden layer in the classification model is used as the multidimensional space vector representation R of the input I.

[0026] Further, S4 specifically involves: using a fully connected layer activated by a non-linear tanh function to learn the potential interactions between different components of the input I; then, the output of the fully connected layer is used to predict the probability l = {0, 1} of the final label; the classifier in the classification model is defined as follows:

[0027] f = tanh(WR + b)

[0028] p(l|<x,x′,e,t> = Softmax(f)

[0029] W and b represent the weight matrix and bias vector, respectively; tanh is the activation function of the Dense layer perceptron; the Softmax function outputs the final probability of label l between 0 and 1; a threshold score of 0.5 is set, the probability of a value greater than the threshold score is output as 1, indicating that an update is needed, and the probability of a value less than the threshold score is output as 0, indicating that an update is not needed.

[0030] Furthermore, S5 specifically involves: when updating test cases, the encoder of the generation model is the same as the encoder of the classification model; using the attention score z = {z1, ..., z...} of I. |I|} serves as the multidimensional space vector representation of the generative model of I, used as input to the decoder.

[0031] Furthermore, S6 specifically involves: test case updating, generating new test cases given the input:

[0032] The decoder generates a new morpheme t corresponding to all previous morphemes based on the encoder's output z and all the test case morphemes generated so far;

[0033] The decoder consists of multiple decoder layers cascaded together. Each layer contains two sub-layers: a self-attention layer and an encoder-decoder cross-attention layer. The self-attention layer only processes the test case morphemes generated so far, while the encoder-decoder cross-attention layer learns the correlation between the currently generated output sequence and the input sequence. For generating the i-th target morpheme, the workflow of each layer of the decoder is as follows:

[0034] S6.1 Input representation calculation: Convert the target word sequence generated at the current time into word vector e. i Then e i and the output vector z of the previous decoder layer i-1 By concatenating the data, we obtain the input representation x of the current decoder layer. i :

[0035] x i =[e i ;z i-1 ]

[0036] Where [;] represents a concatenation operation;

[0037] Special cases:

[0038] When the current moment is the initial state, the target word element is initialized to a special starting character '', indicating that no target word element has been generated yet. The corresponding word vector at this time is denoted as:

[0039] e i = <s> emb

[0040] This is a learnable parameter that needs to be continuously updated with the target lexical and its corresponding word vector representation at the current time step during the generation of the target lexical sequence.

[0041] When we are currently in the first layer of the decoder, there is no output from the previous layer. At this point, a randomly initialized, fixed-length random vector will be used as z. i-1 This fixed length is the dimension of the model's hidden states;

[0042] S6.2 Self-Attention Layer Calculation

[0043] After obtaining the input representation of the current layer, it needs to be used as the Query, Key, and Value, and input into the self-attention function to calculate the attention weights and context vector; specifically, this involves transforming x through a linear transformation. i Mapped to three vectors Q i K i and V i :

[0044] Q i =x i W Q

[0045] K i =x i W K

[0046] V i =x i W V

[0047] Among them, W Q W K W V It is a learnable weight matrix; the self-attention representation of the current decoder layer is computed using the self-attention function, denoted as:

[0048]

[0049]

[0050] Where, d k It is the dimension of vector K;

[0051] S6.3 Encoder-Decoder Cross-Attention Layer Calculation

[0052] When calculating the attention distribution of the input sequence and target lexical, the self-attention representation of the current decoder layer is used. As a query, the encoder's output z is used as the key and value, and input into the cross-attention function to calculate the attention weights and context vectors of the input sequence and target morphemes; specifically, a linear transformation is performed to... And z mapped to Q′ i , K′ i , V′ i :

[0053]

[0054] K′=zW′ K

[0055] V′=zW′ V

[0056] Among them, W′ Q , W′ K , W′ V It is a learnable weight matrix;

[0057] Then, the output representation of the current Decoder layer is computed using the cross-attention function, denoted as:

[0058]

[0059]

[0060] Where, d k It is the dimension of vector K;

[0061] S6.4, Feedforward Neural Network Calculates Decoder Layer Output Vector

[0062] In a feedforward neural network, the output of the current decoder layer is represented as... The representation is mapped to a higher dimension through a fully connected layer, and then mapped back to the original dimension through another fully connected layer; specifically, the output representation of the current decoder layer is mapped... The vector is passed to the feedforward neural network to obtain the output vector z of the current decoder layer. i :

[0063]

[0064] z i =z′ i W′2+b′2

[0065] Where W′1 and W′2 are learnable weight matrices, and b′1 and b′2 are learnable bias vectors;

[0066] Finally, the calculated z i That is, the output vector of the decoder when generating the i-th target word;

[0067] S6.5, Generate target morphemes

[0068] Finally, the decoder output z i After linear transformation and activation function, a probability distribution for the i-th target word is obtained:

[0069] O i =softmax(z i W o +b o )

[0070] Among them, W o and b o It consists of a learnable weight matrix and a learnable bias vector;

[0071] Based on the obtained probability distribution and Robert's dictionary, return the word with the highest probability to obtain the target morpheme;

[0072] S6.6 Repeat S6.1-S6.5 until all target morphemes are generated, resulting in a new test case code.

[0073] (1) Obtain production code and corresponding test case information: Obtain code changes from the commit log. The obtained code information includes the original code sre_code, the modified code tgt_code, the test case code src_test, and record the code change c=<src-code,tgt_code,src_test> .

[0074] (2) Preprocess the code changes to obtain the code representation: Construct the edit code `edit_code` using `sre_code` and `tgt_code`, and expand the original code information to `c′`.<src_code,tgt_code,edit_code,src_test> , denoted as input I =<x,x′,e,t> .

[0075] (3) Input I into the classification model encoder E_C to obtain the multidimensional space vector representation R of c'.

[0076] (4) The obtained input vector representation R is fed into the classification layer to determine the probability value of the current I. 1 means it needs to be updated and 0 means it does not need to be updated.

[0077] (5) When an update is needed, proceed to the next step, i.e., automatically update the test sample, input I into the encoder E_G of the generative model, and obtain the multidimensional space vector representation R_G of I.

[0078] (6) Input R_G into the decoder D_G of the generative model to obtain the generated numerical vector test_G, and restore test_G to the generated test cases through the dictionary.

[0079] The beneficial effects of this invention are:

[0080] Currently, there is no method-level co-evolution method for production test cases. Inspired by existing work on document-level co-evolution of production test cases, this invention uses deep learning technology to implement a production-test case co-evolution method that can both identify and automatically update outdated test cases. This invention can automatically identify and update outdated test cases, providing reliable suggestions to developers, reducing their time and costs, and improving maintenance efficiency and software quality. Attached Figure Description

[0081] Figure 1 This is a flowchart of the present invention;

[0082] Figure 2 This is a diagram showing the results of manual inspection of this invention;

[0083] Figure 3 This is a sample diagram of manual inspection in this invention. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the single embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0085] The specific working process of this invention is as follows:

[0086] S1. Obtain code change information c from the commit log, including the original production code x, the changed production code x′, and the original test case code t;

[0087] S2. Preprocess the code change information c, delete duplicate samples, and construct the edit code sequence e using the original production code x and the changed production code x′. Expand the edit code sequence e to the code change information c to obtain the input sequence I of the classification model; specifically:

[0088] Edit code sequence representation is as follows Where x i and x′ i These are the morphemes representing the original and modified production methods at position i; a i Indicates x i Convert to x′ i Editing actions; editing actions are divided into four types, a i =insert, delete, replace, or equal; To obtain a representation of code changes, the original and modified production codes are first tokenized into sequences of code lexicals, and these sequences are used to construct the edit code sequence; the edit code triple at position i is obtained by calculating word-level alignment, i.e. <x i , x′ i a i > Concatenate the edit code triples to obtain the edit code sequence e;

[0089] The classification model takes as input the original production code, edited code sequences, modified production code, and original test case code. It then segments the original production code, modified production code, and original test case code into a series of morphemes, namely x = {x1, ..., x...}. m }, x′={x′1,...,x′ n }, t={t1,...,t l }, where m, n, and l represent the lengths of the original production code, the modified production code, and the original test case code, respectively; these morphemes are constructed using the Roberta tokenizer of the CodeT5 model; x, x′, e, and t are concatenated into an input I =<x,x′,e,t> .

[0090] S3. Input the input sequence I of the classification model into the classification model to obtain the multidimensional space vector representation R of the classification model with expanded code change information;

[0091] The input sequence I is encoded into a numerical representation using a Transformer-based pre-trained CodeT5 encoder. Then, this encoder converts the numerical representation of the input into a multidimensional vector representation R.

[0092] For each input I in the input sequence I i The Transformer generates three embedding representations for it, namely vector q. i Vector k i sum vector v i In the classification model encoder, I is calculated using the dot product. i Attention score α i，j The method is to use the vector q of the i-th input. i and the j-th input vector k j As shown below:

[0093]

[0094] Where d is q i and k j Dimensions; Attention score α i，j This indicates the level of attention paid to the j-th input when encoding the i-th input;

[0095] Then, attention score α i，j The normalized score is obtained using a softmax function:

[0096]

[0097] To learn relevant or irrelevant tokens in the input sequence, the normalized score of the attention score for each input during encoding of the i-th input is compared with the vector v. j Multiply and add:

[0098]

[0099] Finally, the value z of the last hidden layer in the classification model is used as the multidimensional space vector representation R of the input I.

[0100] S4. Input the obtained multidimensional space vector representation R into the classifier to determine whether the test sample needs to be updated. If no update is needed, end the method; if an update is needed, proceed to the next step.

[0101] Specifically, a fully connected layer activated by a non-linear tanh function is used to learn the potential interactions between different components of the input I; then, the output of the fully connected layer is used to predict the probability l = {0, 1} of the final label; the classifier in the classification model is defined as follows:

[0102] f = tanh(WR + b)

[0103] p(l|<x,x′,e,t> = Softmax(f)

[0104] W and b represent the weight matrix and bias vector, respectively; tanh is the activation function of the Dense layer perceptron; the Softmax function outputs the final probability of label l between 0 and 1; a threshold score of 0.5 is set, the probability of a value greater than the threshold score is output as 1, indicating that an update is needed, and the probability of a value less than the threshold score is output as 0, indicating that an update is not needed.

[0105] S5. Automatically update test cases by inputting I into the encoder of the generative model to obtain the multidimensional space vector representation of I in the generative model.

[0106] Specifically: when updating test cases, the encoder of the generation model is the same as the encoder of the classification model; the attention score z = {z1, ..., z} of I is used. |I| } serves as the multidimensional space vector representation of the generative model of I, used as input to the decoder.

[0107] S6. Input the multidimensional space vector representation of the generative model of I into the decoder in the generative model to obtain the probability distribution. Then, use the dictionary contained in the encoder-decoder to restore the probability distribution to the generated test cases.

[0108] Specifically: Test case updates generate new test cases given the input.

[0109] The decoder generates a new morpheme t corresponding to all previous morphemes based on the encoder's output z and all the test case morphemes generated so far;

[0110] The decoder consists of multiple decoder layers cascaded together. Each layer contains two sub-layers: a self-attention layer and an encoder-decoder cross-attention layer. The self-attention layer only processes the test case morphemes generated so far, while the encoder-decoder cross-attention layer learns the correlation between the currently generated output sequence and the input sequence. For generating the i-th target morpheme, the workflow of each layer of the decoder is as follows:

[0111] S6.1 Input representation calculation: Convert the target word sequence generated at the current time into word vector e. i Then e i and the output vector z of the previous decoder layer i-1 By concatenating the data, we obtain the input representation x of the current decoder layer. i :

[0112] x i =[e i ;z i-1 ]

[0113] Where [;] represents a concatenation operation;

[0114] Special cases:

[0115] When the current moment is the initial state, the target morpheme will be initialized to a special starting character '.< / s> ' indicates that no target morpheme has been generated yet, and its corresponding word vector is denoted as:

[0116] e i = <s> emb

[0117] This is a learnable parameter that needs to be continuously updated with the target lexical and its corresponding word vector representation at the current time step during the generation of the target lexical sequence.

[0118] When we are currently in the first layer of the decoder, there is no output from the previous layer. At this point, a randomly initialized, fixed-length random vector will be used as z. i-1 This fixed length is the dimension of the model's hidden states;

[0119] S6.2 Self-Attention Layer Calculation

[0120] After obtaining the input representation of the current layer, it needs to be used as the Query, Key, and Value, and input into the self-attention function to calculate the attention weights and context vector; specifically, this involves transforming x through a linear transformation. i Mapped to three vectors Q i K i V i :

[0121] Q i =x i W Q

[0122] K i =x i W K

[0123] V i =x i W V

[0124] Among them, W Q W K W V It is a learnable weight matrix; the self-attention representation of the current decoder layer is computed using the self-attention function, denoted as:

[0125]

[0126]

[0127] Where, d k It is the dimension of vector K;

[0128] S6.3 Encoder-Decoder Cross-Attention Layer Calculation

[0129] When calculating the attention distribution of the input sequence and target lexical, the self-attention representation of the current decoder layer is used. As a query, the encoder's output z is used as the key and value, and input into the cross-attention function to calculate the attention weights and context vectors of the input sequence and target morphemes; specifically, a linear transformation is performed to... And z mapped to Q′ i , K′ i , V′ i :

[0130]

[0131] K′=zW′ K

[0132] V′=zW′ V

[0133] Among them, W′ Q , W′ K , W′ V It is a learnable weight matrix;

[0134] Then, the output representation of the current Decoder layer is computed using the cross-attention function, denoted as:

[0135]

[0136]

[0137] Where, d k It is the dimension of vector K;

[0138] S6.4, Feedforward Neural Network Calculates Decoder Layer Output Vector

[0139] In a feedforward neural network, the output of the current decoder layer is represented as... The representation is mapped to a higher dimension through a fully connected layer, and then mapped back to the original dimension through another fully connected layer; specifically, the output representation of the current decoder layer is mapped... The vector is passed to the feedforward neural network to obtain the output vector z of the current decoder layer. i :

[0140]

[0141] z i =z′ i W′2+b′2

[0142] Where W′1 and W′2 are learnable weight matrices, and b′1 and b′2 are learnable bias vectors;

[0143] Finally, the calculated z i That is, the output vector of the decoder when generating the i-th target word;

[0144] S6.5, Generate target morphemes

[0145] Finally, the decoder output z i After linear transformation and activation function, a probability distribution for the i-th target word is obtained:

[0146] O i =softmax(z i W o +b o )

[0147] Among them, W o and b o It consists of a learnable weight matrix and a learnable bias vector;

[0148] Based on the obtained probability distribution and Robert's dictionary, return the word with the highest probability to obtain the target morpheme;

[0149] S6.6 Repeat S6.1-S6.5 until all target morphemes are generated, resulting in a new test case code.

[0150] The present invention will be further described below with reference to the accompanying drawings and examples:

[0151] This invention aims to achieve automatic collaborative evolution of production code and test cases; this method is also known as Ceprot. Figure 1 This paper presents the overall framework of the invention, comprising four phases: input construction, obsolete test case identification, obsolete test case updating, and online production-test co-evolution application. In the input construction phase, input tokens are converted into embedded representations before being fed into a neural network. In the obsolete test case identification phase, this embodiment trains a neural network classification model to identify obsolete test cases that need updating. In the obsolete test case updating phase, a generative model is trained to generate new test cases. Unlike the obsolete test case identification phase, the updating phase is trained using only positive samples and treats new test cases as a given. In an application case, given code changes (a sequence of production and edited code with two versions) and their associated old test cases, the invention first utilizes the trained classification model `Identifier` to predict whether the old test cases need updating. If the answer is yes, then the trained generative model `Updater` is used to generate new tests to replace the old ones.

[0152] This invention can automatically identify and update outdated test cases, providing developers with reliable suggestions, reducing the time and cost spent by developers, and improving maintenance efficiency and software quality.

[0153] Identifying and updating outdated test cases requires understanding changes in the production code. Unlike learning source code, code changes also include editing operations such as insert, delete, replace, or equalize, "add," "delete," and "replace." Furthermore, code changes involve two versions of the production code. Each location marked in the code has its corresponding editing action. Therefore, we should learn fine-grained code changes from editing actions to determine whether and how to update the corresponding tests. In this invention, learning from the two versions of the production code, the editing sequence of code changes, and existing test cases addresses the co-evolution of the production-test case task. This invention comprises two phases: outdated test case identification in Phase 1 and updating in Phase 2. It is necessary to capture the semantic relevance between different inputs, such as source code and editing operations. Furthermore, new test cases need to be generated in the outdated test case update phase, with each lexical having a different relevance to these three inputs. Therefore, to better co-evolve production and test cases, it is necessary to consider the relevance between the two versions of the production code, editing operations, and test cases. To address the above challenges, embodiments of this invention first construct an editing code sequence from the two versions of the production code. The edited code sequence consists of original versions of morphemes and new versions of the code produced at specific positions. We then leverage the CodeT5 encoder-decoder model, pre-trained on a large programming language corpus, to learn semantic relevance for identifying and updating outdated test cases, fine-tuning the model on both the identification and update tasks. Finally, this invention combines these two stages in an online application. To build and evaluate the model of this invention, two datasets were constructed: one for method-level production-test case co-evolution identification, and the other for updating test cases.

[0154] refer to Figure 1 A data flow diagram for the automatic collaborative evolution of production code and test cases is provided, including the following steps:

[0155] 1. Obtain production code and corresponding test case information.

[0156] Obtain production code and corresponding test case information: Retrieve code changes from the commit log. The obtained code information includes the original code (src_code), the modified code (tgt_code), the test case code (src_test), and a record of the code change (c).<src_code,tgt_code,src_test> .

[0157] In this invention, our goal is to identify and update obsolete test cases in method-level production code. This invention constructs two datasets for training and evaluating proposed methods in two phases. These datasets consist of a large number of Java projects with high-quality unit tests. To construct such a dataset, it is necessary to collect co-evolving Java methods annotated with `@Test` and their corresponding test cases; these method-level changes include both non-test methods and unit tests. This embodiment then extracts co-evolving method-level production test samples from the modified methods extracted above. Since the quality of the dataset is crucial for deep learning models, methods and their corresponding tests that change within the same commit can serve as co-evolving production-test pairs. For each non-test method (without the `@Test` annotation), we check for unit tests that have undergone common changes within the same commit. We define a series of heuristic rules to extract unit tests for methods by name matching. Because test case names often resemble their corresponding production methods, the heuristic rules used in this invention are shown in Table 1. For the identification task, we need to build a dataset containing positive and negative samples. If a test case is updated with changes to its corresponding production code, we mark it as a positive sample. Otherwise, we mark it as a negative sample. For update tasks, we only use positive samples from the test cases in the identification task that need to be updated to build the dataset.

[0158] Table 1: Heuristic Rules for Data Filtering

[0159]

[0160] 2. Preprocess the data to obtain the input sequence.

[0161] Preprocessing the code changes yields a code representation: The edit code `edit_code` is constructed using `src_code` and `tgt_code`, expanding the original code information to `c′=`.<src_code,tgt_code,edit_code,src_test> , denoted as input I =<x,x′,e,t> .

[0162] Typically, commits may contain duplicate change samples because developers might make repetitive system or code changes in a single commit. To avoid the poor performance of duplicate samples, we first removed duplicate production method samples from the collected dataset. Then, we preprocessed the source code using TreeSitter by removing comments. Additionally, symbols such as "\n" and "\t" were also removed. The preprocessed code data then served as the input sequence for building edited code sequences and models.

[0163] 3. Input I into the classification model encoder E_C to obtain the multidimensional space vector representation R of c'.

[0164] 4. Feed the obtained input vector representation R into the classification layer and determine the probability value of I at the current time. 1 means that it needs to be updated and 0 means that it does not need to be updated.

[0165] A Dense layer using a non-linear function is used to learn the potential interactions between different components of the input I. The output of the Dense layer is then used to predict the probability l = {0, 1} of the final label. The classifier is defined as follows:

[0166] f = tanh(WR + b)

[0167] p(l|<x,x′,e,t> = Softmax(f)

[0168] W and b represent the weight matrix and bias vector, respectively. tanh is the activation function of the Dense layer perceptron. The Softmax function outputs the final probability of label l between 0 and 1. For the probability score, it should be higher if updating the original test case is needed, and lower if not; in this embodiment, a threshold of 0.5 is set. If the model output is greater than 0.5, we consider that updating the original test case is necessary; otherwise, we consider that updating the original test case is not necessary.

[0169] We evaluated the performance of obsolete test case identification in Phase 1 based on precision, recall, and F1 score on the collected dataset. Table 2 illustrates the evaluation results for the obsolete test case identification task. We can observe that the Ceprot method in this embodiment achieves 91.25% precision and 88.24% recall. Specifically, the method in this embodiment outperforms the baseline methods in terms of precision and F1 score. Among all methods, LSTM performs best in recall. However, its precision is worse than CodeBERT and the method in this embodiment. Since the co-evolution of production and test cases is a two-phase task, we should consider both precision and recall. On the one hand, we should ensure that as many obsolete test cases as possible are identified. On the other hand, we should ensure that the identified test cases are indeed obsolete to avoid redundant updates during the update phase. If a method achieves 100% recall, developers would need to examine many candidates to find an obsolete test case, which is difficult to apply in practice. Considering that the method provided in this embodiment achieves improvements of 54%, 10%, and 5% in F1 score compared to KNN, LSTM, and CodeBERT, it can effectively identify outdated test cases.

[0170] Table 2: Evaluation Results of the Deprecated Test Identification Task

[0171]

[0172] 5. When an update is needed, proceed to the next step, which is to automatically update the test cases by inputting I into the encoder E_G of the generative model to obtain the multidimensional vector representation R_G of I.

[0173] The input to the generative model consists of four parts, namely<x,x′,e,t> Different parts of the input help generate different parts of the test. The correlation between each token I in the input and the target t′ should be understood. Therefore, the attention score z = {z1,…,z} of I is used. |I| Instead of using the last hidden state as input to the decoder.

[0174] 6. Update outdated test cases: Input R_G into the decoder D_G of the generative model to obtain the generated numerical vector test_G, and restore test_G to the generated test cases through the dictionary.

[0175] Table 3 shows the results of updating outdated test cases. Similar to the outdated test identification task, the method provided in this embodiment was compared with three baselines in terms of accuracy and CodeBLEU. We can observe that the method provided in this embodiment outperforms all baselines in all aspects of all metrics. The high CodeBLEU of the method in this embodiment indicates that many generated test cases are syntactically correct, with higher AST matching and data flow matching; this embodiment utilizes deep learning technology to predict the co-evolution of production test cases. The neural network model used by the deep learning technology needs to be trained before it is used for inference, and the training process is carried out offline, which can be used for online prediction. This embodiment also studies the training time cost and testing time cost of the method to better understand the practicality of the method. To measure the time complexity of this method and other baselines, the start and end times of their training and testing processes were recorded, as shown in Table 4.

[0176] Table 3: Results of updating outdated test cases

[0177]

[0178] Table 4: Time Efficiency Chart

[0179]

[0180] Since this invention is an automated method for identifying and updating outdated test cases, it requires joint evaluation of two phases. The effectiveness of this invention depends on both the test identification and update phases. Table 5 shows that this invention significantly outperforms the other three baseline methods across all metrics. In the outdated test identification phase, this method first identifies more outdated tests with higher accuracy. Then, in the update phase, this method updates true positive and false positive samples with higher accuracy, while CodeBERT fails to generate any correct test cases. Furthermore, our method's #TPC is 3 times and 4.2 times that of KNN and NMT, respectively. Specifically, this invention achieves an accuracy of 17.5% (84 / 480) on its predicted true positive samples, which is 150% and 337.5% higher than the accuracy of KNN (7%: 28 / 401) and NMT (4%: 20 / 499), respectively. Although 46 negative samples were predicted as positive, 18 of the generated new test cases were identical to the original test cases.

[0181] Table 5 Evaluation Results

[0182]

[0183] To identify and update obsolete test cases, we use both previous and current versions of production code and an edit sequence as input. Each part of the input has a different impact on the effectiveness of our method. To investigate the effectiveness of each input, we compare our method Ceprot with two variants: Ceprot-with-no-code and Ceprot-with-no-edit. Ceprot-with-no-edit is a testing method that uses two versions of production code and the original code to identify and update obsolete tests without an edit sequence. Furthermore, Ceprot-with-no-code utilizes only the edit sequence to accomplish this task. Table 6 shows the results for Ceprot and its variants. We can see that Ceprot, without using an edit sequence as input, performs similarly to Ceprot, but its performance degrades when removing production code in the identification task. For the update task and the two-phase setup, both Ceprot-with-no-code and Ceprot-with-no-edit show a performance degradation compared to Ceprot. Moreover, both Ceprot-with-no-code and Ceprot-with-no-edit significantly outperform other baseline methods. This indicates that both production code and edit sequences are helpful for updating obsolete tests. Production code provides semantic information about the methods to be tested, while edit sequences help Ceprot learn fine-grained code changes.

[0184] To better analyze why Ceport outperforms other methods, we manually examined the evaluation results. Based on this examination, we identified two main advantages: First, Ceport learns fine-grained code changes from editing operations. Figure 2 An example is shown where none of the baselines generate correct test cases. KNN retrieves tests unrelated to production code, and NMT generates a test case identical to the original without any updates. While CodeBERT generates new and complete tests, they cannot be used to test production code. However, our method learns fine-grained code changes, such as adding the parameter "context" to the interface. It then updates the corresponding method calls in the test case and adds the parameter. These fine-grained code changes are primarily learned from edit sequences.

[0185] Table 6: Evaluation results of Ceprot and its variants

[0186]

[0187] Secondly, Ceport learns important semantic relevance from different types of input. All baselines retrieve or generate test cases by leveraging learned representations. They utilize semantic representations to generate corresponding test cases. In contrast, our method, Ceport, uses CodeT5, which captures the relevance between tests and input production code. For example, Figure 3 This demonstrates how Ceprot can effectively generate boundary conditions that should be tested. However, the tests generated by CodeBERT have no semantic relevance to production code. While KNN can retrieve test cases with overlapping boundary condition tags, such as "state," these test cases are incorrect for testing production code. As for NMT, we can see that it has some limitations in generating test cases:

[0188] 1) Length limitation. LSTM has a drawback in generating long code. Therefore, the test generated by NMT shown in the figure is incomplete.

[0189] 2) Boundary Learning Limitations. From the incomplete tests generated by NMT, we can see that it fails to capture boundary conditions, specifically the state being "null". In contrast, Ceport can accurately generate "null" information to test the boundaries of production code. We also investigated why Ceport cannot co-evolve test cases with changes to production code. We manually examined samples where Ceport failed to make correct predictions. The failure was primarily due to excessively long inputs; when production code is too long, Ceport struggles to generate accurate test cases. Deep neural models cannot capture context from truncated source code. Therefore, when production code is too long, Ceport cannot co-evolve production test cases.

[0190] This invention is the first to use a two-phase framework to identify and update outdated test cases. The two phases include outdated test case detection and updating, with a model graph discriminator and updater reflecting the structure of both phases. Evaluation results show that Ceprot significantly outperforms other baseline methods in co-evolutionary test case generation. Furthermore, we evaluated this method on two separate tasks: identifying and updating outdated test cases. These experimental results demonstrate that Ceprot can effectively identify outdated test cases and generate accurate test cases.

[0191] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.< / s>

Claims

1. A method for automatically co-evolving code and test cases, characterized in that, The method includes the following steps: S1. Obtain code change information c from the commit log, including the original production code. The changed production code Original test case code ; S2. Preprocess the code change information c, delete duplicate samples, and utilize the original production code. and the changed production code Construct an edit code sequence 𝑒, and expand 𝑒 to include code change information c, obtaining the input sequence I for the classification model. The input to the classification model includes the original production code, the edit code sequence, the modified production code, and the original test case code. The original production code, the modified production code, and the original test case code segments are segmented into a series of morphemes, i.e. , , , where , , and l represent the lengths of the original production code, the modified production code, and the original test case code, respectively; These morphemes are constructed using the Roberta segmenter of the CodeT5 model; 𝑥, 𝑥′, 𝑒, and 𝑡 are concatenated into a single input. ; S3. Input the input sequence I of the classification model into the classification model to obtain the multidimensional space vector representation R of the classification model with expanded code change information; The input sequence I is encoded into a numerical representation using a Transformer-based pre-trained CodeT5 encoder. Then, this encoder converts the numerical representation of the input into a multidimensional space vector representation R. For each input of I in the input sequence The Transformer generates three embedding representations for it, namely vectors. ,vector sum vector In the classification model encoder, dot product is used to calculate... Attention score The method is to use the vector of the nth input. and the vector of the j-th input As shown below: in, yes and Dimensions; Attention Score This represents the level of attention paid to the j-th input when encoding the x-th input; Then, attention score The normalized score is obtained using a softmax function: To learn relevant or irrelevant tokens in the input sequence, the normalized score of the attention score for each input is used as a vector when encoding the i-th input. Multiply and add: Finally, the value of the last hidden layer in the classification model is used as the multidimensional space vector representation of input I, R; S4. Input the obtained multidimensional space vector representation R into the classifier to determine whether the test sample needs to be updated. If no update is needed, end the method; if an update is needed, proceed to the next step. S5. Automatically update test cases by inputting I into the encoder of the generative model to obtain the multidimensional space vector representation of I in the generative model. S6. Input the multidimensional space vector representation of the generative model of I into the decoder in the generative model to obtain the probability distribution. Then, use the dictionary contained in the encoder-decoder to restore the probability distribution to the generated test cases.

2. The method for automatically co-evolving and generating code and test cases according to claim 1, characterized in that, Step S2 specifically involves: Edit code sequence representation is as follows , in and These are the morphemes representing the original and modified production methods at position 𝑖; Indicates will Convert to Editing actions; editing actions are divided into four types, To obtain a representation of code changes, the original and modified production codes are first tokenized into sequences of code lexicals, which are then used to construct the edit code sequence. Word-level alignment is then calculated to obtain the edit code triples at position 𝑖, i.e. The edit code triples are concatenated to obtain the edit code sequence 𝑒.

3. The method for automatically co-evolving and generating code and test cases according to claim 1, characterized in that, Specifically, S4 involves using nonlinearity. A fully connected layer activated by a function learns the potential interactions between different components of the input I; then, the output of the fully connected layer is used to predict the probability of the final label. The classifier is defined as follows: W and b represent the weight matrix and bias vector, respectively; tanh is the activation function of the Dense layer perceptron; the Softmax function outputs the final probability of label l between 0 and 1; a threshold score of 0.5 is set, the probability of a value greater than the threshold score is output as 1, indicating that an update is needed, and the probability of a value less than the threshold score is output as 0, indicating that an update is not needed.

4. The method for automatically co-evolving code and test cases according to claim 1, characterized in that, Specifically, S5 involves: when updating test cases, the encoder of the generated model is the same as the encoder of the classification model; using... Attention score As a generative model of I, a multidimensional spatial vector representation is used as input to the decoder.

5. The method for automatically co-evolving code and test cases according to claim 1, characterized in that, Specifically, S6 involves: test case updating, generating new test cases given the input. The decoder is based on the encoder's output. Generate a new morpheme for the test cases corresponding to all the previous morphemes, using all the morphemes generated so far. ; The decoder consists of multiple decoder layers cascaded together. Each layer contains two sub-layers: a self-attention layer and an encoder-decoder cross-attention layer. The self-attention layer only processes the test case morphemes generated so far, while the encoder-decoder cross-attention layer learns the correlation between the currently generated output sequence and the input sequence. For generating the i-th target morpheme, the workflow of each layer of the decoder is as follows: S6.1 Input representation calculation: Convert the target word sequence generated at the current time into word vectors. Then and the output vector of the previous decoder layer By concatenating the data, we obtain the input representation of the current decoder layer. : in, Indicates a splicing operation; Special cases: When the current moment is the initial state, the target word element is initialized to a special starting character '', indicating that no target word element has been generated yet. The corresponding word vector at this time is denoted as: This is a learnable parameter that needs to be continuously updated with the target lexical and its corresponding word vector representation at the current time step during the generation of the target lexical sequence. When we are currently in the first layer of the decoder, there is no output from the previous layer. At this point, a randomly initialized, fixed-length random vector will be used as the output. This fixed length is the dimension of the model's hidden states; S6.2 Self-Attention Layer Calculation After obtaining the input representation of the current layer, it needs to be used as the Query, Key, and Value, and input into the self-attention function to calculate the attention weights and context vector; specifically, this involves using a linear transformation to... Mapped to three vectors , and : in, , and It is a learnable weight matrix; the self-attention representation of the current decoder layer is computed using the self-attention function, denoted as: in, It is a vector The dimension; S6.3 Encoder-Decoder Cross-Attention Layer Calculation When calculating the attention distribution of the input sequence and target lexical, the self-attention representation of the current decoder layer is used. As a query, the encoder's output As the key and value, these are input into the cross-attention function to calculate the attention weights and context vectors of the input sequence and target morphemes; specifically, through a linear transformation... and Mapped to , , : in, , , It is a learnable weight matrix; Then, the output representation of the current Decoder layer is computed using the cross-attention function, denoted as: in, It is a vector The dimension; S6.4, Feedforward Neural Network Calculates Decoder Layer Output Vector In a feedforward neural network, the output of the current decoder layer is represented as... The representation is mapped to a higher dimension through a fully connected layer, and then mapped back to the original dimension through another fully connected layer; specifically, the output representation of the current decoder layer is mapped... The vector is passed into the feedforward neural network to obtain the output vector of the current decoder layer. : in, and The weight matrix is ​​a learnable matrix. and It is a learnable bias vector; Finally, the calculated That is, the output vector of the decoder when generating the i-th target word; S6.5, Generate target morphemes Finally, the decoder output After linear transformation and activation function, a probability distribution for the i-th target word is obtained: in, and It consists of a learnable weight matrix and a learnable bias vector; Based on the obtained probability distribution and Robert's dictionary, return the word with the highest probability to obtain the target morpheme; S6.6 Repeat S6.1-S6.5 until all target morphemes are generated, resulting in a new test case code.

Images