Defect positioning method of program slice facing deep learning framework

Abstract

The invention discloses a deep learning framework-oriented program slice defect positioning method, which comprises the following steps of: inputting an original failure test case causing defects, and extracting a target variation point; a variation rule oriented to a deep learning framework is designed; generating a mutation test case which is highly similar to the mutation execution path of the original failure test case and rejects an error statement; historical iteration data and the state of a current test case are comprehensively considered, the most suitable mutation operator is selected recursively, and code elements containing errors are gradually eliminated; executing the original failure test case and the mutation test case, and collecting coverage information in the execution process; and according to the collected coverage information and the passing / failure state of the suddenly changed test case, carrying out defect positioning by using a defect positioning algorithm based on a frequency spectrum, and outputting an accurate defect positioning result. According to the method, the defects in the codes in the program slices can be quickly locked and repaired, and the overall reliability of the deep learning framework is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of software engineering and software testing, and in particular relates to a software defect location method for a deep learning framework. Background Art

[0002] Deep learning frameworks, such as PyTorch and TensorFlow, have become the backbone of the deep learning field and have greatly promoted the rapid development of artificial intelligence systems. These frameworks provide a comprehensive and systematic application programming interface (API) throughout the development of deep learning applications. Figure 1 As shown, from data preprocessing to model building and training, to model evaluation, optimization and final deployment, each link is supported by a dedicated API. In the development process of deep learning models, engineers can rely on the APIs of these frameworks to flexibly combine components with different functions such as convolutional layers, fully connected layers, pooling layers and normalization layers, so as to build a neural network model with complex structure and excellent performance. Once the model is built, developers can use the relevant training API to start the training process so that the model can learn efficiently on massive data. After the model training is completed, the importance of the reasoning (or prediction) stage becomes prominent. At this stage, developers apply the trained mature model to actual data by calling the reasoning API of the framework to obtain accurate prediction results. The present invention focuses on the two major deep learning frameworks of TensorFlow and PyTorch. These two frameworks are highly praised in academia and industry, and provide strong support for the research and development of deep learning applications with their powerful functions, excellent performance and excellent ease of use.

[0003] Recent studies have shown that some traditional software defect localization techniques have been proven to be efficient, such as defect localization techniques based on program variation analysis, defect localization techniques based on program slicing, and defect localization techniques based on spectrum. This paper will review the above three techniques and explore the potential applicability of these techniques in the deep learning framework environment.

[0004] Defect localization technology based on program mutation analysis is a method derived from in-depth research on software quality and testing processes. This technology aims to provide developers with an effective way to locate defects by simulating the generation of real defects and their behavior patterns. Specifically, for a given program, this technology artificially introduces a series of subtle changes to generate a mutated program version. These changes, or "mutations", involve subtle adjustments to the original program at the syntax or logic level, such as changing variable values, deleting statements, or modifying the results of conditional judgments. Next, by executing a pre-designed set of test cases on the mutated program, the technology observes and records the impact of these mutations on the program execution behavior. The core concept of this technology is based on an assumption: if a mutation triggers the failure of a test case, the location of the mutation is very likely to be the hiding place of the real defect in the program. This assumption is based on the intuitive understanding that most defects in a program come from the failure of the code to execute as expected, and mutations are simulations of this unexpected execution. Therefore, by analyzing the mutations that cause the test case to fail and their exact location in the code, developers can more accurately identify the real defects that may be potential in the program. This method not only significantly improves the accuracy of defect location, but also helps developers to have a deeper understanding of the behavioral characteristics of the program, thereby effectively improving the quality and reliability of the software. However, in the application of deep learning frameworks, a challenge that cannot be ignored is its compilation process. Deep learning frameworks need to be compiled before they can be applied, and this process often consumes a lot of computing resources and time. The key to defect location technology based on program variation analysis is to simulate potential defect situations by generating numerous mutants by modifying the source code. Since a large number of mutants need to be constructed, this technology will incur significant time overhead.

[0005] Defect localization technology based on program slicing is a key tool in the field of software debugging. Its core advantage is that it can deeply analyze the dependencies between program statements, thereby accurately identifying the set of statements closely related to the defect trigger, namely the so-called program slicing. This technology provides strong support for developers to quickly locate problems in large and complex code bases. Program slicing technology is mainly divided into two categories: static slicing and dynamic slicing. Static slicing relies on static dependency analysis, which can identify all sets of statements associated with specific slicing criteria (such as specific statements or variables) without actually executing the program. Dynamic slicing focuses more on the actual execution process of the program. It collects statements directly related to the slicing criteria based on specific execution traces. In addition, dynamic slicing can be further subdivided into backward slicing and forward slicing. The former focuses on statements that cause program state changes, while the latter focuses on statements affected by these state changes. However, the program slicing-based method faces obvious challenges in processing multidimensional tensors in deep learning frameworks. In the process of building and running the computational graph of deep learning, multidimensional tensors are passed between various computing nodes as core data structures. However, potential defects in deep learning frameworks are often not related to the entire tensor, but are closely related to the specific element values ​​in it. Existing technologies have difficulty accurately performing effective slicing operations on multi-dimensional tensors, and therefore cannot accurately identify specific elements directly related to defects.

[0006] Spectrum-based defect location technology has been deeply applied in the field of traditional software testing, and its main goal is to accurately identify and isolate erroneous elements in the program. The core concept of this technology is to analyze the running frequency of each program element during the execution of the test case by collecting the execution results of the test case (including passed and failed cases). Using these frequency data, the present invention can calculate a suspicious value for each program element, which can be used as a reference indicator to determine whether it is the cause of program failure. In addition, in order to more comprehensively and carefully evaluate the suspicious degree of each element, the present invention often adopts a suspicious value aggregation strategy, such as synthesizing the suspicious values ​​of the statements touched by the failed test case, so as to obtain a more accurate analysis result. This aggregation is not only based on a single statement, but can also be extended to different levels such as basic blocks, functions or files, so as to better map the structural characteristics and logical relationships of the program. In summary, the effectiveness of spectrum-based defect localization technology is affected by three core factors: first, the choice of suspicious value calculation formula, which directly determines how to derive the suspiciousness of program elements from execution frequency; second, the aggregation method of suspicious values, which is related to how to comprehensively and accurately evaluate the possibility of problems in each program element; and finally, the quality of test cases, which form the data basis for spectrum information collection and analysis. Optimizing these key factors can significantly improve the accuracy and efficiency of spectrum-based defect localization technology, thereby more effectively revealing and repairing potential problems in the program.

[0007] However, existing defect localization techniques are mainly designed for general software errors (such as errors in C / Java programs), and their implementation methods and mutation operators are not suitable for deep learning frameworks. In addition, the mutation operators in previous studies are not specifically designed for the unique characteristics of deep learning frameworks, such as computational graphs and specific deep learning operations. Summary of the invention

[0008] In order to fill the gap in the defect localization technical tools of deep learning frameworks, the present invention proposes a defect localization method for program slicing of deep learning frameworks. By designing mutation rules for deep learning frameworks, parsing and simulating original failure test cases that cause defects due to mutations, passing mutation test cases are generated, mutation operators are selected, execution coverage information is collected, and finally spectral technology is used to accurately locate defects.

[0009] The present invention is achieved by the following technical solutions:

[0010] A defect location method for program slicing of a deep learning framework, comprising:

[0011] Input the original failed test case that caused the defect, perform in-depth analysis on the original failed test case to accurately identify the code structure, and extract key elements, namely target mutation points, from the identified code;

[0012] Design mutation rules for deep learning frameworks;

[0013] Apply the mutation rule to implement simulated mutations on the original failed test case, so as to generate a mutation test case that is highly similar to the mutated execution path of the original failed test case and has eliminated the erroneous statements;

[0014] The Metropolis-Hastings algorithm is used to select mutation operators in each round of iteration according to the mutation test cases. The most suitable mutation operator is recursively selected by comprehensively considering the historical iteration data and the status of the current test cases to further perform mutation operations on the target original failed test cases, and gradually eliminate the code elements containing errors.

[0015] Execute the original failure test case and the mutation test case, and collect coverage information during the execution of the original failure test case and the mutation test case respectively;

[0016] According to the collected coverage information and the status of the mutation test case, a spectrum-based defect localization algorithm is used to perform defect localization and output an accurate defect localization result.

[0017] In some embodiments, the mutation rules further include API-level mutation operators, control flow-level mutation operators, variable-level mutation operators, map-level mutation operators, and device-level mutation operators, wherein the parameters in the API call are modified by the API-level mutation operator, the logic of the conditional statement is changed by the control flow-level mutation operator, the value in the assignment statement is replaced by the variable-level mutation operator, the computational graph is modified by the map-level mutation operator, and the device allocation instruction is changed by the device-level mutation operator.

[0018] In some implementations, the step of using a spectrum-based defect localization algorithm to perform defect localization based on the collected coverage information and the status of the mutation test case and outputting an accurate defect localization result further includes:

[0019] Based on the collected coverage information and the status of the mutation test case, the N of each code element in the deep learning framework is calculated by analyzing the coverage information. CF 、N CS 、N UF and N US , including three granularities: file, function, and basic block; the suspicious value score of each code element is calculated, and the expression is as follows:

[0020]

[0021] Among them, N CF Indicates the number of failed test cases covering the statement, N CS Indicates the number of passed test cases covering the statement, N UF Indicates the number of failed test cases that do not cover the statement, N US Indicates the number of passed test cases that do not cover the statement; sort the suspicious value scores of each code element in the deep learning framework, and the higher the ranking, the more suspicious the code element.

[0022] In some embodiments, the selection of mutation operators in each iteration process using the Metropolis-Hastings algorithm according to the mutation test case further includes: sampling the next mutation rule MRb according to the probability distribution of the current mutation rule MRa; if MRb performs better than MRa, MRb will be deterministically accepted; conversely, if MRb performs poorly, there is still a probability of being accepted; after obtaining the result of MRb, updating its score and reordering the mutation rules for the next iteration.

[0023] In some embodiments, the coverage information covers the code path during the execution of each test case, and the code path involves code elements at different levels including files, functions, and basic blocks; the coverage information is converted into a spectrum matrix, in which each row represents a specific code element and each column corresponds to N CF 、N CS 、N UF and N US Four key indicators.

[0024] In some implementations, applying the mutation rule to simulate the mutation of the original failed test case further includes: constructing a corresponding abstract syntax tree from the source code of the original failed test case, and modifying the abstract syntax tree in a targeted manner.

[0025] In some embodiments, the probability of accepting mutation rule MRb given mutation rule MRa is calculated as follows:

[0026]

[0027] Among them, k a and k b They respectively represent the positions of mutation rules MRa and MRb in the mutation rule list arranged in descending order of priority scores.

[0028] Compared with the prior art, the beneficial technical effects and technical progress achieved by the present invention are as follows:

[0029] 1) Compare and analyze the similarities and differences between the original failed test cases and the passed test cases in the execution path, generate new test cases with highly similar execution paths to the original failed test cases, but remove the erroneous statements, and deeply explore the potential connection between code elements and defects;

[0030] 2) Comprehensively consider the historical iteration data and the status of the current test case, select the most suitable mutation operator, and gradually eliminate the code elements containing errors;

[0031] 3) Further, the suspicious scores of three code element granularities, namely files, functions and basic blocks, are calculated, and the higher the ranking, the more suspicious the code elements are, so as to output accurate defect location results, thereby helping developers to quickly locate and fix defects in the code of program slices, which can effectively improve the overall quality and reliability of the deep learning framework; BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flow chart of a defect location method for program slicing in a deep learning framework according to the present invention;

[0033] Figure 2A block diagram of a specific implementation method of a defect location method for a deep learning framework of the present invention;

[0034] Figure 3 This is an example diagram of a deep learning framework of the prior art;

[0035] Figure 4 This is an example diagram of the mutation of the original failed test case;

[0036] Figure 5 An example diagram of a general program mutation operator designed for the present invention;

[0037] Figure 6 Example diagram of the feature mutation operator of the deep learning framework library designed for the present invention;

[0038] Figure 7 Architecture diagram for defect localization dataset;

[0039] Figure 8 Workflow diagram for minimizing manual patching;

[0040] Fig. 9 This is a graph showing the distribution of coverage similarity in Pytorch.

[0041] Fig.10 This is a distribution diagram of coverage similarity in Tensorflow. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] like Figure 1 As shown in FIG. 1 , the overall process of the defect location method for program slicing of a deep learning framework of the present invention is shown, and the method is combined with Figure 2 The block diagram of the specific implementation of the present invention shown in the figure, the overall process of the method includes the following steps:

[0044] Step 1: Input the original failed test case that caused the defect, perform in-depth analysis on the original failed test case to accurately identify its code structure, and extract key elements, i.e., target mutation points, from the identified code;

[0045] Step 2: Design mutation rules for deep learning frameworks. The mutation rules further include API-level mutation operators, control flow-level mutation operators, variable-level mutation operators, layer-level mutation operators, and device-level mutation operators. These layer-level mutation operators are mainly applied to Python-level code for failed test cases. Specifically, taking the Pytorch framework as an example, at the API level, the parameters in the API call are modified through the API-level mutation operator, such as changing the dim parameter in x.split(1, dim=-1) from -1 to 2. At the control flow level, the logic of the conditional statement is changed through the control flow-level mutation operator, such as reversing the if statement to ifnot. At the variable level, the value in the assignment statement is replaced through the variable-level mutation operator, for example, replacing a=10 with a=0. At the layer level, some parts of the computational graph are deleted through the layer-level mutation operator to modify the computational graph. At the device level, the device allocation instructions are changed through the device-level mutation operator, such as changing the device parameter in tensor.to('cpu') from 'cpu' to 'cuda:0'. Specifically, for example, the Pytorch or Tensorflow framework is selected as the deep learning framework to implement the above mutation function; Figure 3 As shown, it is an example of a deep learning framework;

[0046] Step 3: Apply the mutation rule of step 2 to simulate the mutation of the original failed test case. In this process, first, start from the source code of the original failed test case and build the corresponding abstract syntax tree. Then, make subtle but targeted modifications to the abstract syntax tree. For example, replace "if" in the conditional statement with "ifnot", or change the data structure from "list" to "tuple", or even set some input parameters to "None". After completing these modifications, re-export the modified abstract syntax tree as source code to generate a mutation test case that is highly similar to the original failed test case mutation execution path and eliminates the erroneous statements. The mutation test case can successfully pass the test execution. Figure 4 Shown is an example diagram of the original failed test case mutation;

[0047] Step 4: For the mutation test cases obtained by mutation rules in step 3, the Metropolis-Hastings algorithm is used to select mutation operators in each round of iteration. This algorithm comprehensively considers the historical iteration data and the status of the current test case, and recursively selects the most suitable mutation operator to further perform mutation operations on the target original failed test cases, gradually eliminating code elements that may contain errors;

[0048] Step 5: Execute the original failed test case and the mutation test case, and collect coverage information in detail during the execution process, respectively. The coverage information includes key data of the granularity of executed code lines, code blocks, functions, and files;

[0049] Step 6: Based on the coverage information collected in step 2 and the pass / fail status of the mutation test case, use the spectrum-based defect localization algorithm to locate the defect and output an accurate defect localization result; the specific process of using the spectrum-based defect localization algorithm to locate the defect is: based on the collected coverage information and the pass / fail status of the mutation test case, by analyzing the coverage information, calculate the N of each code element in the deep learning framework. CF 、N CS 、N UF and N US , including three granularities: file, function and basic block. A basic block specifically refers to a sequence of statements that are executed continuously in a program. It exists in the internal structure of a function and usually contains multiple such basic blocks. The suspicious value Ochiai formula is used as the code suspicious value calculation formula, combined with the N of each code element CF 、N CS 、N UF and N US , thereby calculating the suspicious value score of each code element, the expression is as follows:

[0050]

[0051] Among them, N CF Indicates the number of failed test cases covering the statement, N CS Indicates the number of passed test cases covering the statement, N UF Indicates the number of failed test cases that do not cover the statement, N US Indicates the number of passed test cases that do not cover the statement;

[0052] Sort the suspicious value scores of each code element in the deep learning framework. The higher the ranking, the more suspicious the code element is. It is hoped that the code element where the real defect is located can be ranked as high as possible.

[0053] In spectrum-based technologies, the suspicious value of each code statement is calculated in various ways, as shown in Table 1, which is the defect location suspicious value calculation formula of the existing technology, including formulas such as Tarantula, Ochiai, and DStar.

[0054] Table 1

[0055]

[0056]

[0057] Further, the specific description of the steps involved in the process of the present invention is as follows:

[0058] The present invention first designs a set of mutation rules specifically for deep learning frameworks. These rules are divided into API level, control flow level, variable level, layer level and device level to ensure a comprehensive mutation strategy. Next, the original failed test case that caused the defect will be deeply analyzed, its code structure will be accurately identified, and key elements, namely target mutation points, will be extracted from it. Subsequently, the present invention applies the aforementioned mutation rules to mutate the original failed test case, aiming to generate new test cases that are highly similar to the execution path of the original failed test case, but have eliminated the erroneous statements. These newly generated test cases can successfully pass the test execution.

[0059] The core concept of the present invention is to generate test cases that can pass by performing refined mutation operations on the original failed test cases, thereby locating defects in the deep learning framework. These test cases that pass the test are highly consistent with the failed test cases in the execution path, and the only difference is that the statements containing errors are excluded. In this way, the spectrum-based defect location technology can effectively exclude innocent statements and accurately lock the erroneous statements.

[0060] Therefore, the present invention designs a series of special mutation operators and implements a mutation-based test generation framework for the particularity of the deep learning framework. The framework analyzes the original failed test cases that cause defects and builds the corresponding abstract syntax tree based on the code; then, by traversing all APIs in the program, obtaining their positions and parameter information, identifying which positions and parameters can be used as potential mutation points, and assigning its adapted mutation operator to each mutation point. Specifically, if an API related to the deep learning framework is detected, an API-level mutation operator is assigned; if a control flow element such as if is detected, a control flow-level mutation operator is assigned; for code elements of declaration or assignment type (such as a=True), a variable-level mutation operator is assigned; if a computational graph construction function such as forward is detected, a graph-level mutation operator is assigned; if device parameters such as 'cpu' are monitored, a device-level mutation operator is assigned. Specifically, the framework designs five levels of mutation operators for the deep learning framework: API level, control flow level, variable level, graph level, and device level. In step 2, examples of input and output codes for each mutation operator are added. The above rules are adapted to two specific deep learning frameworks, Pytorch and Tensorflow. Among them, operators at the API level, control flow level, and variable level are developed based on existing research, while operators at the map level and device level are innovatively designed based on the unique characteristics of deep learning programs. Through the comprehensive application of these operators, the framework of the present invention can comprehensively and accurately locate defects in deep learning frameworks, providing strong support and assistance to developers.

[0061] In a deep learning framework, it is usually composed of numerous API calls and branch structures. The framework of the present invention determines the executable part of the code through the mutation operator at the control flow level, and then in each executable part, the mutation operator at the API level is used to change the specific API call, and the mutation operator at the variable level is used to modify the properties of the variable. In addition, in order to adapt to the particularity of deep learning programs, the present invention also uses the mutation operator at the graph level to modify the structure of the calculation graph, and introduces the mutation operator at the device level to change the device properties of the deep learning framework program. Through these hierarchical mutation operations, the present invention can achieve comprehensive positioning and precise repair of deep learning framework errors.

[0062] like Figure 4 and Figure 5 As shown in the figure, the detailed definitions of mutation operators at each level are as follows. These mutation operators can currently be adapted in the mainstream Pytorch and Tensorflow frameworks, thus ensuring their extensiveness and effectiveness in practical applications:

[0063] ① API-level mutation operator: The deep learning framework provides various APIs. When a test case triggers an error in an API, modifying the parameters of the API may turn a failed test case into a passed test case. The present invention can modify API parameters in the following three ways: 1) changing the API parameter setting to its default value; 2) modifying the value of an API parameter to another valid value within its value range; 3) assigning a valid value to an unused API parameter. In addition, the present invention can also replace the API with another API with a similar function signature.

[0064] ② Control flow level mutation operator: By modifying the control flow of the failed test case, the present invention can effectively change its execution path and possibly transform it into a passed test case. The present invention adopts the following two mutation strategies: 1) directly commenting out the code entry that directly causes the trigger error; 2) modifying the branch in the test case. For example, the present invention can modify the branch in the test case to ifTrue to execute the branch that was not executed before.

[0065] ③ Computational graph level mutation operator: Deep learning frameworks usually contain codes for processing various types of computational graphs. The present invention can modify the test case to change the computational graph in the following three ways: 1) modify the forward propagation process; 2) modify the configuration related to the gradient in the computational graph to retain or discard the gradient; 3) modify the type of the computational graph (i.e., static type or dynamic type).

[0066] ④ Variable-level mutation operators: Mutation operators at this level are designed to modify the properties of the original variable (non-tensor variable) in the following ways: 1) change the variable type (for example, change list to tuple, change integer to float); 2) change the data of the original variable (for example, change 2 to 0, change True to False); 3) change the subscript of the variable (for example, change x[:2] to x).

[0067] ⑤ Device-level mutation operator: Given that deep learning frameworks generally support two different types of computing devices, CPU and CUDA, and that the underlying implementation mechanisms on these two devices are different, the present invention is expected to transform originally failed test cases into successful test cases by adjusting the device environment in which the program runs.

[0068] In step 2, the mutation test cases obtained by mutation rules in step 1 are used, and the Metropolis-Hastings algorithm is used to guide the mutation operator selection process in each round of iteration. The algorithm can comprehensively consider the historical iteration data and the specific status of the current test case, and recursively select the most suitable mutation operator. This selection process aims to further implement mutation operations on the target original failed test cases, thereby gradually eliminating potential erroneous code elements. Subsequently, by executing the original failed test cases and the successfully generated passing test cases through mutation operations, the coverage information during the execution is collected in detail. This information includes key data such as executed code lines, code blocks, functions, and files, which provides an important basis for subsequent defect location and repair.

[0069] In view of the huge search space and the limitation of computing resources, the present invention cannot use exhaustive method to generate all possible passing test cases and screen out valid subsets from them. Therefore, the present invention proposes a more cost-effective method: in each generation process, strive to create a passing test case that maximizes the difference from the existing program. For a given original failed test case, the present invention observes that the effects of different mutation rules are different. Mutation rules that can generate diversified passing test cases with high frequency should be given a higher selection probability for subsequent mutation operations. Based on this insight, the present invention will elaborate on a heuristic-based test case generation method in the following section. In particular, the present invention introduces coverage distance as an indicator to measure the diversity between test programs. For a given test program P j and P k , the Jaccard distance between their code coverage is defined as follows to quantify the difference between the two:

[0070]

[0071] in and Respectively represent P j and P k The set of lines of code that are covered.

[0072] After selecting the seed program, the present invention selects mutation rules for mutation. However, these mutation rules are not all equally effective in generating various types of passing test cases for a given failing test case. At the same time, the same mutation rule also behaves differently for different initial failing test cases. Therefore, the present invention designs an adaptive process to select mutation rules to construct an effective set of passing test cases. If a mutation rule can more frequently generate passing test cases with greater diversity than existing test cases, then the rule should be selected with a higher probability for subsequent mutation.

[0073] The mutation rules can be arranged in descending order according to their calculated priority scores. However, since the ranking is only based on the historical performance of these mutation rules, it is difficult to accurately predict their future effects. Therefore, the present invention does not directly select the mutation rule with the highest ranking for subsequent mutation. In view of this, the present invention adopts a probabilistic selection mechanism to ensure that each mutation rule has a chance to be selected for the next mutation, while ensuring that the mutation rules with higher positions in the ranking list have a relatively higher probability of being selected. In other words, the problem of selecting mutation rules can be transformed into a problem of sampling from a specific probability distribution. Based on this logic, the present invention calculates a priority score for each mutation rule MR, and after each iteration, updates the score of each mutation operator r according to the established rules:

[0074]

[0075] Where n represents the number of passed test cases generated up to the current iteration, P represents the number of new passed test cases generated during the current iteration, and Rate success It is the success rate of passing test cases among the test cases generated by r.

[0076] Given that the selection of mutation rules depends on their recent performance, this problem can be regarded as a typical Markov Chain (MC) application scenario. In order to solve the problem of sampling from probability distribution, the Markov Chain Monte Carlo Metropolis-Hastings (MH) algorithm is used, and it is assumed that the required distribution is an equilibrium distribution. Within this framework, the MH algorithm is used to obtain random samples from a specific probability distribution. In the case of the present invention, the algorithm samples the next mutation rule (denoted as MRb) according to the probability distribution of the current mutation rule (denoted as MRa). If MRb performs better than MRa, that is, MRb has a higher priority score than MRa, then MRb will be deterministically accepted; conversely, if MRb performs poorly, it still has a certain probability of being accepted. The present invention selects geometric distribution as the probability distribution model, which refers to the probability distribution law of the number of trials required to obtain a success in a series of Bernoulli trials. If the probability of success of each trial is p, then the probability of the first success in the kth trial can be calculated as:

[0077] P s (X=k)=(1-p) k-1 ·p

[0078] In this process, the mutation rule is randomly selected, and the probability of accepting the mutation rule MRb given the known mutation rule MRa is calculated as follows:

[0079]

[0080] Among them, ka and k b Respectively represent the positions of mutation rules MRa and MRb in the mutation rule list arranged in descending order according to priority scores. This mechanism ensures the flexibility and diversity of mutation rule selection. After obtaining the result of MRb (i.e., acceptance or rejection), the present invention updates its score and reorders these mutation rules for the next iteration.

[0081] After generating a variety of passing test cases, the present invention further uses the professional tool Coverage.py to execute failed test cases and passed test cases. This is intended to collect coverage information of these test cases on the source code of the deep learning framework in a detailed and systematic manner. By combining Coverage.py and Scalpel tools, the present invention can accurately obtain multi-dimensional detailed coverage data including code lines, code blocks, functions, and files.

[0082] In step 3, the present invention adopts spectrum-based defect location technology based on the detailed execution coverage information and test case pass / fail status obtained in step 2. With the help of a carefully designed defect location formula, the present invention is able to deeply compare and analyze the differences and commonalities between the original failed test cases and the passed test cases in the execution path, and then explore the deep connection between code elements and defects. Furthermore, by accurately calculating the suspicious scores of three types of code elements such as files, functions and basic blocks, the present invention can output highly accurate defect location results. This is intended to provide developers with strong support for quickly locating and fixing defects in the code, thereby significantly improving the overall performance and stability of the deep learning framework.

[0083] The coverage information covers the code paths touched during the execution of each test case, which involve code elements at different levels such as files, functions, and basic blocks. This type of coverage data provides a solid data foundation for spectrum analysis. Specifically, the present invention converts the collected coverage information into a spectrum matrix, and combines the pass / fail status of the mutation test case to calculate in detail the number of times each code element is executed by a passed test case, a failed test case, or not executed. In the constructed spectrum matrix, each row represents a specific code element, and the four columns correspond to the N mentioned in the background technology. CF 、N CS 、N UF and N US Four key indicators.

[0084] After the spectrum matrix is ​​constructed, the present invention draws on the practical experience of traditional software defect location and selects the Ochiai formula as the basis for calculating the suspicious value of the code to accurately locate the defective code in the deep learning framework. The Ochiai formula is a classic algorithm in the field of defect location. It determines the suspicious degree of the code element by quantifying the correlation between the frequency of test cases covering specific code elements and the test case results. More specifically, the Ochiai formula reveals the possible connection between certain code elements and potential defects by comparing and analyzing the similarities and differences between failed test cases and passed test cases in the execution path. The specific calculation method of the suspicious value suspiciousness of the Ochiai formula is as follows:

[0085]

[0086] In addition, the present invention also considers the calculation of suspicious values ​​of different program element granularities (including code blocks, functions, and files). The suspicious values ​​of these elements are obtained by aggregating the suspicious values ​​of their internal statements. To achieve this goal, the present invention adopts an average aggregation algorithm, which can calculate the suspicious scores of code elements of different granularities (such as files, functions, and basic blocks) respectively. Specifically, the suspicious value of a function can be expressed as the average value of the suspicious values ​​of all its internal statements, and the calculation formula is as follows:

[0087]

[0088] After calculating the suspicious score of each code element, the present invention further arranges these elements in reverse order according to the suspicious score. This arrangement can give priority to displaying code elements with higher suspicious scores, thereby helping developers to more efficiently screen out potential defect locations. Through this method, developers can quickly locate the code area that is most likely to contain defects, thereby significantly reducing the scope and time cost of manual investigation.

[0089] In order to verify the effectiveness of the present invention, the experimental results were scientifically analyzed.

[0090] The present invention uses the test cases provided by developers in the source code of the deep learning framework as a baseline method. When locating defects in the deep learning framework, it is an efficient and direct method to collect test cases directly from the source code of the framework. This is because developers provide a large number of test cases to ensure the correctness and stability of the code in the process of writing the framework. These test suites cover various API calls of the deep learning framework and can cover more potential defect scenarios. Therefore, the present invention implements a defect location method based on the test suite in the source code of the deep learning framework, which the present invention refers to as Developer. For each defect, the present invention identifies and extracts relevant test cases from the corresponding deep learning framework source code, and executes the collected and passed test cases in the corresponding runtime environment.

[0091] The present invention targets the PyTorch framework and identifies and extracts all matching Python files using the regular expression pattern test_*.py. To ensure that the collected test cases are limited to the code of PyTorch itself, the present invention excludes files located in the "third_party" directory because these files belong to third-party dependencies rather than PyTorch core code. For the TensorFlow framework, the present invention adopts a similar strategy, but the matching rules are slightly different. The present invention collects all Python files whose names end with *_test.py, which typically contain test cases for TensorFlow functions.

[0092] However, it should be noted that directly executing the entire test file may be very time-consuming. Therefore, in order to improve efficiency and pertinence, the present invention does not consider the entire file as a test case, but treats each test function in the file as an independent test case. In order to automatically execute these test cases and collect runtime data, the present invention adopts the Pytest framework. Pytest is a powerful Python testing framework that allows the present invention to conveniently execute test functions and collect spectrum data at runtime.

[0093] Since there is no defect localization technology specifically for deep learning frameworks, the corresponding real and reliable defect localization datasets are also extremely scarce. In order to fill this gap, Figure 6 As shown in the figure, the present invention carefully collected 80 real-world crash defects from GitHub and evenly distributed them in two mainstream deep learning frameworks, PyTorch and TensorFlow, for experimental analysis. Specifically, these defect data cover 12 different versions of PyTorch and 11 versions of TensorFlow, which fully demonstrates the diversity and complexity of deep learning frameworks between different versions.

[0094] In order to make the defect localization dataset include defects from different versions of deep learning frameworks, the present invention crawled all merged and closed pull requests (PRs) from the official GitHub repositories of PyTorch and TensorFlow between "2019-01-01" and "2023-12-31". At this stage, the present invention obtained 1,802 PRs for PyTorch and 9,868 PRs for TensorFlow. The present invention randomly extracted 200 PRs from TensorFlow and PyTorch to determine the keywords that frequently appear in the PR information of fixing defects. The present invention obtained four keywords related to fixing defects, including fix, bug, error, and fault. Then, the present invention used the above keywords to filter PRs that did not contain these keywords in the title, and finally obtained 435 PRs for PyTorch and 2,181 PRs for TensorFlow. Since manually configuring the runtime environment is time-consuming and laborious, it is almost impossible to include defects in all PRs in the defect localization dataset. Therefore, the present invention randomly extracts 40 reproducible defects of each framework from all PRs and constructs a defect localization dataset.

[0095] For each extracted PR, the present invention first checks whether the PR contains defects that cause crashes or incorrect functions, and whether a script that triggers the defect (i.e., a failed test case) is provided. Then, the present invention extracts the merged commits from the PR page as fix commits. The commit containing the defect is usually the closest commit to the fix commit (i.e., the parent node of the same branch) that introduced the defect. The present invention determines whether a defect should be included in the defect localization dataset according to the following rules:

[0096] ① Defect reproduction program: Defect reproduction program is a key component of dynamic defect location technology. These programs are responsible for executing test cases to trigger potential defects in the deep learning framework and collect corresponding spectrum data for subsequent analysis. The lack of defect reproduction program will lead to incomplete testing process, and thus the spectrum data cannot be generated for effective dynamic defect location. Therefore, when constructing the data set, the present invention filters out the defects of the lack of defect reproduction program to ensure that the data set of the present invention contains defect samples that can provide sufficient information to support dynamic defect location technology.

[0097] ② Compilation status: For each defect in the defect location data set, the present invention needs to perform Linux-level source code compilation according to the defect submission (that is, the defect includes the defect submission) to build a corresponding defect reproduction environment. Failed compilation and installation processes will result in the inability to perform subsequent testing and analysis work, making it impossible to accurately assess the nature and impact of the defect. If the defect submission cannot be successfully compiled and installed, the present invention will exclude the defect from the defect location data set to ensure that all defects contained in the data set of the present invention can be reproduced.

[0098] ③ Patch correctness: The correctness of the patch is one of the key indicators for evaluating the effect of defect location. If the defect triggering test still fails after applying the repair patch, the present invention believes that the defect has not actually been successfully repaired and excludes it from the defect location data set of the present invention. This is because even if the present invention applies a seemingly correct repair patch to the defect, if the same defect can still be triggered in subsequent tests, it means that the repair patch has not successfully solved the corresponding problem. This may be because the repair patch does not fully cover the root cause of the problem, or it may be that the repair patch introduces new problems, causing the original defect to still exist.

[0099] ④ Reproducibility: Reproducibility is a prerequisite for evaluating the effectiveness of defect location technology. If the defect cannot be reproduced reliably, that is, the same defect behavior cannot be triggered again under different environments or conditions, the present invention excludes it from the defect location data set. Defects that lack reproducibility may be caused by differences in factors such as test environment, configuration, or dependencies. If the present invention cannot reproduce the same defect in different environments, it cannot be accurately analyzed and evaluated, thereby failing to ensure that the research and improvement work of the present invention is based on a reliable data foundation.

[0100] Since developers may fix multiple defects or introduce unrelated changes (e.g., refactoring) in the same commit, marking all code changes as root causes may inaccurately evaluate the effectiveness of defect localization techniques. To address this issue, the present invention performs patch minimization in a semi-automated manner to accurately identify the defect location of each defect.

[0101] Figure 6The process of root cause marking is demonstrated. The present invention first removes all document-related changes and test case changes from the patch because they are not related to fixing the defect. Then, the present invention executes the failed test cases and collects the corresponding code spectrum information. If any subset in the patch is not involved by the failed test cases, the present invention excludes them from the patch because they are also not related to the defect. Finally, the present invention manually checks the remaining code changes to determine which part of the code is related to fixing the defect. Then, the present invention marks these code changes as the root cause of the defect. In some cases, the developer may only insert new elements in the patch instead of modifying or deleting old elements. In order to determine the root cause when the developer inserts an element in the patch, the present invention follows previous work and marks the element immediately following the inserted element as the root cause. After the patch is minimized, the present invention manually marks the code blocks, functions, and files containing the patch as defective code blocks, defective functions, and defective files, respectively.

[0102] The present invention adopts the worst ranking strategy to generate the ranking list, which has been used in previous studies. Specifically, when two elements have the same suspicious value and are ranked in the kth position, the present invention assigns the k+1th position as their ranking. The present invention uses a widely used evaluation method as follows:

[0103] ①Top-N: Top-N refers to the number of defects successfully located in the top N positions. The larger the value of Top-N, the better the performance of the defect localization technology.

[0104] ②Mean First Rank (MFR): First rank refers to the ranking of the first defect element of each defect. MFR represents the average ranking of the first defect elements of all defects. The smaller the MFR value, the better the performance.

[0105] ③MeanAverage Rank (MAR): Average rank refers to the average rank of all defect elements for each defect. MAR represents the average of the average ranks of all defects. The smaller the MAR value, the better the performance.

[0106] The project of the present invention is implemented using Python 3.11 as the main programming language. The experimental environment is configured on a high-performance server, which is equipped with an Intel Xeon Gold 6326 CPU chip, an RTX 3090 GPU, 512G memory, and runs the Ubuntu 18.04.6LTS operating system. To ensure the precise positioning of defects, the present invention sets a 60-minute termination condition for the positioning process of each defect. At the same time, in order to reduce the impact of randomness on the experimental results during the test generation process, the present invention adopts a strategy of repeated experiments, that is, each experiment is repeated five times, and the average result is finally reported, thereby ensuring the stability of the experimental data and the reliability of the results of the present invention.

[0107] Tables 2 and 3 compare in detail the defect localization method for deep learning frameworks proposed by the present invention, which the present invention calls TenMutFL, with the application effect of the existing baseline method Developer in deep learning frameworks. Specifically, in Table 2, the present invention records in detail the specific number of defects successfully located in the Top-N position on the two major mainstream deep learning frameworks of PyTorch and TensorFlow when a specific test generation method is adopted. These data provide a quantitative basis for evaluating the effectiveness of the localization method. Furthermore, in Table 3, the present invention presents the average value of the ranking of successfully located defective elements under different defect granularities of the PyTorch and TensorFlow frameworks by different test generation methods. This statistic helps to more comprehensively evaluate the accuracy and stability of the localization method. By combining the data in Tables 2 and 3, the present invention can make a comprehensive performance comparison of the TenMutFL method and the Developer method.

[0108] By deeply analyzing the data in Table 2 and Table 3, the present invention draws a clear conclusion: the TenMutFL method shows significant effectiveness in locating defects in deep learning frameworks. In a comprehensive evaluation covering two deep learning frameworks (PyTorch and Tensorflow), three different granularities, and four top-N positions, the performance of the TenMutFL method is better than the comparative method Developer. Specifically, under the PyTorch framework, the TenMutFL method can accurately locate 62.5% and 80% of the defects in the top 10 functions and the top 10 files, respectively; while in the Tensorflow framework, the method has a defect location accuracy of 57.5% and 72.5% in the top 10 functions and the top 10 files, respectively. It is worth mentioning that even with finer granularity division and more Top position considerations, the TenMutFL method can still maintain a high defect location effect, which fully verifies its reliability and superiority in practical applications.

[0109] The superiority of the TenMutFL method over the comparative method Developer is explored in depth, and a detailed manual test case analysis is conducted. Through comparative analysis, the present invention reveals the dual advantages of the TenMutFL method in generating test cases:

[0110] 1) The execution spectrum of the test cases generated by this method on error-free code elements is highly similar to that of failed test cases. This feature effectively reduces the noise interference in the ranking process of error elements and improves the accuracy of ranking.

[0111] 2) On the erroneous code elements, the test cases generated by the TenMutFL method show significant execution spectrum differences compared with the failed test cases, which provides strong support for accurately locating defects.

[0112] In summary, the TenMutFL method achieves more efficient defect localization by optimizing the similarities and differences of execution spectra.

[0113] The Top-N results of the control method Developer and the method of the present invention TenMutFL are shown in Table 2. The MFR / MAR results of the control method Developer and the method of the present invention TenMutFL are shown in Table 3.

[0114] Table 2

[0115]

[0116] Table 3

[0117]

[0118] In order to further explore the effectiveness of the TenMutFL method in deep learning framework defect localization, this study compares the code coverage of test cases generated by the TenMutFL method and the Developer method, and also considers the code coverage of the original failed test cases on all crash defects. Specifically, the present invention calculates the number of code blocks covered by the failed test cases and the passed test cases, and calculates the ratio of it to the total number of code blocks touched by the failed test cases.

[0119] Figure 6 and Figure 7 The distribution of coverage similarity between the two test generation techniques is shown in detail. The box plot clearly shows the median, quartiles and outliers of the data, while the violin plot further reveals the distribution density of the data in different value ranges. Figure 6 is the distribution of coverage similarity in Pytorch, Figure 7is the distribution of coverage similarity in Tensorflow, through Figure 6 and Figure 7 Through careful observation, the present invention finds that compared with the Developer method, the mutation-based test generation method TenMutFL performs well in constructing passing test cases with higher coverage similarity to failed test cases. This advantage provides the SBFL technology with a more accurate ability to distinguish between error codes and error-free codes. Therefore, it can be clearly concluded that the test cases generated by the TenMutFL method can significantly improve the effectiveness of the SBFL technology in identifying defective elements.

[0120] It should be noted that, although the present invention has been shown and described with reference to specific exemplary embodiments of the present invention, those skilled in the art should understand that the present invention is not limited to the above-mentioned embodiments and all kinds of changes to the present invention fall within the scope of protection of the present invention.

Claims

1. A defect location method for program slicing in a deep learning framework, characterized in that: include: Input the original failed test case that caused the defect, perform in-depth analysis on the original failed test case to accurately identify the code structure, and extract the target variation point from the identified code; Design mutation rules for deep learning frameworks; Apply the mutation rule to implement simulated mutations on the original failed test case, so as to generate a mutation test case that is highly similar to the mutated execution path of the original failed test case and has eliminated the erroneous statements; The mutation operator selection in each round of iteration is implemented according to the mutation test case, and the most suitable mutation operator is recursively selected by comprehensively considering the historical iteration data and the status of the current test case to further perform mutation operations on the target original failed test case, and gradually eliminate the code elements containing errors; Execute the original failure test case and the mutation test case, and collect coverage information during the execution of the original failure test case and the mutation test case respectively; According to the collected coverage information and the status of the mutation test case, a spectrum-based defect localization algorithm is used to perform defect localization and output an accurate defect localization result.

2. The defect location method for program slicing in a deep learning framework according to claim 1, characterized in that: The mutation rules further include API-level mutation operators, control flow-level mutation operators, variable-level mutation operators, map-level mutation operators and device-level mutation operators, wherein the API-level mutation operator is used to modify parameters in an API call, the control flow-level mutation operator is used to change the logic of a conditional statement, the variable-level mutation operator is used to replace the value in an assignment statement, the map-level mutation operator is used to modify a computational graph, and the device-level mutation operator is used to change a device allocation instruction.

3. The defect location method for program slicing in a deep learning framework according to claim 1, characterized in that: The step of using a spectrum-based defect localization algorithm to perform defect localization based on the collected coverage information and the status of the mutation test case and outputting an accurate defect localization result further includes: Based on the collected coverage information and the status of the mutation test case, the N of each code element in the deep learning framework is calculated by analyzing the coverage information. CF 、N CS 、N UF and N US , including three granularities: file, function, and basic block; the suspicious value score of each code element is calculated, and the expression is as follows: Among them, N CF Indicates the number of failed test cases covering the statement, N CS Indicates the number of passed test cases covering the statement, N UF Indicates the number of failed test cases that do not cover the statement, N US Indicates the number of passed test cases that do not cover the statement; sort the suspicious value scores of each code element in the deep learning framework, and the higher the ranking, the more suspicious the code element.

4. The defect location method for program slicing in a deep learning framework according to claim 1, characterized in that: The mutation operator selection in each round of iteration process is implemented according to the mutation test case using the Metropolis-Hastings algorithm, and the implementation process further includes: sampling the next mutation rule MRb according to the probability distribution of the current mutation rule MRa; if MRb performs better than MRa, MRb will be deterministically accepted; conversely, if MRb performs poorly, it still has a probability of being accepted; after obtaining the result of MRb, its score is updated, and the mutation rules are reordered for the next iteration.

5. The defect location method for program slicing in a deep learning framework according to claim 1, characterized in that: The coverage information covers the code path during the execution of each test case, and the code path involves code elements at different levels including files, functions and basic blocks; The coverage information is converted into a spectrum matrix, where each row represents a specific code element and each column corresponds to N CF 、N CS 、N UF and N US Four key indicators.

6. The defect location method for program slicing in a deep learning framework according to claim 1, characterized in that: It further includes performing the defect location evaluation according to the defect reproduction procedure, compilation status, patch correctness, and reproducibility, and incorporating defects that meet the requirements into the defect location data set.

7. The defect location method for program slicing in a deep learning framework according to claim 1, characterized in that: The probability of accepting mutation rule MRb given mutation rule MRa is calculated as follows: Among them, k a and k b They respectively represent the positions of mutation rules MRa and MRb in the mutation rule list arranged in descending order of priority scores.

8. The defect location method for program slicing in a deep learning framework according to claim 6, characterized in that: Further including, removing all document-related changes and test case changes from the patch, then executing the original failing test cases and collecting the corresponding code spectrum information; Exclude from the patch any subset of the code that was not covered by the original failing test cases, and finally inspect the remaining code changes to identify the portion of code that is relevant to fixing the defect, and mark this portion of the code as the cause of the defect.

9. The defect location method for program slicing in a deep learning framework according to claim 6, characterized in that: In case of inserting new elements in the patch, the element immediately following the inserted element is marked as the root cause. After the patch is minimized, the code block, function and file containing the patch are marked as defective code block, defective function and defective file respectively.

Images