A Defect Localization Method for Program Slices in Deep Learning Frameworks

Abstract

This invention discloses a defect localization method for program slices in deep learning frameworks. The method includes: inputting the original failed test cases that caused the defect; extracting the target mutation point; designing mutation rules for deep learning frameworks; generating mutated test cases with execution paths highly similar to the original failed test cases but with erroneous statements removed; recursively selecting the most suitable mutation operator by comprehensively considering historical iteration data and the current test case state, gradually eliminating code elements containing errors; executing the original failed test cases and mutated test cases, collecting coverage information during execution; and using a spectrum-based defect localization algorithm to locate the defect based on the collected coverage information and the pass / fail status of the mutated test cases, outputting accurate defect localization results. This invention can quickly locate and repair defects in the code of program slices, effectively improving the overall reliability of deep learning frameworks.

Description

Technical Field

[0001] This invention belongs to the fields of software engineering and software testing, and in particular relates to a method for locating software defects for deep learning frameworks. Background Technology

[0002] Deep learning frameworks, such as PyTorch and TensorFlow, have become the backbone of the deep learning field, greatly driving the rapid development of artificial intelligence systems. These frameworks provide a comprehensive and systematic set of Application Programming Interfaces (APIs) that are integral to the entire process of deep learning application development. For example... Figure 2 As shown, each step, from data preprocessing to model building and training, and then to model evaluation, optimization, and final deployment, is supported by dedicated APIs. During the development of deep learning models, engineers can leverage these framework APIs to flexibly combine components with different functions, such as convolutional layers, fully connected layers, pooling layers, and normalization layers, to construct complex and high-performance neural network models. Once the model is built, developers can use the relevant training APIs to start the training process, enabling the model to learn efficiently on massive amounts of data. After model training, the importance of the inference (or prediction) stage becomes apparent. In this stage, developers apply the trained, mature model to real-world data by calling the framework's inference API to obtain accurate prediction results. This invention specifically focuses on two major deep learning frameworks: TensorFlow and PyTorch. These two frameworks are highly praised in both academia and industry, providing strong support for the research and development of deep learning applications due to their powerful functionality, excellent performance, and outstanding ease of use.

[0003] Recent studies have shown that some traditional software defect localization techniques have proven to be efficient, such as defect localization techniques based on program mutation analysis, defect localization techniques based on program slicing, and defect localization techniques based on spectrum. This invention will review these three techniques and explore their potential applicability in deep learning framework environments.

[0004] Defect localization technology based on program mutation analysis is a method derived from in-depth research into software quality and the testing process. This technology aims to provide developers with an effective way to locate defects by simulating the generation and behavioral patterns of real defects. Specifically, for a given program, this technology artificially introduces a series of subtle changes, generating a mutated version of the program. These changes, or "mutations," involve minor adjustments to the original program at the syntactic or logical level, such as changing variable values, deleting statements, or modifying the results of conditional judgments. Next, by executing a pre-designed set of test cases on the mutated program, the technology observes and records the impact of these mutations on the program's execution behavior. The core concept of this technology is based on the assumption that if a mutation triggers the failure of a test case, the location of that mutation is very likely to be a hidden location of a real defect in the program. This assumption is based on the intuitive understanding that defects in programs often stem from code not executing as expected, and mutations are simulations of this unexpected execution. Therefore, by analyzing the mutations that cause test case failures and their exact locations in the code, developers can more accurately identify potential real defects in the program. This method not only significantly improves the accuracy of defect localization but also helps developers gain a deeper understanding of program behavior, thereby effectively improving software quality and reliability. However, a significant challenge in the application of deep learning frameworks lies in their compilation process. Deep learning frameworks require compilation before application, a process that often consumes substantial computational resources and time. Defect localization techniques based on program mutation analysis rely on generating numerous mutants by modifying the source code to simulate potential defect scenarios. Due to the need to construct a large number of mutants, this technique incurs significant time overhead.

[0005] Defect localization based on program slicing is a key tool in software debugging. Its core advantage lies in its ability to deeply analyze the dependencies between program statements, thereby accurately identifying the set of statements closely related to defect triggering—the so-called program slice. This technology provides powerful support for developers to quickly locate problems in large and complex codebases. Program slicing technology is mainly divided into two categories: static slicing and dynamic slicing. Static slicing relies on static dependency analysis, which can identify all sets of statements associated with a specific slicing criterion (such as a specific statement or variable) without actually executing the program. Dynamic slicing, on the other hand, focuses more on the actual execution flow of the program, collecting statements directly related to the slicing criterion based on a specific execution trajectory. In addition, dynamic slicing can be further subdivided into backward slicing and forward slicing. The former focuses on statements that cause changes in program state, while the latter focuses on statements affected by these state changes. However, program slicing-based methods face significant challenges when dealing with multidimensional tensors in deep learning frameworks. In the construction and operation of deep learning computational graphs, multidimensional tensors serve as core data structures passed between various computational nodes. However, potential defects in deep learning frameworks are often unrelated to the entire tensor, but are closely related to the values ​​of specific elements within it. Existing techniques struggle to perform effective slicing operations on multidimensional tensors with precision, thus failing to accurately identify specific elements directly related to defects.

[0006] Spectrum-based defect localization technology has been widely applied in traditional software testing, with the primary goal of accurately identifying and isolating erroneous elements in a program. The core concept of this technology lies in analyzing the execution frequency of each program element during test case execution by collecting the execution results of test cases (including both successful and failed cases). Using this frequency data, this invention can calculate a suspicious value for each program element, which can serve as a reference indicator for determining whether it is the cause of program failure. Furthermore, to more comprehensively and meticulously assess the suspiciousness of each element, this invention often employs a suspicious value aggregation strategy, such as combining the suspicious values ​​of statements touched by failed test cases to obtain more accurate analysis results. This aggregation is not only based on single statements but can also be extended to different levels such as basic blocks, functions, or files, thereby better mapping the structural characteristics and logical relationships of the program. In summary, the effectiveness of spectrum-based defect localization technology is influenced by three core factors: first, the choice of the suspicious value calculation formula, which directly determines how to deduce the suspiciousness of program elements from execution frequency; second, the aggregation method of suspicious values, which concerns how to comprehensively and accurately assess the problem probability of each program element; and finally, the quality of test cases, which constitute the data foundation for spectrum information collection and analysis. Optimizing these key factors can significantly improve the accuracy and efficiency of spectrum-based defect localization technology, thereby more effectively revealing and fixing potential problems in programs.

[0007] However, existing defect localization techniques are primarily designed for general software errors (such as errors in C / Java programs), and their implementation methods and mutation operators are not suitable for deep learning frameworks. Furthermore, the mutation operators in previous studies were not specifically designed for the unique characteristics of deep learning frameworks, such as computational graphs and specific deep learning operations. Summary of the Invention

[0008] To fill the gap in defect localization tools for deep learning frameworks, this invention proposes a defect localization method for program slices in deep learning frameworks. By designing mutation rules for deep learning frameworks, parsing and simulating original failed test cases that cause defects due to mutations, passing mutation test cases are generated. Mutation operators are selected, execution coverage information is collected, and finally, spectrum technology is used to accurately locate defects.

[0009] This invention is achieved using the following technical solution:

[0010] A method for defect localization in program slices for deep learning frameworks includes:

[0011] Input the original failed test cases that caused the defect, perform in-depth analysis on the original failed test cases to accurately identify the code structure, and extract key elements, i.e. target mutation points, from the identified code.

[0012] Design mutation rules for deep learning frameworks;

[0013] The mutation rules are applied to simulate mutations on the original failed test cases, thereby generating mutated test cases that are highly similar to the mutation execution paths of the original failed test cases and have had erroneous statements removed.

[0014] The Metropolis-Hastings algorithm is used to select mutation operators in each iteration based on the mutation test cases. Taking into account historical iteration data and the current test case status, the most suitable mutation operator is recursively selected to further mutate the target original failed test cases and gradually eliminate code elements containing errors.

[0015] Execute the original failure test cases and the mutation test cases, and collect coverage information during the execution of the original failure test cases and the mutation test cases respectively;

[0016] Based on the collected coverage information and the status of the mutation test cases, a spectrum-based defect localization algorithm is used to locate defects and output accurate defect localization results.

[0017] In some implementations, the mutation rules further include API-level mutation operators, control flow-level mutation operators, variable-level mutation operators, layer-level mutation operators, and device-level mutation operators. Specifically, API-level mutation operators modify parameters in API calls, control flow-level mutation operators change the logic of conditional statements, variable-level mutation operators replace values ​​in assignment statements, layer-level mutation operators modify the computation graph, and device-level mutation operators change device allocation instructions.

[0018] In some implementations, the step of using a spectrum-based defect localization algorithm to locate defects based on the collected coverage information and the status of the mutation test cases, and outputting accurate defect localization results, further includes:

[0019] Based on the collected coverage information and the state of the mutation test cases, the coverage information is analyzed to calculate the coverage of each code element in the deep learning framework. , , and Three granularities; calculate the suspicious value score for each code element, as shown in the following expression:

[0020] ;

[0021] in, This indicates the number of failed test cases that cover this statement. This indicates the number of test cases that passed and covered this statement. This indicates the number of failed test cases that did not cover the statement. This indicates the number of pass test cases that do not cover the statement; it sorts the suspiciousness scores of each code element in the deep learning framework, with the code element ranked higher being more suspicious.

[0022] In some implementations, the selection of mutation operators in each iteration using the Metropolis-Hastings algorithm based on the mutation test cases further includes: sampling the next mutation rule MRb according to the probability distribution of the current mutation rule MRa; if MRb performs better than MRa, then MRb will be accepted deterministically; otherwise, if MRb performs poorly, it will still be accepted with a probability; after obtaining the result of MRb, its score is updated to reorder the mutation rules for the next iteration.

[0023] In some implementations, the coverage information encompasses the code path during the execution of each test case, involving different levels of code elements including files, functions, and basic blocks; the coverage information is transformed into a spectrum matrix, where each row represents a specific code element, and each column corresponds to... , , and Four key indicators.

[0024] In some implementations, the application of the mutation rules to simulate mutations on the original failed test cases further includes: constructing a corresponding abstract syntax tree from the source code of the original failed test cases, and making targeted modifications to the abstract syntax tree.

[0025] In some implementations, the probability of accepting the mutation rule MRb given the mutation rule MRa is calculated as follows:

[0026] ;

[0027] in, and These represent the positions of mutation rules MRa and MRb in the list of mutation rules arranged in descending order of priority score, respectively.

[0028] Compared with the prior art, the beneficial technical effects and technical progress achieved by the present invention are as follows:

[0029] 1) By comparing and analyzing the differences in execution paths between the original failed test cases and the passed test cases, new test cases with execution paths highly similar to the original failed test cases but with erroneous statements removed can be generated, enabling in-depth exploration of the potential connections between code elements and defects;

[0030] 2) Taking into account both historical iteration data and the current state of test cases, select the most suitable mutation operator to gradually eliminate code elements containing errors;

[0031] 3) Further, the suspiciousness scores of three code element granularities—files, functions, and basic blocks—are calculated. The code elements with higher rankings are analyzed and judged as more suspicious, so as to output accurate defect location results. This helps developers quickly locate and fix defects in the code in the program slice, which can effectively improve the overall quality and reliability of the deep learning framework. Attached Figure Description

[0032] Figure 1 This is a flowchart of a method for defect localization of program slices for deep learning frameworks according to the present invention.

[0033] Figure 2 Example diagram of existing deep learning frameworks;

[0034] Figure 3 This is a block diagram illustrating a specific implementation of a defect localization method for deep learning frameworks according to the present invention.

[0035] Figure 4Example diagram of variations of the original failed test cases;

[0036] Figure 5 Example diagram of the general program mutation operator designed for this invention;

[0037] Figure 6 Example diagram of the feature mutation operator of the deep learning framework library designed for this invention;

[0038] Figure 7 Architecture diagram of the defect localization dataset;

[0039] Figure 8 A flowchart of a minimal manual patching workflow;

[0040] Figure 9 This is a graph showing the distribution of coverage similarity in PyTorch.

[0041] Figure 10 This is a graph showing the distribution of coverage similarity in TensorFlow. Detailed Implementation

[0042] The present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0043] like Figure 1 The diagram shows the overall flow of a defect localization method for program slices in a deep learning framework according to the present invention, and how it is combined with... Figure 3 The diagram shown illustrates a specific embodiment of the present invention. The overall process of this method includes the following steps:

[0044] Step 1: Input the original failed test cases that caused the defect, perform in-depth analysis on the original failed test cases to accurately identify their code structure, and extract key elements, i.e. target mutation points, from the identified code.

[0045] Step 2: Design mutation rules for deep learning frameworks. These mutation rules further include API-level mutation operators, control flow-level mutation operators, variable-level mutation operators, layer-level mutation operators, and device-level mutation operators. These level mutation operators are mainly applied to the Python-level code of failed test cases. Specifically, taking the PyTorch framework as an example, at the API level, API-level mutation operators modify parameters in API calls, such as changing the dim parameter in x.split(1, dim=-1) from -1 to 2. At the control flow level, control flow-level mutation operators change the logic of conditional statements, such as reversing an if statement to if not. At the variable level, variable-level mutation operators replace values ​​in assignment statements, such as replacing a=10 with a=0. At the layer level, layer-level mutation operators delete certain parts of the computation graph to modify the computation graph. At the device level, device allocation instructions are modified using device-level mutation operators, such as changing the device parameter in `tensor.to('cpu')` from 'cpu' to 'cuda:0'. Specifically, for example, PyTorch or TensorFlow frameworks are chosen as deep learning frameworks to implement the above mutation functionality; Figure 3 The image shown is an example of a deep learning framework.

[0046] Step 3: Apply the mutation rules described in Step 2 to simulate mutations on the original failed test cases. In this process, firstly, starting from the source code of the original failed test cases, construct the corresponding abstract syntax tree (AST). Then, make subtle but targeted modifications to this AST, such as replacing "if" in conditional statements with "if not," changing the data structure from "list" to "tuple," or even setting some input parameters to "None." After these modifications, re-export the modified AST as source code, thereby generating mutated test cases that are highly similar to the original failed test cases in their mutation execution path and have had erroneous statements removed. These mutated test cases can successfully pass the test execution. Figure 4 The image shown is an example diagram of variations of the original failed test cases;

[0047] Step 4: For the mutated test cases obtained through the mutation rules in Step 3, the Metropolis-Hastings algorithm is used to select the mutation operator in each iteration. This algorithm will comprehensively consider the historical iteration data and the current test case status, and recursively select the most suitable mutation operator to further perform mutation operations on the target original failed test cases and gradually eliminate code elements that may contain errors.

[0048] Step 5: Execute the original failure test cases and the mutation test cases, and collect the coverage information in detail during the execution process. The coverage information includes key data at the granularity of executed code lines, code blocks, functions and files.

[0049] Step 6: Based on the coverage information collected in Step 5 and the pass / fail status of the mutation test cases, use a spectrum-based defect localization algorithm to locate the defect and output accurate defect localization results. The specific process of using the spectrum-based defect localization algorithm for defect localization is as follows: Based on the collected coverage information and the pass / fail status of the mutation test cases, by analyzing the coverage information, calculate the frequency response of each code element in the deep learning framework. , , and It includes three granularities: files, functions, and basic blocks. A basic block specifically refers to a sequence of continuously executed statements in a program, existing within the internal structure of a function, and typically contains multiple such basic blocks. The Ochiai formula for calculating code suspicion values ​​is used, combined with the information of each code element... , , and This allows us to calculate the suspicious value score for each code element, as shown in the following expression:

[0050]

[0051] in, This indicates the number of failed test cases that cover this statement. This indicates the number of test cases that passed and covered this statement. This indicates the number of failed test cases that did not cover the statement. This indicates the number of test cases that passed but did not cover the statement;

[0052] The suspiciousness score of each code element in the deep learning framework is sorted, and the code element with the higher the ranking is considered more suspicious. It is hoped that the code element where the real defect is located can be ranked as high as possible.

[0053] In spectrum-based technologies, the calculation of suspicious values ​​for each code statement varies, as shown in Table 1, which presents formulas for calculating suspicious values ​​in existing technologies for defect localization. These formulas include those from Tarantula, Ochiai, and DStar.

[0054] Table 1

[0055]

[0056] Furthermore, the specific steps involved in the process of this invention are described below:

[0057] This invention first designs a set of mutation rules specifically for deep learning frameworks. These rules are meticulously divided into API level, control flow level, variable level, layer level, and device level to ensure a comprehensive mutation strategy. Next, the original failed test cases that cause defects are deeply analyzed to accurately identify their code structure and extract key elements, i.e., target mutation points. Subsequently, this invention applies the aforementioned mutation rules to mutate the original failed test cases, aiming to generate new test cases that are highly similar to the execution path of the original failed test cases but have eliminated erroneous statements. These newly generated test cases can successfully pass test execution.

[0058] The core concept of this invention lies in generating passable test cases by performing refined mutation operations on the original failed test cases, thereby locating defects in the deep learning framework. These passable test cases maintain a high degree of consistency with the failed test cases in their execution paths; the only difference is the exclusion of statements containing errors. In this way, the spectrum-based defect localization technique can effectively eliminate innocent statements, achieving precise identification of erroneous statements.

[0059] Therefore, this invention, considering the specific characteristics of deep learning frameworks, designs a series of specialized mutation operators and implements a mutation-based test generation framework. This framework analyzes the original failed test cases that lead to defects and constructs a corresponding abstract syntax tree based on the code. Then, by traversing all APIs in the program, it obtains their locations and parameter information, identifies which locations and parameters can serve as potential mutation points, and assigns a suitable mutation operator to each mutation point. Specifically, if an API related to the deep learning framework is detected, an API-level mutation operator is assigned; if a control flow element such as `if` is detected, a control flow-level mutation operator is assigned; for code elements of declaration or assignment type (such as `a = True`), a variable-level mutation operator is assigned; if a computation graph construction function such as `forward` is detected, a layer-level mutation operator is assigned; and if a device parameter such as 'cpu' is detected, a device-level mutation operator is assigned. Specifically, this framework designs five levels of mutation operators for deep learning frameworks: API level, control flow level, variable level, layer level, and device level. Step 2 includes examples of input and output code for each mutation operator. The rules described above are adapted to the two specific deep learning frameworks, PyTorch and Tensorflow. The API-level, control flow-level, and variable-level operators are developed based on existing research, while the layer-level and device-level operators are innovatively designed according to the unique characteristics of deep learning programs. Through the comprehensive application of these operators, the framework of this invention can comprehensively and accurately pinpoint the deficiencies in deep learning frameworks, providing strong support and assistance to developers.

[0060] Deep learning frameworks typically consist of numerous API calls and branching structures. This invention's framework uses control flow-level mutation operators to determine the executable parts of the code. Within each executable part, API-level mutation operators are used to modify specific API calls, while variable-level mutation operators are used to modify variable attributes. Furthermore, to accommodate the specific characteristics of deep learning programs, this invention also employs layer-level mutation operators to modify the computation graph structure and introduces device-level mutation operators to change the device attributes of the deep learning framework program. Through these hierarchical mutation operations, this invention can achieve comprehensive localization and precise repair of errors in deep learning frameworks.

[0061] like Figure 5 and Figure 6 As shown, the detailed definitions of mutation operators at each level are as follows. These mutation operators can currently be adapted to mainstream PyTorch and Tensorflow frameworks, thus ensuring their wide applicability and effectiveness in practical applications:

[0062] ① API-level mutation operator: Deep learning frameworks provide various APIs. When a test case triggers an error in an API, modifying the API's parameters may turn a failing test case into a passing one. This invention can modify API parameters in three ways: 1) changing the API parameter setting to its default value; 2) modifying the API parameter's value to another valid value within its value range; 3) assigning a valid value to an unused API parameter. Furthermore, this invention can also replace the API with another API having a similar function signature.

[0063] ② Control Flow Level Mutation Operator: By modifying the control flow of a failed test case, this invention can effectively change its execution path and potentially transform it into a passing test case. This invention employs the following two mutation strategies: 1) Directly commenting out the code entry point that directly triggers the error; 2) Modifying the branches in the test case. For example, this invention can modify a branch in the test case to `if True` to execute a previously unexecuted branch.

[0064] ③ Computation layer-level mutation operators: Deep learning frameworks typically contain code for handling various types of computation graphs. This invention can modify test cases to change the computation graph in the following three ways: 1) Modify the forward propagation process; 2) Modify the gradient-related configuration in the computation graph to retain or discard gradients; 3) Modify the type of the computation graph (i.e., static or dynamic type).

[0065] ④ Variable-level mutation operators: Mutation operators at this level are designed to modify the properties of the original variable (non-tensor variable) in the following ways: 1) Change the variable type (e.g., change list to tuple, change integer to float); 2) Change the data of the original variable (e.g., change 2 to 0, change True to False); 3) Change the index of the variable (e.g., change x[:2] to x).

[0066] ⑤ Device-level mutation operator: Given that deep learning frameworks generally support two different types of computing devices, CPU and CUDA, and that there are differences in the underlying implementation mechanisms on these two devices, this invention is expected to transform originally failed test cases into successful test cases by adjusting the device environment in which the program runs.

[0067] In step 4, the mutated test cases obtained through the mutation rules in step 3 are used, and the Metropolis-Hastings algorithm is employed to guide the selection of mutation operators in each iteration. This algorithm comprehensively considers historical iteration data and the specific state of the current test cases, recursively selecting the most suitable mutation operator. This selection process aims to further mutate the original failed test cases, thereby gradually eliminating potential erroneous code elements. Subsequently, by executing the original failed test cases and the successfully generated test cases after mutation, detailed coverage information during execution is collected. This information includes key data such as executed lines of code, code blocks, functions, and files, providing crucial information for subsequent defect localization and repair.

[0068] Given the vast search space and computational resource limitations, this invention cannot exhaustively generate all possible passable test cases and then select a valid subset. Therefore, this invention proposes a more cost-effective method: in each generation process, it focuses on creating a passable test case that maximizes the difference from the existing program. For a given initial failed test case, this invention observes that different mutation rules have varying effects. Mutation rules that can generate diverse passable test cases with high frequency should be given a higher selection probability for subsequent mutation operations. Based on this insight, this invention will elaborate on a heuristic test case generation method in the following sections. In particular, this invention introduces coverage distance as a metric for measuring the diversity among test programs. For a given test program... and The Jaccard distance between their code coverage areas is defined as follows, which is used to quantify the difference between the two:

[0069] ;

[0070] in and They represent and The set of lines of code covered.

[0071] After selecting a seed program, this invention selects mutation rules for mutation. However, these mutation rules are not all equally effective in generating various types of pass test cases for a given set of failed test cases. Furthermore, the same mutation rule may perform differently for different initial failed test cases. Therefore, this invention designs an adaptive process to select mutation rules to construct an effective set of pass test cases. If a mutation rule can generate pass test cases with greater diversity than existing test cases more frequently, then that rule should be selected with a higher probability for subsequent mutations.

[0072] Mutation rules can be sorted in descending order based on their calculated priority scores. However, since the ranking is based solely on the historical performance of these mutation rules, it is difficult to accurately predict their future effects. Therefore, this invention does not directly select the top-ranked mutation rule for subsequent mutations. In view of this, this invention employs a probabilistic selection mechanism to ensure that each mutation rule has a chance to be selected for the next mutation, while guaranteeing that mutation rules with higher rankings have a relatively higher probability of being selected. In other words, the problem of selecting mutation rules can be transformed into a problem of sampling from a specific probability distribution. Based on this logic, this invention calculates a priority score for each mutation rule MR, and after each iteration, updates the scores of each mutation operator r according to predetermined rules:

[0073] ;

[0074] in, This indicates the number of passed test cases generated up to the current iteration. Indicates the current iteration period The newly generated test cases passed. yes The success rate of passing test cases in the generated test cases.

[0075] Given that the selection of mutation rules depends on their recent performance, this problem can be considered a typical application scenario of Markov Chains (MC). To address the problem of sampling from a probability distribution, the Markov Chain Monte Carlo (Megapolis-Hastings) algorithm is employed, assuming the required distribution is in equilibrium. Within this framework, the MH algorithm is used to obtain random samples from a specific probability distribution. In this invention, the algorithm samples the next mutation rule (denoted as MRb) based on the probability distribution of the current mutation rule (MRa). If MRb performs better than MRa, i.e., MRb has a higher priority score than MRa, then MRb will be accepted deterministically; conversely, if MRb performs poorly, it still has a certain probability of being accepted. This invention selects the geometric distribution as the probability distribution model. The geometric distribution refers to the probability distribution of the number of trials required to achieve success in a series of Bernoulli trials. If the success probability of each trial is p, then the probability of achieving success on the k-th trial can be calculated as:

[0076] p;

[0077] In this process, the mutation rule is randomly selected. Given the mutation rule MRa, the probability of accepting the mutation rule MRb is calculated as follows:

[0078] ;

[0079] in, and These represent the positions of mutation rules MRa and MRb in the list of mutation rules arranged in descending order of priority score. This mechanism ensures flexibility and diversity in the selection of mutation rules. After obtaining the result of MRb (i.e., acceptance or rejection), this invention updates its score and reorders these mutation rules for the next iteration.

[0080] After generating diverse pass / fail test cases, this invention further utilizes the specialized tool Coverage.py to execute both fail and pass test cases. This aims to collect comprehensive and systematic coverage information of these test cases within the deep learning framework's source code. By combining Coverage.py and Scalpel, this invention can accurately obtain detailed, multi-dimensional coverage data, including lines of code, code blocks, functions, and files.

[0081] In step 6, based on the detailed execution coverage information and test case pass / fail status obtained in step 5, this invention adopts a spectrum-based defect localization technique. With the help of a carefully designed defect localization formula, this invention can deeply compare and analyze the differences and commonalities in the execution paths of the original failed test cases and passed test cases, thereby exploring the deep correlation between code elements and defects. Furthermore, by accurately calculating the suspicious scores of three types of code elements—files, functions, and basic blocks—this invention can output highly accurate defect localization results. This aims to provide developers with powerful support for quickly locating and fixing defects in code, thereby significantly improving the overall performance and stability of deep learning frameworks.

[0082] Coverage information encompasses the code paths reached during the execution of each test case, involving code elements at different levels, such as files, functions, and basic blocks. This coverage data provides a solid foundation for spectrum analysis. Specifically, this invention transforms the collected coverage information into a spectrum matrix and, combined with the pass / fail status of mutation test cases, meticulously calculates the number of times each code element was executed by passed test cases, failed test cases, and not executed. In the constructed spectrum matrix, each row represents a specific code element, while the four columns correspond to the elements mentioned in the background art. , , and Four key indicators.

[0083] After constructing the spectrum matrix, this invention draws on traditional software defect localization practices and selects the Ochiai formula as the basis for calculating code suspiciousness values ​​to accurately locate defective code within deep learning frameworks. The Ochiai formula is a classic algorithm in defect localization; it determines the suspiciousness of a code element by quantifying the correlation between the frequency of test cases covering a specific code element and the test case results. More specifically, the Ochiai formula reveals potential connections between certain code elements and latent defects by comparing and analyzing the differences in execution paths between failed and passed test cases. The suspiciousness value calculated using the Ochiai formula... The specific calculation method is as follows:

[0084]

[0085] Furthermore, this invention also considers the calculation of suspicious values ​​at different program element granularities (including code blocks, functions, and files). The suspicious values ​​of these elements are derived by aggregating the suspicious values ​​of their internal statements. To achieve this, this invention employs an average aggregation algorithm, which can calculate the suspicious score for code elements of different granularities (such as files, functions, and basic blocks). Specifically, the suspicious value of a function can be expressed as the average of the suspicious values ​​of all its internal statements, calculated using the following formula:

[0086]

[0087] After calculating the suspicious score for each code element, this invention further arranges these elements in reverse order of their suspicious scores. This arrangement prioritizes displaying code elements with higher suspicious scores, thereby helping developers more efficiently identify potential defect locations. Using this method, developers can quickly locate the code area most likely to contain defects, significantly reducing the scope and time cost of manual investigation.

[0088] To verify the effectiveness of this invention, the experimental results were scientifically analyzed.

[0089] This invention uses test cases provided by developers in the source code of deep learning frameworks as a baseline method. When locating defects in deep learning frameworks, directly collecting test cases from the framework's source code is an efficient and direct approach. This is because developers provide a large number of test cases during the framework development process to ensure the correctness and stability of the code. These test suites cover various API calls of the deep learning framework, covering more potential defect scenarios. Therefore, this invention implements a defect location method based on test suites in the deep learning framework source code, which this invention calls the Developer. For each defect, this invention identifies and extracts relevant test cases from the corresponding deep learning framework source code and executes the collected and passed test cases in the corresponding runtime environment.

[0090] This invention targets the PyTorch framework, identifying and extracting all matching Python files using the regular expression pattern `test_*.py`. To ensure that the collected test cases are limited to PyTorch code itself, files located in the "third_party" directory are excluded, as these are third-party dependencies rather than PyTorch core code. For the TensorFlow framework, a similar strategy is employed, but the matching rules differ slightly. This invention collects all Python files whose names end with `*_test.py`, which typically contain test cases for TensorFlow functionalities.

[0091] However, it should be noted that directly executing the entire test file can be very time-consuming. Therefore, to improve efficiency and specificity, this invention does not consider the entire file as a test case, but rather treats each test function in the file as an independent test case. To automate the execution of these test cases and collect runtime data, this invention employs the Pytest framework. Pytest is a powerful Python testing framework that allows this invention to easily execute test functions and collect spectral data at runtime.

[0092] Currently, there is no specific defect localization technology for deep learning frameworks, resulting in a severe shortage of realistic and reliable defect localization datasets. To fill this gap, such as... Figure 7 As shown, this invention meticulously collected 80 real-world crashes from GitHub and evenly distributed them across the two major deep learning frameworks, PyTorch and TensorFlow, for experimental analysis. Specifically, these crash data cover 12 different versions of PyTorch and 11 versions of TensorFlow, thus fully demonstrating the diversity and complexity of deep learning frameworks across different versions.

[0093] To ensure the defect localization dataset includes defects from different versions of deep learning frameworks, this invention crawled all merged and closed pull requests (PRs) from the official PyTorch and TensorFlow GitHub repositories between January 1, 2019, and December 31, 2023. In this phase, this invention obtained 1,802 PRs from PyTorch and 9,868 PRs from TensorFlow. This invention randomly extracted 200 PRs from TensorFlow and PyTorch to identify frequently occurring keywords in defect-fixing PR information. This invention obtained four keywords related to defect fixing: fix, bug, error, and invalid. Then, this invention used these keywords to filter PRs whose titles and page descriptions did not contain these keywords, ultimately obtaining 965 PRs from PyTorch and 4,308 PRs from TensorFlow. Since manually configuring the runtime environment is both time-consuming and laborious, it is almost impossible to include all defects from all PRs in the defect localization dataset. Therefore, this invention randomly selected 40 reproducible defects from each frame from all PRs and constructed a defect localization dataset.

[0094] For each extracted PR, this invention first checks whether the PR contains a defect that causes a crash or malfunction, and whether it provides a script that triggers the defect (i.e., a failing test case). Then, this invention extracts the merged commits from the PR page as fix commits. The commit containing the defect is typically the commit closest to the fix commit (i.e., the parent node of the same branch) that introduced the defect. This invention determines whether a defect should be included in the defect localization dataset based on the following rules:

[0095] ① Defect Reproduction Procedures: Defect reproduction procedures are a key component of dynamic defect localization technology. These procedures are responsible for executing test cases to trigger potential defects in the deep learning framework and collecting corresponding spectral data for subsequent analysis. The lack of defect reproduction procedures leads to an incomplete testing process, thus preventing the generation of spectral data for effective dynamic defect localization. Therefore, this invention filters out defects lacking defect reproduction procedures when constructing the dataset to ensure that the dataset contains defect samples that provide sufficient information to support dynamic defect localization technology.

[0096] ② Compilation Status: For each defect in the defect localization dataset, this invention requires Linux-level source code compilation based on the defect submission (i.e., the submission containing the defect), thereby constructing the corresponding defect reproduction environment. Failed compilation and installation processes will prevent subsequent testing and analysis, making it impossible to accurately assess the nature and impact of the defect. If a defect submission cannot be successfully compiled and installed, this invention will exclude that defect from the defect localization dataset to ensure that all defects included in the dataset can be reproduced.

[0097] ③ Patch Correctness: Patch correctness is one of the key indicators for evaluating defect localization effectiveness. If the defect triggering test still fails after applying the repair patch, this invention considers the defect not to have been successfully repaired and excludes it from the defect localization dataset. This is because even if this invention applies a seemingly correct repair patch to the defect, if the same defect is still triggered in subsequent tests, it indicates that the repair patch has not successfully solved the corresponding problem. This could be because the repair patch does not fully cover the root cause of the problem, or it could be that the repair patch introduces new problems, causing the original defect to persist.

[0098] ④ Reproducibility: Reproducibility is a prerequisite for evaluating the effectiveness of defect localization techniques. If a defect cannot be reliably reproduced, i.e., the same defect behavior cannot be triggered again under different environments or conditions, this invention excludes it from the defect localization dataset. Defects lacking reproducibility may be due to differences in factors such as testing environment, configuration, or dependencies. If this invention cannot reproduce the same defect in different environments, then it cannot be accurately analyzed and evaluated, thus failing to ensure that the research and improvement work of this invention is based on a reliable data foundation.

[0099] Because developers may fix multiple defects or introduce unrelated changes (e.g., refactoring) in the same commit, labeling all code changes as root causes may not accurately assess the effectiveness of defect localization techniques. To address this issue, this invention performs patch minimization in a semi-automatic manner to precisely identify the defect location for each defect.

[0100] Figure 7The process of root cause identification is demonstrated. This invention first removes all documentation-related changes and test case changes from the patch, as they are irrelevant to fixing the defect. Then, this invention executes the failed test cases and collects the corresponding code spectrum information. If any subset of the patch is not involved by the failed test cases, this invention excludes them from the patch, as they are also irrelevant to the defect. Finally, this invention manually examines the remaining code changes to determine which parts of the code are relevant to fixing the defect. These code changes are then marked as the root cause of the defect. In some cases, developers may only insert new elements into the patch, rather than modifying or deleting old elements. To determine the root cause when a developer inserts an element into the patch, this invention follows previous work and marks the element immediately following the inserted element as the root cause. After the patch is minimized, this invention manually marks the code block, function, and file containing the patch as defective code block, defective function, and defective file, respectively.

[0101] This invention employs the worst-case ranking strategy to generate the ranking list, which has been used in previous studies. Specifically, when two elements have the same question value and are ranked at position k, this invention assigns their ranking to position k+1. This invention uses a widely used evaluation method, as follows:

[0102] ①Top-N: Top-N refers to the number of defects successfully located in the first N positions. The larger the value of Top-N, the better the performance of the defect location technology.

[0103] ②Mean First Rank (MFR): First rank refers to the ranking of the first defect element of each defect. MFR represents the average ranking of the first defect elements of all defects. The smaller the MFR value, the better the performance.

[0104] ③Mean Average Rank (MAR): Average rank refers to the average rank of all defect elements for each defect. MAR represents the average of the average ranks of all defects. The smaller the MAR value, the better the performance.

[0105] This invention project is implemented using Python 3.11 as the primary programming language. The experimental environment is configured on a high-performance server equipped with an Intel Xeon Gold 6326 CPU, an RTX 3090 GPU, 512GB of RAM, and running Ubuntu 18.04.6 LTS. To ensure accurate defect localization, this invention sets a 60-minute termination condition for each defect localization process. Furthermore, to reduce the impact of randomness during test generation on the experimental results, this invention adopts a strategy of repeated experiments, i.e., each experiment is repeated five times, and the average result is reported, thereby ensuring the stability of the experimental data and the reliability of the results.

[0106] Tables 2 and 3 provide a detailed comparison of the defect localization method for deep learning frameworks proposed in this invention, termed TenMutFL, and its application performance against the existing baseline method, Developer, within deep learning frameworks. Specifically, Table 2 records the number of successfully located defects in the Top-N positions on the two major deep learning frameworks, PyTorch and TensorFlow, when using a specific test generation method. This data provides a quantitative basis for evaluating the effectiveness of the localization method. Furthermore, Table 3 presents the average ranking of successfully located defect elements at different defect granularities in PyTorch and TensorFlow frameworks using different test generation methods. This statistic helps to more comprehensively evaluate the accuracy and stability of the localization method. By combining the data from Tables 2 and 3, this invention allows for a comprehensive performance comparison between the TenMutFL and Developer methods.

[0107] By thoroughly analyzing the data in Tables 2 and 3, this invention draws a clear conclusion: the TenMutFL method demonstrates significant effectiveness in locating defects in deep learning frameworks. In a comprehensive evaluation covering two deep learning frameworks (PyTorch and Tensorflow), three different granularities, and four top-N positions, the TenMutFL method outperforms the comparative method Developer. Specifically, in the PyTorch framework, the TenMutFL method can accurately locate 62.5% and 80% of defects in the first 10 functions and first 10 files, respectively; while in the Tensorflow framework, the method achieves defect location accuracy of 57.5% and 72.5% in the first 10 functions and first 10 files, respectively. It is worth mentioning that even with finer granularity and more consideration of top positions, the TenMutFL method still maintains high defect location performance, which fully verifies its reliability and superiority in practical applications.

[0108] This paper delves into the advantages of the TenMutFL method compared to the Developer method, conducting a detailed analysis of manual test cases. Through comparative analysis, this invention reveals the dual advantages of the TenMutFL method in generating test cases:

[0109] 1) The execution spectrum of the test cases generated by this method on elements without error codes is highly similar to that of the failed test cases. This feature effectively reduces noise interference in the ranking process of error elements and improves the accuracy of the ranking.

[0110] 2) Regarding error code elements, the test cases generated by the TenMutFL method show a significant difference in execution spectrum compared to the failed test cases. This difference provides strong support for accurately locating defects.

[0111] In summary, the TenMutFL method achieves more efficient defect localization by optimizing the similarity and difference of the execution spectrum.

[0112] Table 2 shows the Top-N results for the Developer method and the TenMutFL method of this invention. Table 3 shows the MFR / MAR results for the Developer method and the TenMutFL method of this invention.

[0113] Table 2

[0114]

[0115] Table 3

[0116]

[0117] Table 3

[0118]

[0119] To explore the effectiveness of the TenMutFL method in defect localization within deep learning frameworks, this study compares the code coverage of test cases generated by the TenMutFL method with that generated by the Developer method. It also considers the code coverage of original failed test cases across all crash defects. Specifically, this invention calculates the number of code blocks jointly covered by failed and passed test cases and then uses this number as a ratio to the total number of code blocks covered by the failed test cases.

[0120] Figure 9 and Figure 10 The data provides a detailed overview of the distribution of coverage similarity between the two test generation techniques. Box plots clearly present the median, quartiles, and outliers, while violin plots further reveal the distribution density of the data across different value ranges. Figure 9 This shows the distribution of coverage similarity in PyTorch. Figure 10 This refers to the distribution of coverage similarity in TensorFlow, through... Figure 9 and Figure 10 Through meticulous observation, this invention reveals that, compared to the Developer method, the mutation-based test generation method TenMutFL excels in constructing pass test cases with higher coverage similarity to failed test cases. This advantage provides SBFL technology with a more accurate ability to distinguish between error codes and error-free codes. Therefore, it can be clearly concluded that the test cases generated by the TenMutFL method can significantly improve the effectiveness of SBFL technology in defect element identification.

[0121] It should be noted that although the present invention has been shown and described with reference to specific exemplary embodiments thereof, those skilled in the art should understand that the present invention is not limited to the above embodiments, and all modifications to the present invention fall within the scope of protection of the present invention.

Claims

1. A method for defect localization in program slices for deep learning frameworks, characterized in that, include: Input the original failed test cases that caused the defect, perform in-depth analysis on the original failed test cases to accurately identify the code structure, and extract the target mutation points from the identified code. Design mutation rules for deep learning frameworks; The mutation rules are applied to simulate mutations on the original failed test cases, thereby generating mutated test cases that are highly similar to the mutation execution paths of the original failed test cases and have had erroneous statements removed. Based on the mutation test cases, the mutation operator is selected in each iteration process. Taking into account the historical iteration data and the current test case status, the most suitable mutation operator is recursively selected to further perform mutation operations on the target original failed test cases and gradually eliminate code elements containing errors. Execute the original failure test cases and the mutation test cases, and collect coverage information during the execution of the original failure test cases and the mutation test cases respectively; Based on the collected coverage information and the status of the mutation test cases, a spectrum-based defect localization algorithm is used to locate defects and output accurate defect localization results.

2. The defect localization method for program slices oriented towards deep learning frameworks according to claim 1, characterized in that, The mutation rules further include API-level mutation operators, control flow-level mutation operators, variable-level mutation operators, layer-level mutation operators, and device-level mutation operators. Specifically, API-level mutation operators modify parameters in API calls, control flow-level mutation operators change the logic of conditional statements, variable-level mutation operators replace values ​​in assignment statements, layer-level mutation operators modify the computation graph, and device-level mutation operators change device allocation instructions.

3. The defect localization method for program slices oriented towards deep learning frameworks according to claim 1, characterized in that, The step of using a spectrum-based defect localization algorithm to locate defects based on the collected coverage information and the status of the mutation test cases, and outputting accurate defect localization results, further includes: Based on the collected coverage information and the state of the mutation test cases, the coverage information is analyzed to calculate the coverage of each code element in the deep learning framework. , , and It includes three granularities: file, function, and basic block; it calculates the suspicious value score for each code element, using the following expression: ； in, This indicates the number of failed test cases that cover this statement. This indicates the number of test cases that passed and covered this statement. This indicates the number of failed test cases that did not cover the statement. This indicates the number of pass test cases that do not cover the statement; it sorts the suspiciousness scores of each code element in the deep learning framework, with the code element ranked higher being more suspicious.

4. The defect localization method for program slices oriented towards deep learning frameworks according to claim 1, characterized in that, The selection of mutation operators in each iteration based on the mutation test cases is implemented using the Metropolis-Hastings algorithm. The implementation process further includes: sampling the next mutation rule MRb according to the probability distribution of the current mutation rule MRa; if MRb performs better than MRa, MRb will be accepted deterministically; otherwise, if MRb performs poorly, it still has a probability of being accepted; after obtaining the result of MRb, its score is updated, and the mutation rules are reordered for the next iteration.

5. The defect localization method for program slices oriented towards deep learning frameworks according to claim 1, characterized in that, The coverage information encompasses the code path during the execution of each test case, involving different levels of code elements including files, functions, and basic blocks. The coverage information is transformed into a spectrum matrix, where each row represents a specific code element, and each column corresponds to... , , and Four key indicators.

6. The defect localization method for program slices oriented towards deep learning frameworks according to claim 1, characterized in that, Further, it includes evaluating the defect location based on the defect reproduction procedure, compilation status, patch correctness, and reproducibility, and including defects that simultaneously meet these criteria into the defect location dataset.

7. The defect localization method for program slices oriented towards deep learning frameworks according to claim 1, characterized in that, Given the mutation rule MRa, the probability of accepting the mutation rule MRb is calculated as follows: ； in, and These represent the positions of mutation rules MRa and MRb in the list of mutation rules arranged in descending order of priority score, respectively.

8. The defect localization method for program slices oriented towards deep learning frameworks according to claim 6, characterized in that, Further steps included removing all documentation-related changes and test case modifications from the patch, then executing the original failing test cases, and collecting the corresponding code spectrum information. Exclude any subset of the patch that was not covered by the original failing test cases. Finally, examine the remaining code changes, identify the parts of the code that are relevant to fixing the defect, and mark these changes as the cause of the defect.

9. A method for defect localization of program slices for deep learning frameworks according to claim 6, characterized in that, When a new element is inserted into a patch, the element immediately following the inserted element is marked as the root cause. After the patch is minimized, the code block, function, and file containing the patch are marked as defective code block, defective function, and defective file, respectively.

Images