A code coverage analysis system and method for a deep learning framework

Abstract

This invention discloses a code coverage analysis system and method for deep learning frameworks. Taking into full account the characteristics of deep learning frameworks, this invention designs a lightweight code coverage analysis system and method for deep learning frameworks. The main idea is to instrument the deep learning framework using an instrumentation module before testing, and to obtain coverage using different instrumentation methods for multiple programming languages ​​within the deep learning framework. Then, the coverage information is stored in a shared code coverage analysis library using block coverage granularity. Algorithms and engineering improvements are made to the shared algorithm library, enabling the coverage analysis process to be completed with extremely low time overhead when the code coverage analysis system needs to access the coverage information.

Description

Technical Field

[0001] This invention belongs to the field of software testing technology, specifically relating to a code coverage analysis system and method for deep learning frameworks. Background Technology

[0002] As artificial intelligence (AI) systems mature, technologies such as facial recognition, autonomous driving, and the Internet of Things (IoT) are emerging in everyday life. With the continued rapid development of AI systems, deep learning frameworks, as the platforms for building and supporting them, have received increasing attention. However, vulnerabilities in popular international deep learning frameworks like TensorFlow and PyTorch, and domestic frameworks like MindSpore and PaddlePaddle, have seriously threatened the application and development of AI systems. For example, in 2016, due to incorrect assumptions about road conditions made by a deep learning system, Google's self-driving car collided with the side of a bus at low speed, causing a traffic accident. Furthermore, PyTorch versions below 2.3.24 used an insecure YAML file loading method, leading to model resource interruptions. These vulnerabilities in deep learning frameworks seriously threaten the application and development of AI systems. Therefore, testing deep learning frameworks is a crucial technology for ensuring their security.

[0003] In the early stages of testing deep learning frameworks, differential testing and fuzz testing were the most commonly used methods. Differential testing typically selects two or more functionally identical test objects, and simultaneously injects the same test cases that meet the requirements of each test object into the test objects and executes them to observe whether there are differences in the execution results. Fuzz testing, on the other hand, is a method for discovering software vulnerabilities by providing unexpected input to the target system and capturing anomalous results. Due to its high degree of automation, this technique has become the most effective method in the field of vulnerability discovery and is widely used by software testers.

[0004] Compared to traditional software, deep learning frameworks are characterized by complex structures and cumbersome execution processes. Testing deep learning frameworks requires inputting multi-layered, stacked deep learning models composed of numerous neurons and connection weights. These models are trained on vast amounts of data. However, due to high training costs and limited training data, building a large number of models that can trigger framework vulnerabilities is extremely difficult.

[0005] Recently, coverage, as an important metric in traditional software testing, has gradually gained attention in deep learning framework testing. Coverage techniques are a method used in software testing to quantify test completeness; it measures the ratio of executed code to the total code during testing. Specifically, coverage includes various types such as line coverage, branch coverage, condition coverage, and path coverage, focusing on whether statements in the source code are executed, whether branches are traversed, whether the results of conditional expressions are tested, and whether all possible code paths are executed, respectively. These techniques help evaluate the depth and breadth of testing, ensuring software quality and stability.

[0006] Currently, many top academic conferences widely utilize coverage techniques in research on deep learning framework testing. However, these studies typically use coverage as the sole criterion for evaluating test performance, lacking in the application and analysis of coverage information. The current field of coverage analysis technology faces two significant challenges. First, existing code coverage analyzers suffer from significant incompatibility with deep learning frameworks. This deficiency limits their ability to evaluate deep learning framework code, preventing them from working seamlessly with the frameworks. Second, these analyzers often introduce substantial runtime overhead, impacting developer efficiency and increasing project time costs. Therefore, resolving compatibility issues with deep learning frameworks and reducing runtime overhead have become crucial directions for the development of current coverage analysis techniques.

[0007] 1) Compatibility Issues: Popular deep learning frameworks today are primarily composed of Python and C programming languages. For Python, existing coverage analysis tools include Coverage.py and SlipCover, which support testing deep learning frameworks. However, because these tools are command-line based, they require adaptation for different frameworks to function correctly. For C, existing coverage analysis tools mainly include the AFL Fuzzer series and Gcov. afl-gcc and afl-clang are two crucial components of the AFL toolset, providing customized versions of the GCC and Clang compilers that AFL supports, allowing developers to generate programs instrumented by AFL. Gcov is a code coverage testing tool tightly integrated with GCC. It's a command-line console program distributed with GCC, used to analyze statement and branch coverage in C / C++ code. Gcov can count the execution frequency of each line of code, helping developers understand which code was actually executed and the execution time of each section. This information is crucial for optimizing code, improving program performance, and ensuring code quality. Instrumentation refers to adding extra monitoring code to the source code or binary code of a program. This code helps AFL track which code blocks are executed, thus making the fuzzing process more efficient. However, these tools are primarily designed for traditional software with single-language and simple structures, which can lead to compilation failures when testing deep learning frameworks. In summary, existing coverage analyzers cannot be directly applied to deep learning framework testing, thus necessitating an urgent solution to the poor compatibility problem of coverage analyzers.

[0008] 2) Time Cost Issue: To verify the time cost issue, we set up an experiment. Based on the official documentation of deep learning frameworks, we used FreeFuzz to generate 100 Python scripts as test samples for each deep learning framework. Then, we executed these test samples 100 times on each deep learning framework using different coverage analyzers. After each execution, we immediately performed coverage analysis using the selected coverage analyzer. In the experiment, we chose Coverage.py, SlipCover, and Gcov for testing on TensorFlow, PyTorch, and PaddlePaddle frameworks, respectively. The experimental results are shown in Table 1. The data shows that using existing coverage analyzers for Python code introduces an average time cost of 2-180 times compared to not using a coverage analyzer. For C language, the Gcov tool is only compatible with TensorFlow and introduces a significant time cost. The significant time cost introduced by existing code coverage analyzers in deep learning framework testing has become a common problem with existing tools.

[0009] Table 1: Average Cost of Code Coverage Analysis (Unit: milliseconds)

[0010] Summary of the Invention

[0011] To address the aforementioned shortcomings in existing technologies, the code coverage analysis system and method for deep learning frameworks provided by this invention analyzes the characteristics of deep learning frameworks and uses an interface-based instrumentation method to design targeted instrumentation methods according to different programming languages, thus approximating the compatibility issues of existing code coverage analyzers for deep learning frameworks. Simultaneously, by modeling the code coverage analysis process for deep learning frameworks and making algorithmic and engineering improvements to address performance bottlenecks, the problem of excessive time overhead is solved.

[0012] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: a code coverage analysis system for deep learning frameworks, comprising:

[0013] The instrumentation module is used to instrument deep learning frameworks, and uses the corresponding instrumentation methods for the programming language of the deep learning framework to obtain coverage information.

[0014] The code coverage analysis module is used to obtain coverage information based on instrumentation, call Bitmap statistics and recording information according to user needs, and then call an improved shared algorithm library based on Bitmap statistics. Combined with coverage granularity optimized based on Bitmap recording information, it completes the recording, statistics and analysis of coverage information.

[0015] Furthermore, the insertion mode of the insertion module is an interface-type insertion module;

[0016] The instrumentation methods of the instrumentation module include runtime bytecode instrumentation for Python code and compile-time instrumentation for C code.

[0017] Furthermore, the code coverage analysis module includes:

[0018] Bitmap recording unit, used to record the coverage information of the current code block when stub code is executed;

[0019] Bitmap statistics unit is used to collect all coverage information obtained from testing deep learning frameworks that have completed instrumentation using the tester.

[0020] The coverage granularity optimization unit is used to analyze the cost metrics of different coverage granularities based on Bitmap record information, and obtain the optimized coverage granularity for coverage analysis.

[0021] A shared algorithm library for counting code coverage based on Bitmap statistics, using concurrency optimizations and the hacker_popcount algorithm.

[0022] A shared code coverage analysis library is used to track code coverage information through function calls and to calculate the total code coverage in the shared algorithm library in real time, thereby realizing code coverage analysis.

[0023] Furthermore, in the coverage granularity optimization unit, different coverage granularities include row coverage, branch coverage, block coverage, edge coverage, and path coverage;

[0024] The overhead metrics include the time required for instrumentation, execution overhead, and analysis overhead.

[0025] The optimized coverage granularity is block coverage.

[0026] A code coverage analysis method includes the following steps:

[0027] S1. Based on the programming language of the deep learning framework, instrument the deep learning framework using the interface instrumentation mode through the instrumentation module, and package and integrate the instrumented framework code back into the deep learning framework to record code coverage information.

[0028] S2. Test the instrumented deep learning framework using a tester and collect test information using Bitmap.

[0029] S3. Record the stub code after instrumentation by the deep learning framework using Bitmap;

[0030] S4. Analyze the overhead indicators of different coverage granularities in Bitmap record information through the coverage granularity optimization unit to determine the optimal coverage granularity;

[0031] S5. Code coverage counting is performed based on a shared algorithm library that incorporates concurrency optimization and the hacker_popcount algorithm;

[0032] S6. In the shared code coverage analysis library, the optimization coverage granularity is tracked through function calls, and the total code coverage in the shared algorithm library is calculated in real time to realize code coverage analysis.

[0033] Furthermore, in step S1, the programming languages ​​of the deep learning framework include C code and Python code;

[0034] When the programming language is Python, runtime bytecode instrumentation is used to instrument deep learning frameworks.

[0035] When the programming language is C code, compile-time instrumentation is used to instrument deep learning frameworks.

[0036] Furthermore, in step S4, the coverage granularity is optimized as block coverage.

[0037] Furthermore, in step S5, a compiler-level concurrent optimization based on OpenMP instructions is introduced into the shared algorithm library; at the same time, the hacker_popcount algorithm is introduced to accelerate bit statistics in the code coverage bitmap.

[0038] Furthermore, in step S6, code coverage information is stored in the shared code coverage analysis library by setting and maintaining a pointer named mdata as a shared memory block.

[0039] The beneficial effects of this invention are as follows:

[0040] 1) Compatibility with most versions of popular frameworks: The code coverage analysis method for deep learning frameworks proposed in this invention is compatible with multiple versions of different frameworks. It uses an interface-based instrumentation approach, making the stub code more flexible and adaptable to the multi-language nature of deep learning frameworks. Targeted instrumentation methods are designed for the characteristics of different programming languages. Compared to existing tools that only work on fixed versions of frameworks, this method is compatible with multiple versions of frameworks such as TensorFlow, PyTorch, and PaddlePaddle. Experiments show that the frameworks and versions compatible with this invention are listed in Table 2.

[0041] Table 2: Compatibility of this invention with multiple versions of different deep learning frameworks

[0042]

[0043] 2) Significantly Reduced Time Cost: The lightweight code coverage analysis method proposed in this invention significantly reduces the time cost of existing instrumentation tools by improving the parallel loop algorithm and using the hacker_popcount algorithm. Specific results, measured by time consumption, are shown in Table 3. The average analysis cost across multiple frameworks is only 2.19ms, which is 4.45% of SlipCover and 2.86% of coverage.py. It still outperforms existing tools in terms of overall cost, with a total time cost of 44.2% of SlipCover and 24.40% of coverage.py. In TensorFlow, DCOV performs coverage analysis 28 times and 23 times faster than Coverage.py and SlipCover, respectively. In PyTorch, DCOV performs coverage analysis 4.7 times and 2.1 times faster than Coverage.py and SlipCover, respectively. In PaddlePaddle, DCOV performs coverage analysis 32 times and 26 times faster than Coverage.py and SlipCover, respectively.

[0044] Table 3: Time cost of the present invention on different frameworks (unit: milliseconds)

[0045] Attached Figure Description

[0046] Figure 1 This is an overall architecture diagram of the code coverage analysis system for deep learning frameworks provided by the present invention.

[0047] Figure 2 This is an example diagram of a C-type insertion stake provided by the present invention.

[0048] Figure 3 This is an example diagram of Python instrumentation provided by the present invention.

[0049] Figure 4 This is an example diagram of the insertion mode provided by the present invention.

[0050] Figure 5 The code coverage analysis module architecture diagram provided for this invention.

[0051] Figure 6 This is a schematic diagram of the shared code coverage analysis library provided by the present invention.

[0052] Figure 7A schematic diagram of the improved algorithm in the shared algorithm library provided by this invention.

[0053] Figure 8 This is a schematic diagram of the algorithm for resolving race risk in the shared code coverage analysis library provided by this invention. Detailed Implementation

[0054] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0055] Example 1:

[0056] This invention provides a code coverage analysis system for deep learning frameworks, such as... Figure 1 As shown, it includes:

[0057] The instrumentation module is used to instrument deep learning frameworks, and uses the corresponding instrumentation methods for the programming language of the deep learning framework to obtain coverage information.

[0058] The code coverage analysis module is used to obtain coverage information based on instrumentation, call Bitmap statistics and recording information according to user needs, and then call an improved shared algorithm library based on Bitmap statistics. Combined with coverage granularity optimized based on Bitmap recording information, it completes the recording, statistics and analysis of coverage information.

[0059] In this embodiment of the invention, the instrumentation module uses interface-based instrumentation and instrumentes the Python and C code in the deep learning framework according to different instrumentation methods. The instrumented framework code is then packaged and integrated back into the deep learning framework. This method is used to record and obtain coverage information.

[0060] Specifically, instrumentation methods refer to inserting instrumentation code into the actual target program and the encoded form of the inserted instrumentation code. Instrumentation methods can be roughly divided into static instrumentation and dynamic instrumentation, where static instrumentation is completed before program execution and dynamic instrumentation is completed during program execution. Granularity further considers the encoded form of the program at the time of detection, i.e., source code, intermediate representation, and binary. Based on this classification, afl-gcc's compile-time instrumentation is based on static assembly instrumentation, while SlipCover is based on dynamic Python bytecode instrumentation. The lifecycle of a deep learning framework is a process of converting high-level programming languages ​​(Python and C) into machine-executable low-level programming languages. During compilation, only C code is compiled into .so format library files and packaged into a Python package along with the Python source code. The Python source code is compiled into bytecode at runtime and executed by the Python virtual machine along with the .so files from C to complete the user's development tasks. Based on this, we designed different detection methods according to different programming languages. For Python code, because Python is an interpreted language, its code is automatically compiled into Python bytecode during program execution and then executed by the Python virtual machine. To ensure the performance of Python programs, Python bytecode files are gradually generated and cached during Python script execution, and the cached Python bytecode is directly executed during subsequent repeated code executions. Therefore, checking the Python bytecode the first time it is generated is the most appropriate time. For C code, because C is a compiled language, this means that deep learning frameworks use dynamic library files that have already been compiled into stable binaries at runtime. Dynamically editing library files compiled from millions of lines of code at runtime is extremely challenging. Therefore, compile-time instrumentation with compatibility optimizations (such as afl-gcc) is a more feasible option.

[0061] Based on this, the instrumentation method of the instrumentation module in this embodiment includes a runtime bytecode instrumentation method for Python code and a compile-time instrumentation method for C code, examples of which are shown below. Figure 2 and Figure 3 As shown.

[0062] In this embodiment, the instrumentation mode of the instrumentation module can be divided into embedded instrumentation and interface instrumentation, such as... Figure 4As shown, embedded instrumentation inserts a stub code containing all the statements needed to perform code coverage analysis, while interface instrumentation inserts call statements into specific functions in an external library without directly implementing code coverage analysis. Embedded instrumentation is more intuitive due to its WYSIWYG nature, but once the stub code is inserted, it cannot be changed. On the other hand, interface instrumentation has more assembly instructions at the assembly level than embedded instrumentation, but after instrumentation, the probe's functionality can still be adjusted by changing the external library. Considering the multi-programming-language code structure of deep learning frameworks, if embedded instrumentation is used, a set of independent stub codes with the same logical functionality but using different programming languages ​​needs to be implemented for Python and C. Conversely, if interface instrumentation is used, only one call statement to an external function needs to be inserted in both Python and C. The actual code coverage analysis functionality can then be implemented in the external library. Therefore, considering flexibility and compatibility with deep learning frameworks, interface instrumentation is used in this embodiment.

[0063] In this embodiment, the compatibility issue between the code coverage analyzer and the deep learning framework is solved by using an interface-based instrumentation pattern and targeted instrumentation methods.

[0064] In embodiments of the present invention, such as Figure 5 As shown, the code coverage analysis module includes:

[0065] Bitmap recording unit, used to record the coverage information of the current code block when stub code is executed;

[0066] Bitmap statistics unit is used to collect all coverage information obtained from testing deep learning frameworks that have completed instrumentation using the tester.

[0067] The coverage granularity optimization unit is used to analyze the cost metrics of different coverage granularities based on Bitmap record information, and obtain the optimized coverage granularity for coverage analysis.

[0068] A shared algorithm library for counting code coverage based on Bitmap statistics, using concurrency optimizations and the hacker_popcount algorithm.

[0069] A shared code coverage analysis library is used to track code coverage information through function calls and to calculate the total code coverage in the shared algorithm library in real time, thereby realizing code coverage analysis.

[0070] In this embodiment, the term "bitmap" originates from Google's famous fuzz testing tool, American FuzzyLoop (AFL). Bitmap is used to check whether a record with only one bit has been hit, and it has a small memory footprint.

[0071] In the coverage granularity optimization unit of this embodiment, different coverage granularities include row coverage, branch coverage, block coverage, edge coverage, and path coverage; the overhead indicators include instrumentation time, execution overhead, and analysis overhead; the optimized coverage granularity is block coverage.

[0072] Specifically, in this embodiment, regarding coverage granularity, in software testing, different code coverage standards provide a measure of the coverage of the tested program at different levels within the test set. The different coverage granularities mainly include line coverage, branch coverage, block coverage, edge coverage, and path coverage.

[0073] Line coverage, also known as statement coverage, measures whether each line of code in a program is executed at least once. This is the most basic coverage standard, and its goal is to ensure that every line of code in the program is executed by test cases at least once in order to discover potential errors.

[0074] Branch coverage measures that every possible result of each branch in a program is executed at least once. It requires that every possible result of every conditional expression be covered; compared to line coverage, branch coverage examines different paths of code logic more deeply.

[0075] Block coverage is the coverage of basic code blocks in a program. A basic block is a set of instructions that are executed sequentially in the program code, without any jump instructions. Block coverage ensures that each code block is executed at least once. Block coverage does not focus on every line of code or every branch, but rather on whether the code blocks in the program are covered.

[0076] Edge coverage focuses on the coverage of edges connecting nodes in the program's control flow graph. This means it guarantees that every edge in the control flow graph representing a jump from one block to another in the program is executed at least once. Edge coverage is generally more stringent than simple branch coverage because it considers the execution paths through control structures in the program.

[0077] Path coverage testing is one of the most stringent coverage standards, designed to ensure that all possible paths in a program are executed. A path is the execution path from the program's entry point to its exit point, including all possible branches at decision points. Because the number of possible paths in modern software can be enormous, achieving complete path coverage is generally impractical in practice.

[0078] Generally, different coverage standards are used in different situations depending on project requirements, resources, and risk assessments. The choice of which coverage standard to use depends on the specific project's quality assurance objectives and constraints. Typically, line coverage and branch coverage are more commonly used, while path coverage, due to its complexity and numerous test cases, is usually only used when a very high level of test assurance is required.

[0079] In this embodiment, from an overhead perspective, the granularity of coverage directly affects the detection frequency, the operations performed by the stub code during execution, and the analysis scope of the analyzer; specifically, the first two factors increase execution overhead, while the latter increases analysis overhead. Recent state-of-the-art software testing research has adopted various types of coverage granularity, and the various indicators for different coverage granularities are shown in Table 4:

[0080] Table 4 Coverage Granularity Cost Table

[0081]

[0082] In this embodiment, through experiments and analysis, block coverage is selected as the coverage granularity of the present invention because block coverage measures the execution of each basic block from the perspective of the control flow graph and has the same ability to reflect execution information as line coverage under low overhead, while using the least amount of consumption to obtain all necessary information.

[0083] This invention fully considers the characteristics of deep learning frameworks and designs a lightweight code coverage analysis system for them. The main idea is to instrument the deep learning framework using an instrumentation module before testing, employing different instrumentation methods for different programming languages ​​within the framework to obtain coverage. Then, the coverage information is stored in a shared code coverage library using block coverage as the code coverage granularity. Finally, the shared code coverage library is improved in terms of algorithms and engineering, enabling the coverage analysis process to be completed with extremely low time overhead when the code coverage analyzer needs to access coverage information.

[0084] Example 2:

[0085] This invention provides a code coverage analysis method based on the code coverage analysis system for deep learning frameworks in Embodiment 1, comprising the following steps:

[0086] S1. Based on the programming language of the deep learning framework, instrument the deep learning framework using the interface instrumentation mode through the instrumentation module, and package and integrate the instrumented framework code back into the deep learning framework to record code coverage information.

[0087] S2. Test the instrumented deep learning framework using a tester and collect test information using Bitmap.

[0088] S3. Record the stub code after instrumentation by the deep learning framework using Bitmap;

[0089] S4. Analyze the overhead indicators of different coverage granularities in Bitmap record information through the coverage granularity optimization unit to determine the optimal coverage granularity;

[0090] S5. Code coverage counting is performed based on a shared algorithm library that incorporates concurrency optimization and the hacker_popcount algorithm;

[0091] S6. In the shared code coverage analysis library, the optimization coverage granularity is tracked through function calls, and the total code coverage in the shared algorithm library is calculated in real time to realize code coverage analysis.

[0092] In step S1 of this embodiment of the invention, the programming languages ​​of the deep learning framework include C code and Python code; when the programming language is Python code, the deep learning framework is instrumented using runtime bytecode instrumentation; when the programming language is C code, the deep learning framework is instrumented using compile-time instrumentation.

[0093] Specifically, in this embodiment, the lifecycle of the deep learning framework is a process of converting high-level programming languages ​​(Python and C) into machine-executable low-level programming languages. During compilation, only C code is compiled into .so format library files and packaged into a Python package along with the Python source code. The Python source code is compiled into bytecode at runtime and executed by the Python virtual machine along with the .so files from C to complete the user's development tasks. Based on this, we designed different detection methods for different programming languages. For Python code, because Python is an interpreted language, its code is automatically compiled into Python bytecode during program execution and then executed by the Python virtual machine. To ensure the performance of Python programs, Python bytecode files are gradually generated and cached during Python script execution, and the cached Python bytecode is directly executed during subsequent repeated code executions. Therefore, detecting the Python bytecode the first time it is generated is the most appropriate time. For C language code, since C is a compiled language, this means that the deep learning framework uses dynamic library files that have already been compiled into stable binary files at runtime. Dynamically editing library files compiled from millions of lines of code at runtime is extremely challenging. Therefore, compile-time instrumentation with compatibility optimizations (such as afl-gcc) is a more viable option.

[0094] In step S4 of this embodiment, the coverage granularity is optimized as block coverage because block coverage measures the execution of each basic block from the perspective of the control flow graph and has the same ability to reflect execution information as line coverage under low overhead, using the least amount of consumption when obtaining all necessary information.

[0095] In step S5 of this embodiment of the invention, a compiler-level concurrent optimization based on OpenMP instructions is introduced into the shared algorithm library; at the same time, the hacker_popcount algorithm is introduced to accelerate bit statistics in the code coverage bitmap.

[0096] Specifically, in this embodiment, based on such Figure 6 The shared code coverage analysis library shown requires calculating the sum of all non-zero bits in `mdata` in `dcov_info.cpp` of the shared algorithm library to implement code coverage counting. However, Figure 6 The code shown only considers traditional software testing scenarios, where the program under test is small, thus requiring only a very small bitmap to map all the basic blocks. In deep learning framework scenarios, even using bits as the smallest unit, such as the 64KB bitmap used in AFL, is insufficient to map all the basic blocks. To address this issue, this invention proposes an improved algorithm, such as... Figure 7 As shown in the diagram. In section A, this invention implements compile-level concurrency optimization based on OpenMP instructions. In section B, the hacker_popcount algorithm is introduced to accelerate bit statistics in the coverage bitmap, reducing time overhead.

[0097] In step S6 of this embodiment of the invention, code coverage information is stored in the shared code coverage analysis library by setting and maintaining a pointer named mdata as a shared memory block;

[0098] Specifically, such as Figure 6 The shared code coverage analysis library shown bridges the gap between probes and code coverage data. Specifically, probes call the `on_bb_hit` function to implement unified code coverage tracking, while testers use the `get_bb_cnt` function to calculate the current total coverage in real time. The code in the diagram is given as a simplified implementation and aims to illustrate how code coverage data is updated and calculated as simply as possible. To implement code coverage tracking features, the shared code coverage analysis library maintains a large shared memory block called `mdata` to store code coverage information. `mdata` is a pointer with the smallest unit of bytes, but each bit corresponds to the execution of a unique basic block in the deep learning framework, which initially defaults to 0. When a basic block (whether Python or C) is executed, it calculates the position in `mdata` based on the hash code written when it was detected and sets the bit at that position to 1.

[0099] However, such a process might be suitable for traditional software testing, where the test objects are often small, single-processor, single-threaded programs. However, due to considerations of data and computational parallelism, deep learning frameworks tend to run multiple threads or even multiple processes simultaneously, thus exposing the above code to significant race conditions. To address this issue, we will... Figure 6 The original dcov_trace.cpp in the file has been improved to Figure 8 As shown in the diagram. The code execution process is as follows: Lines 9-10 perform atomic reads on `mdata` and `compute` to generate a new value. Line 12 performs an atomic comparison and swap operation, which swaps the value of `mdata+idx` with the value of `data`. If the two values ​​are not equal, it returns `false`. The value in `new_byte` is then assigned to the byte in `mdata+idx`. If the two values ​​are equal, it returns `true`. The new algorithm ensures that `new_byte` is only updated when there is no contention on `mdata+idx`.

[0100] Example 3:

[0101] This invention provides a specific example of the code coverage analysis method in Embodiment 2.

[0102] In this embodiment, the software tool integrating the method in Embodiment 2 can be deployed and run on a host machine.

[0103] The code coverage analysis method in this embodiment has strong extensibility and theoretically supports compatibility with most existing deep learning framework testing tools. FreeFuzz is a dynamic vulnerability discovery method for deep learning software frameworks, focusing on the security of deep learning software libraries. It has a unique dynamic execution and input mutation mechanism, capable of constructing model inputs and API inputs that conform to the syntax specifications of specific programming languages, thereby triggering potential defects in deep learning libraries. Furthermore, FreeFuzz supports multiple deep learning frameworks, such as TensorFlow and PyTorch, and can run on different operating systems and hardware platforms. The following will detail the process of using FreeFuzz to test TensorFlow, thereby obtaining code coverage and analyzing it.

[0104] (1) According to the official documentation provided by FreeFuzz, first collect API call specification data, and then instrument it.

[0105] (2) Python dynamic instrumentation: TensorFlow gray-box test coverage information is obtained based on the name of the API under test and the interpreter dynamic instrumentation information.

[0106] C static instrumentation: This instrumentation method uses interface-based instrumentation, leveraging the intermediate representation GIMPLE used by GCC to perform instrumentation within GIMPLE during compilation.

[0107] (3) Configure test parameters for TensorFlow and start the fuzzer: Create a configuration file to specify test parameters, including database connection, output settings, mutation parameters, etc.; configure database connection information to store and retrieve data during testing; specify output configuration, including the save path of test results, report format, etc.; configure mutation parameters to determine the generation method and mutation strategy of test cases.

[0108] (4) Generate API call scripts: Based on the API information in the database, automatically generate multiple API call scripts and add these scripts to the seed queue as the initial input for testing.

[0109] (5) Code coverage analysis module algorithm: Based on OpenMP instructions, the compiler-level concurrent optimization is implemented and the hacker_popcount algorithm is introduced to accelerate bit statistics in the coverage bitmap and reduce time overhead.

[0110] (6) Coverage information retrieval and analysis: The analyzer will guide the testing work and draw current experimental conclusions based on the statistical coverage information. When the coverage is further improved, the analyzer will provide the coverage data and results analysis back to the R&D personnel, which can be used to guide the next step of the work.

[0111] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0112] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A code coverage analysis system for deep learning frameworks, characterized in that, include: The instrumentation module is used to instrument deep learning frameworks, and uses the corresponding instrumentation methods for the programming language of the deep learning framework to obtain coverage information. The insertion mode of the insertion module is an interface-type insertion module; The instrumentation methods of the instrumentation module include runtime bytecode instrumentation for Python code and compile-time instrumentation for C code. The code coverage analysis module is used to obtain coverage information based on instrumentation, call Bitmap statistics and record information according to user needs, and then call the improved shared algorithm library based on the Bitmap statistics. Combined with the coverage granularity optimized based on Bitmap record information, it completes the recording, statistics and analysis of coverage information. The code coverage analysis module includes: Bitmap recording unit, used to record the coverage information of the current code block when stub code is executed; Bitmap statistics unit is used to collect all coverage information obtained from testing deep learning frameworks that have completed instrumentation using the tester. The coverage granularity optimization unit is used to analyze the overhead indicators of different coverage granularities based on Bitmap record information, and obtain the optimized coverage granularity for coverage analysis, wherein the optimized coverage granularity is block coverage; A shared algorithm library is used to count code coverage based on Bitmap statistics using concurrency optimization and the hacker_popcount algorithm. The shared algorithm library introduces compiler-level concurrency optimization based on OpenMP instructions. At the same time, the hacker_popcount algorithm is introduced to accelerate bit statistics in the code coverage bitmap. A shared code coverage analysis library is used to track code coverage information through function calls and to calculate the total code coverage in the shared algorithm library in real time, thereby realizing code coverage analysis.

2. The code coverage analysis system for deep learning frameworks according to claim 1, characterized in that, In the coverage granularity optimization unit, different coverage granularities include row coverage, branch coverage, block coverage, edge coverage, and path coverage; The overhead metrics include the time required for instrumentation, execution overhead, and analysis overhead. The optimized coverage granularity is block coverage.

3. A code coverage analysis method based on the code coverage analysis system for deep learning frameworks as described in any one of claims 1 to 2, characterized in that, Includes the following steps: S1. Based on the programming language of the deep learning framework, instrument the deep learning framework using the interface instrumentation mode through the instrumentation module, and package and integrate the instrumented framework code back into the deep learning framework to record code coverage information. S2. Test the instrumented deep learning framework using a tester and collect test information using Bitmap. S3. Record the stub code after instrumentation by the deep learning framework using Bitmap; S4. Analyze the overhead indicators of different coverage granularities in Bitmap record information through the coverage granularity optimization unit to determine the optimal coverage granularity; S5. Code coverage counting is performed based on a shared algorithm library that incorporates concurrency optimization and the hacker_popcount algorithm; S6. In the shared code coverage analysis library, the optimization coverage granularity is tracked through function calls, and the total code coverage in the shared algorithm library is calculated in real time to realize code coverage analysis.

4. The code coverage analysis method according to claim 3, characterized in that, In step S1, the programming languages ​​of the deep learning framework include C code and Python code; When the programming language is Python, runtime bytecode instrumentation is used to instrument deep learning frameworks. When the programming language is C code, compile-time instrumentation is used to instrument deep learning frameworks.

5. The code coverage analysis method according to claim 3, characterized in that, In step S6, code coverage information is stored in the shared code coverage analysis library by setting and maintaining a pointer named mdata as a shared memory block.

Images