Code coverage rate analysis system and method for deep learning framework

Abstract

The invention discloses a code coverage rate analysis system and a code coverage rate analysis method for a deep learning framework, and designs a lightweight code coverage rate analysis system and a lightweight code coverage rate analysis method for the deep learning framework by fully considering the characteristics of the deep learning framework. The deep learning framework is subjected to instrumentation by using an instrumentation module, and the coverage rate is obtained by using different instrumentation methods for multiple programming languages in the deep learning framework. Afterwards, the code coverage rate granularity of the block coverage rate is used for storing the coverage rate information into a shared code coverage rate analysis library, algorithm and engineering improvement is carried out on a shared algorithm library, and when a code coverage rate analysis system needs to call the coverage rate information, the coverage rate analysis process can be completed under the condition of using extremely low time overhead.

Description

Technical Field

[0001] The present invention belongs to the technical field of software testing, and in particular relates to a code rate coverage analysis system and method for a deep learning framework. Background Art

[0002] Artificial intelligence systems are becoming more and more mature, and various technologies that can be seen everywhere in life, such as face recognition, automatic driving, and IoT home, are emerging one after another. With the continuous and rapid development of artificial intelligence systems, deep learning frameworks have received more and more widespread attention as their construction and support platforms. For today's relatively popular foreign deep learning frameworks such as TensorFlow and PyTorch, and domestic frameworks MindSpore and PaddlePaddle, their own vulnerabilities have seriously threatened the application and development of artificial intelligence systems. For example, in 2016, due to the deep learning system's incorrect assumptions about road conditions, Google's self-driving vehicle crashed into the side of a bus at a low speed, causing a car accident. In addition, PyTorch versions below 2.3.24 use an unsafe YAML file loading method, which leads to interruption of model resources. These deep learning framework vulnerabilities seriously threaten the application and development of artificial intelligence systems. Therefore, testing for deep learning frameworks is a key technology to ensure the security of deep learning frameworks.

[0003] In the early days of testing deep learning frameworks, differential testing and fuzz testing were the most commonly used methods. Differential testing usually selects two or more objects with the same functions, and selects the same test cases that meet the requirements of the objects to be tested and injects them into the objects to be tested and executes them to observe whether there are differences in the execution results. Fuzz testing is a method of discovering software vulnerabilities by providing unexpected inputs to the target system and capturing abnormal results. Due to its high degree of automation, this technology has become the most effective method in the field of vulnerability mining and is widely used by software testers.

[0004] Compared with traditional software, deep learning frameworks have complex structures and cumbersome execution processes. Testing deep learning frameworks requires taking multi-layer stacked deep learning models consisting of a large number of neurons and connection weights as input. These deep learning models are trained with a large amount of data. However, due to the high cost of training and limited training data, it is very difficult to build a large number of models that can trigger framework vulnerabilities.

[0005] Recently, coverage has gradually attracted the attention of deep learning framework testing as an important indicator in traditional software testing. Coverage technology is a method used in software testing to quantify the completeness of testing. It measures the ratio of the code executed to the total code during the test. Specifically, coverage includes line coverage, branch coverage, conditional coverage, path coverage and other types, which focus on whether the statements in the source code are executed, whether all branches are traversed, whether the results of conditional expressions are tested, and whether all possible code paths are executed. These technologies help to evaluate the depth and breadth of testing and ensure the quality and stability of software.

[0006] At present, coverage technology is widely used in the research on deep learning framework testing in many top academic conferences. However, such research usually uses coverage as the basis for the quality of testing capabilities, and still lacks the application and analysis of coverage information. In the current field of coverage analysis technology, there are two significant challenges. First, the existing code coverage analyzers have significant deficiencies in compatibility with deep learning frameworks. This deficiency leads to the limitations of these analyzers in evaluating deep learning framework code, making it impossible for them to work smoothly with deep learning frameworks. Secondly, these analyzers often introduce a lot of time overhead at runtime, which not only affects the developer's work efficiency, but also increases the time cost of the project. Therefore, solving the compatibility problem with the deep learning framework and reducing the time overhead have become important directions for the development of current coverage analysis technology:

[0007] 1) Compatibility issues: Today's popular deep learning frameworks are mainly composed of two programming languages, Python and C. Among them, the existing coverage analysis tools corresponding to the Python language are mainly Coverage.py and SlipCover, which support the testing of deep learning frameworks. However, during the testing process, since the original tools are used in the form of command lines, these two tools need to be adapted to different frameworks in order to perform normal testing. For the C language, the existing coverage analysis tools are mainly the AFL Fuzzer series and Gcov. afl-gcc and afl-clang are two very critical components in the AFL tool set. They are GCC and Clang compiler versions that AFL supports customization, allowing developers to use these compilers to generate programs that have been instrumented by AFL. Gcov is a code coverage testing tool that is closely integrated with GCC. It is a command-line console program that is released with GCC and is used to analyze the statement coverage and branch coverage of C / C++ code. Gcov can count the execution frequency of each line of code, thereby helping developers understand which code is actually executed and the execution time of each code segment. This information is crucial for optimizing code, improving program performance, and ensuring code quality. Instrumentation refers to adding additional monitoring code to the source code or binary code of the program. These codes can help AFL track which code blocks are executed, making the fuzz testing process more efficient. However, the object-oriented nature of these tools is mainly traditional software with a single language and a single structure, which may cause compilation failures when testing deep learning frameworks. In summary, existing coverage analyzers cannot be directly applied to deep learning framework testing, so there is an urgent need to solve the problem of poor compatibility of coverage analyzers.

[0008] 2) Time overhead problem: In order to verify the time overhead problem, we set up an experiment. Based on the official documentation of the deep learning framework, we used FreeFuzz to generate 100 Python scripts for each deep learning framework as test samples. Then, we used different coverage analyzers to execute these test samples 100 times on each deep learning framework. After each execution, we immediately performed coverage analysis using the selected coverage analyzer. In the experiment, we chose to use Coverage.py, SlipCover and Gcov to test on three frameworks: TensorFlow, PyTorch and PaddlePaddle. The experimental results are shown in Table 1. From the data, it can be seen that when using the existing coverage analyzer for Python code, the use of the analyzer brings an average of 2-180 times the time overhead compared to the time overhead without using the coverage analyzer. For C language, the Gcov tool is only compatible with TensorFlow and brings a huge time overhead. Existing code coverage analyzers introduce huge time overhead in deep learning framework testing, which has become a common problem of existing tools.

[0009] Table 1: Average cost of code coverage analysis (in milliseconds)

[0010] Summary of the invention

[0011] In view of the above-mentioned deficiencies in the prior art, the code coverage analysis system and method for deep learning frameworks provided by the present invention analyze the characteristics of the deep learning framework, use an interface-based plug-in method, and design targeted plug-in methods according to different programming languages, thereby approaching the compatibility problem of existing code coverage analyzers for deep learning frameworks; at the same time, by modeling the code coverage analysis process of the deep learning framework, algorithmic and engineering improvements are made to performance bottlenecks, thereby solving the problem of excessive time overhead.

[0012] In order to achieve the above-mentioned purpose of the invention, the technical solution adopted by the present invention is: a code coverage analysis system for a deep learning framework, comprising:

[0013] The instrumentation module is used to instrument the deep learning framework and obtain coverage information using the corresponding instrumentation method for the programming language of the deep learning framework;

[0014] The code coverage analysis module is used to call Bitmap statistics and record information according to user needs based on the coverage information obtained by instrumentation, and then call the improved shared algorithm library according to the Bitmap statistical information, and combine the coverage granularity optimized according to the Bitmap recording information to complete the recording, statistics and call analysis of the coverage information.

[0015] Furthermore, the plugging mode of the plugging module is an interface plugging module;

[0016] The plugging method of the plugging module includes a runtime bytecode plugging method for a programming language that is C code, and a compile-time plugging method for a programming language that is Python code.

[0017] Furthermore, the code coverage analysis module includes:

[0018] Bitmap recording unit, used to record the coverage information of the current code block when the stub code is executed;

[0019] Bitmap statistics unit, used to count all coverage information obtained by testing the deep learning framework that uses the tester to complete the instrumentation process;

[0020] The coverage granularity optimization unit is used to analyze the overhead indicators of different coverage granularities according to the Bitmap record information, and obtain the optimized coverage granularity for coverage analysis;

[0021] Shared algorithm library for counting code coverage using concurrent optimization and hacker_popcount algorithm based on Bitmap statistics;

[0022] The shared code coverage analysis library is used to track code coverage information through function calls and calculate the total code coverage in the shared algorithm library in real time to implement code coverage analysis.

[0023] Further, in the coverage granularity optimization unit, different coverage granularities include row coverage, branch coverage, block coverage, edge coverage and path coverage;

[0024] The overhead indicators include the time required for instrumentation, execution overhead, and analysis overhead;

[0025] The optimized coverage granularity is block coverage.

[0026] A code coverage analysis method comprises the following steps:

[0027] S1. According to the programming language of the deep learning framework, the deep learning framework is instrumented by using the instrumentation module using the interface instrumentation mode, and the instrumented framework code is packaged and integrated back into the deep learning framework, thereby recording the code coverage information;

[0028] S2. Test the deep learning framework after the instrumentation process through the tester, and use Bitmap to count the test information;

[0029] S3, record the stub code after the deep learning framework is plugged in through Bitmap;

[0030] S4, analyzing the overhead indicators of different coverage granularities in the Bitmap record information through the coverage granularity optimization unit, and determining the optimized coverage granularity;

[0031] S5. Code coverage counting based on the shared algorithm library that introduces concurrency optimization and hacker_popcount algorithm;

[0032] S6. In the shared code coverage analysis library, the optimization coverage granularity is tracked through function calls, and the total code coverage in the shared algorithm library is calculated in real time to implement code coverage analysis.

[0033] Further, in step S1, the programming language of the deep learning framework includes C code and Python code;

[0034] When the programming language is C code, the runtime bytecode instrumentation method is used to instrument the deep learning framework;

[0035] When the programming language is Python code, the deep learning framework is instrumented using the compile-time instrumentation method.

[0036] Furthermore, in step S4, the coverage granularity is optimized to be block coverage.

[0037] Furthermore, in step S5, in the shared algorithm library, OpenMP instructions are introduced to implement compilation-level concurrency optimization; at the same time, a hacker_popcount algorithm is introduced to accelerate bit statistics in the code coverage bitmap.

[0038] Furthermore, in step S6, in the shared code coverage analysis library, the code coverage information is stored by setting and maintaining a pointer named mdata as a shared memory block.

[0039] The beneficial effects of the present invention are:

[0040] 1) Compatible with most versions of popular frameworks: The code coverage analysis method of deep learning frameworks with different versions that are compatible with multiple frameworks proposed in the present invention uses an interface-based plug-in method to make the stub code more flexible, which can fit the multi-language characteristics of the deep learning framework. According to the characteristics of different programming languages, targeted plug-in methods are designed; compared with existing tools that can only work on fixed versions of the framework, this method can be compatible with multiple versions of multiple frameworks such as TensorFlow, PyTorch, and PaddlePaddle. After experiments, the frameworks and versions that the present invention can be compatible with are shown in Table 2:

[0041] Table 2: Compatibility of the present invention with multiple versions of different deep learning frameworks

[0042] Deep Learning Frameworks Version Number Compiler compatibility TensorFlow v2.10.0 bazel compatible TensorFlow v2.11.0 bazel compatible PaddlePaddle v2.1.0 cmake+make compatible PaddlePaddle v2.4.0 cmake+ninja compatible PaddlePaddle v2.5.0 cmake+ninja compatible PaddlePaddle v2.6.0 cmake+ninja compatible PaddlePaddle v2.6.1 cmake+ninja compatible PyTorch v2.1.0 cmake+make compatible PyTorch v2.2.0 cmake+make compatible

[0043] 2) Greatly reduce time overhead: The lightweight code coverage analysis method proposed in this invention greatly reduces the time overhead of existing instrumentation tools by improving the parallel loop algorithm and using the hacker_popcount algorithm. Taking time consumption as the measurement indicator, the specific results are shown in Table 3. The average analysis overhead of multiple framework tests is only 2.19ms, which is 4.45% of SlipCover and 2.86% of coverage.py. The total overhead is still better than the existing tools, with a total time overhead of 44.2% of SlipCover and 24.40% of coverage.py. In TensorFlow, DCOV performs coverage analysis 28 times and 23 times faster than Coverage.py and SlipCover, respectively. In PyTorch, DCOV performs coverage analysis 4.7 times and 2.1 times faster than Coverage.py and SlipCover, respectively. In PaddlePaddle, DCOV performs coverage analysis 32 times and 26 times faster than Coverage.py and SlipCover, respectively.

[0044] Table 3: Time cost of the present invention on different frameworks (unit: milliseconds)

[0045]

[0046] BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The overall architecture diagram of the code coverage analysis system for deep learning framework provided by the present invention.

[0048] Figure 2 This is an example diagram of the C plug-in provided by the present invention.

[0049] Figure 3 This is an example diagram of the Python instrumentation provided by the present invention.

[0050] Figure 4 This is an example diagram of the insertion mode provided by the present invention.

[0051] Figure 5 This is an architecture diagram of the code coverage analysis module provided by the present invention.

[0052] Figure 6 A schematic diagram of a shared code coverage analysis library provided by the present invention.

[0053] Figure 7 A schematic diagram of the improved algorithm in the shared algorithm library provided by the present invention.

[0054] Figure 8 A schematic diagram of an algorithm for resolving competition risk in a shared code coverage analysis library provided by the present invention. DETAILED DESCRIPTION

[0055] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0056] Embodiment 1:

[0057] The embodiment of the present invention provides a code coverage analysis system for a deep learning framework, such as Figure 1 As shown, including:

[0058] The instrumentation module is used to instrument the deep learning framework and obtain coverage information using the corresponding instrumentation method for the programming language of the deep learning framework;

[0059] The code coverage analysis module is used to call Bitmap statistics and record information according to user needs based on the coverage information obtained by instrumentation, and then call the improved shared algorithm library according to the Bitmap statistical information, and combine the coverage granularity optimized according to the Bitmap recording information to complete the recording, statistics and call analysis of the coverage information.

[0060] In an embodiment of the present invention, the instrumentation module uses interface-based instrumentation and instruments the Python and C codes in the deep learning framework respectively according to different instrumentation methods, packages the instrumented framework code and integrates it back into the deep learning framework, and records and obtains coverage information through this method.

[0061] Specifically, for the instrumentation method, which refers to the actual insertion of the instrumentation code into the target program and the coding form of the inserted instrumentation code, the instrumentation method can be roughly divided into static instrumentation and dynamic instrumentation, where static instrumentation is completed before the program is executed and dynamic instrumentation is completed during the program execution. The granularity division is to further consider the coding form of the program when it is detected, that is, source code, intermediate representation and binary. According to this classification, the compile-time instrumentation of afl-gcc is based on static assembly instrumentation, while SlipCover is based on dynamic Python bytecode instrumentation. The life cycle of a deep learning framework is a process of converting high-level programming languages ​​(Python and C) into machine-executable low-level programming languages. During the compilation process, only the C code is compiled into a library file in .so format and packaged into a Python package together with the Python source code. The Python source code is compiled into bytecode at runtime and executed by the Python virtual machine together with the .so file from C to complete the user's development task. On this basis, we designed different detection methods according to different programming languages. For Python code, because Python is an interpreted language, its code is automatically compiled into Python bytecode during program execution and then executed by the Python virtual machine. In order to ensure the performance of Python programs, Python bytecode files are gradually generated and cached when Python scripts are executed, and the cached Python bytecode is directly executed during subsequent repeated code execution. Therefore, it is the most appropriate time to detect Python bytecode when it is first generated. For C language code, since C is a compiled language, this means that deep learning frameworks use dynamic library files that have been compiled into stable binary files at runtime. It is extremely challenging to dynamically edit library files compiled from millions of lines of code at runtime. Therefore, compile-time instrumentation with compatibility optimization (such as afl-gcc) is a more feasible option.

[0062] Based on this, the plugging method of the plugging module in this embodiment includes a runtime bytecode plugging method for C code and a compile-time plugging method for Python code, and examples thereof are respectively as follows: Figure 2 and Figure 3 shown.

[0063] In this embodiment, the plug-in mode of the plug-in module can be divided into embedded plug-in and interface plug-in, such as Figure 4As shown, the embedded instrumentation inserts a pile code, which contains all the statements required to complete the code coverage analysis function, and the interface-based instrumentation inserts the call statement into a specific function in the external library without directly implementing the code coverage analysis function. The embedded instrumentation is more intuitive due to its WYSIWYG characteristics, but once the pile code is inserted, it cannot be changed. On the other hand, the interface-based instrumentation has more assembly instructions at the assembly level than the embedded instrumentation, but after the instrumentation is completed, the function of the probe can still be adjusted by changing the external library. Considering that the deep learning framework has a code structure with multiple programming languages, if the embedded instrumentation is used, it is necessary to implement a set of independent pile codes with the same logical functions but different programming languages ​​for Python and C. On the contrary, if the interface-based instrumentation is used, only a call statement to the external function needs to be inserted in Python and C. Then the actual function of the code coverage analysis can be implemented in the external library. Therefore, considering the flexibility and compatibility with the deep learning framework, the interface-based instrumentation is used in this embodiment.

[0064] In this embodiment, the compatibility problem between the code coverage analyzer and the deep learning framework is solved through the interface-based plug-in mode and the targeted plug-in method.

[0065] In the embodiment of the present invention, Figure 5 As shown, the code coverage analysis module includes:

[0066] Bitmap recording unit, used to record the coverage information of the current code block when the stub code is executed;

[0067] Bitmap statistics unit, used to count all coverage information obtained by testing the deep learning framework that uses the tester to complete the instrumentation process;

[0068] The coverage granularity optimization unit is used to analyze the overhead indicators of different coverage granularities according to the Bitmap record information, and obtain the optimized coverage granularity for coverage analysis;

[0069] Shared algorithm library for counting code coverage using concurrent optimization and hacker_popcount algorithm based on Bitmap statistics;

[0070] The shared code coverage analysis library is used to track code coverage information through function calls and calculate the total code coverage in the shared algorithm library in real time to implement code coverage analysis.

[0071] The term "bitmap" in this embodiment originates from American FuzzyLoop (AFL), a famous fuzz testing tool of Google. Bitmap is used to check whether only one bit record has been hit, and the memory usage is small.

[0072] In the coverage granularity optimization unit of this embodiment, different coverage granularities include row coverage, branch coverage, block coverage, edge coverage and path coverage; overhead indicators include the time required for insertion, execution overhead and analysis overhead; the optimized coverage granularity is block coverage.

[0073] Specifically, in this embodiment, for coverage granularity, in software testing, different code coverage standards provide measurements of coverage of different levels of the tested program in the test set. Different coverage granularities mainly include line coverage, branch coverage, block coverage, edge coverage and path coverage.

[0074] Among them, line coverage, also known as statement coverage, measures whether each line of code in the program is executed at least once; this is the most basic coverage standard, and its goal is to ensure that every line of the code is executed at least once by the test case to discover possible errors.

[0075] Branch coverage measures whether every possible outcome of every branch in the program has been executed at least once. It requires that every possible outcome of every conditional expression be covered; compared to line coverage, branch coverage examines different paths of code logic more deeply.

[0076] Block coverage is the coverage of basic code blocks in a program. A basic block is a group of instructions executed sequentially in the program code without any jump instructions. Block coverage ensures that each code block is executed at least once. Block coverage does not focus on every line of code or every branch, but on whether the code blocks in the program are covered.

[0077] Edge coverage is concerned with the coverage of the edges connecting nodes in a program's control flow graph. This means that it ensures that all edges in the control flow graph that represent jumps from one block to another in the program are executed at least once. Edge coverage is usually more stringent than simple branch coverage because it takes into account the execution paths through the control structures in the program.

[0078] Path coverage testing is one of the most stringent coverage criteria and aims to ensure that all possible paths in a program are exercised. A path is an execution path from the entry point to the exit point of a program, including all possible branches at decision points. Since the number of possible paths in modern software can be very large, it is usually not feasible to achieve complete path coverage in practice.

[0079] Generally speaking, different coverage standards are used in different situations according to different project requirements, resources and risk assessment. The choice of which coverage standard depends on the quality assurance goals and constraints of the specific project. Usually, line coverage and branch coverage are more commonly used, while path coverage is usually only used when a very high level of test assurance is required due to its complexity and numerous test cases.

[0080] In this embodiment, from the perspective of overhead, the coverage granularity directly affects the detection frequency, the operations taken by the stub code during execution, and the analysis scope of the analyzer; specifically, the first two factors increase the execution overhead, while the latter factor increases the analysis overhead; recent state-of-the-art software testing research has adopted various types of coverage granularity, and the indicators of different coverage granularities are shown in Table 4:

[0081] Table 4 Coverage granularity overhead table

[0082] Coverage granularity Time required for insertion Execution overhead Analyze the overhead Row Coverage high high Low Branch Coverage Low Low Low Block Coverage Low Low Low Edge Cover Low middle Low Path coverage Low Low / Medium high

[0083] In this embodiment, through experiments and analysis, block coverage is selected as the coverage granularity selected by the present invention, because block coverage measures the execution of each basic block from the perspective of the control flow graph, and has the same ability to reflect execution information as line coverage under low overhead, and uses the least consumption when obtaining all necessary information.

[0084] The embodiment of the present invention fully considers the characteristics of the deep learning framework and designs a lightweight code coverage analysis system for the deep learning framework. The main idea is to use the plug-in module to plug the deep learning framework before testing, and use different plug-in methods to obtain coverage for multiple programming languages ​​in the deep learning framework. Afterwards, the coverage information is stored in the shared code coverage library using the code coverage granularity of block coverage. Finally, the shared code coverage library is improved in algorithms and engineering, so that when the code coverage analyzer needs to call the coverage information, the coverage analysis process can be completed with extremely low time overhead.

[0085] Embodiment 2:

[0086] The present invention provides a code coverage analysis method based on the code coverage analysis system for a deep learning framework in Example 1, comprising the following steps:

[0087] S1. According to the programming language of the deep learning framework, the deep learning framework is instrumented by using the instrumentation module using the interface instrumentation mode, and the instrumented framework code is packaged and integrated back into the deep learning framework, thereby recording the code coverage information;

[0088] S2. Test the deep learning framework after the instrumentation process through the tester, and use Bitmap to count the test information;

[0089] S3, record the stub code after the deep learning framework is plugged in through Bitmap;

[0090] S4, analyzing the overhead indicators of different coverage granularities in the Bitmap record information through the coverage granularity optimization unit, and determining the optimized coverage granularity;

[0091] S5. Code coverage counting based on the shared algorithm library that introduces concurrency optimization and hacker_popcount algorithm;

[0092] S6. In the shared code coverage analysis library, the optimization coverage granularity is tracked through function calls, and the total code coverage in the shared algorithm library is calculated in real time to implement code coverage analysis.

[0093] In step S1 of the embodiment of the present invention, the programming language of the deep learning framework includes C code and Python code; when the programming language is C code, the deep learning framework is instrumented using a runtime bytecode instrumentation method; when the programming language is Python code, the deep learning framework is instrumented using a compile-time instrumentation method.

[0094] Specifically, in this embodiment, the life cycle of the deep learning framework is a process of converting high-level programming languages ​​(Python and C) into machine-executable low-level programming languages. During the compilation process, only C code is compiled into a library file in .so format and packaged into a Python package together with the Python source code. The Python source code is compiled into bytecode at runtime and executed by the Python virtual machine together with the .so file from C to complete the user's development task. On this basis, we designed different detection methods according to different programming languages. For Python code, because Python is an interpreted language, its code is automatically compiled into Python bytecode during program execution and then executed by the Python virtual machine. In order to ensure the performance of the Python program, the Python bytecode file will be gradually generated and cached when the Python script is executed, and the cached Python bytecode will be directly executed when the code is repeated in the subsequent execution. Therefore, it is the most appropriate time to detect the Python bytecode when it is generated for the first time. For C language code, since C is a compiled language, this means that the deep learning framework uses dynamic library files that have been compiled into stable binary files at runtime. It is extremely challenging to dynamically edit library files compiled from millions of lines of code at runtime. Therefore, compile-time instrumentation with compatibility optimizations (such as afl-gcc) is a more viable option.

[0095] In step S4 of this embodiment, the coverage granularity is optimized to block coverage, because block coverage measures the execution of each basic block from the perspective of the control flow graph, and has the same ability to reflect execution information as line coverage under low overhead, using the least consumption when obtaining all necessary information.

[0096] In step S5 of the embodiment of the present invention, in the shared algorithm library, OpenMP instructions are introduced to implement compilation-level concurrency optimization; at the same time, the hacker_popcount algorithm is introduced to accelerate the bit statistics in the code coverage bitmap.

[0097] Specifically, in this embodiment, based on Figure 6 In order to implement the code coverage counting function, the sum of all non-zero bits in mdata must be calculated in dcov_info.cpp of the shared algorithm library. However, Figure 6 The code shown in only considers the traditional software testing scenario, where the size of the program under test is small, so only a very small bitmap is needed to map all basic blocks. In the deep learning framework scenario, even if bits are used as the minimum unit, such as the 64KB bitmap used in AFL, it is still not enough to map all basic blocks. In order to solve this problem, the present invention proposes an improved algorithm, such as Figure 7 As shown in the figure, in the part marked as A, the present invention implements concurrent optimization at the compilation level based on OpenMP instructions. In the part marked as B in the figure, the hacker_popcount algorithm is introduced to accelerate the bit statistics in the coverage bitmap and reduce the time overhead.

[0098] In step S6 of the embodiment of the present invention, in the shared code coverage analysis library, a pointer named mdata is set and maintained as a shared memory block to store code coverage information;

[0099] Specifically, Figure 6 The shared code coverage analysis library shown builds a bridge between the probe and the code coverage data. Specifically, the probe calls the on_bb_hit function to implement the unified code coverage tracking function, while the tester calculates the current total coverage in real time through the get_bb_cnt function. The code in the figure is given as a simple implementation and is intended to illustrate how to update and calculate code coverage data as simply as possible. The shared code coverage analysis library implements the code coverage tracking feature, so it maintains a huge shared memory block named mdata to store code coverage information. mdata is a pointer with the smallest byte unit, but each of its bits corresponds to the execution of a unique basic block in the deep learning framework, and initially the default values ​​of these basic blocks are all 0. When a basic block (whether in Python or C) is executed, it calculates the position in mdata based on the hash code written when it was instrumented, and sets the bit of that position to 1.

[0100] However, such a process may be applicable to the traditional software testing field, as the test objects in traditional testing work are often small programs with single processing and single thread. However, due to data and computational parallelism, deep learning frameworks tend to run multiple threads or even multiple processes at the same time, so the above code must be exposed to huge competition risks. To solve this problem, we will Figure 6 The original dcov_trace.cpp in was improved to Figure 8 As shown. The code execution process is as follows: Lines 9-10 perform atomic reads on mdata and compute to generate a new value. Line 12 performs an atomic compare and exchange operation, which swaps the value of mdata+idx with the value of data, returns false if the two values ​​are not equal, and assigns the value in new_byte to the byte of mdata+idx, and returns true if the two values ​​are equal. The new algorithm ensures that new_byte is only updated when there is no competition on mdata+idx.

[0101] Embodiment 3:

[0102] The embodiment of the present invention provides a specific example of the code coverage analysis method in Embodiment 2.

[0103] In this embodiment, the software tool integrating the method in Embodiment 2 is deployed on a host and runs.

[0104] The code coverage analysis method in this embodiment has strong scalability and theoretically supports compatibility with most existing deep learning framework testing tools. FreeFuzz is a dynamic vulnerability mining method for deep learning software frameworks, focusing on the security of deep learning software libraries. It has a unique dynamic execution and input mutation mechanism, and can build model inputs and API inputs that conform to the syntax specifications of specific programming languages, thereby triggering potential defects in deep learning libraries. At the same time, FreeFuzz supports a variety of deep learning frameworks, such as TensorFlow, PyTorch, etc., and can run on different operating systems and hardware platforms. The following will describe in detail the process of using FreeFuzz to test TensorFlow in the present invention, and then obtaining code coverage and analyzing it.

[0105] (1) According to the official documentation provided by FreeFuzz, first collect API call specification data and then perform instrumentation.

[0106] (2) Python dynamic instrumentation: Obtain the gray box test coverage information of TensorFlow based on the API name under test and the interpreter dynamic instrumentation information.

[0107] C static instrumentation: Use interface-based instrumentation, using the intermediate representation GIMPLE used by GCC, and perform instrumentation in GIMPLE during compilation.

[0108] (3) Configure the test parameters for TensorFlow and start the fuzzer: Create a configuration file to specify the test parameters, including database connection, output settings, mutation parameters, etc.; configure database connection information to store and retrieve data during testing; specify output configuration, including the save path of test results, report format, etc.; configure mutation parameters to determine the test case generation method and mutation strategy.

[0109] (4) Generate API call scripts: Based on the API information in the database, multiple API call scripts are automatically generated and added to the seed queue as the initial input for the test.

[0110] (5) Code coverage analysis module algorithm: Based on OpenMP instructions, it implements compilation-level concurrent optimization and introduces the hacker_popcount algorithm to accelerate the bit statistics in the coverage bitmap and reduce time overhead.

[0111] (6) Coverage information call and analysis: The statistical coverage information will guide the test work and draw the current experimental conclusions. When the coverage is further improved, the analyzer will feedback the coverage data and result analysis to the R&D personnel, which can be used for the next step of guidance.

[0112] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0113] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A code coverage analysis system for deep learning frameworks, characterized in that: include: The instrumentation module is used to instrument the deep learning framework and obtain coverage information using the corresponding instrumentation method for the programming language of the deep learning framework; The code coverage analysis module is used to call Bitmap statistics and record information according to user needs based on the coverage information obtained by instrumentation, and then call the improved shared algorithm library according to the Bitmap statistical information, and combine the coverage granularity optimized according to the Bitmap recording information to complete the recording, statistics and call analysis of the coverage information.

2. The code coverage analysis system for deep learning framework according to claim 1, characterized in that: The plugging mode of the plugging module is an interface plugging module; The plugging method of the plugging module includes a runtime bytecode plugging method for a programming language that is C code, and a compile-time plugging method for a programming language that is Python code.

3. The code coverage analysis system for deep learning framework according to claim 1, characterized in that: The code coverage analysis module includes: Bitmap recording unit, used to record the coverage information of the current code block when the stub code is executed; Bitmap statistics unit, used to count all coverage information obtained by testing the deep learning framework that uses the tester to complete the instrumentation process; The coverage granularity optimization unit is used to analyze the overhead indicators of different coverage granularities according to the Bitmap record information, and obtain the optimized coverage granularity for coverage analysis; Shared algorithm library for counting code coverage using concurrent optimization and hacker_popcount algorithm based on Bitmap statistics; The shared code coverage analysis library is used to track code coverage information through function calls and calculate the total code coverage in the shared algorithm library in real time to implement code coverage analysis.

4. The code coverage analysis system for deep learning framework according to claim 3, characterized in that: In the coverage granularity optimization unit, different coverage granularities include row coverage, branch coverage, block coverage, edge coverage and path coverage; The overhead indicators include the time required for instrumentation, execution overhead, and analysis overhead; The optimized coverage granularity is block coverage.

5. A code coverage analysis method based on the code coverage analysis system for deep learning framework according to any one of claims 1 to 4, characterized in that: The following steps are involved: S1. According to the programming language of the deep learning framework, the deep learning framework is instrumented by using the instrumentation module using the interface instrumentation mode, and the instrumented framework code is packaged and integrated back into the deep learning framework, thereby recording the code coverage information; S2. Test the deep learning framework after the instrumentation process through the tester, and use Bitmap to count the test information; S3, record the stub code after the deep learning framework is plugged in through Bitmap; S4, analyzing the overhead indicators of different coverage granularities in the Bitmap record information through the coverage granularity optimization unit, and determining the optimized coverage granularity; S5. Code coverage counting based on the shared algorithm library that introduces concurrency optimization and hacker_popcount algorithm; S6. In the shared code coverage analysis library, the optimization coverage granularity is tracked through function calls, and the total code coverage in the shared algorithm library is calculated in real time to implement code coverage analysis.

6. The code coverage analysis method according to claim 4, characterized in that: In step S1, the programming language of the deep learning framework includes C code and Python code; When the programming language is C code, the runtime bytecode instrumentation method is used to instrument the deep learning framework; When the programming language is Python code, the deep learning framework is instrumented using the compile-time instrumentation method.

7. The code coverage analysis method according to claim 5, characterized in that: In step S4, the coverage granularity is optimized to be block coverage.

8. The code coverage analysis method according to claim 5, characterized in that: In step S5, in the shared algorithm library, OpenMP instructions are introduced to implement compilation-level concurrent optimization; at the same time, the hacker_popcount algorithm is introduced to accelerate the bit statistics in the code coverage bitmap.

9. The code coverage analysis method according to claim 5, characterized in that: In the step S6, in the shared code coverage analysis library, a pointer named mdata is set and maintained as a shared memory block to store code coverage information.

Images