Deep learning compiler fuzzing method and system based on large language model

Abstract

The application provides a large language model-based deep learning compiler fuzzing method and system, the method comprising: generating multiple code segments capable of calling one or more specified deep learning APIs based on a large language model to form an initial code seed set for fuzz testing; extracting parameter passing constraints according to the documents of the specified deep learning APIs to obtain API constraint files; mutating the initial code seeds based on the API constraint files to obtain test code segments; and obtaining the test results of the deep learning compiler under test by comparing the compilation results of the test code segments in the deep learning compiler under test and a reference compiler. The application can find vulnerabilities in the deep learning compiler, ensure that the semantics of the mutated deep learning model are the same as those of the original model, and thus guarantee the correctness of downstream application programs.

Description

Technical Field

[0001] This invention belongs to the field of computer technology and relates to a fuzzing method and system for deep learning compilers based on large language models. Background Technology

[0002] Over the past decade, deep learning (DL) systems have been widely applied in various fields such as natural language processing, healthcare, activity recognition, and autonomous driving, bringing efficiency and convenience to our lives. However, ensuring the correctness of deep learning systems is extremely challenging due to their inherent uncertainties. Errors in deep learning systems can have serious consequences, even threatening human lives. In past research, researchers have explored various techniques to test, analyze, and validate deep learning models, as their quality directly affects the behavior of the corresponding systems. Recently, researchers have also proposed new techniques for testing low-level operator-level deep learning libraries (such as TensorFlow and PyTorch). These libraries provide general binary implementations for each high-level deep learning operator, forming the basis for running deep learning models on different hardware platforms. However, work on the reliability of emerging deep learning compilers (also known as tensor compilers) remains limited. Their goal is to automatically compile high-level tensor computation graphs directly into high-performance binaries, achieving better efficiency, portability, and scalability than traditional operator-level libraries.

[0003] Currently, while some existing fuzzing techniques can be used to test deep learning compilers, they fail to adequately resolve complex compiler infrastructures and are too coarse-grained to generate lightweight yet valuable inputs. They can only uncover very shallow front-end vulnerabilities, and their code coverage growth converges early on. Furthermore, traditional fuzzing techniques struggle to effectively handle such a challenging domain because the input deep learning program must simultaneously satisfy the syntax / semantics of the input language (e.g., Python) and the input / shape constraints of deep learning APIs for tensor computation. Summary of the Invention

[0004] The purpose of this invention is to provide a fuzz testing method and system for deep learning compilers based on a large language model. This method generates deep learning code by using a large language model, extracts API parameter constraints from the deep learning API documentation, uses the constraints to mutate the generated code, and performs differential testing. This can effectively assist security analysts in discovering vulnerabilities in deep learning compilers, improve the accuracy of vulnerability detection by security personnel, and ensure the security and reliability of downstream deep learning applications.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A fuzzing method for a deep learning compiler based on a large language model includes:

[0007] Multiple code snippets capable of calling one or more specified deep learning APIs are generated based on a large language model to form the initial code seed set for fuzz testing;

[0008] Extract the constraints of the parameters passed according to the documentation of the specified deep learning API to obtain the API constraint file;

[0009] The initial code seed is mutated based on the API constraint file to obtain the test code snippet;

[0010] The test results of the deep learning compiler under test are obtained by comparing the compilation results of the test code snippets with those of the reference compiler.

[0011] Furthermore, the generation of multiple code snippets calling specified deep learning APIs based on the large language model to form the initial code seed set for fuzzing includes:

[0012] Use an HTML crawler to automatically retrieve the API signature from the API documentation.

[0013] By combining two different specified deep learning libraries, Tensorflow and PyTorch, a prompt is constructed that can call one or more specified deep learning APIs; wherein the prompt includes the specified deep learning library and the API signature of the specified deep learning library.

[0014] The Codex model is guided to generate code by providing step-by-step prompts, resulting in multiple code snippets that call specified deep learning APIs.

[0015] Furthermore, the tasks corresponding to the prompt include: importing deep learning libraries, generating input data, and calling one or more specified deep learning APIs.

[0016] Furthermore, the loss function for training the Codex model Where n represents the token training sequence T = {t1, t2, ..., t} n The length of}, t i T represents the i-th token. <i Let P represent the sequence of tokens generated so far, and let P be the Codex model outputting the token generation probability.

[0017] Furthermore, the step of extracting parameter constraints based on the documentation of the specified deep learning API to obtain an API constraint file includes:

[0018] Obtain the constraint documentation from the open-source documentation repository of the deep learning library;

[0019] For the annotation section of the constraint document, after deep learning API classification using tags in natural language text blocks, the parameter attribute description text is extracted according to the indentation specification of the text blocks to obtain the API constraint preprocessing file;

[0020] After normalizing the API constraint preprocessing file into a dependency tree based on the Stanford parsing library, the dependency tree is then divided into subtrees;

[0021] The optimal mapping between subtrees and parameter constraints is identified by designing optimization problems to obtain constraint extraction rules;

[0022] The constraint extraction rules are filtered according to a preset threshold, and the frequently obtained tree constraint set is used as the constraint extraction rule with higher confidence.

[0023] When extracting constraints for a specified deep learning API, a parse tree of the specified deep learning API is generated. The tree pattern in the constraint extraction rules with high confidence is matched with the parse tree of the specified deep learning API. The matching result is mapped to the corresponding abstract constraint and instantiated in conjunction with the context described by the specified deep learning API to obtain the API constraint file of the specified deep learning API.

[0024] Furthermore, the API constraint file includes data structure constraints, data type constraints, parameter shape constraints, valid value range constraints, and context information constraints for specifying the deep learning API.

[0025] Furthermore, the mutation includes: constraint-compliant mutation and constraint-violation mutation; wherein, the constraint-compliant mutation refers to all mutation parameters satisfying the extracted structure, data type, shape, and valid value constraints, and the constraint-violation mutation refers to randomly selecting a parameter and violating the value of one or more related constraints of that parameter for mutation.

[0026] Furthermore, the initial code seed is mutated based on the API constraint file to obtain a test code snippet, including:

[0027] The initial code seed is mutated maxIter times, and one mutation is performed in each Iteration iteration.

[0028] When performing a constraint-compliant mutation, if a specific value is specified in the API constraint file, the mutation is selected from that specific value; otherwise, a dtype is selected from the list of dtypes specified in the API constraint file, and a shape is created based on the API constraint file.

[0029] When performing constraint violation mutation, a parameter is randomly selected and its value is mutated by violating one or more related constraints. For parameters other than the parameter that violates the constraint mutation, constraint-compliant mutation or constraint-violation mutation can be performed in subsequent iterations.

[0030] Fuzz testing is performed on the mutated code snippet. A fitness function is designed based on the code coverage of the deep learning compiler under test. The fitness score of the mutated code snippet is then used as the probability of subsequent mutation selection through a softmax operation.

[0031] Furthermore, the step of obtaining the test results of the deep learning compiler under test by comparing the compilation results of the test code snippets with those of the reference compiler includes:

[0032] Run the test code snippet on the deep learning compiler under test to obtain the first running result;

[0033] Run the test code snippet on the reference compiler to obtain the second result;

[0034] Calculate the difference between the first and second running results, and if the difference is greater than a threshold, proceed to the manual review stage; otherwise, determine that the test code snippet has not found any vulnerabilities in the deep learning compiler under test.

[0035] During the manual review phase, the specific code locations that cause significant differences in the running results are analyzed. If it is determined that there is a bug in the deep learning compiler under test, it is reported to the developer of the deep learning compiler under test through Github. If the developer accepts it, it is considered that a vulnerability has been found in the deep learning compiler under test.

[0036] Track the code coverage of the deep learning compiler under test and the number of bugs found.

[0037] A deep learning compiler fuzzing system based on a large language model includes:

[0038] The code generation module is used to generate multiple code snippets based on a large language model that can call one or more specified deep learning APIs, in order to form the initial code seed set for fuzz testing;

[0039] The constraint extraction module is used to extract the constraints passed to the specified deep learning API based on the documentation, and obtain the API constraint file.

[0040] The mutation module is used to mutate the initial code seed based on the API constraint file to obtain test code snippets.

[0041] The differential testing module is used to obtain the test results of the deep learning compiler under test by comparing the compilation results of the test code snippets with those of the reference compiler.

[0042] Compared to existing technologies, this invention is the first to apply large language models and constraint extraction techniques to the seed generation and mutation stages of fuzz testing for deep learning compilers. Specifically, firstly, it utilizes the rich API syntax / corpus information contained in the large language model and its powerful generation capabilities to generate diverse and accurate input models for the deep learning compiler. Secondly, it uses constraint extraction techniques to assist the mutation process, further enhancing the diversity and effectiveness of the generated models, ensuring that the deep code logic of the deep learning compiler can be tested, and achieving higher code coverage. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of a deep learning compiler fuzzing method and system framework based on a large language model.

[0044] Figure 2 This is a flowchart of the code generation module generating the initial seed set.

[0045] Figure 3 This is a flowchart illustrating the process by which the constraint extraction module extracts API constraint files from deep learning library documentation.

[0046] Figure 4 This is a flowchart of how the mutation module mutates the generated code according to constraints.

[0047] Figure 5 This is a flowchart illustrating how the differential testing module performs differential testing on the mutated code snippet between the deep learning compiler under test and the reference compiler, and discovers bugs. Detailed Implementation

[0048] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0049] To address the limitations and shortcomings of existing fuzzing methods for deep learning compilers, this invention proposes a fuzzing method for deep learning compilers based on large language models. This invention leverages the conclusion that modern large language models can contain numerous code snippets calling deep learning library APIs within their training corpora, thereby implicitly learning language syntax / semantics and complex deep learning API constraints to generate effective deep learning programs. Furthermore, by utilizing constraints extracted from official API documentation, the mutation process of fuzzing can be optimized, generating a more effective seed library for fuzzing. Compared to existing fuzzing methods for deep learning compilers, this invention achieves higher code coverage and vulnerability detection effectiveness, thus contributing to the development of deep learning compilers.

[0050] This embodiment is based on a deep learning compiler fuzzing system with a large language model, and the overall system framework is as follows: Figure 1 As shown, it mainly includes the following modules:

[0051] 1) Code generation module.

[0052] This invention uses a Large Language Model (LLM) for code generation, generating multiple code snippets that call one or more specified deep learning APIs. These snippets form the initial seed set for fuzzing. To generate the initial seed set for the target deep learning API, this invention first queries the Codex model using a step-by-step prompt approach and performs multiple sampling completion operations. When using Codex for code generation, prompts that can call one or more specified deep learning prompts are constructed to guide Codex in code generation, thus providing code snippets that meet the requirements. These generated code snippets serve as the initial seed set for fuzzing. Specifically, the prompts designed for generating the initial seed set using Codex include three parts: Task 1: Importing the deep learning library; Task 2: Generating input data; and Task 3: Calling one or more specified APIs. The designed prompts guide the model to perform these three tasks accordingly.

[0053] The code generation module uses LLM to generate the initial seed set as follows: Figure 2 As shown, the specific explanation is as follows:

[0054] 1a) Use an HTML crawler to automatically obtain the API signature from the API documentation (e.g., tf.nn.conv2d(input,filters,...)), go to 1b).

[0055] 1b) Construct a prompt that can call one or more specified deep learning APIs to guide Codex in code generation, wrapping the task description in docstring form. The prompt includes the target library (e.g., TensorFlow) and the target API signature (e.g., tf.nn.conv2d(input, filter, ...)), see 1c).

[0056] 1c) Design a step-by-step instruction (e.g., Task 1:…, Task 2:…, Task 3:…) to improve the performance of the Codex model. More precisely, guide the model to perform three tasks sequentially: (1) import the target library; (2) generate input data; (3) call the target API. An example instruction could be Task 1: importTensorFlow 2.10.0, Task 2: Generate input data, Task 3: Call the APItf.as_dtype(type_value), go to 1d).

[0057] 1d) Codex uses causal language modeling for training, where the model aims to predict the next token using all previous generations of token sequences. Given a token training sequence T = {t1, t2, ..., t...} n Let T <m ={t1,t2,…,t m-1 Let} represent the sequence of tokens generated so far (m≤n), and P be the Codex model that outputs the token generation probability. The Codex loss function is defined as: Use the prompts constructed in 1c) as input to the Codex model to generate the initial seed set, then proceed to 1e).

[0058] 1e) The final generated code snippet will look like the following:

[0059] #Task 1:import TensorFlow 2.10.0

[0060] import tensorflow as tf

[0061] #Task 2: Generate input data

[0062] data = [1, 2, 3, 4, 5]

[0063] #Task 3:Call the API tf.as_dtype(type_value)

[0064] dtype = tf.as_dtype(data)

[0065] print(dtype)

[0066] 2) Constraint Extraction Module.

[0067] The constraint extraction module extracts constraints from the documentation of deep learning APIs. First, it categorizes APIs using tags from the natural language text blocks of the annotation documentation. Then, it extracts the descriptions of each parameter attribute based on the indentation specifications of the text blocks, resulting in an API constraint preprocessing file. Next, a condition transformer is built, employing a combination of natural language processing techniques and pattern matching. Using the Stanford Parser Library (SPL), the constraint preprocessing file is normalized into a dependency tree (DT), which is then segmented into subtrees. For a small subset of API dependency trees and corresponding manually annotated abstract constraints (ACs), an optimal mapping between the subtrees and parameter constraints—the constraint extraction rules—is designed, and a threshold is used for confidence filtering. In the specific constraint extraction stage, given an API description, the constraint extractor matches the tree patterns in the rules with the parse tree of that description. The match is then mapped to the corresponding AC and further instantiated within the context of the description to derive the specific constraints.

[0068] Specifically, the constraint extraction module's process for extracting API constraint files from deep learning library documentation is as follows: Figure 3 As shown, it includes:

[0069] 2a) Download the constraint document (DL doc) to be extracted from the open-source documentation repository of the deep learning library. The annotation part of the DLdoc is a semi-structured natural language text block. Then, in the natural language text block, use the tags introduced by "@" (e.g., @tf_export, @dispatch, @deprecation) to divide the document structure and obtain the text block corresponding to each API. Proceed to 2b).

[0070] 2b) Analyze the text block corresponding to a single API, and extract the description of parameter types, return types, and abnormal behaviors according to the indentation specifications. This description is called the API description. i exc, where the index i is the API to be extracted, go to 2c).

[0071] 2c) Design a conditional translator that combines NLP techniques with pattern matching. Use the Stanford Parser Library (SPL) to translate the API. i The exc function performs preprocessing tasks such as text cleaning, terminology standardization, and text conversion, and then converts the API. i The expression `exc` is transformed into a dependency tree (DT), and then the dependency tree is decomposed into subtrees. (Go to 2d).

[0072] 2d) For a small subset of APIs' dependency trees and corresponding manually labeled abstract constraints (AC), design the optimal mapping between the optimization problem identification subtree and the parameter constraints, i.e., the constraint extraction rule DT→AC. (Go to 2e).

[0073] 2e) Filter the extracted rule mappings according to the preset threshold min_support to obtain the frequent tree constraint set FreqD f These are constraint extraction rules with high confidence. (Go to 2f).

[0074] 2f) In the concrete constraint extraction stage, given an API description, the tree patterns in the constraint extraction rules with high confidence are matched against the parse tree of that description. The matches are then mapped to the corresponding ACs and further instantiated within the context of the description to derive the concrete constraints and generate the constraint file.

[0075] 3) Mutation module.

[0076] The mutation module uses the extracted API parameter constraints to mutate the generated code, uses the mutated code as input to the deep learning compiler for fuzzing, and adds interesting seeds that help discover vulnerabilities to the seed set.

[0077] In other words, the initial code seed set generated by Codex is mutated based on the extracted API constraints. Mutation types include constraining mutations (CM) and constraining violating mutations (VM). Constraining mutations are designed to test the core functionality of the compiler under test; all mutation parameters satisfy the extracted structure, data type, shape, and valid value constraints. Constraining violating mutations are designed to test the compiler under test's input validity checking code; a parameter is randomly selected and its value violates one or more related constraints. The proposed method performs maxIter mutations on each initial code seed, where the proportion allocated to each mutation type (CM and VM) is determined by the confirm_ratio. The mutated code is then used as input to a deep learning compiler for fuzz testing, and interesting seeds that help discover bugs are added to the seed set.

[0078] Specifically, the process of mutating the generated code according to constraints is as follows: Figure 4 As shown, it includes:

[0079] 3a) Perform maxIter mutations on each initial code seed. Mutation is performed once in each Iteration. Given the extracted constraints, this method performs mutations in the order determined by the constraint dependencies. If the mutation performed is a constraint-compliant mutation, proceed to 3b); if the mutation performed is a constraint-violation mutation, proceed to 3e).

[0080] 3b) For a mutator that meets the constraints, all parameters of the mutator satisfy the extracted structure, data type, shape, and valid value constraints, i.e., structure, dtype, shape, and valid value. If specific (e.g., enumerated values) values ​​are specified in the constraints, the mutator selects from these values ​​and proceeds to 3c); otherwise, it selects a dtype from the list of dtypes specified in the constraints and creates a shape according to the constraints, proceeding to 3d).

[0081] If a specific value (e.g., an enumerated value) is specified in the constraint, the mutator randomly selects one of these values ​​as the mutated parameter value and proceeds to 3f).

[0082] If no specific value (e.g., an enumerated value) is specified in the constraint (3d), the mutator selects a dtype from the list of dtypes specified in the constraint, creates a shape based on the constraint, and proceeds to 3f).

[0083] 3e) For constraint-violation mutations, randomly select a parameter and mutate it by violating one or more related constraints. Constraint violation specifically refers to a constraint requiring a parameter's dtype to be either dtype1 or dtype2. For the generated code, mutate the parameter's dtype to a dtype other than dtype1 or dtype2 in the dtype list. For constraint-violation mutations, for parameters other than the one violating the constraint, subsequent iterations can also perform constraint-compliant or constraint-violation mutations to further expand the number of code snippets under test. (Go to 3f).

[0084] 3f) Perform fuzz testing on the mutated code snippets, design a fitness function based on the code coverage of the compiler under test, and use the softmax operation based on the fitness score of the code snippets as the probability of subsequent mutation selection. Add interesting seeds that are helpful in discovering vulnerabilities to the seed set.

[0085] 4) Differential test module.

[0086] The mutated code snippets were subjected to differential testing on the test deep learning compiler and a reference compiler. The compilation results of the generated code snippets were compared between the test and reference compilers to further identify security vulnerabilities in the test compiler. During differential testing, code snippets calling Tensorflow / PyTorch APIs were generated and their compilation results were compared between the test compiler and Tensorflow / PyTorch. This comparison, along with manual review, determined whether the test compiler contained vulnerabilities.

[0087] Specifically, the process for conducting differential testing and discovering bugs is as follows: Figure 5 As shown, it includes:

[0088] 4a) Run the modified PyTorch / Tensorflow deep learning code snippet on the deep learning compiler under test (e.g., Apache TVM, ONNX) and the reference compiler (PyTorch / Tensorflow), obtain the results, and then proceed to 4b).

[0089] 4b) Design a function to calculate the difference between the running result of the deep learning compiler under test and the running result of the reference compiler. If the difference is greater than the threshold, it will enter the manual review stage and proceed to 4c). Otherwise, it will be determined that the seed has not found any vulnerabilities in the compiler under test.

[0090] 4c) Analyze the specific code location that causes the significant difference in the running results. If it is determined that there is a bug in the compiler under test, report it to the compiler developer via Github. If the developer accepts the report, it can be considered that a vulnerability in the compiler under test has been discovered, and proceed to 4d).

[0091] 4d) Track the code coverage and the number of bugs found in the compiler under test.

[0092] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A fuzzing method for a deep learning compiler based on a large language model, characterized in that, The method includes: Multiple code snippets capable of calling one or more specified deep learning APIs are generated based on a large language model to form the initial code seed set for fuzz testing; Extract the constraints of the parameters passed according to the documentation of the specified deep learning API to obtain the API constraint file; The initial code seed is mutated based on the API constraint file to obtain a test code snippet; The test results of the deep learning compiler under test are obtained by comparing the compilation results of the test code snippets with those of the reference compiler. This involves extracting parameter constraints based on the documentation of a specified deep learning API to obtain an API constraint file, including: Obtain the constraint documentation from the open-source documentation repository of the deep learning library; For the annotation section of the constraint document, after deep learning API classification using tags in natural language text blocks, the parameter attribute description text is extracted according to the indentation specification of the text blocks to obtain the API constraint preprocessing file; After normalizing the API constraint preprocessing file into a dependency tree based on the Stanford parsing library, the dependency tree is then divided into subtrees; The optimal mapping between subtrees and parameter constraints is identified by designing optimization problems to obtain constraint extraction rules; The constraint extraction rules are filtered according to a preset threshold, and the frequently obtained tree constraint set is used as the constraint extraction rule with higher confidence. When extracting constraints for a specified deep learning API, a parse tree of the specified deep learning API is generated. The tree pattern in the constraint extraction rule with high confidence is matched with the parse tree of the specified deep learning API. The matching result is mapped to the corresponding abstract constraint and instantiated in combination with the context of the specified deep learning API description to obtain the API constraint file of the specified deep learning API. The mutations include: constraint-compliant mutations and constraint-violation mutations. Constraint-compliant mutations refer to mutations in which all parameters of the mutations satisfy the extracted structure, data type, shape, and valid value constraints. Constraint-violation mutations refer to mutations in which a parameter is randomly selected and its value violates one or more related constraints. The step of mutating the initial code seed based on the API constraint file to obtain the test code snippet includes: Perform initial code seed The mutation occurs once in each Iteration iteration. When performing a constraint-compliant mutation, if a specific value is specified in the API constraint file, the mutation is selected from that specific value; otherwise, a dtype is selected from the list of dtypes specified in the API constraint file, and a shape is created based on the API constraint file. When performing constraint violation mutation, a parameter is randomly selected and its value is mutated by violating one or more related constraints. For parameters other than the parameter that violates the constraint mutation, constraint-compliant mutation or constraint-violation mutation can be performed in subsequent iterations. Fuzz testing is performed on the mutated code snippet. A fitness function is designed based on the code coverage of the deep learning compiler under test. The fitness score of the mutated code snippet is then used as the probability of subsequent mutation selection through a softmax operation.

2. The method according to claim 1, characterized in that, The generation of multiple code snippets that call specified deep learning APIs based on a large language model to form the initial code seed set for fuzzing includes: Use an HTML crawler to automatically retrieve the API signature from the API documentation. By combining two different specified deep learning libraries, Tensorflow and PyTorch, a prompt is constructed that can call one or more specified deep learning APIs; wherein the prompt includes the specified deep learning library and the API signature of the specified deep learning library. The Codex model is guided to generate code by providing step-by-step prompts, resulting in multiple code snippets that call specified deep learning APIs.

3. The method according to claim 2, characterized in that, The tasks corresponding to the prompt include: importing deep learning libraries, generating input data, and calling one or more specified deep learning APIs.

4. The method according to claim 2, characterized in that, Loss function for training the Codex model ;in, Represents the token training sequence Length, Indicates the first One token, This represents the sequence of tokens generated so far. The Codex model outputs the probability of token generation.

5. The method according to claim 1, characterized in that, The API constraint file contains data structure constraints, data type constraints, parameter shape constraints, valid value range constraints, and context information constraints for specifying the deep learning API.

6. The method according to claim 1, characterized in that, The test results of the deep learning compiler under test are obtained by comparing the compilation results of the test code snippets with those of the reference compiler. These results include: Run the test code snippet on the deep learning compiler under test to obtain the first running result; Run the test code snippet on the reference compiler to obtain the second result; Calculate the difference between the first and second running results, and if the difference is greater than a threshold, proceed to the manual review stage; otherwise, determine that the test code snippet has not found any vulnerabilities in the deep learning compiler under test. During the manual review phase, the specific code locations that cause significant differences in the running results are analyzed. If it is determined that there is a bug in the deep learning compiler under test, it is reported to the developer of the deep learning compiler under test through Github. If the developer accepts it, it is considered that a vulnerability has been found in the deep learning compiler under test. Track the code coverage of the deep learning compiler under test and the number of bugs found.

7. A fuzz testing system for a deep learning compiler based on a large language model, characterized in that, The system includes: The code generation module is used to generate multiple code snippets based on a large language model that can call one or more specified deep learning APIs, in order to form the initial code seed set for fuzz testing; The constraint extraction module is used to extract the constraints passed to the specified deep learning API based on the documentation, and obtain the API constraint file. The mutation module is used to mutate the initial code seed based on the API constraint file to obtain test code snippets. The differential testing module is used to obtain the test results of the deep learning compiler under test by comparing the compilation results of the test code snippets with those of the reference compiler. This involves extracting parameter constraints based on the documentation of a specified deep learning API to obtain an API constraint file, including: Obtain the constraint documentation from the open-source documentation repository of the deep learning library; For the annotation section of the constraint document, after deep learning API classification using tags in natural language text blocks, the parameter attribute description text is extracted according to the indentation specification of the text blocks to obtain the API constraint preprocessing file; After normalizing the API constraint preprocessing file into a dependency tree based on the Stanford parsing library, the dependency tree is then divided into subtrees; The optimal mapping between subtrees and parameter constraints is identified by designing optimization problems to obtain constraint extraction rules; The constraint extraction rules are filtered according to a preset threshold, and the frequently obtained tree constraint set is used as the constraint extraction rule with higher confidence. When extracting constraints for a specified deep learning API, a parse tree of the specified deep learning API is generated. The tree pattern in the constraint extraction rule with high confidence is matched with the parse tree of the specified deep learning API. The matching result is mapped to the corresponding abstract constraint and instantiated in combination with the context of the specified deep learning API description to obtain the API constraint file of the specified deep learning API. The mutations include: constraint-compliant mutations and constraint-violation mutations. Constraint-compliant mutations refer to mutations in which all parameters of the mutations satisfy the extracted structure, data type, shape, and valid value constraints. Constraint-violation mutations refer to mutations in which a parameter is randomly selected and its value violates one or more related constraints. The step of mutating the initial code seed based on the API constraint file to obtain the test code snippet includes: Perform initial code seed The mutation occurs once in each Iteration iteration. When performing a constraint-compliant mutation, if a specific value is specified in the API constraint file, the mutation is selected from that specific value; otherwise, a dtype is selected from the list of dtypes specified in the API constraint file, and a shape is created based on the API constraint file. When performing constraint violation mutation, a parameter is randomly selected and its value is mutated by violating one or more related constraints. For parameters other than the parameter that violates the constraint mutation, constraint-compliant mutation or constraint-violation mutation can be performed in subsequent iterations. Fuzz testing is performed on the mutated code snippet. A fitness function is designed based on the code coverage of the deep learning compiler under test. The fitness score of the mutated code snippet is then used as the probability of subsequent mutation selection through a softmax operation.

Images