Multi-language compiler fuzz test input generation method based on large language model

Abstract

The invention discloses a multi-language compiler fuzzy test input generation method based on a large language model, and belongs to the technical field of software testing and compiler verification. According to the method, the illusion problem when a large language model processes a long text is relieved by introducing a thinking chain prompt method, the long text input by a user is optimized, a plurality of candidate prompts are generated, and the optimal prompt is automatically screened out by utilizing a scoring function; and diversified fuzzy test input code snippets are generated in combination with a loop iteration framework and four generation strategies (creation, variation, semantic transformation and compilation options). The problem that the cross-language expansion difficulty of a traditional fuzzy test technology is large is solved, test input generation of the multi-language compiler can be supported, meanwhile, the diversity of test input and the code coverage rate are improved, and an efficient and flexible technical means is provided for reliability verification of the compiler.

Description

Technical Field

[0001] This invention relates to the implementation of a method for generating fuzzy test inputs for a multilingual compiler based on a large language model, belonging to the field of software testing and compiler verification technology. Background Technology

[0002] Fuzzing is a testing method that monitors the behavior of a target system by inputting a large amount of random, anomalous, invalid, or unexpected data to discover potential software vulnerabilities. Specifically, fuzzing utilizes program analysis techniques such as taint analysis, instrumentation, and symbolic execution to collect information about the target system. Depending on its dependence on the target system's internal information, it employs different testing methods such as white-box, black-box, and gray-box testing. It automatically or semi-automatically generates a large number of expected or unexpected test cases based on seed generation strategies, and finally analyzes the output of the target system to detect security vulnerabilities. Compiler fuzzing is a commonly used and effective technique for testing the reliability and functionality of compilers. By generating syntactically correct test cases, the fuzzer can test the compiler's deep code. The compiler is a crucial component of the software development toolchain and a critical infrastructure for computer system software; its stability and correctness are paramount. If the compiler itself has problems, the correctness of the program cannot be guaranteed. All analyses and judgments of program correctness are based on the correctness of the compiler itself. In recent years, compiler technology has continued to develop, and the total amount of code in production-level compilers has also been increasing. Currently, mainstream compiler verification techniques mainly include three methods: random testing, formal verification, and translation verification. However, applying these methods to production-level compilers with their massive codebases presents significant challenges. Therefore, fuzzing remains one of the primary techniques used in compiler testing. Against this backdrop, semantically aware gray-box compiler fuzzing, which introduces the ability to perceive program semantics, relies on specific language parsers and only supports C / C++ compilers. Furthermore, fuzzing methods for Just-In-Time (JIT) compilers in JavaScript engines are only applicable to specific compilers for specific languages ​​and cannot be applied to other languages ​​or even other versions of the same language.

[0003] To address this issue, this invention proposes a fuzzy test input generation method based on a large language model and a multilingual compiler. Specifically, firstly, to address the issue of large language models easily generating illusions when processing long texts, a thought chain-based suggestion method is used to optimize the large language model's processing of long user-input text, generating multiple candidate suggestions. Then, a scoring function is used to automatically select the best suggestion. Finally, a loop iterative framework and four generation strategies are employed to enable the code generation model to generate as diverse a fuzzy test input code fragment as possible. Summary of the Invention

[0004] The technical problem this invention aims to solve is that fuzzing, due to its independence from the internal structure of the system under test, is widely used in compiler fuzzing. However, existing compiler fuzzing techniques face significant challenges in cross-language extension. Therefore, this invention provides a multi-language compiler fuzzing input generation method based on a large language model to address this issue. In this invention, a thought chain-based suggestion method is used to mitigate the illusion problem caused by large language models by distilling long text input by the user, obtaining numerous candidate suggestions. These suggestions are then automatically filtered based on a scoring function, and a cyclic iterative framework and four generation strategies are used to generate more diverse fuzzing inputs.

[0005] This invention discloses a method for generating fuzzy test inputs for a multilingual compiler based on a large language model. The process is as follows: Figure 1 As shown, it mainly includes long text optimization processing, optimal suggestion selection, and iterative steps.

[0006] Step 1: Optimizing Long Texts to Alleviate Hallucinations The flowchart for the long text optimization process to alleviate hallucinations is as follows: Figure 2 As shown.

[0007] The text generation model mines actions based on a given set of questions and action mining prompt templates, and then formulates a corresponding plan using the mined actions according to instructions. After the user inputs long text into the text generation model, the model executes the pre-defined plan based on the text, distilling the user's input text to transform the text content into prompts that are more suitable for use as a fuzzy generator.

[0008] Step 2: Optimal suggestion selection based on scoring function and automated filtering The flowchart of the best suggestion selection based on scoring functions and automated filtering is as follows: Figure 3 As shown.

[0009] The candidate prompts are fed one by one into the code generation model, which generates several fuzzy test input code snippets based on each prompt, and scores each candidate prompt according to a scoring function. The candidate prompt with the highest score is selected as the best prompt, also known as the base prompt.

[0010] Step 3: Generation of Fuzzy Test Input Based on Iterative Loop and Large Language Model The flowchart for the fuzzy test input generation part based on iterative loops and a large language model is as follows: Figure 4 As shown.

[0011] The basic hints are combined with the creation strategy and fed into the code generation model to generate several fuzzy test inputs, completing the first round of iteration. In each subsequent round of iteration, a valid fuzzy test input generated in the previous round is randomly selected as an example. Then, one of four generation strategies—creation, mutation, semantic transformation, and compilation options—is randomly selected and combined with the example. This example is then combined with the basic hints to generate new hints, which are fed into the code generation model to generate several fuzzy test inputs. This process is repeated for a user-specified number of times or until a user-specified duration is reached.

[0012] The advantages and beneficial effects of this invention are as follows: This invention, based on a multi-language compiler fuzzing input generation method using a large language model, effectively alleviates the illusion problem when large language models process long texts by introducing mind chain hint technology, significantly improving the accuracy and diversity of generated test inputs. This method utilizes long text optimization, automated hint filtering, and a loop iteration framework, combined with four generation strategies (creation, mutation, semantic transformation, and compilation options), to efficiently generate syntactically correct and widely covered test cases, solving the problem of the difficulty in cross-language expansion of traditional fuzzing techniques. Compared with existing technologies, this invention not only supports test input generation for multiple language compilers but also significantly improves code coverage through scoring functions and iterative optimization, providing an efficient, flexible, and automated technical means for compiler reliability verification, while reducing manual intervention and resource consumption. Experiments demonstrate that this method has significant advantages in discovering deep compiler vulnerabilities and improving testing efficiency. Attached Figure Description

[0013] Figure 1 This is a flowchart of a fuzzy test input generation method for a multilingual compiler based on a large language model.

[0014] Figure 2 Flowchart for optimizing long text processing to alleviate hallucinations.

[0015] Figure 3 A flowchart for selecting the best suggestion based on a scoring function and automated filtering.

[0016] Figure 4A flowchart for generating fuzzy test input based on iterative loops and a large language model.

[0017] Figure 5 This is a schematic diagram of the prompt structure for the action mining phase.

[0018] Figure 6 A schematic diagram illustrating the structure of the planning phase.

[0019] Figure 7 This is a schematic diagram illustrating the structure of the planning and execution phase. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This section uses a C++ compiler as an example to illustrate the implementation process of generating fuzzy test input using this invention. The specific steps are as follows: Step 1: Optimize long text and generate candidate suggestions This step can be divided into three stages: action discovery, plan formulation, and plan execution.

[0022] In the action mining phase, a manually compiled set of seed actions, instructions, and questions are combined to form an action mining prompt structure, which is then input into the text generation model to extract actions. The prompt structure in this phase is as follows: Figure 5 As shown. Here, an action refers to a basic unit in the long text processing process. Each action is formatted as a procedural function with input parameters, followed by the function definition generated by the natural language model. An example format of an action generated by a text generation model is as follows: FIND_DEFINITION(CTX, X) # Locates and extracts the definition or purpose of term X from the input text CTX. In the planning phase, the identified actions, user questions (summarizing the above information as comprehensively and concisely as possible, describing the language's usage and functions), and instructions are combined to form a planning prompt structure. This structure is then input into the text generation model to generate a plan. The prompt structure in this phase is as follows: Figure 6 As shown. A plan is a guiding framework or outline for answering complex questions, which may involve multi-step reasoning and require a global understanding of long documents. Given a question, the text generation model is prompted to generate a plan based on a previously mined set of actions. Each step of the plan is written in the following format: output = ACTION(arg1, arg2, ...), The `output` variable stores the result of the current action (ACTION), while the `arg` parameter can be the input document (CTX), a string, or the output variable of a previous step. Actions used in the plan must be actions generated in the action mining step, and the number of parameters for each action must match the number of previously mined corresponding actions. The generated plan needs to be validated. If the above conditions are not met, the generated plan is invalid and cannot be used, and an error will be automatically reported. An example of a generated plan is shown below:

[0023] During the plan execution phase, the user-provided long text input in C++ language, combined with previously mined action definitions, the current step of the ongoing plan, and parameter assignments, is fed into the text generation model as the structure input for plan execution. This allows the model to execute the plan step by step, ultimately generating numerous candidate hints. First, a temperature of 0 is set to generate a highly credible answer. Then, a temperature of 1 is set to generate several more diverse answers. These answers are all considered candidate hints. The hint structure for this phase is as follows: Figure 7 As shown.

[0024] Step Two: Best Suggestion Selection Each candidate suggestion generated in the previous step is fed into the code generation model, which generates an equal number of fuzzy test input code snippets for each suggestion. Next, the fuzzy test input code snippets generated for each candidate suggestion are scored using a scoring function. Based on the scores, the candidate suggestion with the highest score—that is, the one that generates the most unique fuzzy test input code snippets—is selected as the best suggestion.

[0025] The scoring function of this invention uses the proportion of unique and valid fuzzy test input code fragments generated as the evaluation index, which is the ratio of the number of non-repeating fuzzy test input code fragments that can be accepted by the C++ compiler of the software system under test to the total number of fuzzy test input code fragments generated.

[0026] The scoring function used in this invention can be expressed as:

[0027] in, Here, p is the code generation model, SUT is the C++ compiler of the software system under test, N is the total number of generated fuzzy test input code snippets, and isValid() is a function that determines whether the fuzzy test input code snippet C generated by the code generation model is valid. If the fuzzy test input code snippet C is valid, it returns 1; otherwise, it returns 0.

[0028] Step 3: Iterate through the loop to generate fuzzy test input for the C++ compiler. Before each iteration, one of the fuzzy test inputs generated in the previous round is randomly selected as an example and combined with the initial prompt and generation strategy to expand the initial prompt, generating new prompts and guiding the large language model to generate richer and more diverse code snippets. In the very first iteration, based on the previously selected best prompt and combined with the generation strategy, new prompt inputs are directly generated for the code generation model, generating several fuzzy test inputs.

[0029] The generation strategy used in the above process is a type of instruction. As an instruction, the generation strategy specifies how the code generation model should generate new code snippets based on the example code.

[0030] The generation strategies used in this invention include four types: creation, mutation, semantic transformation, and compilation options. Specifically, the creation strategy's instruction is "create a program that conforms to the language syntax for {user-given} compiler," where "{user-given}" is "C++"; the mutation strategy's instruction is "generate a mutated program that modifies the previously generated code"; the semantic transformation strategy's instruction is "create a different program that is semantically equivalent to the previously generated program"; and the compilation options strategy's instruction is "insert, change, or delete the #pragma compiler option."

[0031] The following are specific examples: In the action mining stage of generating candidate suggestions through long text optimization processing in step one, the manually compiled seed action set is as follows:

[0032] The manually compiled question set is as follows:

[0033] The instructions are as follows:

[0034] These inputs are combined into an action mining cue structure, which is then fed into the text generation model. The mined actions are as follows:

[0035]

[0036] During the planning phase, the user's questions were:

[0037] The instruction is:

[0038] The action set mined in the previous stage is combined with user questions and instructions to form a plan formulation prompt structure, which is then input into the text generation model to generate the following action plan:

[0039] During the planning and execution phase, the long text describing the C++ language provided by the user is as follows:

[0040]

[0041] This long text input is combined with the previously mined action definitions, the current step of the ongoing plan, and parameter assignments to form a prompt structure for plan execution. This structure is then fed into the text generation model, which executes the plan step by step, ultimately producing several candidate prompts as follows: High-confidence candidate prompt (greedy prompt):

[0042]

[0043] Diverse candidate prompt 0 (No.0 prompt):

[0044]

[0045] Diverse candidate prompt 1 (No.1 prompt)

[0046] Diverse candidate prompt 2 (No. 2 prompt)

[0047]

[0048] ... In this way, multiple candidate suggestions were obtained through step one.

[0049] In step two, optimal suggestion selection, each candidate suggestion generated in step one is input into the code generation model, causing it to generate 20 fuzzy test input code snippets for each candidate suggestion. Next, the fuzzy test input code snippets generated for each candidate suggestion are scored according to a scoring function as follows:

[0050] Based on the ratings, suggestion #1 received the highest score and was selected as the best suggestion.

[0051] In step three, which iteratively generates fuzzy test inputs for the C++ compiler, the fuzzy test input generated in the previous round based on the best hint as the initial hint is:

[0052] The above output, combined with semantic equivalent transformation instructions:

[0053] The newly generated fuzz test input is:

[0054] The above output combined with the mutation command:

[0055] The newly generated fuzz test input is:

[0056] The above output, combined with compiler option directives:

[0057] The newly generated fuzz test input is:

[0058]

[0059] The above output, combined with the instruction to create new code:

[0060] The newly generated fuzzy test input, which is completely different from the previous code, is:

[0061]

[0062] In this way, by iteratively selecting different instructions, the large language model is guided to continuously generate code snippets. These code snippets can be used as fuzzing inputs for the compiler. The fuzzing inputs generated through the above steps have diversity, improving the code coverage of the C++ compiler's fuzzing tests. The automatically generated test code snippets are as follows:

[0063] This caused the G++ compiler version 13.3.0 to fail to correctly instantiate function templates during recursion, leading to a crash. This revealed a new explicit template parameter bug in the compiler.

Claims

1. A method for generating fuzzy test inputs for a multilingual compiler based on a large language model, characterized in that, The steps include the following: Step 1: Optimizing Long Texts to Alleviate Hallucinations The text generation model mines actions based on a given set of questions and action mining prompt templates, and then formulates a corresponding plan based on the instructions to use the mined actions. After the user inputs long text into the text generation model, the text generation model executes the previously formulated plan based on the text, distills the text input by the user, and transforms the text content into prompts that are more suitable for use as a fuzzy generator. Step 2: Optimal suggestion selection based on scoring function and automated filtering The candidate prompts are fed into the code generation model one by one, which generates several fuzzy test input code snippets based on each prompt, and scores each candidate prompt according to the scoring function. The candidate suggestion with the highest score is selected as the best suggestion, which is the basic suggestion; Step 3: Generation of Fuzzy Test Input Based on Iterative Loop and Large Language Model The basic hints are combined with the creation strategy and fed into the code generation model to generate several fuzzy test inputs, completing the first round of iteration. In each subsequent round of iteration, a valid fuzzy test input generated in the previous round is randomly selected as an example. Then, one of the four generation strategies—creation, mutation, semantic transformation, and compilation options—is randomly selected and combined with the example. This is then combined with the basic hints to generate new hints, which are fed into the code generation model to generate several fuzzy test inputs. This process is repeated for the number of times specified by the user, or until the specified duration is reached.

2. The method for generating fuzzy test input based on a large language model for a multilingual compiler according to claim 1, characterized in that: Step one is divided into three stages: action discovery, plan formulation, and plan execution.

3. The method for generating fuzzy test input based on a large language model for a multilingual compiler according to claim 2, characterized in that: In the action mining phase, a manually compiled set of seed actions, instructions, and questions are combined into an action mining prompt structure, which is then input into the text generation model to extract actions. An action is a basic unit in long text processing, and each action is formatted as a procedural function with input parameters, followed by the function definition generated by the model in natural language. An example format of actions generated by the text generation model is shown below: FIND_DEFINITION(CTX, X) # Locate and extract the definition or purpose of term X in the input text CTX.

4. The method for generating fuzzy test input based on a large language model for a multilingual compiler according to claim 2, characterized in that: In the planning phase, the mined actions are combined with user questions and instructions to form a planning prompt structure, which is then input into the text generation model to generate a plan. The plan serves as a guiding framework or outline for answering complex questions that involve multi-step reasoning and require a global understanding of long documents. Given a question, the text generation model is prompted to generate a plan based on the previously mined action set. Each step of the plan is written in the following format: output = ACTION(arg1, arg2, ...), The `output` variable stores the result of the current action (ACTION), while the parameter `arg` can be the input document (CTX), a string, or the output variable of the previous step. Actions used in the plan must be actions generated in the action mining step, and the number of their respective parameters must be consistent with the corresponding actions mined previously. For the generated plan, it is necessary to determine whether it is valid. If the above conditions are not met, the generated plan is invalid and cannot be used, and an error will be automatically reported.

5. The method for generating fuzzy test input based on a large language model for a multilingual compiler according to claim 2, characterized in that: During the execution phase, the long text input provided by the user in C++ language is combined with the previously mined action definitions, the current steps of the plan being executed, and parameter assignments, and fed into the text generation model as the structure input for plan execution. This allows the model to execute the plan step by step, ultimately generating numerous candidate hints. First, the temperature is set to 0 to generate highly credible answers, and then the temperature is set to 1 to generate several more diverse answers. All of these answers are considered candidate hints.

6. The method for generating fuzzy test input based on a large language model for a multilingual compiler according to claim 1, characterized in that: In step two, the scoring function uses the percentage of unique and valid fuzzy test input code snippets generated as the evaluation metric, which is the ratio of the number of non-repeating fuzzy test input code snippets accepted by the C++ compiler of the software system under test to the total number of fuzzy test input code snippets generated.

7. The method for generating fuzzy test input based on a large language model for a multilingual compiler according to claim 6, characterized in that: The scoring function used is expressed as follows: ； in, Here, p is the code generation model, SUT is the C++ compiler of the software system under test, N is the total number of generated fuzzy test input code snippets, and isValid() is a function that determines whether the fuzzy test input code snippet C generated by the code generation model is valid. If the fuzzy test input code snippet C is valid, it returns 1; otherwise, it returns 0.

8. The method for generating fuzzy test input based on a large language model for a multilingual compiler according to claim 1, characterized in that: In step three, the generation strategy used is an instruction; as an instruction, the generation strategy specifies how the code generation model should generate new code snippets based on the example code.

9. The method for generating fuzzy test input based on a large language model for a multilingual compiler according to claim 1 or 8, characterized in that: There are four generation strategies: creation, mutation, semantic transformation, and compiler options. The creation strategy's instruction is "create a program that conforms to the language syntax for the {user-given} compiler," where "{user-given}" is "C++". The mutation strategy's instruction is "generate a mutated program that changes the code generated last time". The semantic transformation strategy's instruction is "create a different program that is semantically equivalent to the code generated last time". The compiler options strategy's instruction is "insert, change, or delete the #pragma compiler option".

Images