Identifier renaming method in mutation operator and compiler test program generation method

Abstract

The invention provides a method for renaming an identifier in a mutation operator on the first aspect, which comprises the following steps of: acquiring a name and a position of the identifier in the mutation operator, and acquiring an available identifier; sorting the available identifiers according to the names and the positions; calculating the relationship strength of each sequence according to a set rule; traversing and calculating the maximum value of the sum of the relation strength to obtain a sorting result corresponding to the maximum value, and taking the sorting result as a renaming result. The invention also provides a compiler test program generation method, which comprises the following steps of: acquiring a mutation operator capable of inserting a seed program, and renaming an identifier of the mutation operator to obtain an insertion operator; determining insertion information according to the seed program; and inserting the insertion operator into the seed program according to the insertion information so as to generate the compiler test program.

Description

Technical Field

[0001] The present invention belongs to the technical field of compiler test generation, and in particular relates to an identifier renaming method in a mutation operator and a compiler test program generation method. Background Art

[0002] Compilers are core components in computer science, and their correctness, efficiency, and security are crucial for the reliable operation of software systems. Compiler testing aims to automatically generate test cases to detect potential errors or vulnerabilities in compiler implementations. This paper introduces new test case generation techniques and methods to improve the efficiency and accuracy of compiler testing, particularly in handling complex programs and compiler optimization.

[0003] In the field of compiler testing, various techniques and tools have been developed for generating and evaluating compiler test cases. Csmith is a well-known random program generator that specializes in generating large, complex C programs to test compiler robustness and error handling. Over the years, Csmith has helped discover numerous bugs in GCC and Clang / LLVM. Building on the success of Csmith, Lidbury et al. further developed CLsmith, which focuses on testing the correctness of the OpenCL compiler.

[0004] In addition to Csmith, several other random program generators have been used for compiler testing. For example, Orange3 focuses on testing arithmetic optimizations in C compilers, while CCG aims to discover compiler crashes. Epiphon employs randomization techniques to detect flaws in compiler warning diagnostics. Furthermore, there is a significant amount of work on compiler testing for functional languages, including the development of exhaustive or random test case generators.

[0005] In recent years, compiler testing methods based on equivalent modular input (EMI) have gained increasing popularity. This approach modifies programs by inserting and deleting statements in unexecuted branches, thereby generating more diverse test cases. Orion and Athena are typical representatives of this approach, using random deletion and Markov Chain Monte Carlo (MCMC) methods to guide statement insertion and deletion, respectively. Hermes focuses on inserting code snippets into active regions, while Proteus applies EMI technology to test link-time optimizers. These EMI-based frameworks have achieved significant results in compiler testing, discovering numerous serious bugs. In addition to random generation and mutation testing, compiler testing also involves advanced techniques such as translation verification and verified compilers. Translation verification aims to verify that the compiler's output is consistent with expectations, while verified compilers guarantee the correctness of the compilation process through rigorous mathematical proofs. Differential testing is also an important method in compiler testing, detecting differences by comparing the output of the same program using different compilers or different compiler versions.

[0006] Before submitting a bug report, test programs need to be reduced and optimized to improve the quality and readability of the bug report. Tools such as C-Reduce and Berkeley Delta play an important role in this regard. Finally, bounded exhaustive testing is another testing approach worth considering. It generates test cases by enumerating all possible input structures of a given size, ensuring comprehensiveness and accuracy. However, this approach can be prohibitively expensive and impractical for large programs.

[0007] In summary, while existing compiler testing technologies have achieved some success, they still face challenges in handling complex programs and optimizing compilers. Therefore, this invention aims to improve the efficiency and accuracy of compiler testing by introducing new test case generation technologies and methods, thereby providing a more robust guarantee for compiler reliability and security.

[0008] Existing technologies for C++ compiler testing primarily rely on traditional compiler testing frameworks and program generators, such as Csmith and YarpGen. These technologies have the following significant drawbacks when testing C++ compilers:

[0009] 1) Limited test coverage:

[0010] Existing program generators (such as Csmith and YarpGen) primarily generate C++ test programs that strictly conform to syntax and type checking rules and typically do not include advanced C++ features (such as templates and classes). This makes it difficult for these test programs to trigger potential compiler bugs related to advanced features. Due to the lack of complexity in the test programs, the compiler may quickly pass these tests, while potential advanced feature bugs cannot be effectively detected.

[0011] 2) Insufficient test input generation capabilities:

[0012] Generating highly structured test programs (i.e., test inputs in compiler testing) is crucial to test effectiveness. However, existing techniques struggle to generate test programs that are more likely to trigger compiler bugs. This limits the ability of testing frameworks to discover deep-seated compiler bugs.

[0013] 3) Insufficient processing power for complex compiler output:

[0014] Different compilers (such as GCC and Clang) have different diagnostic systems, which can result in different output when compiling the same program. Furthermore, existing methods cannot effectively handle complex compiler output, making it difficult to identify potential errors. This increases the difficulty of identifying compiler errors and reduces the practicality of testing frameworks. Summary of the Invention

[0015] To solve the above problem, the first aspect of the present invention provides a method for renaming identifiers in a mutation operator, comprising:

[0016] Get the name and position of the identifier in the mutation operator and get the available identifiers;

[0017] sorting the available identifiers according to the name and the position;

[0018] Calculate the relationship strength of each of the rankings according to the set rules;

[0019] The maximum value of the sum of the relationship strengths is traversed and calculated to obtain a sorting result corresponding to the maximum value, and the sorting result is used as the renaming result.

[0020] Furthermore, the identifier is an identifier of a variable.

[0021] Furthermore, the available identifier is an identifier within a scope to which the insertion position in the seed program of the mutation operator belongs.

[0022] Furthermore, the setting rules are obtained based on the assignment relationship and decision relationship between the identifiers.

[0023] A second aspect of the present invention provides a compiler test program generation method, comprising:

[0024] Get mutation operators that can be plugged into the seed program,

[0025] Renaming the identifier of the mutation operator to obtain an insertion operator;

[0026] Determining insertion information according to the seed program;

[0027] The insertion operator is inserted into the seed program according to the insertion information, thereby generating the compiler test program.

[0028] Furthermore, the mutation operator that can be inserted into the target program is obtained by preprocessing and slicing the program that triggers the compiler defect.

[0029] Furthermore, the insertion information includes an insertion position, an insertion timing, and an insertion method.

[0030] Furthermore, the insertion position is determined based on a set position point.

[0031] A third aspect of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing the above method.

[0032] A fourth aspect of the present invention provides a storage medium storing a computer program, wherein the computer program implements the above method when executed by a computer.

[0033] The beneficial effects of the present invention are as follows:

[0034] 1) Increase the complexity of the test program:

[0035] The test programs generated by existing technologies are too simple and cannot effectively trigger errors in advanced features of the compiler. By implementing a practical program generator, we can generate test programs that include advanced C++ features to improve test coverage and make it easier to trigger potential errors in the compiler.

[0036] 2) Enhanced test input generation capabilities:

[0037] Existing techniques are insufficient in generating highly structured test programs, making it difficult to discover deep-seated compiler errors. This paper utilizes an insertion mutation strategy and an identifier enumeration technique based on the strength of identifier relationships to generate test programs that are more likely to trigger compiler errors.

[0038] 3) Improve the ability to handle complex compiler output:

[0039] Existing methods cannot effectively handle complex compiler outputs, making it difficult to identify potential errors. A differential testing strategy is adopted to identify different types of compiler errors by comparing the output inconsistencies of different compilers.

[0040] The present invention designs a method for counting program identifier relationships, which can accurately count the relationships between identifiers in a program and generate a numerical value that measures the strength of the identifier relationship. This helps to deeply understand the internal structure and complexity of the program. Furthermore, the present invention aims to design a skeletal program enumeration method based on the above-mentioned identifier relationship strength, which ensures that the generated program is neither equivalent in identifier relationship nor contains equivalent identifier relationships, thereby effectively pruning programs with equivalent identifier relationships, reducing the size of the compiler test set, and improving test efficiency. The technical solution of the present invention effectively solves the problems of insufficient program diversity and low test efficiency in compiler testing by introducing a program identifier relationship counting method, a skeletal program enumeration method based on identifier relationship strength, and a program generation method based on mutation, and achieves the beneficial technical effect of improving test coverage and fault detection rate, thereby fully demonstrating the creativity and practicality of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The flowchart of identifier renaming in an embodiment of the present invention.

[0042] Figure 2 This is an example diagram of identifier position identification in an embodiment of the present invention.

[0043] Figure 3 This is an example of a C language program fragment in an embodiment of the present invention.

[0044] Figure 4 This is an example diagram of program enumeration in an embodiment of the present invention.

[0045] Figure 5 Flowchart of a test program generation method in an embodiment of the present invention.

[0046] Figure 6 This is an example diagram of the insertion timing selection strategy in an embodiment of the present invention.

[0047] Figure 7 This is an example diagram of the process of inserting an insert operator into a target program in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0049] A first embodiment of the present invention provides a method for renaming identifiers in a mutation operator.

[0050] During the scope-based program mutation process, to ensure the validity and diversity of the program after the mutation operator is inserted into the seed program, the identifiers of the inserted mutation operators must be renamed. This is because the source program used for mutation sometimes uses the same identifiers. For example, the identifiers f or fun may be used as function names, and the identifiers a, b, and c may be used as variable names. This results in duplicate names if two identifiers of the same name and type appear in the same scope, resulting in compiler failure. This wastes compiler testing resources and reduces compiler testing efficiency. During the scope-level mutation operator insertion process, identifier renaming is required to prevent this problem.

[0051] In addition, for the mutation operators at the scope level of the same grammatical structure, the present invention adopts an identifier renaming strategy based on identifier relationship enumeration, which can obtain different renamed mutation operators. The prior art has shown that even for programs with the same grammatical structure, different identifier naming methods can play different roles during compiler testing. Therefore, when renaming, the present invention not only considers the legality of the mutation operator of the scope after insertion, but also extracts the grammatical skeleton of the mutation operator of this scope, specifically, extracts the positions of all identifiers in the mutation operator. In order to make the association relationship between the inserted mutation operator and the seed program as diverse and close as possible, the present invention uses an identifier relationship enumeration strategy to obtain a set of all identifier selection sequences that can be used for naming scope fragment identifiers, and then prunes those identifier selection sequences with equivalent relationships from this set to obtain a streamlined set.

[0052] The specific method for renaming identifiers in the mutation operator in the first embodiment of the present invention is as follows: ① randomly select an original mutation operator containing some identifiers; ② identify the name and position of the identifier in the mutation operator to obtain a position sequence; ③ obtain the available identifiers in the scope of the seed program P at the insertion position D, and collect all identifiers of the same type in the current scope and the parent scope of the current scope as the available identifier set; ④ use the available identifiers to number and sort these positions to obtain a series of identifier selection sequences, count the identifier relationships of these identifier selection sequences, delete those sequences with low identifier relationship strength, and only retain those sequences with the largest identifier relationship strength; ⑤ use the identifier selection sequences obtained after screening to fill the mutation operator one by one to obtain a series of renamed mutation operators; ⑥ then insert these renamed mutation operators into the insertion position D of the current seed program P to obtain a series of mutated test programs.

[0053] The present invention provides a detailed description of the first embodiment, the flow chart is as follows Figure 1 shown.

[0054] Step 101: Identification of Identifier Location

[0055] The identifier relationship enumeration strategy proposed in the present invention needs to identify the position of the identifier from the mutation operator to be inserted.

[0056] The present invention utilizes the parser generated by the syntax analyzer generation tool Antlr4 (ANother Tool for Language Recognition) for C++ grammar to parse the abstract syntax tree AST (Abstract Syntax Tree) in the source program. In each scope node in the AST, the identifiers contained therein are identified, and the identifier names and the positions of the identifiers are recorded. These identifiers are numbered and renamed according to vi from top to bottom so that the identifiers at different positions can be distinguished later. This is because the same identifier may appear at different positions, but for this field, the identifiers at different positions are different. By renaming the originally identical identifiers at different positions into different identifiers, the resulting program will change in both data and control flow, thereby enhancing the diversity of the test program generated subsequently.

[0057] refer to Figure 2 , Figure 2 The example illustrates a mutation operator that needs to identify the position of each identifier. Figure 2 The first three lines should return the scope mutation operators renamed using the identifier numbers v1, v2, v3, and v4 shown in the third program snippet in the figure, as well as a list of identifier location information:

[0058] {[1,”v1”,1,1],[2,”v2”,2,1],[3,”v3”,2,2],[4,”v4”,2,3]}.

[0059] [3,"v3",2,2] means that the third variable v3 is located in the second row and the second variable.

[0060] Step 102: Identifier relationship count

[0061] This step introduces the types of relationships between seed program identifiers and source program identifiers. First, we'll explain the possible relationships between identifiers. The identifiers mentioned in this document refer to variable identifiers. For identifiers of functions, structures, classes, etc., enumeration is not performed. This is because it's often difficult to ensure that enumerated identifiers remain valid.

[0062] When inserting a mutation operator into a seed program, the relationship between the mutation operator and the variables of the seed program may be varied. In the identifier relationship enumeration strategy of the embodiment of the present invention, only the two types that have the greatest relationship with data flow and control flow are concerned, one is the assignment relationship, and the other is the decision relationship. An assignment relationship refers to a variable participating in the assignment operation of another variable. A decision relationship means that when deciding how a variable is assigned, the value of another variable must meet certain constraints. If there is an assignment relationship, there is an assigned relationship, and if there is a decision relationship, there is a decided relationship. Relationships have their own attributes, such as the strength of the relationship, the specific content of the relationship, etc. In the present invention, only the following 16 basic relationships are considered for calculation:

[0063] (1) Direct assignment: a=>b, b∈V a1 , a∈V b5 .

[0064] (2) Complete direct assignment: a=>>b, b∈V a2 , a∈V b6 ,|V b6 |=1.

[0065] (3) Indirect assignment: a*=>b, b∈V a3 , a∈V b7 .

[0066] (4) Fully indirect assignment: a*=>>b, b∈V a4 , a∈V b8 .

[0067] (5) Directly assigned: a<=b, b∈V a5 , a∈V b1 .

[0068] (6) Completely directly assigned: a<<=b, b∈V a6 ,|V a6 |=1,a∈V b2 .

[0069] (7) Indirect assignment: a<=b*, b∈V a7 , a∈V b3 .

[0070] (8) Completely indirectly assigned: a<<=b*, b∈V a8 , a∈V b4 .

[0071] (9) Direct decision: c(a)=>b, b∈V a9 , a∈V b13 .

[0072] (10) Complete direct decision: c(a)=>>b, b∈Va10 , a∈V b14 ,|V b14 |=1.

[0073] (11) Indirect decision: c(a)*=>b, b∈V a11 , a∈V b15 .

[0074] (12) Completely indirect decision: c(a)*=>>b, b∈V a12 , a∈V b16 .

[0075] (13) Direct decision: a <= c(b), b∈V a13 , a∈V b9 .

[0076] (14) Completely directly decided: a<<=c(b), b∈V a14 ,|V a14 |=1,a∈V b10 .

[0077] (15) Indirect decision: a <= c(b)*, b∈V a15 , a∈V b11 .

[0078] (16) Completely indirect decision: a<<=c(b)*, b∈V a16 , a∈V b12 .

[0079] Operators: *=>>, *=>, *, ∨, =>>, =>, ∧, ￠,∈,

[0080] refer to Figure 3 If an identifier has more than one relationship with identifier v, then the number of types of relationships between the two identifiers is more than 1. This is because there is sometimes more than one relationship between two variables. Figure 3 The following example illustrates a C program fragment in which there are four relationships between variables a and b:

[0081] (1) a=>b, this direct assignment relationship can be obtained according to the statement "b=a+c";

[0082] (2) a*=>b, according to the statement "int c=a*5;" we can get a=>c, according to the statement "b=a+c;" we can get c=>

[0083] b, according to the third basic relation, the definition of indirect assignment, we can get a*=>b;

[0084] (3) c(a)=>b. According to the if statement “if(a>0&&c>0)” and the statement “b=a+c;” in the if block, we can get the direct decision relationship c(a)=>b.

[0085] (4) c(a)*=>b. According to the statement “int c=a*5;”, a=>c is obtained. According to the if statement “if(a>0&&c>0)” and the statement “b=a+c;” in the if block, c(c)=>b is obtained. According to the definition of the 11th basic relation indirect decision, c(a)*=>b is obtained.

[0086] Step 103: Identifier renaming based on identifier relationship strength

[0087] When renaming identifiers in inserted mutation operators, this paper proposes an optimization function based on the strength of identifier relationships, giving priority to using identifiers in the available identifier set AI (Available Identifier) ​​in the current scope, and maximizing the value of the optimization function:

[0088] Opt identifier-relation =∑ v∈AIV (|V v1 |+|V v2 |+|V v3 |+…+|V v16 |),

[0089] where |V vi |(1≤i≤16) represents the number of identifiers that have the i-th relationship with the currently available identifier v. (|V v1 |+|V v2 |+|V v3 |+…+|V v16 |) adds up to the total number of identifiers that have a relationship with the currently available identifier v.

[0090] The goal of this invention's identifier selection is to maximize the optimization function so that, after the mutation operator is inserted into the program, the number of identifiers within the aforementioned 16 relationships increases as much as possible. This approach has the advantage of creating a closer connection between the inserted mutation operator and the seed program in terms of data flow and control flow. When multiple identifier selection strategies maximize the optimization function, all sequences selected from these identifiers are used as the result of identifier renaming.

[0091] refer to Figure 4 , Figure 4 An example illustrating the strength of an identifier relationship. Figure 4For example, consider inserting the While mutation operator from the source program p1.cpp into the third line of the main function scope of the seed program p2.cpp. First, extract the syntax skeleton of the While mutation operator and its variable list. It is found that four variables require identifier renaming. Otherwise, the program will fail to compile after the insertion because the identifiers index, num, num1, and num2 do not exist.

[0092] In p3.cpp, the four variables extracted above that require identifier renaming are renamed using the identifier of variable a, which matches the type of the variable in the current scope. In this case, variable a is directly assigned to variable a (a = a + a), and a also directly determines a (if (a++ ≤ 10)). Therefore, the value of the identifier relationship strength optimization function is 4.

[0093] It should be noted that although the full direct assignment includes direct assignment, it is only counted once in the statistics of the optimized function. a2 |=1, but do not remember |V a1 = 1. If counted twice, the number of times the assignment relationship is repeated will result in this complete class relationship being given more weight in the optimization function, which is not the case with which the present invention is applicable.

[0094] Step 103 can ensure that the generated programs are not equivalent in terms of identifier relationships and do not contain equivalent relationships, thereby achieving pruning of programs with equivalent identifier relationships. By introducing the strength of identifier relationships as the criterion for enumeration, the present invention can generate more diverse skeleton programs and avoid generating a large number of equivalent program variants, thereby significantly reducing the set size of compiler tests. This enumeration method improves the coverage and efficiency of the test and reduces the test cost.

[0095] In p4.cpp, the four variables extracted above that require identifier renaming are renamed using the identifiers of the matching variables a and i in the current scope. In this case, variable a is directly assigned to variable a (a = a + a), and variable i directly determines a (if (i++ ≤ 10)). Therefore, the value of the identifier relationship strength optimization function is 4.

[0096] In p5.cpp, the four variables requiring identifier renaming are renamed using the identifiers of the currently scoped variables a, b, c, and i. In this case, variables b and c are directly assigned to variable a (a = b + c), and variable i completely determines a (if (i++ ≤ 10)). Therefore, the identifier relationship strength optimization function has a value of 5.

[0097] Therefore, among p3.cpp, p4.cpp, and p5.cpp, p5.cpp with a larger optimization function value will be generated first because it can reflect more identifier relationships. The present invention uses the above method to enumerate the identifiers in the mutation operators to be inserted into the seed program, and then outputs multiple mutation operators with the same program skeleton but different data flows and control flows as insertion operators.

[0098] This embodiment can accurately count the relationships between identifiers in a given program and generate a numerical value to measure the strength of the identifier relationships in the program. Through accurate identifier relationship counting, the correlation and dependency between identifiers within the program can be quantified, providing basic data support for subsequent program enumeration and variant program generation. This quantitative analysis helps to deeply understand the program structure and improve the pertinence and efficiency of compiler testing. Experiments show that compared with traditional identifier relationship identification methods based on static analysis, the method of the present invention has significant improvements in computational complexity and accuracy. Especially when processing large programs, the counting method of the present invention can more effectively identify the complex relationships between identifiers, thereby generating more accurate identifier relationship strength values.

[0099] A second embodiment of the present invention provides a compiler test program generation method.

[0100] The present invention uses a mutation method guided by identifier relationship counting to generate complex and diverse legal C++ programs. Figure 5 , Figure 5The overall process of the compiler test program generation method according to an embodiment of the present invention is shown. First, during the grammar-based random program generation process, the present invention determines where a mutation operator can be inserted without destroying the program's grammatical structure. This process is referred to as determining the insertion position and timing. After determining the insertion position, the content to be inserted must be determined. The present invention extracts multiple mutation operators from a source program collection and randomly selects one that can be inserted at the current position. This step is referred to as mutation operator selection. To generate diverse compiler test programs and ensure that inserting this mutation operator into the current generated position does not cause issues such as duplicate identifiers or undefined identifiers, the program skeleton is extracted for this insertion mutation operator. This process obtains the positions of all identifiers appearing in this mutation operator. Then, using a method based on identifier relationship counting strength, the identifiers in this mutation operator are renamed to obtain an insertion operator. Finally, this embodiment inserts the renamed insertion operator into the current generated position, making it part of the generated test program. The program generated by the present invention can ensure the diversity while ensuring the legality of the program, so that it can be used to test the back end of the compiler, especially the compiler optimization part. Each step of the second embodiment of the present invention will be described in detail in the following part.

[0101] Step 201: Insertion location selection

[0102] In order to insert the mutation operator into the seed program, it is necessary to add some position marks in the seed program so that the insertion operator can be inserted into the appropriate position to ensure the legitimacy of the seed program after insertion. Add a line of comments at the starting position of each scope to represent the type of mutation operator that can be inserted at this position. Although the mutation operator can be inserted into many positions, considering the convenience of actual operation, the present invention currently only performs the insertion operation of the insertion operator on the first and last rows of the scope in the seed program, and does not perform this operation at other positions. This is because whether the insertion operator is inserted into the first or second row in the same scope does not have a great impact on the diversity of the program. The diversity of the test program generated by the embodiment of the present invention is mainly achieved by the diversity of the selected mutation operators and the diversity of renaming the mutation operators.

[0103] To illustrate the insertion position selection strategy, the embodiment of the present invention defines the following three types of position points:

[0104] Global Insertion Position (GIP): The variables or functions declared at this position are globally available.

[0105] Class Insertion Position (CIP): The variables or functions declared at this position are available in the class.

[0106] Scope Insertion Position (SIP): A point within a specific scope where variables declared are only available within that scope.

[0107] Corresponding to the three types of positions, there are also three types of scopes. That is, global scope, class scope and local scope. The mutation operators inserted in different scopes are different. Therefore, the insertion position is marked for each type of scope, and the types of mutation operators that can be inserted in the current scope are recorded. One type is a mutation operator that can only be inserted at a global position point, such as a header file or a class declaration. The other type is a mutation operator that can only be inserted in a global or class, such as a function declaration. The last type is a mutation operator that can be inserted in a local scope, such as a function call statement, arithmetic operation statement, etc. The last type of scope is further distinguished between the main function scope and other local scopes, because the present invention hopes that function calls can be inserted as much as possible in the main function and other scopes can be inserted into other local scopes as much as possible, such as the scope of loop statements or if statements.

[0108] Each location type is listed below, along with the location and type in which an outer scope fragment can be inserted, as shown in Table 1.

[0109] Table 1. Location types and where outer-scope fragments can be inserted

[0110]

[0111]

[0112] In the global scope, the present invention designs a location point "#insertion-include-available" for inserting header file statements after the header file of the test program. And a location point "#insertion-global-available" for inserting global declarations is designed before the start of the main function of the test program. In the class scope, the present invention inserts "#insertion-class-first-available" and "#insertion-class-last-available" at the start and end positions of the class respectively. For the local scope main function, the present invention designs two location points "#insertion-main-first-available" and "#insertion-main-last-available" for inserting outer scopes. For other local scopes, the present invention not only designs the location points "#insertion-scope-first-available" and "#insertion-scope-last-available" for inserting outer scopes, but also designs the location points "#insertion-scope-first-statement-available" and "#insertion-scope-last-statement-available" for inserting outer statements.

[0113] Step 202: Determine the insertion timing

[0114] At present, in the process of test program generation, it is necessary to determine when to insert the mutation operator into the current insertion position, that is, when to perform the insertion operation. The present invention designs three insertion timing strategies, such as Figure 6 As shown in the figure. The first method is to insert the mutation operator after the test program is fully generated. The second method is to directly insert the mutation operator when an insertion position is generated during the test program generation process, and insert the function declaration, variable declaration, class declaration, and header file statements involved in the inserted mutation operator at the same time. The last method is to insert part of the mutation operator during the test program generation process and the rest at the end of the test program generation. For example, some mutation operators are inserted during the generation process, and the declarations of global variables, functions, templates, structures, etc. are inserted after the program generation is completed.

[0115] Different strategies have different implementation methods. Considering that for the first strategy, when the mutation operator is inserted into the target program after the generation is complete, scope control and symbol table management will be more difficult. Because when the insertion operation is performed, the seed program is already just a simple program, and does not have the scope and symbol table information in the generation process. If the second strategy is a generation-variant insertion strategy, since the test program generation of the present invention is based on the depth-first method of grammar, the test program is not complete during the generation process. After inserting some mutation operators into a certain scope, the insertion positions of the declarations of variables, functions, templates, structures, etc. associated with the mutation operator may not have been generated yet. Therefore, the second strategy is not suitable either.

[0116] Therefore, the present invention adopts a third insertion method. That is, during the test program generation process, a generation program is maintained, along with a list of mutation operators that should be inserted but have not yet been inserted. This list contains each mutation operator that should be inserted, along with the scope S in which it should be inserted. Although all statements that affect program validity can be inserted into the global scope, to ensure the diversity of generated programs and minimize identifier conflicts, the present invention inserts the required variable declarations into the nearest scope outside the insertion scope S. Functions, structures, classes, and the like are inserted into the global scope.

[0117] Step 203: Select mutation operator

[0118] More than 7,000 C++ test programs that triggered compiler defects and have been fixed were obtained from the open source defect lists of GCC and LLVM compilers, and the C++ programs that triggered these defects were collected for scope-level slicing. These programs were preprocessed before slicing. The preprocessing step is mainly used to standardize the content of the program, such as deleting blank lines and comments, etc. The slicing part is mainly used to slice out each scope and the mutation operators contained in it from the preprocessed program. Finally, a lot of mutation operators that can be used to insert into the seed program were obtained. During the slicing process, the header files contained in the original program that triggered the compiler defect are also retained. If a fragment of the program is inserted into the C++ program to be synthesized, these header files will also be added to the header file list of the C++ program to be synthesized to ensure the validity and legality of the program. The embodiment of the present invention sets different mutation operators for different scopes. The details are shown in Table 2 below:

[0119] Table 2. Scope-level mutation operator types and where they can be inserted

[0120]

[0121]

[0122] Step 204: Rename the identifier in the mutation operator

[0123] In the second embodiment of the present invention, the method of the first embodiment is used to rename the identifier in the mutation operator in step 203 to obtain an insertion operator.

[0124] Step 205: Insertion of the Insert Operator

[0125] After obtaining the insertion operator and the seed program, the present invention designs a combined scope-level insertion strategy, that is, at each insertable position point in the scope S, there is a certain probability R of choosing whether to insert the insertion operator, for example, R = 50%. If not, then the insertion operation is not performed and the subsequent program content is generated; if so, then a scope IS is randomly selected from the sliced ​​scope set, and the mutation operator F corresponding to the scope IS is inserted into the position after enumeration and renaming of the identifier, and the external declaration O required by F is added to the list of those to be inserted but not yet inserted, and the scope corresponding to O is set as the current scope. This embodiment also updates the identifier contained in F to the S node in the current scope tree.

[0126] refer to Figure 7 , Figure 7 The figure shows an example of the degradation of the insertion process when the target scope S = main function scope, R = 50%, and the source scope IS = main function scope. That is, the main function in the source program source.cpp in the upper left corner of the figure is merged into the main function of the target program destination.cpp in the lower left corner of the figure. In order to ensure the legality and validity of the program, all function declarations outside the main function in the source program source.cpp are also added outside the main function of the target program destination.cpp, forming the synthesized program destination.cpp on the right side of the figure. The present invention randomly generates a large number of seed programs, selects appropriate insertion operators from programs that have triggered compiler defects, and inserts them into the seed programs, synthesizing a large number of test programs to test the compiler.

[0127] A third embodiment of the present invention provides a method for performing differential testing on a C++ compiler using a generated test program.

[0128] Compiler differential testing ensures that compilers implemented according to the same specification have identical functionality. If these functionally identical compilers are given the same input and the output of one (such as target code, assembly code, error messages, or warning messages) differs from the output of the others, then one of these compilers must have a functional error. Compiler differential testing is based on the following principle: compilers implemented strictly according to the same specification (such as C99, C++11, etc.) may use different implementation algorithms due to differences in application scenarios, optimization strategies, or target platforms, but the functionality they ultimately achieve should be consistent.

[0129] After the present embodiment generates a large amount of test programs, these test programs are utilized to test the compiler.The present embodiment generates test programs on the Linux operating system Ubuntu 20.04.5LTS equipped with AMD Ryzen 9 5950X processor (16 cores, 32 threads) and tests the latest versions of compiler LLVM (10, 11, 12, 13, 14 and 15), GCC (10, 111, 12 and 13).By compiling each variant program and using each standard -O0, -O1, -O2, -O3 and -Os optimization level, this has found the defects in GCC and LLVM on some Linux.The present embodiment also uses a small part of the variant program generated on Linux to carry out a brief evaluation on Windows, to test the Microsoft Visual Studio compiler (MSVC 19.28.29915) with / Od (no optimization) and / O2 (maximum speed) optimization settings. In addition to these three commonly used compilers, this embodiment also tests the latest version of icc ("Intel C / C++ Compiler Classic Edition") and the latest version of icx ("Intel(R) oneAPI DPC++ / C++ Compiler 2023.0.0") using the -O0, -O1, -O2, -O3, and -Os options.

[0130] This embodiment implements the aforementioned compiler test program generation method in the ScopeFuzz tool. ScopeFuz is an automatic compiler test generation tool. ScopeFuzz is implemented in Java. It requires a directory containing grammar files and a directory containing source programs to be mutated as input. It outputs a directory containing valid and legal test programs. ScopeFuzz consists of three parts: program generation, scope enumeration, and program compilation.

[0131] The program generation component takes as input a C++ grammar file from the grammar collection GrammarV4, which is a collection of various ANTLR4 grammars. To generate an executable program, complex language features are manually removed from the C++ grammar and special grammar rules are marked in the grammar, such as function rules, structure rules, class rules, template rules, if rules, while rules, for rules, variable rules, pointer rules, array rules, scope-related rules, identifier rules, advanced feature rules, and corresponding basic feature rules. This embodiment addresses issues such as deep recursion and undefined identifiers in the grammar, preventing the generation of test programs containing undefined behavior and reducing test efficiency. Subsequently, the C++ test program is generated by randomly expanding the grammar. Furthermore, to ensure the dynamic correctness of the program, this embodiment addresses issues such as division by zero, array index out-of-bounds, null pointers, infinite loops, and expression type checking. The program generation component constructs a scope tree for the program during the generation process, which is used to manage identifiers and types in the program to generate legal and valid C++ programs.

[0132] The scope enumeration component first selects a location in the source program to insert a scope evaluator. It then mutates the scope fragment by enumerating the available identifiers in the current scope and inserting it into the location. When enumerating the available identifiers to rename the identifiers in the scope fragment, the mutation results with identifiers that have an equivalence relationship are pruned. To implement this part, an enumeration tree of scope fragments is first constructed and the identifier relationship is applied to generate the test program. Specifically, scope information is collected based on AST blocks (such as functions and classes). These holes, namely load holes and storage holes, are distinguished by the mutable node attributes in the seed program AST. A two-dimensional list is used to store the path. In each derivation, the path list is updated and the entire enumeration tree is eventually stored.

[0133] In the compiler test component of this embodiment, the derived program is fed to a given compiler. The whole process follows a dynamic strategy: once a test program is derived from the path, ScopeFuzz executes it with the target C++ compiler by calling os.system() in the os module. This function returns a system code that reflects the execution status of the test program, such as normal termination or crash. ScopeFuzz monitors a set of system codes related to different crash signals, such as Linux system code 134-segmentation fault. If the crash signal is captured, ScopeFuzz saves the test program in the output directory. Otherwise, ScopeFuzz will execute the compiled executable file, compare the execution results on different compilers and optimization options, and save the test programs with different direct results in the output directory. ScopeFuzz will then process the next derived test program. In addition, in order to avoid generating too many test programs, this embodiment sets the maximum processing time of the seed program to 2 hours. The maximum execution time of the test program is set to 60 seconds to avoid infinite loops.

[0134] This example uses a total of 7369 C++ programs from the latest GCC and LLVM test suites as mutation source programs. Table 3 shows the sources and specific data of the source programs collected in this example.

[0135] Table 3

[0136]

[0137] The fourth embodiment of the present invention provides a comparative experiment between the method of the present invention (ScopeFuzz) and the prior art.

[0138] Comparative Experiment 1: Comparison of tool throughput and static effectiveness (based on GCC14 / Clang20 compilation measurement, 10 runs / 24 hours).

[0139] As shown in Table 4, this method is significantly superior to existing methods in terms of test program generation efficiency. The generation rate of the method of the present invention is 526,253 programs / hour, far exceeding GrayC (101,142 programs / hour), Csmith (7,125 programs / hour) and YarpGen (18,067 programs / hour). Although the static effectiveness of ScopeFuzz is 26.86%, slightly lower than YarpGen (92.35%) and GrayC (100%), its high throughput indicates that it can generate more candidate test programs per unit time, providing a wider input space for compiler stress testing. In comparison, the throughput and effectiveness of traditional methods such as Csmith are at a low level, making it difficult to meet large-scale testing needs.

[0140] Table 4

[0141] tool Generation rate (programs / hour) Static effectiveness (%) ScopeFuzz 526,253 26.86 GrayC 101,142 100.00 Csmith 7,125 9.82 YarpGen 18,067 92.35

[0142] Comparative Experiment 2: Comparison of compiler backend coverage (based on Clang-20.1.1 and GCC-14.2.0).

[0143] Coverage metrics measure the extent to which a test program covers the compiler's code structure. Line coverage refers to the percentage of source code lines covered by the test program compared to the total number of source code lines, while function coverage refers to the percentage of functions called by the test program compared to the total number of functions.

[0144] As shown in Table 5, our method outperforms existing methods (such as Csmith, YarpGen, and GrayC) in terms of line coverage (22.98%), function coverage (17.00%), and area coverage (12.69%). In particular, ScopeFuzz achieves nearly four times higher area coverage than GrayC (3.18%), indicating that its generated test programs can more comprehensively trigger different code paths in the compiler backend. While GrayC excels in static validity, its coverage improvement is limited (line coverage is only 2.65%), making it difficult to deeply test the complex logic of the compiler. Furthermore, in the GCC-14.2.0 environment, ScopeFuzz's coverage metric (13.6% line coverage) remains ahead, further verifying its robustness across compiler versions.

[0145] Table 5

[0146]

[0147] The above alternatives are all intended to ensure that, while achieving the same invention purpose, a broader scope of technical protection is achieved through different technical means and combinations. These alternatives not only cover the replacement of some structures, devices, and method steps, but also cover different implementation paths for the overall technical solution. By adopting these alternatives, the technical implementation of the present invention can be ensured to be more flexible, diverse, and robust, thereby effectively preventing others from circumventing this technology to achieve the same invention purpose.

Claims

1. A method for renaming identifiers in a mutation operator, comprising: Get the name and position of the identifier in the mutation operator and get the available identifiers; sorting the available identifiers according to the name and the position; Calculate the relationship strength of each of the sortings according to the set rules; The maximum value of the sum of the relationship strengths is traversed and calculated to obtain a sorting result corresponding to the maximum value, and the sorting result is used as the renaming result.

2. The method according to claim 1, characterized in that The identifier is an identifier of a variable.

3. The method according to claim 1, characterized in that The available identifier is an identifier within the scope to which the insertion position in the seed program of the mutation operator belongs.

4. The method according to claim 1, wherein The setting rules are obtained based on the assignment relationship and decision relationship between the identifiers.

5. A method for generating a compiler test program, comprising: Obtaining a mutation operator that can be inserted into a seed program, and determining insertion information according to the seed program; Renaming the identifier of the mutation operator to obtain an insertion operator; The insertion operator is inserted into the seed program according to the insertion information, thereby generating the compiler test program.

6. The method according to claim 5, characterized in that The mutation operator that can be inserted into the target program is obtained by preprocessing and slicing the program that triggers the compiler defect.

7. The method according to claim 5, characterized in that The insertion information includes an insertion position, an insertion timing, and an insertion method.

8. The method according to claim 7, characterized in that The insertion position is determined based on a set position point.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, the computer program being configured to be executed by the processor, the computer program comprising instructions for executing the method according to any one of claims 1 to 8.

10. A storage medium storing a computer program, wherein when the computer program is executed by a computer, the method according to any one of claims 1 to 8 is implemented.

Images