A compiler automation testing method based on specification inference and loop consistency verification

Abstract

This invention provides an automated compiler testing method based on specification inference and loop consistency verification to address the problems of traditional compiler testing, which relies on differential testing, requires reference implementations, and cannot effectively test domain-specific compilers. The method first extracts a syntactic skeleton set and a detailed reference set from heterogeneous specification documents; it then uses a large language model to infer type-safe formal grammar rules through multi-agent collaboration; based on the inferred grammar, it generates semantically valid test cases using a seed-preserving abstract syntax tree-level mutation strategy; it executes a compile-decompile-recompile loop on the test cases and detects compiler defects through binary-centralized comparison and noise normalization. This invention achieves systematic automated testing of compilers in the absence of reference implementations and execution oracles, improving compiler reliability and system security.

Description

Technical Field

[0001] This invention relates to the field of software testing and compiler verification technology, specifically to an automated testing method, system, and storage medium that uses a large language model to infer syntax rules from heterogeneous specifications and detects compiler defects through cycle consistency testing. Background Technology

[0002] The Advanced Configuration and Power Interface (ACPI) is an open standard that defines the hardware-software interface for power management and device configuration in modern computer systems. The ACPI Source Language (ASL) compiler, as the de facto reference implementation, is widely integrated into mainstream operating systems. It compiles ASL code written by firmware developers into ACPI Machine Language (AML) bytecode, which is then interpreted and executed by the operating system kernel. The correctness of the compiler is crucial to system reliability and security. Compiler defects can lead to memory safety vulnerabilities or semantic compilation errors, which can affect every platform built using that toolchain. In particular, silent semantic compilation errors can introduce potential backdoors into cryptographically signed and deployed firmware binaries, constituting a unique attack vector within the firmware supply chain. However, systematically verifying compilers faces three major challenges: (1) Syntactic barriers: Inputs generated by bit-level mutations in traditional fuzzers are usually rejected by the compiler front-end, making it difficult to access deep compilation logic; (2) Specification complexity: Specification documents are voluminous, with formal syntax tables intertwined with natural language descriptions, making manual extraction and maintenance of accurate syntax rules time-consuming and error-prone; (3) Semantic gap: ASL code directly generated by large language models, although syntactically sound, often violates semantic constraints and is rejected by the compiler front-end. Existing compiler testing techniques are difficult to apply. Differential testing requires independent reference implementations, but there are no alternative implementations in this field. Equivalent modular input techniques rely on profile-guided mutations, but analyzing AML execution requires a complete operating system or a heavyweight simulation environment. These limitations have resulted in long-term inefficiency in the automated testing of related toolchains. This invention provides an automated compiler testing method based on specification inference and cycle consistency verification to solve the problem that existing technologies cannot effectively test compilers in the absence of reference implementations and execution oracles. Summary of the Invention

[0003] This invention provides an automated compiler testing method based on canonical inference and cycle consistency verification to address the problems of traditional compiler testing, such as reliance on differential testing, the need for reference implementations, and the inability to effectively test domain-specific compilers. This method automatically infers type-safe syntax from heterogeneous specifications using a large language model, generates semantically valid test cases based on the inferred syntax, and establishes cycle-consistent metamorphic test relationships, systematically detecting compiler defects without relying on hardware execution. To achieve automated compiler testing, this invention discloses an automated compiler testing method based on canonical inference and cycle consistency verification, specifically including the following steps:

[0004] Step 1, Specification Preprocessing: Obtain the ACPI specification document and organize it into a syntax skeleton set and a detailed reference set according to its structure. The syntax skeleton set comes from the syntax summary table at the beginning of the specification, and the detailed reference set comes from the detailed operator reference section.

[0005] Step 2, LLM-assisted syntax inference: A large language model with multi-agent collaboration is used to process the grammar skeleton set and detailed reference set in three stages. In the first stage, the generator agent extracts the basic formal grammar from the grammar skeleton set. In the second stage, the optimizer agent traverses the detailed reference set to perform type refinement and structure explicitation. In the third stage, the closed-loop verification mechanism verifies and corrects syntax errors through the syntax analysis toolchain, and finally outputs type-safe formal grammar rules.

[0006] Step 3, Syntax-guided test case synthesis: Based on the inferred formal grammar, initialize a fuzz tester containing ASL seed programs that satisfy global semantic constraints, parse the seed into an abstract syntax tree (AST), apply mutation operators of grammatical constraints to transform the AST, including subtree update, structural cardinality mutation, topological transformation and isomorphic recombination, and generate semantically valid ASL test cases.

[0007] Step 4, Cyclic Consistency Verification: Perform a compile-decompile-recompile cycle on the generated ASL test cases. First, use the compiler to compile the ASL test cases into the first AML binary file. Then, use the decompiler to decompile the binary file into the refactored source code. Finally, use the compiler to recompile the refactored source code into the second AML binary file.

[0008] Step 5, Binary Centralized Comparison: Before recompiling, external declaration synchronization processing is performed on the refactored source code. The normalized list of external declarations is signed and re-injected according to the original source code position. Then, a binary comparison is performed on the first AML binary file and the second AML binary file to detect compiler crashes, idempotency violations, or semantic inconsistency defects.

[0009] In step 1, the ACPI specification presents the language definition in a heterogeneous format, consisting of two parts: a syntax summary table and a detailed operator reference. The syntax summary table, located in the opening chapter of the specification, provides a high-level syntactic representation but relies on generic type rules and lacks specific constraints. The detailed operator reference describes the specific type requirements, optional parameters, and boundary conditions of each operator in prose form.

[0010] In step 2, the multi-agent collaboration mechanism refines the grammar through three stages. The first stage extracts the grammatical skeleton, capturing the hierarchical structure of ASL but using generic definitions. The second stage replaces generic tags with concrete type rules through type refinement and formalizes the optional parameters of the prose description into correct grammatical representations through explicit structure. The third stage, a closed-loop verification mechanism, inputs the generated grammar into the grammar analysis toolchain. If errors such as ambiguity or undefined symbols are reported, the error log is fed back to the repair agent for iterative repair until a valid and conflict-free grammar rule is generated.

[0011] In step 3, the mutation strategy for seed preservation ensures that the generated test cases inherit the global semantic context of the seed. The initialization of the seed satisfies context-related semantic constraints such as scope hierarchy and object declaration. Mutation operators operate at the Abstract Syntax Tree (AST) level and are strictly subject to inference syntax constraints: subtree update operators replace the target subtree with a newly generated subtree based on the same syntax rules; structural cardinality mutation operators modify container sizes to test dynamic memory allocation and boundary conditions; topology transformation operators modify the structural relationships between nodes to expose execution order and symbolic scope-related defects; and isomorphic recombination operators exchange subtrees from different ASTs but derived from the same syntax rules to propagate valid logical patterns.

[0012] In step 4, the cycle consistency verification leverages the assumption that compilation and decompilation should be semantically reversible. Since direct comparison of ASL source code involves syntactic variations and format differences, binary comparison abstraction sugar is used to provide semantic equivalence checks.

[0013] In step 5, the external declaration synchronization process addresses the noise introduced by the decompiler's normalization. While the compiler preserves the original order of external declarations, the decompiler collects, sorts, and relocates these scattered declarations to the file header, resulting in bit-level differences between semantically equivalent binary files. By extracting the precise location signatures of the external declarations from the original source code, the external declarations of the reconstructed source code are re-injected into their original locations before recompiling, eliminating spurious differences and ensuring that any remaining differences reflect genuine compiler defects.

[0014] This invention effectively generates semantically valid test cases by combining the syntax inference capabilities of large language models with structure-aware mutation strategies; it establishes a reliable defect detection oracle in the absence of a reference implementation by establishing a consistent metamorphic relationship of compilation-decompilation-recompilation; and it significantly reduces the false positive rate by binary-centered comparison and noise normalization, thus achieving effective automated testing of domain-specific compilers.

[0015] Beneficial effects: This invention implements an end-to-end automated testing framework for the ACPI compiler, enabling systematic detection of compiler defects without relying on reference implementations or hardware execution environments. Through LLM-assisted specification inference, type-safe syntax is automatically extracted from heterogeneous specifications, overcoming the difficulty of manually maintaining syntax. An AST-level mutation strategy with seed retention generates test cases that satisfy semantic constraints, effectively reaching the compiler's backend logic. Through cycle consistency verification and noise normalization, a reliable metamorphic test oracle is established, significantly reducing the false positive rate. Attached Figure Description

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention as described above or otherwise will become clearer.

[0017] Figure 1 This is an overall flowchart of the compiler automated testing method based on specification inference and cycle consistency verification provided in this embodiment of the invention.

[0018] Figure 2 This is a flowchart of LLM-assisted syntax inference multi-agent collaboration provided in an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of the seed retention AST-level mutation strategy provided in an embodiment of the present invention.

[0020] Figure 4 This is a flowchart of cycle consistency verification and noise normalization provided in an embodiment of the present invention. Detailed Implementation

[0021] To more clearly describe the purpose, technical solution, and advantages of this invention, this chapter provides a further detailed description of the invention in conjunction with the accompanying drawings.

[0022] Figure 1 The overall flowchart of the compiler automated testing method based on canonical inference and cycle consistency verification in this invention is shown, including 5 core steps, as follows:

[0023] Step 1-1, Specification Preprocessing: Parse the ACPI specification document and separate it into a syntactic skeleton set and a detailed reference set based on its structure. The syntactic skeleton set comes from the syntactic summary table at the beginning of the specification, providing the high-level hierarchical structure of the language; the detailed reference set comes from the detailed operator reference section, describing the constraints, types, and boundary conditions of each operator in prose form.

[0024] Steps 1-2, LLM-assisted syntax inference: Through the three-stage collaboration of generator proxy, optimizer proxy, and repair proxy, type-safe formal syntax rules are inferred from heterogeneous specifications. The generator proxy constructs the basic syntax skeleton, the optimizer proxy refines type constraints, and the closed-loop verification mechanism iteratively repairs syntax errors;

[0025] Steps 1-3, Syntax-Guided Test Case Synthesis: Based on the inferred formal grammar, initialize an ASL seed set that satisfies global semantic constraints. Parse the seed as an Abstract Syntax Tree (AST), and apply four mutation operators: subtree update, structural cardinality mutation, topological transformation, and isomorphic recombination. While maintaining syntactic validity, inherit the semantic context of the seed to generate diverse test cases.

[0026] Steps 1-4, Cyclic Consistency Verification: Perform a compile-decompile-recompile cycle for each generated ASL test case. First, use the compiler to compile it into an AML binary file, then use the decompiler to decompile it into refactored source code, and finally use the compiler to recompile it into a second binary file;

[0027] Steps 1-5, Binary Centralized Comparison: Before recompiling, external declaration synchronization is performed to eliminate noise, and then the two binary files are compared. This detects three types of defects: compiler crashes, idempotency violations, and semantic inconsistencies.

[0028] Figure 2 The flowchart of LLM-assisted syntax inference multi-agent collaboration is shown in detail to illustrate the specific implementation of steps 1-2, which is as follows:

[0029] Step 2-1: The generator proxy processes the grammar skeleton set and extracts the basic formal grammar. This grammar captures the hierarchical structure of ASL but depends on generic type definitions;

[0030] Step 2-2: The optimizer agent traverses the detailed reference set for each operator description, identifying type polymorphism and structural ambiguity issues in the specification. Type refinement and explicit structuralization are performed on each operator rule to generate an updated syntax.

[0031] Steps 2-3 involve the closed-loop verification mechanism inputting the syntax into the syntax analysis toolchain. If the toolchain reports errors such as ambiguity or undefined symbols, it will send the error log and the current syntax back to the repair agent.

[0032] Steps 2-4: The repair agent generates the repaired syntax based on the error log and re-executes step 2-3 for verification. Iterate until the toolchain reports success and outputs the final type-safe syntax.

[0033] Figure 3 A schematic diagram of the seed-preserving AST-level mutation strategy is shown, illustrating the specific operations of the four mutation operators in steps 1-3, including:

[0034] Step 3-1, Subtree Update Operator: Select the target subtree node, generate a new subtree according to the syntax rules corresponding to that node, and replace the original subtree. Since the new subtree is derived from the same rules, syntax validity is guaranteed;

[0035] Step 3-2, Structural cardinality mutation operator: For variable-length structures, perform injection, pruning, and copying operations to test the compiler's handling of dynamic memory allocation and boundary conditions;

[0036] Step 3-3, Topological Transformation Operators: Modify the structural relationships between AST nodes without changing the node content. For example, the promotion operation moves nested nodes to higher-level compatible ancestor nodes, forcing the compiler to recalculate symbol visibility;

[0037] Steps 3-4, Isomorphic Recombination Operator: Exchange subtrees between two ASTs, requiring that the root nodes of the exchanged subtrees derive from the same grammatical rules. This operation propagates valid logical patterns and detects semantic defects.

[0038] Figure 4 A flowchart illustrating the cycle consistency verification and noise normalization process is provided to demonstrate the specific implementation of steps 1-4 and 1-5, as follows:

[0039] Step 4-1: Compile the original ASL source code to generate the first AML binary file;

[0040] Step 4-2 involves decompiling the first binary file to generate reconstructed ASL source code. The decompiler performs code normalization processes, such as reordering and relocating external declarations.

[0041] Step 4-3: Before recompiling, perform external declaration synchronization. Extract the position signatures of all external declarations from the original source code. Remove the normalized list of external declarations from the refactored source code. Re-inject the external declarations according to the position signatures of the original source code to generate the synchronized source code;

[0042] Step 4-4: Compile the synchronized source code to generate a second AML binary file;

[0043] Step 5-1: Perform a binary comparison on the two binary files. If they are equal, the test passes; if they are not equal, proceed to step 5-2.

[0044] Step 5-2: Categorize and detect defects: compiler crashes, idempotency violations, and semantic inconsistencies. Report the detected defects to the developers.

[0045] This invention provides a method for testing domain-specific compilers using large language models and metamorphic testing. Many methods and approaches exist for implementing this technical solution. The above description is merely a preferred embodiment of this invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. An automated compiler testing method based on canonical inference and cycle consistency verification, characterized in that, Includes the following steps: Step 1: Obtain the ACPI specification document and decompose it into a syntax skeleton set and a detailed reference set; Step 2: Use a large language model to perform multi-stage processing on the grammatical skeleton set and detailed reference set to infer type-safe formal grammar rules; Step 3: Based on the formal syntax rules, generate semantically valid ASL test cases through the seed-preserving Abstract Syntax Tree (AST) level mutation strategy; Step 4: Perform a compile-decompile-recompile loop on the ASL test cases to obtain the first AML binary file and the second AML binary file; Step 5: Perform a binary-centered comparison of the first AML binary file and the second AML binary file to detect compiler stability faults or semantic inconsistency defects.

2. The method according to claim 1, characterized in that, Step 2 includes: Phase 1: The generator agent extracts the basic formal grammar from the set of grammar skeletons, which captures the hierarchical structure of ASL but uses generic type definitions; The second stage: The optimizer agent traverses the detailed reference set, and through type refinement and explicit structuring, replaces generic tags with specific type rules and formalizes the syntactic representation of optional parameters; The third stage: The closed-loop verification mechanism inputs the inferred syntax into the syntax analysis toolchain. If an error is reported, the error log is fed back to the repair agent for syntax repair until a valid and conflict-free syntax rule is generated.

3. The method according to claim 1, characterized in that, Step 3 includes: Initialize the fuzz tester and provide an ASL seed set that satisfies global semantic constraints; Parse the seed input into an abstract syntax tree (AST); The AST is transformed by mutation operators that apply grammatical constraints. These mutation operators include subtree update operators, structural cardinality mutation operators, topological transformation operators, and isomorphic recombination operators. While ensuring that the mutated AST satisfies the grammatical rules, it inherits the semantic context of the seed.

4. The method according to claim 1, characterized in that, Step 4 includes: The generated ASL test cases are compiled into a first AML binary file using a compiler. The first AML binary file was decompiled into reconstructed source code using a decompiler. The refactored source code is then recompiled into a second AML binary using a compiler.

5. The method according to claim 1, characterized in that, Before step 5, the following also applies: Perform external declaration synchronization processing on the refactored source code and extract the location signatures of external declarations in the original source code; Before recompiling, the list of normalized external declarations in the refactored source code is removed, and the external declarations are re-injected in the same signature as in the original source code.

6. The method according to claim 1, characterized in that, Step 5 includes: Detect compiler crashes: If the compiler terminates abnormally while compiling ASL test cases, it is marked as a memory safety vulnerability; Detecting idempotency violations: If the ASL test cases in step 4 compile successfully, but the refactored source code in step 4 fails to compile, then it is marked as a defect where the decompiler outputs invalid code or the compiler rejects valid code. Detect semantic inconsistency faults: If compilation is complete but the first AML binary file is not equal to the second AML binary file, it is marked as an opcode generation error or instruction decoding defect.

7. The method according to claim 3, characterized in that, The subtree update operator replaces the target subtree with a newly generated subtree according to the same syntax rules, while maintaining syntax validity; The structural cardinality mutation operator modifies the container size of long structures through injection, pruning, and copying operations; The topology transformation operator modifies the structural relationships between AST nodes without changing the node content, including promoting nested nodes to higher-level compatible ancestor nodes; The isomorphic recombination operator exchanges subtrees derived from the same syntax rules between two ASTs.

8. The method according to claim 1, characterized in that, In step 1: The set of grammatical skeletons is derived from the grammatical summary table at the beginning of the ACPI specification document, providing the high-level hierarchical structure of the language; The detailed reference set is derived from the detailed operator reference section of the ACPI specification document, and contains a prose description of the constraints, valid data types, and boundary cases for each operator.

9. An electronic device, characterized in that, include: Processor; and A memory for storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images