AI compiler-oriented optimization sensitive test system and method

Abstract

The invention discloses an optimization sensitive test system and method for an AI compiler, and belongs to the technical field of software testing. The method mainly comprises the following steps: firstly, automatically extracting an IR sub-graph mode capable of triggering specific optimization Pass from an existing test of a compiler through dynamic instrumentation; secondly, through a strategy of multiplexing compatible nodes or creating new nodes, the optimization modes are intelligently synthesized into a diversified seed calculation graph context, and a large number of test cases capable of effectively triggering compiler optimization are generated; and finally, efficiently revealing the defects of the AI compiler in the optimization stage by taking crash detection and reasoning consistency comparison as test predictions. The defect that an existing random generation or grammar generation method is insufficient in test coverage rate in the optimization stage is overcome, the deep optimization defect in the AI compiler can be automatically and efficiently detected, and the method has the high error detection rate and good universality.

Description

Technical Field

[0001] This invention belongs to the field of AI compiler testing technology, specifically relating to a method for detecting AI compiler optimization defects by generating optimization-sensitive test cases. Background Technology

[0002] 1) AI Compiler:

[0003] AI compilers (such as TVM, ONNXRuntime, and TensorRT) enable efficient deployment of AI models on diverse hardware platforms by converting AI models into intermediate representations (IR) and applying optimization strategies (such as operation fusion, memory planning, and subgraph reuse). These optimizations are typically designed for specific computation graph structures to ensure performance improvements. However, the complexity of the optimization process makes compilers susceptible to introducing errors, potentially leading to model execution anomalies or hardware runtime failures. Existing research indicates that the optimization component is the most error-prone part of an AI compiler, and its errors can have serious consequences, such as service unavailability or inconsistent model output. Therefore, ensuring the reliability and robustness of AI compiler optimizations has become a critical industry requirement.

[0004] 2) Test case generation:

[0005] Test case generation is a crucial research area in software testing. Its core objective is to automatically construct inputs that effectively reveal program defects, thereby reducing manual testing costs and improving test coverage. In traditional software systems, test case generation primarily relies on methods such as syntax analysis, symbolic execution, or search algorithms to ensure that the generated inputs meet program semantic constraints and cover different execution paths. With the widespread application of artificial intelligence (AI) systems, research on test case generation has gradually expanded to the model and compiler levels. As a key component connecting high-level models and low-level hardware, the AI ​​compiler's complex optimization logic and cross-layer dependencies make it difficult for random or syntax-driven generation methods to discover deep-seated defects. Therefore, test case generation for AI compilers not only needs to consider input validity but also needs to possess "optimization sensitivity," meaning that the generated tests can cover and trigger execution paths of specific optimization passes, thereby effectively detecting logical errors or inconsistent behavior that may exist during the optimization phase. This optimization-sensitive test generation has become an important direction for ensuring the reliability of AI compilers. Summary of the Invention

[0006] To address the specific technical challenge of defect detection during the optimization phase of AI compilers and improve test coverage and compiler reliability, this invention proposes an optimization-sensitive testing method and system for AI compilers. By generating optimization-sensitive test cases, it directly tests the triggering conditions of optimization passes, thus solving the problem of insufficient coverage in existing random generation methods.

[0007] To achieve the above-mentioned objectives, the present invention proposes the following technical solution:

[0008] Firstly, this invention proposes an optimization-sensitive testing method for AI compilers, comprising the following steps:

[0009] By instrumenting code, the AI ​​compiler monitors the execution process of optimization passes when running optimization-related tests, dynamically records the intermediate representation IR subgraphs that trigger specific optimization passes, and extracts the structural features and parameter constraints of the IR subgraphs to build an optimization pattern database.

[0010] The optimization pattern is selected from the optimization pattern database and embedded into the seed computation graph context. The mismatch between the pattern and the context is repaired by reusing compatible nodes in the seed computation graph or creating new nodes, and a complete computation graph with semantic correctness that can trigger the target optimization pass is generated.

[0011] The generated computation graph is input into the AI ​​compiler. By monitoring whether the compiler crashes and / or comparing whether the inference output of the computation graph before and after optimization is consistent, it can be determined whether a compiler defect has been detected.

[0012] Among them, the generated computation graphs that do not trigger defects will be added to the seed pool for use in the synthesis of computation graphs in subsequent iterations.

[0013] In some implementations, the optimization pattern extraction step further includes:

[0014] Structural analysis is performed on the recorded IR subgraph to extract information including operator type, data dependency relationship, and input-output tensor constraints, which are used as the triggering conditions and structural templates for the optimization pass to be invoked.

[0015] Based on the granularity of the optimization pass, optimization patterns are adaptively extracted: for subgraph-level optimization, patterns are extracted in units of the set of operators sharing input and output tensors; for block-level optimization, computational blocks are divided according to operator dependencies and topological order to form patterns.

[0016] In some implementations, the node reuse strategy in the computation graph synthesis step includes:

[0017] Analyze the metadata of existing tensor nodes in the seed computation graph, including shape, data type, and tensor rank;

[0018] Calculate the compatibility between the existing tensor nodes and the input-output constraints of the optimization pattern to be embedded;

[0019] Select nodes with high compatibility for reuse, and connect context nodes and pattern nodes by inserting new edges. Then perform static type checking and shape reasoning to verify the legality of the data flow.

[0020] In some implementations, the strategy for creating new nodes in the computation graph synthesis step includes:

[0021] When the context lacks a tensor that satisfies the optimization mode constraints, a bridging operator node is automatically created based on the constraints.

[0022] The bridging operator node includes one or more of the following: Reshape operator for adjusting dimensions, Cast operator for type conversion, Pad operator for padding operator element count, and Crop operator for cutting off element count.

[0023] After a new node is created, semantic verification at the IR level is performed, including topology sorting verification, cycle detection, and type checking, to ensure the correctness of the graph structure.

[0024] In some implementations, the inference consistency check in the test verification step specifically includes:

[0025] For the same generated computation graph, obtain its IR versions before and after optimization;

[0026] Run two versions under the same input conditions and obtain the inference output;

[0027] Calculate the Chebyshev distance between the two outputs. If the distance exceeds a preset threshold, an inconsistency defect is detected.

[0028] Secondly, this invention proposes an optimization-sensitive testing system for AI compilers that implements any of the methods described above, comprising:

[0029] The optimization pattern extraction module is configured to dynamically extract and record the IR subgraphs that trigger compiler optimization passes through code instrumentation, analyze and store their structural templates, and form an optimization pattern database.

[0030] The computation graph synthesis module is configured to embed patterns from the optimization pattern database into the seed computation graph. By reusing compatible nodes or creating new nodes, it generates a semantically correct test computation graph that can trigger optimization.

[0031] The test oracle module is configured to input the generated computation graph into the compiler, determine whether to reveal compiler defects through crash detection and inference consistency comparison, and add test cases that do not trigger defects to the seed pool for iterative synthesis.

[0032] The computational graph synthesis module is further configured as follows:

[0033] When implementing the node reuse strategy, a compatibility score is calculated based on shape, data type, and semantic similarity, and nodes that are most similar to the original context of the pattern are selected first.

[0034] When executing the strategy for creating a new node, random but compliant parameters are generated for the bridging operator, and candidate path switching and alternative connection paths are automatically performed when loops or conflicts are detected.

[0035] Thirdly, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0036] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0037] 1) By extracting real optimization trigger patterns from the compiler's own tests, especially ensuring that the generated test cases map to each optimization pass of the compiler; and through experiments demonstrating that, under the same test budget, this invention (OATest) detects significantly more unknown defects than baseline methods based on syntax generation (NNSmith) and large language models (WhiteFox, LLMTest), revealing up to 42 officially confirmed defects. High optimization trigger rate and defect detection capability:

[0038] 2) By using node reuse and new node creation strategies, the structural and semantic correctness of the synthesized computation graph is ensured, and invalid tests caused by errors are avoided, thereby improving testing efficiency.

[0039] 3) It can deeply cover the code paths related to compiler optimization. Experimental data shows that its branch coverage and line coverage in optimization-related code are significantly improved compared with the baseline method (average improvement of 60.20% and 66.98%, respectively), indicating that it can explore the compiler's optimization logic more fully.

[0040] 4) This invention does not depend on the implementation details of a specific compiler and has been successfully applied to two mainstream compilers, TVM and ONNXRuntime. It also supports intermediate representations such as Relax IR and ONNX IR, indicating that it has good versatility and can be widely used in the testing scenarios of other AI compilers. Attached Figure Description

[0041] Figure 1This is a schematic diagram of the optimized sensitive testing system structure for AI compilers according to the present invention.

[0042] Figure 2 This is a schematic diagram of the optimized sensitive testing technology process for AI compilers according to the present invention.

[0043] Figure 3 This is a roadmap for an implementation of the AI ​​compiler-oriented optimization-sensitive testing system and method of the present invention.

[0044] Figure 4 This is a flowchart of the node reuse strategy of the present invention.

[0045] Figure 5 This is a flowchart illustrating the new node creation strategy of the present invention.

[0046] Figure 6 Example image of a detected defect. Detailed Implementation

[0047] The technical solution proposed in this invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] This invention addresses the challenge of defect detection during the optimization phase of AI compilers, improving the reliability and efficiency of the compiler through automated testing, demonstrating its technical sophistication.

[0049] Example 1: Figure 1 The present invention illustrates an optimization-sensitive testing system for AI compilers, comprising an optimization pattern extraction module 100, a computation graph synthesis module 200, and a test oracle module 300.

[0050] The optimization pattern extraction module 100 is configured to dynamically extract and record the IR subgraph that triggers the compiler optimization pass through code instrumentation, analyze and store its structural template, and form an optimization pattern database.

[0051] The computation graph synthesis module 200 is configured to embed patterns from the optimization pattern database into the seed computation graph, and generate a semantically correct test computation graph that can trigger optimization by reusing compatible nodes or creating new nodes.

[0052] The test oracle module 300 is configured to input the generated computation graph into the compiler, determine whether a compiler defect is revealed by crash detection and inference consistency comparison, and add test cases that do not trigger defects to the seed pool for iterative synthesis.

[0053] The computational graph synthesis module 200 is further configured as follows:

[0054] When implementing the node reuse strategy, a compatibility score is calculated based on shape, data type, and semantic similarity, and nodes that are most similar to the original context of the pattern are selected first.

[0055] When executing the strategy for creating a new node, random but compliant parameters are generated for the bridging operator, and candidate path switching and alternative connection paths are automatically performed when loops or conflicts are detected.

[0056] Example 2: Figure 2 This invention illustrates an optimized sensitive testing technique for AI compilers, comprising the following steps:

[0057] Step 1: Optimization Pattern Extraction: This step involves monitoring the execution process of optimization passes by the AI ​​compiler during optimization-related tests using code instrumentation. It dynamically records the intermediate representation (IR) subgraphs that trigger specific optimization passes, extracts the structural features and parameter constraints of these IR subgraphs, and constructs an optimization pattern database. The specific details are as follows:

[0058] In the optimization pattern extraction step, optimization patterns are automatically identified and extracted from the optimization-related tests of the AI ​​compiler. These optimization-related tests are typically written by the AI ​​compiler developers to verify the correctness of a specific optimization pass; their input is often an IR subgraph that triggers a specific optimization. Dynamic instrumentation is performed during compiler execution to monitor the execution of each optimization pass in real time. When an optimization is triggered, the system automatically records the IR subgraph that triggered it and establishes a mapping relationship between it and the corresponding optimization pass, forming an optimization trigger sample. Optimization passes are key technologies in compilers or AI frameworks used to improve code performance or efficiency; their core lies in the rational arrangement of the execution order and dependencies of passes. A pass is an operation in the compiler or framework that analyzes and optimizes intermediate representations (such as LLVM IR, computation graphs).

[0059] Next, structural analysis is performed on the recorded IR subgraphs to extract operator types, data dependencies, input / output tensor constraints, and parameter settings related to the optimization logic. For example, for constant folding optimizations, operator nodes involving constant tensors and their arithmetic relationships are extracted; for fusion optimizations, operator sequences and their shared input / output connection methods are analyzed. Through these analyses, an optimization pattern database is generated, where each record corresponds to the triggering conditions and structural template of an optimization pass to be invoked.

[0060] Furthermore, an adaptive extraction mechanism supporting different optimization granularities was designed. When an optimization pass operates at the subgraph level, the system extracts optimization patterns on a per-unit basis, using the set of operators sharing input-output tensors. When an optimization pass operates on a single computational block, the system divides the computational block into patterns based on operator dependencies and topological order. This granularity-adaptive strategy accurately captures the triggering structures of various optimization types and compresses the pattern size while maintaining semantic integrity. Finally, the extracted optimization patterns are saved as structural templates, including operator graphs, parameter constraints, and tensor type information. These patterns will serve as the basis for the next stage of computational graph synthesis to support pattern embedding in diverse contexts.

[0061] Step 2, Computational Graph Synthesis Step: Select an optimization pattern from the optimization pattern database and embed it into the seed computational graph context. Repair the mismatch between the pattern and the context by reusing compatible nodes in the seed computational graph or creating new bridging nodes, generating a semantically correct and complete computational graph that can trigger the target optimization pass. The relevant description is as follows:

[0062] Furthermore, in order to achieve effective connection between the optimization mode and the context while maintaining the semantic correctness of the graph structure, two complementary synthesis strategies are designed: a strategy of reusing compatible nodes and a strategy of creating new nodes.

[0063] 2.1 The node reuse strategy in the computational graph synthesis step includes:

[0064] Analyze the metadata of existing tensor nodes in the seed computation graph, including shape, data type, and tensor rank;

[0065] Calculate the compatibility between the existing tensor nodes and the input-output constraints of the optimization pattern to be embedded;

[0066] Select nodes with high compatibility for reuse, and connect context nodes and pattern nodes by inserting new edges. Then perform static type checking and shape reasoning to verify the legality of the data flow.

[0067] Figure 3 The node reuse strategy flow of the present invention is shown below:

[0068] Analyze the existing tensor node metadata in the seed computation graph. In this step, scan and obtain the key attributes of all existing nodes in the current computation graph to prepare for subsequent matching.

[0069] The compatibility of the nodes with the input and output constraints of the optimization pattern is calculated by comparing the node metadata obtained in the previous step with the constraints required by the optimization pattern to be embedded, and calculating a compatibility score for each node.

[0070] Based on the calculated compatibility, candidate nodes with high compatibility are selected for reuse, that is, one or more nodes that best meet the constraints are selected from the candidate node pool.

[0071] After selecting a node, insert a new edge to connect the context node and the pattern node, thereby integrating the optimization pattern into the existing computation graph structure.

[0072] Subsequently, static type checking and shape reasoning are performed to validate the new connections, ensuring that the data flow type and dimension are compatible, thus achieving quality verification. If the verification passes, the data flow validity strategy is completed; if the verification fails, the process will backtrack to selecting a candidate node with high compatibility for reuse and then reselecting the node, trying other candidate nodes again, forming an iterative optimization closed loop until the strategy is successfully completed or all possibilities are exhausted.

[0073] In summary, the core value of this strategy lies in its ability to efficiently and reliably integrate optimization patterns into new contexts by intelligently identifying and reusing semantically compatible nodes in existing computation graphs. This avoids creating redundant nodes while maximizing the semantic correctness of the graph structure and its optimization triggering capabilities.

[0074] Related concepts: A computation graph represents the computational logic of an AI model. It is a directed graph structure, usually represented as a directed acyclic graph, where nodes represent computational operations (called operator nodes or operation nodes), and edges represent data dependencies (i.e., tensors).

[0075] Operator Nodes: Each operator node represents a specific computational operation, such as convolution, matrix multiplication, addition, or activation function ReLU. Operator nodes contain an identifier for their operation type and the parameters required for the operation (such as kernel size, stride, etc.).

[0076] Tensors: Tensors are the flowing data carriers in a computation graph, a generalized concept of multidimensional arrays. Edges connect operator nodes and carry tensors. The output tensor of one operator becomes the input tensor of subsequent operators. Tensors have key metadata attributes, including: Data type: Defines the type of elements in the tensor, such as float32, int64, etc.

[0077] Shape: Defines the length of each dimension of the tensor, for example, [batch_size, height, width, channels]. Shape information is crucial for memory allocation and computational correctness.

[0078] Rank: The number of dimensions of a tensor.

[0079] Graph Structure: A computation graph defines the data flow of computation through nodes and edges. It explicitly specifies the execution order (topological order) of computations and the dependencies between operations. The AI ​​compiler receives the model represented in the form of a computation graph, performs a series of optimizations and transformations on it, and ultimately generates efficient code for specific hardware.

[0080] Specifically, the node reuse strategy aims to analyze existing tensor nodes in the context computation graph (a context computation graph refers to any semantically correct complete computation graph, which can be automatically generated using existing syntax-based computation graph generation tools such as NNSmith and HirGen) and select those nodes that are compatible with the input and output of the optimization pattern in terms of shape, data type, and semantics, thus achieving seamless integration between the pattern and the context. This strategy is based on the observation that in the IR of an AI compiler, multiple operators often share similar input-output constraints, such as the same tensor shape, data type, or dimensional relationships. If these compatible nodes can be identified and reused, the integrity of the graph structure can be maintained, the creation of invalid nodes can be effectively reduced, and the efficiency of graph synthesis can be improved. In implementation, the meta-information of all reusable tensors is first extracted from the context computation graph. This includes its shape, data type, and rank. Subsequently, for each optimization pattern to be embedded, the system analyzes its input-output constraints, which were determined during the pattern extraction phase using operator parameters and tensor inference rules. A compatibility score is calculated between the context tensor and the pattern's input and output using a matching algorithm. This score comprehensively considers whether the shape matches perfectly, whether the types are consistent, and whether the operator semantics are reusable. When multiple candidate nodes simultaneously meet the compatibility conditions, the system prioritizes the node most similar to the original context structure of the pattern to maintain the stability of optimization triggering. After successful reuse, new edges are inserted into the computation graph to connect the context nodes and the pattern nodes, forming an expanded computation graph structure. At this point, the system automatically triggers a static type check and shape inference to verify the validity of the connected data flow.

[0081] 2.2 The strategy for creating new nodes in the computational graph synthesis step includes:

[0082] When the context lacks a tensor that satisfies the optimization mode constraints, a bridging operator node is automatically created based on the constraints.

[0083] The bridging operator node includes one or more of the following: Reshape operator for adjusting dimensions, Cast operator for type conversion, Pad operator for padding operator element count, and Crop operator for cutting off element count.

[0084] After a new node is created, semantic verification at the IR level is performed, including topology sorting verification, cycle detection, and type checking, to ensure the correctness of the graph structure.

[0085] Figure 5 The process for creating a new node is shown below:

[0086] The judgment is based on the current computation graph context, specifically checking if the required tensor exists. This is the trigger condition for the entire strategy.

[0087] When a node needs to be created, the system selects an appropriate operator from a predefined bridging operator library (including Reshape, Cast, Pad, Crop, etc.) and automatically generates random parameters that conform to semantic constraints to bridge the gap between the context and the optimization mode.

[0088] After a new node is created, it must undergo rigorous semantic verification at the IR level. This step is crucial to ensuring that the generated computation graph is syntactically and logically correct, effectively preventing issues such as loops and type conflicts.

[0089] The flowchart includes the crucial "candidate path switching" mechanism. When validation fails, the system does not terminate directly but automatically performs the failed candidate path switching operation and attempts to select other types of bridging nodes or connection paths. This greatly enhances the robustness and success rate of the strategy, achieving fault tolerance and iteration.

[0090] Specifically, when reusable nodes are absent from the context computation graph, or when reusing nodes results in type or shape incompatibility, a new node creation strategy is employed. This strategy repairs the mismatch between the schema and the context by automatically inserting intermediate bridging operator nodes into the computation graph, thus ensuring the semantic validity of the generated graph. In this strategy, the input and output tensor constraints of the optimization schema are first analyzed, including shape, rank, dtype, and numerical range. When a tensor node satisfying these constraints is found to be missing in the context, the system automatically creates new operator nodes to generate matching input tensors based on the constraint information. For example, when the schema requires input with a specific dimension, a Reshape operator is inserted to adjust the dimension; when the data types are mismatched, a Cast operator is automatically created for type conversion; when the number of tensor elements is mismatched, Pad or Crop nodes are inserted to fill or cut elements to ensure element alignment. During the new node creation process, random but compliant parameter values ​​are generated for each node, and necessary input and output connections are automatically filled in based on tensor dependencies. To prevent overly complex graph structures or the generation of invalid dependencies, the system performs semantic verification at the IR level after each node creation, including topology sorting verification, cycle detection, and type checking. If a new node insertion causes a loop or conflict, the system automatically switches the candidate path to the previous synthesis state and retrys different connection paths. Furthermore, to enhance the diversity of the generated graph, the synthesis process is executed multiple times under the same optimization mode, randomly selecting the context insertion point and node creation strategy each time. This allows for the generation of diverse test cases with significantly different structures under the same mode, covering more potential optimization trigger scenarios. This mechanism not only ensures the semantic correctness of each generated graph but also improves overall test coverage and optimization trigger rate. The strategy for creating new nodes is particularly crucial for optimizations that rely on complex input structures.

[0091] Step 3, Testing and Verification Steps: Input the generated computation graph into the AI ​​compiler, and determine whether a compiler defect has been detected by monitoring whether the compiler crashes and / or comparing whether the inference output of the computation graph before and after optimization is consistent.

[0092] Among them, the generated computation graphs that do not trigger defects will be added to the seed pool for use in the synthesis of computation graphs in subsequent iterations.

[0093] The inference consistency check in the test verification step specifically includes:

[0094] For the same generated computation graph, obtain its IR versions before and after optimization;

[0095] Run two versions under the same input conditions and obtain the inference output;

[0096] Calculate the Chebyshev distance between the two outputs. If the distance exceeds a preset threshold, an inconsistency defect is detected.

[0097] Example 3: The present invention proposes a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method described in any one of Examples 1.

[0098] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] The beneficial effects are verified as follows:

[0100] a) Experimental setup:

[0101] Test Objects: To systematically evaluate the effectiveness of the proposed OATest method, empirical studies were conducted on several mainstream AI compilers. The test objects include two widely used open-source compilers: TVM and ONNXRuntime. The experiment aims to systematically detect defects in the optimization phase; therefore, the focus is on test inputs that can trigger graph-level optimization passes. TVM is an end-to-end compiler that can optimize and deploy models built with various frameworks to various platforms, containing 24 block-level optimizations and 41 subgraph-level optimizations distributed across more than 140 source files. The latest version of TVM (commit id: 292ecfd) was used to evaluate the OATest. ONNXRuntime is a graph optimization compiler for ONNX models, supporting almost all pre-trained models (converted through various frameworks), containing 21 block-level optimizations and 25 subgraph-level optimizations distributed across more than 130 source files. The latest version of ONNXRuntime (commit id: 5c1b7cc) was used for testing. Both involve two mainstream computational graph formats, Relax IR and ONNX IR, and are compatible with almost all mainstream AI compilers. Therefore, OATest can be directly used for fuzz testing with other compatible compilers, making it highly versatile.

[0102] Baseline Methods: Three mainstream test generation tools were selected as baseline methods: NNSmith, WhiteFox, and LLMTest. NNSmith is a state-of-the-art syntax-based AI compiler testing tool capable of randomly generating multi-operator models to cover different computational paths. WhiteFox utilizes a large language model to directly infer optimization rules from the compiler source code and generate tests, while LLMTest uses the latest DeepSeek-v3 and Qwen2.5-Coder models to generate optimization-related test inputs. These three baseline methods represent the current mainstream approaches: random generation, LLM semantic generation, and generation based on manual constraints.

[0103] b) Experimental results:

[0104] The method proposed in this invention was systematically evaluated on two mainstream AI compilers, TVM and ONNXRuntime. Experimental results show that, under a 12-hour testing budget, a total of 56 previously unknown defects were detected, of which 42 were officially confirmed and 24 have been fixed (defect information is as follows). Figure 6 (As shown). In contrast, NNSmith and WhiteFox detected only 5 and 4 defects, respectively. The detected defects mainly manifested as two symptoms: crashes and inconsistencies. Crashes refer to unexpected termination of the compiler during compilation (36 out of 42 defects), and inconsistencies refer to mismatches in inference results or IR in differential testing (6 confirmed defects). These defects are serious and can lead to flawed optimizations that are difficult to detect. Experiments show that both test predictions can effectively detect the corresponding defects. Next, based on developer discussions and patch fixes, the 42 confirmed defects were analyzed, revealing four different root causes. The results show that this invention can reveal multiple root cause defects, including: erroneous code logic, tensor shape problems, improper exception handling, and type errors. In the comparative experiment on code coverage with the baseline method, the average branch coverage of optimization-related code was improved by an average of 60.20%, and the line coverage was improved by 66.98%. In addition, ablation experiments further verified the role of each component. When the adaptive granularity extraction mechanism was removed, the number of defect detections decreased by about half, indicating that granularity control is crucial to the accuracy of optimization pattern extraction. In summary, the OATest technology proposed in this invention achieves efficient testing and defect detection during the optimization phase of the AI ​​compiler through automated optimization pattern extraction, structured computation graph synthesis, and a reliable testing and verification mechanism. It overcomes the problems of insufficient test coverage and unstable generation in the optimization phase of existing technologies, and has high research value and engineering application potential.

[0105] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention. The above description is merely a preferred embodiment of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered to fall within the protection scope of the present invention.

Claims

1. An optimization-sensitive testing method for AI compilers, characterized in that, The method performs the following steps: By instrumenting code, the AI ​​compiler monitors the execution process of optimization passes when running optimization-related tests, dynamically records the intermediate representation IR subgraphs that trigger specific optimization passes, and extracts the structural features and parameter constraints of the IR subgraphs to build an optimization pattern database. The optimization pattern is selected from the optimization pattern database and embedded into the seed computation graph context. The mismatch between the pattern and the context is repaired by reusing compatible nodes in the seed computation graph or creating new bridging nodes, and a complete computation graph with semantic correctness that can trigger the target optimization pass is generated. The generated computation graph is input into the AI ​​compiler. By monitoring whether the compiler crashes and / or comparing whether the inference output of the computation graph before and after optimization is consistent, it can be determined whether a compiler defect has been detected. Among them, the generated computation graphs that do not trigger defects will be added to the seed pool for use in the synthesis of computation graphs in subsequent iterations.

2. The method according to claim 1, characterized in that, The optimization mode extraction step further includes: Structural analysis is performed on the recorded IR subgraph to extract information including operator type, data dependency relationship, and input-output tensor constraints, which are used as the triggering conditions and structural templates for the optimization pass to be invoked. Based on the granularity of the optimization pass, optimization patterns are adaptively extracted: for subgraph-level optimization, patterns are extracted in units of the set of operators sharing input and output tensors; for block-level optimization, computational blocks are divided according to operator dependencies and topological order to form patterns.

3. The method according to claim 1, characterized in that, The node reuse strategy in the computation graph synthesis step includes: Analyze the metadata of existing tensor nodes in the seed computation graph, including shape, data type, and tensor rank; Calculate the compatibility between the existing tensor nodes and the input-output constraints of the optimization pattern to be embedded; Select nodes with high compatibility for reuse, and connect context nodes and pattern nodes by inserting new edges. Then perform static type checking and shape reasoning to verify the legality of the data flow.

4. The method according to claim 1, characterized in that, The strategies for creating new nodes in the computational graph synthesis step include: When the context lacks a tensor that satisfies the optimization mode constraints, a bridging operator node is automatically created based on the constraints. The bridging operator node includes one or more of the following: Reshape operator for adjusting dimensions, Cast operator for type conversion, Pad operator for padding operator element count, and Crop operator for cutting off element count. After a new node is created, semantic verification at the IR level is performed, including topology sorting verification, cycle detection, and type checking, to ensure the correctness of the graph structure.

5. The method according to claim 1, characterized in that, The inference consistency check in the test verification step specifically includes: For the same generated computation graph, obtain its IR versions before and after optimization; Run two versions under the same input conditions and obtain the inference output; Calculate the Chebyshev distance between the two outputs. If the distance exceeds a preset threshold, an inconsistency defect is detected.

6. The method according to claim 3, characterized in that, The steps for calculating compatibility include: calculating a compatibility score based on the shape, data type, and semantic similarity of tensor nodes, and setting a threshold to prioritize the node with the highest score; when verification fails, automatically backtracking to the node selection step for iterative retry.

7. The method according to claim 4, characterized in that, The steps for creating bridging operator nodes include: generating random but compliant parameters for the Reshape operator, Cast operator, Pad operator, or Crop operator; and automatically switching candidate paths to try alternative connection paths when a loop or conflict is detected during semantic verification.

8. An optimization-sensitive testing system for AI compilers that implements the method as described in any one of claims 1-7, characterized in that, include: The optimization pattern extraction module is configured to dynamically extract and record the IR subgraphs that trigger compiler optimization passes through code instrumentation, analyze and store their structural templates, and form an optimization pattern database. The computation graph synthesis module is configured to embed patterns from the optimization pattern database into the seed computation graph. By reusing compatible nodes or creating new nodes, it generates a semantically correct test computation graph that can trigger optimization. The test oracle module is configured to input the generated computation graph into the compiler, determine whether to reveal compiler defects through crash detection and inference consistency comparison, and add test cases that do not trigger defects to the seed pool for iterative synthesis.

9. The system according to claim 8, characterized in that, The computational graph synthesis module is further configured as follows: When implementing the node reuse strategy, a compatibility score is calculated based on shape, data type, and semantic similarity, and nodes that are most similar to the original context of the pattern are selected first. When executing the strategy for creating a new node, random but compliant parameters are generated for the bridging operator, and candidate path switching and alternative connection paths are automatically performed when loops or conflicts are detected.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images