A graph compiling method and device, electronic equipment and storage medium

Abstract

The application provides a graph compiling method and device, electronic equipment and storage medium, comprising obtaining an initial computing graph corresponding to a to-be-compiled model, the to-be-compiled model being deployed on a target hardware, and the target hardware adopting a block format data layout; replacing a target operator node in the initial computing graph with a hardware-specific subgraph, the hardware-specific subgraph being obtained by extracting an operator distribution logic of the target hardware through a graph capturing mechanism, so as to explicitly introduce a data format conversion operation between a linear format and the block format in the initial computing graph, and obtaining a first computing graph; performing redundant format conversion elimination on the first computing graph to obtain a second computing graph, the redundant format conversion elimination comprising at least one of constant folding, common subexpression elimination and cross-operation round trip conversion elimination; and compiling the second computing graph into executable code of the target hardware. The application reduces data carrying and computing kernel startup overhead, and improves inference performance of a model on an artificial intelligence chip.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a graph compilation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Some artificial intelligence (AI) chips use a tiled data layout, where matrix multiplication units require data to be stored in tiles of a specific size (e.g., 32×32). In contrast, general-purpose graphics processing units (GPUs) use a linear (row-major) data layout. Therefore, data must be converted from linear to tiled format before entering the computation units of such AI chips, and converted back to linear format after computation.

[0003] In Large Language Models (LLMs) based on the Transformer architecture, matrix multiplication is the most frequent computational operation, resulting in a large number of format conversions. Taking a 2-layer Llama model as an example, 18 matrix multiplication operations introduce a total of 90 format conversions. These operations only change the physical arrangement of the data and do not generate computational value, but they consume memory bandwidth and computational kernel startup resources, affecting the model's inference performance.

[0004] Existing graph compilation technologies struggle to eliminate the aforementioned redundancy. Mainstream AI compilers are designed for linear data layouts and lack the ability to identify and optimize block format conversions. While compilation toolchains for dedicated AI chips implement format conversions at the operator level, their optimization granularity is limited to individual operators, and redundancy such as repeated conversions of constant weights, repeated conversions of shared inputs, and cross-operation round-trip conversions still exist. Furthermore, the format conversion logic is hard-coded into the operator distribution implementation, making it difficult to perform global analysis and optimization at the compiler level.

[0005] Therefore, a graph compilation method that starts from the global computation graph and systematically eliminates redundant format conversions is needed to solve at least one of the above problems. Summary of the Invention

[0006] To overcome at least one of the above problems, the present invention provides a graph compilation method, apparatus, electronic device, and storage medium.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] In a first aspect, the present invention provides a graph compilation method, comprising: obtaining an initial computation graph corresponding to a model to be compiled, wherein the model to be compiled is deployed on target hardware and the target hardware adopts a block format data layout; replacing target operator nodes in the initial computation graph with a hardware-specific subgraph, wherein the hardware-specific subgraph is extracted from the operator distribution logic of the target hardware through a graph capture mechanism, so as to explicitly introduce a data format conversion operation between linear format and block format in the initial computation graph to obtain a first computation graph; performing redundant format conversion elimination on the first computation graph to obtain a second computation graph, wherein the redundant format conversion elimination includes at least one of constant folding, common subexpression elimination, and cross-operation round-trip conversion elimination; and compiling the second computation graph into executable code of the target hardware.

[0009] In some optional embodiments, the hardware-specific subgraph is extracted by: extracting the processing flow function of the target operator from the operator distribution logic of the target hardware; and using the graph capture mechanism to trace the processing flow function to generate the hardware-specific subgraph corresponding to the processing flow function.

[0010] In some optional embodiments, the step of using the graph capture mechanism to trace the processing flow function includes: registering a custom compilation backend, wherein the custom compilation backend is used to receive and save the graph object generated during the tracing process, and throw an exception after saving to short-circuit the actual execution of the processing flow function.

[0011] In some optional embodiments, replacing the target operator node in the initial computation graph with a hardware-specific subgraph includes: obtaining the input edge of the target operator node; mapping the input placeholder node of the hardware-specific subgraph to the input edge of the target operator node; copying the internal node of the hardware-specific subgraph to the initial computation graph to replace the position of the target operator node, and redirecting the successor edge of the output node of the hardware-specific subgraph to the successor node of the target operator node; and deleting the target operator node.

[0012] In some optional embodiments, the method further includes: caching the generated hardware-specific subgraph using a combination of operation type, shape and data type of each input tensor as a cache key; and when a target operator node matching the cache key exists, reusing the cached hardware-specific subgraph to replace the target operator node.

[0013] In some optional embodiments, the constant folding includes: in response to the input placeholder node having a value type of model parameter type and existing in the model parameter list, promoting the input placeholder node to a constant attribute read node pointing to the constant storage of the model module; starting from the constant attribute read node, tracing a format conversion operation chain that sequentially includes a padding operation node and a linear to block format conversion operation node; executing the format conversion operation chain on the compiler's host machine to obtain a pre-computation result, writing the pre-computation result into the constant storage, replacing the format conversion operation chain with the constant attribute read node that reads the pre-computation result, and connecting the output edge of the replaced constant attribute read node to the successor node at the end of the format conversion operation chain.

[0014] In some optional embodiments, the common subexpression elimination includes: eliminating type conversion nodes in the first computation graph where the source data type and the target data type are the same; performing common subexpression elimination on the view transformation nodes associated with each of the multiple operator nodes sharing the same input data, and merging view transformation nodes with the same input source and the same target shape; based on the merging of the view transformation nodes, performing common subexpression elimination on the fill nodes associated with each of the multiple operator nodes, and merging fill nodes with the same input source; based on the merging of the fill nodes, performing common subexpression elimination on the linear format to block format conversion nodes associated with each of the multiple operator nodes, and merging linear format to block format conversion nodes with the same input source.

[0015] In some optional embodiments, the cross-operation round-trip conversion elimination includes: identifying a data flow path in the first computation graph where there is a round-trip conversion from block format to linear format and back to block format, and the successor operator after the round-trip conversion supports block format data input; deleting block format to linear format conversion nodes and linear format to block format conversion nodes in the data flow path; directly connecting the block format data output of the predecessor node at the entrance of the data flow path to the successor operator at the exit of the data flow path; retaining one linear format to block format conversion at the entrance of the data flow path, and retaining one block format to linear format conversion at the exit of the data flow path.

[0016] In some optional embodiments, identifying a data flow path in the first computation graph where there is a round-trip conversion from block format to linear format and back to block format, and where subsequent operators after the round-trip conversion support block format data input, includes: marking computation nodes in the first computation graph that perform calculations using block format data; starting from the output of the computation node, tracing along the data flow direction, and grouping downstream nodes that meet the following conditions into the same operation chain: the downstream node is a block format to linear format conversion node and its output is directly or after element-wise operation into a linear format to block format conversion node, or the downstream node is an element-wise operation node or a shape transformation operation node; the starting point of the chain is the position where data first enters the block format from the linear format, and the ending point of the chain is the position where data last returns from the block format to the linear format.

[0017] In some optional embodiments, when there is a shape transformation operation node in the data flow path, the cross-operation round-trip conversion elimination further includes: determining whether the shape transformation operation node meets the condition for direct execution in the block format; if it does, then by remapping the data access index relationship of the shape transformation operation node, the shape transformation is directly completed in the block format without reverting the data to the linear format.

[0018] In some optional embodiments, determining whether the shape transformation operation node meets the conditions for direct execution in the block format includes: determining whether the transformation boundary of the shape transformation operation node is aligned with the block boundary of the block format; determining whether the dimension involved in the shape transformation operation node is a dimension participating in the block format, so as to determine the adjustment method of inter-block index and intra-block index when performing the shape transformation in the block format; determining whether the shape transformation operation node introduces irregular data dependencies across block boundaries; if the transformation boundary is aligned with the block boundary and the shape transformation operation node does not introduce irregular data dependencies across block boundaries, then the shape transformation operation node is determined to meet the conditions for direct execution in the block format, and the shape transformation is performed according to the determined adjustment method of inter-block index and intra-block index.

[0019] In some optional embodiments, the shape transformation operation node is a tensor shape reshaping operation node. The step of directly completing the shape transformation in the block format by remapping the data access index relationship of the shape transformation operation node includes: recalculating the element index mapping relationship in the block format to make the physical storage addresses before and after the transformation equivalent, so as to perform zero-copy shape reinterpretation of the tensor shape reshaping operation node.

[0020] In some optional embodiments, the shape transformation operation node is a tensor dimension transpose operation node. The step of directly completing the shape transformation in the block format by remapping the data access index relationship of the shape transformation operation node includes: modifying the source address step size parameter and the target address step size parameter of the data transport descriptor to complete the transpose corresponding to the tensor dimension transpose operation node during the data transport process.

[0021] In some optional embodiments, the shape transformation operation node is an element-wise operation node. The step of directly completing the shape transformation in the block format by remapping the data access index relationship of the shape transformation operation node includes: determining that the element-wise operation node does not need to remap the data access index relationship, keeping the block format label of the input data and output data of the element-wise operation node unchanged, and directly passing through for execution.

[0022] In some optional embodiments, compiling the second computation graph into executable code for the target hardware includes at least one of the following: merging adjacent element-wise operations in the second computation graph into a single computation kernel; generating corresponding computation kernel code based on the target hardware; and generating call encapsulation code for orchestrating the execution order of each computation kernel.

[0023] Secondly, the present invention provides a graph compilation apparatus, comprising: a graph acquisition module, configured to acquire an initial computation graph corresponding to a model to be compiled, wherein the model to be compiled is deployed on target hardware and the target hardware adopts a block format data layout; a subgraph replacement module, configured to replace target operator nodes in the initial computation graph with hardware-specific subgraphs, wherein the hardware-specific subgraphs are extracted from the operator distribution logic of the target hardware through a graph capture mechanism, thereby explicitly introducing a data format conversion operation between linear and block formats in the initial computation graph to obtain a first computation graph; a redundancy elimination module, configured to perform redundancy format conversion elimination on the first computation graph to obtain a second computation graph, wherein the redundancy format conversion elimination includes at least one of constant folding, common subexpression elimination, and cross-operation round-trip conversion elimination; and a compilation module, configured to compile the second computation graph into executable code for the target hardware.

[0024] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the graph compilation method as described in any of the first aspects above.

[0025] Fourthly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the graph compilation method as described in any of the first aspects above.

[0026] The technical solution of the present invention has the following advantages or beneficial effects:

[0027] (1) This invention explicitly exposes the format conversion operations implicit in the operator distribution logic to the computation graph, and performs at least one of the following redundant format conversion eliminations on the computation graph: constant folding, common subexpression elimination, and cross-operation round-trip conversion elimination. This solves the problem of a large number of redundant executions of format conversion operations, which consume memory bandwidth and computation kernel startup resources in artificial intelligence chips that adopt block format data layout. It achieves the technical effect of reducing the number of format conversion operations from 90 to 18 times in a 2-layer Llama model as an example, with an elimination ratio of 80%, and the optimization effect expands linearly with the model size without decaying with the number of model layers. The optimization idea can also be extended to any artificial intelligence chip architecture that requires non-standard data layout.

[0028] (2) This invention solves the problems of needing to manually write graph transformation code for hardware-specific subgraphs and being prone to inconsistency with the underlying distribution logic by extracting the processing flow function of the target operator from the operator distribution logic of the target hardware, using graph capture mechanism to perform graph tracing to generate hardware-specific subgraphs, and using the combination of operation type, input shape and data type as cache key to reuse subgraphs. It achieves the effects of automatically extracting subgraphs, ensuring consistency between subgraphs and underlying distribution logic, automatically adapting when adding new hardware operation support by only adding distribution logic, and reducing compilation time and code maintenance costs.

[0029] (3) This invention solves the problem of constant weights repeatedly performing format conversion at runtime by promoting the input placeholder node to a constant attribute reading node, executing the format conversion operation chain on the host machine of the compiler and replacing the runtime operation chain with the pre-calculated result. It achieves the effect of completely eliminating the runtime format conversion of constant weights (28 times for 2-layer models, 448 times for 32-layer models, and 1120 times for 80-layer models).

[0030] (4) This invention eliminates type conversion nodes where the source data type and the target data type are the same, and performs multi-stage common subexpression elimination in the order of view transformation node, fill node, and linear format to block format conversion node. This solves the problem of semantically equivalent format conversion operations being repeatedly executed when multiple operators share the same input data, and achieves the effect of reusing the same conversion result and eliminating repeated format conversions (12 times in 2-layer model).

[0031] (5) This invention uses the alignment of the transformation boundary with the block boundary, whether the involved dimension is a dimension participating in the block, and whether irregular data dependencies across the block boundary are introduced as judgment conditions to determine whether the shape transformation operation node can be directly executed in the block format. This solves the problem of the full format conversion round trip triggered by the shape transformation operation in the chain and ensures the correctness of the computational semantics of the shape transformation directly executed in the block format.

[0032] (6) This invention solves the problem of introducing additional format conversion in the shape transformation operation within the chain by recalculating the element index mapping relationship for the tensor shape reshaping operation node, modifying the step size parameter of the data transport descriptor for the tensor dimension transpose operation node, and directly passing through the block format annotation for the element-by-element operation node, thereby directly completing the corresponding shape transformation under the block format. It achieves the effect of zero copy shape reinterpretation, single data transport to complete transpose, and zero additional format conversion overhead. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a graph compilation method according to an embodiment of the present invention;

[0034] Figure 2 This is a flowchart of the overall optimized pipeline for the graph compilation method according to an embodiment of the present invention;

[0035] Figure 3 This is a flowchart illustrating the format conversion process for a single matrix multiplication operation according to an embodiment of the present invention. Detailed Implementation

[0036] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0037] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0038] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0039] To overcome at least one of the technical problems described in the background section, the first aspect of the present invention provides a graph compilation method, comprising: obtaining an initial computation graph corresponding to a model to be compiled, the model to be compiled being deployed on target hardware, the target hardware adopting a block format data layout; replacing target operator nodes in the initial computation graph with a hardware-specific subgraph, the hardware-specific subgraph being extracted from the operator distribution logic of the target hardware through a graph capture mechanism, so as to explicitly introduce a data format conversion operation between linear format and block format in the initial computation graph, thereby obtaining a first computation graph; performing redundant format conversion elimination on the first computation graph to obtain a second computation graph, the redundant format conversion elimination including at least one of constant folding, common subexpression elimination, and cross-operation round-trip conversion elimination; and compiling the second computation graph into executable code for the target hardware.

[0040] In this embodiment, for ease of understanding, tile format and linear format (row-major order) are used as examples of data formats. Tile format refers to a data layout method that arranges and stores matrix data in fixed-size sub-blocks, typically T×T, where T is a positive integer, commonly 32. Linear format refers to a layout method that stores data row-by-row. In the following embodiments, the conversion between tile format and linear format includes conversion from linear format to tile format and from tile format to linear format. The target hardware is an artificial intelligence chip that uses a tile format data layout, and its computing unit requires input data to be stored in tile format.

[0041] See Figure 1The diagram shows a flowchart of the graph compilation method. The graph compilation method includes the following steps: obtaining an initial computation graph corresponding to the model to be compiled, wherein the model to be compiled is deployed on target hardware, and the target hardware uses a block-format data layout; replacing the target operator nodes in the initial computation graph with a hardware-specific subgraph, which is extracted from the operator distribution logic of the target hardware through a graph capture mechanism, to explicitly introduce data format conversion operations between linear and block formats in the initial computation graph, thus obtaining a first computation graph; performing redundant format conversion elimination on the first computation graph to obtain a second computation graph, wherein the redundant format conversion elimination includes at least one of constant folding, common subexpression elimination, and cross-operation round-trip conversion elimination; and compiling the second computation graph into executable code for the target hardware.

[0042] like Figure 2 As shown, the graph compilation method is executed by a graph compiler, which runs on the host machine (the central processing unit (CPU) environment where the compilation process resides). It receives the model code submitted by the user and outputs the compiled executable code, which is then deployed and run on the target hardware. Based on the problem modeling of block format conversion, this invention constructs a multi-level compilation optimization pipeline. This optimization pipeline is executed sequentially according to a preset order to gradually complete the explicitization of format conversion nodes, redundancy identification and elimination, and the final backend code generation.

[0043] Step S1 is the graph acquisition step. The input is the user model code (e.g., Python code). The Python code is captured using the compiler's graph capture mechanism (e.g., the Dynamo mechanism, which is a mechanism that converts dynamic model code into a static computation graph based on Python bytecode tracing). This converts the model computation process into an FX computation graph (an intermediate representation that records operator nodes and tensor data streams in the form of a directed acyclic graph), thus obtaining the initial computation graph. ,in Represents a set of nodes. The initial computation graph represents operators with nodes and tensor data streams with directed edges. However, the initial computation graph does not contain explicit data format conversion nodes. Format conversion operations are hidden within the operator dispatch logic in related technologies and are not visible in the computation graph, thus they cannot be recognized and optimized by the compiler.

[0044] Step S2 is the subgraph replacement step. The target operator node in the initial computation graph is replaced with a hardware-specific subgraph. This hardware-specific subgraph is not manually written but extracted from the operator distribution logic of the target hardware through a graph capture mechanism. This extraction method is referred to as "meta-compilation subgraph extraction" in this application. The hardware-specific subgraph contains data format conversion operation nodes and the hardware computation nodes corresponding to the target operator node. Therefore, after the replacement, the format conversion operations originally implicit in the hardware distribution logic are explicitly exposed on the computation graph, resulting in the first computation graph. Specifically, the nodes explicitly introduced in the computation graph include fill nodes, linear format to block format conversion nodes, block format computation nodes, and block format to linear format conversion nodes. It should be noted that the target operator node refers to the operator node whose execution requires block format data. Exemplarily, it can be a matrix multiplication operator node, a batch matrix multiplication operator node, or a linear layer operator node, but it is not limited to these. Any operator node that requires data format conversion before execution falls within the scope of the target operator node.

[0045] Step S3 is the weight constant folding step. Weight constant nodes in the first computation graph are identified. Format conversion chains related to this type of weight are pre-computed during compilation, and the pre-computed results replace the repeatedly executed format conversion operations at runtime, thereby eliminating redundant format conversions on the weight side and obtaining the computation graph after weight constant folding.

[0046] Step S4 is the activation common subexpression elimination step. For cases where multiple operators share the same activation input, common subexpression elimination is performed on related view transformation operations, fill operations, and linear-to-block format conversion operations to reuse identical intermediate results and reduce repetitive format conversions caused by shared activation inputs, resulting in a computational graph after activation common subexpression elimination processing.

[0047] Step S5 is the block format chain propagation step. Chain operation paths in the computation graph are identified. Under the condition that subsequent operators can directly consume block format data, intermediate results are kept in block format within the chain, thereby eliminating the round-trip conversion from block format to linear format and back to block format across operation boundaries. This results in a second computation graph after redundant format conversion is eliminated, where the number of format conversion operations is reduced from the initial value before optimization to the final value after optimization.

[0048] Step S6 is the backend compilation step. Backend compilation processing is performed on the second computation graph after redundant format conversion and elimination, including operator fusion, scheduling optimization, and code generation, ultimately generating executable code for the target AI chip. Specifically, operator fusion merges adjacent element-wise operations into a single computation kernel, reducing the number of kernel startups and memory read / write operations for intermediate results; code generation generates corresponding executable code based on the target hardware (such as the Triton computation kernel or hardware-specific instructions); and Python call wrapper code is generated to orchestrate the execution order of the computation kernels.

[0049] After subgraph replacement, matrix multiplication-related operations in the computation graph are expanded into explicit block format transformation chains, including input-side padding, linear-to-block format transformation, block format computation, and output-side block-to-linear format transformation. Analysis of this type of computation graph structure reveals that format transformation redundancy primarily arises from three scenarios. The first type of redundancy involves repeated format transformations of constant weights. Model weights typically remain unchanged during inference and are constant data that can be determined at compile time. However, without optimization, the compiler still retains the padding and linear-to-block format transformations corresponding to the weights and repeats them at runtime. For a 2-layer Llama model, this type of redundancy can eliminate 28 format transformations. The second type of redundancy involves repeated format transformations of shared activation inputs. When the same activation tensor is used by multiple successor operators, each successor operator performs view transformation, padding, and linear-to-block format transformations on the activation tensor, resulting in the repeated construction of semantically equivalent intermediate results. For a 2-layer Llama model, this type of redundancy can eliminate 12 format transformations. The third type of redundancy is the round-trip format transformation across operation boundaries. When multiple matrix multiplication operations form a chained execution path, the output of the previous operation is first transformed from block format back to linear format, and then immediately transformed back into block format by the next operation, forming a round-trip transformation chain from block format to linear format and back to block format. For a 2-layer Llama model, this type of redundancy can eliminate 32 format transformations. The above three types of redundancy are eliminated through weight constant folding, activation common subexpression elimination, and block format chain propagation, respectively.

[0050] See Figure 3 This illustrates the format conversion process for a single matrix multiplication operation. Assuming the block size of the AI ​​chip is T (e.g., T=32), for any matrix... The block format conversion process includes input-side padding and alignment, linear-to-block format conversion, matrix multiplication calculation in block format, and output-side block-to-linear format conversion. Specifically, for matrix multiplication C=A×B, where... For an m×k matrix, Given a k×n matrix, inputs A and B are independently padded and aligned (padding the matrix rows and columns to integer multiples of the block size T) and converted from linear to block format. The number of rows in A after padding and alignment is... The number of columns is B has k rows after padding and alignment. padded The column number of B After performing matrix multiplication in block format, the output side performs a block format to linear format conversion on the calculation result. Therefore, each matrix multiplication operation introduces 5 format conversion operations (2 on side A, 2 on side B, and 1 on the output side). Taking a 2-layer Llama model as an example, which contains 18 matrix multiplication operations (7 matrix multiplication operations per layer and 2 batch matrix multiplication operations, for a total of 2 layers), the baseline generates 18 × 5 = 90 format conversions.

[0051] In the aforementioned redundancy format conversion elimination, constant folding, common subexpression elimination, and cross-operation round-trip conversion elimination can be implemented independently or in any combination. When implemented in combination, there is no strict functional dependency between the three; executing any one of them individually can eliminate the corresponding format conversion redundancy and achieve the corresponding technical effect. Specifically, constant folding refers to pre-compiling the format conversion operation chain associated with constant data (such as model weight parameters, but not limited to these) whose values ​​can be determined at compile time in the computation graph, and replacing the repeatedly executed format conversion operation chain at runtime with the pre-computation result. Common subexpression elimination refers to merging the semantically equivalent format conversion operations associated with each operator when multiple operators share the same input data, reusing the same format conversion result. Cross-operation round-trip conversion elimination refers to deleting redundant conversion nodes in the round-trip conversion of data flow paths in the computation graph that involve block format to linear format and back to block format, ensuring that intermediate data propagates in block format within the data flow path.

[0052] In this embodiment, the graph compilation method extracts meta-compilation subgraphs, explicitly exposing format conversion operations in the hardware distribution logic onto the computation graph. This allows the compiler to identify and eliminate redundancy from a global perspective. Related techniques either completely ignore block format conversion or only process it locally at the operator level, failing to achieve global optimization. Simultaneously, constant folding, common subexpression elimination, and cross-operation round-trip conversion elimination are organically integrated, eliminating redundancy from three different sources: weight constants, shared activation inputs, and cross-operation chains, respectively, achieving systematic optimization. Taking a 2-layer Llama model as an example, the number of format conversion operations is reduced from 90 to 18, an elimination rate of 80%; for a 32-layer Llama-7B model, it is reduced from 1440 to 288; and for an 80-layer Llama-70B model, it is reduced from 3600 to 720, also an elimination rate of 80%. The optimization effect scales linearly with model size and does not diminish with increasing model layer count. Furthermore, the number of format conversions is independent of the sequence length and depends only on the number of matrix multiplication operations. When the sequence lengths are 64, 128, 256, and 512, the number of format conversions for the 2-layer Llama model is reduced from the baseline of 90 to 18, and the optimization effect remains consistent under different sequence lengths.

[0053] In some optional embodiments, the hardware-specific subgraph is extracted by: extracting the processing flow function of the target operator from the operator distribution logic of the target hardware; and using the graph capture mechanism to trace the processing flow function to generate the hardware-specific subgraph corresponding to the processing flow function.

[0054] Specifically, this embodiment defines the specific extraction method for hardware-specific subgraphs. The operator distribution logic of the target hardware typically resides in the torch_dispatch implementation of the TileTensor class or in the hardware backend distribution logic. Taking the matrix multiplication operator as an example, the core distribution logic for matrix multiplication-related operations extracted from the torch_dispatch implementation of TileTensor (a hook method in PyTorch used by the TileTensor class to overload operator distribution) is the independent function mma_linear2linear(). The processing flow corresponding to this function includes: padding alignment of the input tensor, conversion from linear format to block format, matrix multiplication calculation in block format, conversion of the output result from block format back to linear format, and removal of padding regions. After extracting this function, the compiler's Dynamo graph capture mechanism is used to trace its graph. Specifically, calling torch.compile(mma_linear2linear, backend=subgraph_capture_backend) triggers Dynamo to trace the function. Dynamo traces the function as an FX computation graph, ultimately obtaining the FX subgraph G_sub. The node sequence of this subgraph is as follows: First input path placeholder(A) (placeholder is a graph node representing the input tensor of the model) → pad_to_tileformat → linear_to_tileformat; Second input path placeholder(B) → pad_to_tileformat → linear_to_tileformat; Convergence computation path mm_t2t(A_tile, B_tile) (mm_t2t represents matrix multiplication from block format to block format, i.e., both input and output are stored in block format) → tileformat_to_linear → slice([0:m, 0:n]) → contiguous.

[0055] This embodiment avoids manually writing graph transformation code by automatically extracting processing flow functions from the hardware dispatch logic and performing graph tracing. When the hardware is upgraded or new operators are added, only the underlying hardware dispatch logic needs to be updated, and the graph compiler can automatically adapt, significantly reducing code maintenance costs.

[0056] In some optional embodiments, the step of using the graph capture mechanism to trace the processing flow function includes: registering a custom compilation backend, wherein the custom compilation backend is used to receive and save the graph object generated during the tracing process, and throw an exception after saving to short-circuit the actual execution of the processing flow function.

[0057] Specifically, a custom compilation backend, `subgraph_capture_backend`, is registered. This backend receives the `GraphModule` object generated by Dynamo, saves it to the variable `captured_graph`, and then throws a `SubgraphCaptured` exception for short-circuit execution. When Dynamo traces the processing flow functions, it calls this custom backend. After saving the graph structure, the backend throws an exception. Dynamo catches this exception and returns the generated FX graph without actually performing computation. In this way, the graph capture process does not incur actual computational overhead; it only acquires the graph structure.

[0058] This embodiment uses a custom compilation backend and an exception short-circuiting mechanism to ensure that the graph capture process only acquires the computation graph structure without actually performing computations, thus avoiding computational resource consumption during the graph capture phase and improving compilation efficiency.

[0059] In some optional embodiments, replacing the target operator node in the initial computation graph with a hardware-specific subgraph includes: obtaining the input edge of the target operator node; mapping the input placeholder node of the hardware-specific subgraph to the input edge of the target operator node; copying the internal node of the hardware-specific subgraph to the initial computation graph to replace the position of the target operator node, and redirecting the successor edge of the output node of the hardware-specific subgraph to the successor node of the target operator node; and deleting the target operator node.

[0060] Specifically, this embodiment defines the specific operations for subgraph inlining. For the initial computation graph... For each target operator node n (e.g., mm, bmm, linear), perform the following operations: obtain the input edge of node n {(src_node, src_output_idx)→n}; map the placeholder nodes of the FX subgraph G_sub to the corresponding input edges; copy all internal nodes of G_sub to... In the process, the position of n is replaced; all successor edges of the output node of G_sub are redirected to the successor node of n; the original node n is deleted. Through the above operations, the original operator node is replaced by a complete hardware-specific subgraph, and the format conversion operation is explicitly included in the computation graph.

[0061] This embodiment achieves precise replacement of the original operator nodes in the hardware-specific subgraph through standard graph transformation operations such as node mapping, edge redirection, and original node deletion. This ensures that the replaced computation graph is semantically equivalent to the original computation graph, while explicitly including format conversion operations, providing a foundation for subsequent redundancy elimination.

[0062] In some optional embodiments, the method further includes: caching the generated hardware-specific subgraph using a combination of operation type, shape and data type of each input tensor as a cache key; and when a target operator node matching the cache key exists, reusing the cached hardware-specific subgraph to replace the target operator node.

[0063] Specifically, this embodiment defines a subgraph caching mechanism. A cache key is defined as `cache_key=(op_type, shape_A, shape_B, dtype)`, where `op_type` is the operation type (mm, bmm, linear), `shape_A` is the shape of input A (e.g., [m, k]), `shape_B` is the shape of input B (e.g., [k, n]), and `dtype` is the data type (e.g., bf16, fp16). Operations with the same `cache_key` reuse the same subgraph `G_sub`, avoiding duplicate generation. For example, a two-layer Llama model may contain multiple matrix multiplication operations with the same shape. Through the caching mechanism, only one subgraph needs to be generated to reuse all operations with the same shape. It should be noted that subgraph caching is a compilation efficiency optimization technique and not a necessary condition for eliminating redundant format conversions; in alternative embodiments, subgraph caching may not be required, and the subgraph can be regenerated each time a replacement occurs, still achieving the purpose of this invention.

[0064] This embodiment avoids repeatedly performing graph capture and subgraph generation for operator nodes with the same operation type, input shape, and data type by using a subgraph caching mechanism, which significantly reduces compilation time.

[0065] In some optional embodiments, the constant folding includes: in response to the input placeholder node having a value type of model parameter type and existing in the model parameter list, promoting the input placeholder node to a constant attribute read node pointing to the constant storage of the model module; starting from the constant attribute read node, tracing a format conversion operation chain that sequentially includes a padding operation node and a linear to block format conversion operation node; executing the format conversion operation chain on the compiler's host machine to obtain a pre-computation result, writing the pre-computation result into the constant storage, replacing the format conversion operation chain with the constant attribute read node that reads the pre-computation result, and connecting the output edge of the replaced constant attribute read node to the successor node at the end of the format conversion operation chain.

[0066] Specifically, this embodiment defines the specific implementation of constant folding. It iterates through all input placeholder nodes (graph nodes representing model input tensors) in the first computation graph G1, and promotes nodes that meet the following conditions: the node's meta value type is torch.nn.Parameter, and the node has a corresponding reference in the model's parameter list. The promotion operation replaces the placeholder node with a get_attr node, which points to the buffer attribute of the model module, semantically indicating that the constant tensor is read from the module attribute, rather than obtained from the runtime input. For each promoted get_attr(W) node, the data flow is traced backward to identify the format conversion operation chain starting from that node: Chain(W) = [get_attr(W), pad_to_tileformat, linear_to_tileformat]. The identification conditions are: the input of each node in the chain comes only from the predecessor node in the chain, and the end point of the chain is followed by a computation operation (such as mm_t2t). For the identified format conversion chain Chain(W), the operation chain is executed on the compiler host: W_padded = pad_to_tileformat(W), W_tile = linear_to_tileformat(W_padded). After execution, the result W_tile is written to the model's buffer storage; the original operation chain Chain(W) is replaced with a single node get_attr(W_tile); and the output edge of get_attr(W_tile) is connected to the successor node of the original chain's endpoint. Taking a 2-layer Llama model as an example, the model contains 14 weight tensors. Each weight tensor introduces one padding operation and one linear-to-block format conversion operation. 28 runtime format conversions can be eliminated through constant folding.

[0067] This embodiment completely eliminates redundant format conversions of constant weights at runtime by moving the chain of format conversion operations associated with weight constants from runtime to compile time. For example, in a 2-layer Llama model, 28 redundant format conversions are eliminated; for a 32-layer Llama-7B model, 448 are eliminated; and for an 80-layer Llama-70B model, 1120 are eliminated. Since the weights remain unchanged during inference, the pre-computation results are numerically equivalent to the runtime successive execution results, without introducing any semantic bias.

[0068] In some optional embodiments, the common subexpression elimination includes: eliminating type conversion nodes in the first computation graph where the source data type and the target data type are the same; performing common subexpression elimination on the view transformation nodes associated with each of the multiple operator nodes sharing the same input data, and merging view transformation nodes with the same input source and the same target shape; based on the merging of the view transformation nodes, performing common subexpression elimination on the fill nodes associated with each of the multiple operator nodes, and merging fill nodes with the same input source; based on the merging of the fill nodes, performing common subexpression elimination on the linear format to block format conversion nodes associated with each of the multiple operator nodes, and merging linear format to block format conversion nodes with the same input source.

[0069] Specifically, this embodiment defines the implementation of common subexpression elimination, targeting scenarios where multiple operators share the same activation input (e.g., Q / K / V projection sharing activation input in Transformer, gate / up projection sharing activation input in FFN). First, pre-cleaning is performed: all _to_copy nodes (type conversion operations) in computation graph G2 are scanned. Nodes where the source data type equals the target data type (e.g., bf16→bf16) are directly eliminated, the node is deleted, and the input edges of its successor nodes are redirected to the input source of that node. Then, view node deduplication is performed: common subexpression elimination is performed on the aten.view.default node. For two view nodes v1 and v2 in the graph, if v1.input_source == v2.input_source and v1.target_shape == v2.target_shape, they are merged. The node with the earlier topological order is selected as the canonical node v_canonical. The input edges of all successor nodes of the other node v_merged are redirected to v_canonical, and v_merged is deleted. Based on view deduplication, common subexpression elimination is performed on the pad_to_tileformat node. For two padding nodes p1 and p2 in the diagram, if p1.input == p2.input (this condition depends on the deduplication result of the view nodes), they are merged. Based on padding deduplication, common subexpressions are eliminated for the linear_to_tileformat nodes. For two linear-to-tileformat conversion nodes l1 and l2 in the diagram, if l1.input == l2.input (this condition depends on the deduplication result of the padding nodes), they are merged. Taking Q / K / V projection as an example, the input activation X executes the view→pad_to_tileformat→linear_to_tileformat format conversion chain independently in each of the three projection operations. Through the multi-stage deduplication in this embodiment, only one format conversion chain needs to be executed for use by the three projections. Taking a 2-layer Llama model as an example, 12 repeated format conversions can be eliminated.

[0070] This embodiment employs a multi-stage directed acyclic graph-aware common subexpression elimination strategy, using a strict dependency sequence of view deduplication → padding deduplication → linear format to block format conversion deduplication, effectively eliminating repetitive format conversions caused by shared activation inputs. Taking a 2-layer Llama model as an example, it eliminates 12 repetitive conversions; and this elimination ratio scales linearly with the model size.

[0071] In some optional embodiments, the cross-operation round-trip conversion elimination includes: identifying a data flow path in the first computation graph where there is a round-trip conversion from block format to linear format and back to block format, and the successor operator after the round-trip conversion supports block format data input; deleting block format to linear format conversion nodes and linear format to block format conversion nodes in the data flow path; directly connecting the block format data output of the predecessor node at the entrance of the data flow path to the successor operator at the exit of the data flow path; retaining one linear format to block format conversion at the entrance of the data flow path, and retaining one block format to linear format conversion at the exit of the data flow path.

[0072] Specifically, this embodiment defines the specific implementation of cross-operation round-trip transformation elimination. After optimization in steps S3 and S4, repetitive format transformations related to weight constants and shared activation inputs in the computation graph have been eliminated. However, a large number of round-trip transformations from block format to linear format and back to block format may still remain between multiple consecutive operations. In the Transformer model, consecutive operations such as attention chains and feedforward network chains naturally constitute computation chains. Intermediate results within the chain do not necessarily need to return to linear format at each operation boundary, provided that the successor operator can directly consume block format data. Specific operation: Identify data flow paths in computation graph G3 that contain round-trip transformations from block format to linear format and back to block format, and where the successor operator after the round-trip transformation supports block format data input. Delete the block format to linear format transformation node (tileformat_to_linear) and the linear format to block format transformation node (pad_to_tileformat→linear_to_tileformat) within this data flow path, and directly connect the block format data output of the ingress predecessor node to the outgress successor operator. The data flow path retains one conversion from linear to block format at the entry point and one conversion from block format to linear format at the exit point. Taking the round-trip conversion between two adjacent matrix multiplications in the attention chain as an example, the original format is: mm_t2t tileformat_to_linear [Element-by-element operations / shape transformation nodes] pad_to_tileformat linear_to_tileformat mm_t2t; after transformation, it becomes: mm_t2t [Element-by-element operations / shape transformation nodes (executed directly in block format)] mm_t2t. Taking a 2-layer Llama model as an example, it can eliminate 32 round-trip format conversions.

[0073] This implementation eliminates round-trip format conversions across operation boundaries by identifying operation chains and maintaining block format propagation within the chains. For example, in a 2-layer Llama model, 32 round-trip conversions are eliminated; for a 32-layer Llama-7B model, 512 are eliminated; and for an 80-layer Llama-70B model, 1280 are eliminated. This elimination significantly reduces data transport overhead and computational kernel startup overhead.

[0074] In some optional embodiments, identifying a data flow path in the first computation graph where there is a round-trip conversion from block format to linear format and back to block format, and where subsequent operators after the round-trip conversion support block format data input, includes: marking computation nodes in the first computation graph that perform calculations using block format data; starting from the output of the computation node, tracing along the data flow direction, and grouping downstream nodes that meet the following conditions into the same operation chain: the downstream node is a block format to linear format conversion node and its output is directly or after element-wise operation into a linear format to block format conversion node, or the downstream node is an element-wise operation node or a shape transformation operation node; the starting point of the chain is the position where data first enters the block format from the linear format, and the ending point of the chain is the position where data last returns from the block format to the linear format.

[0075] Specifically, this embodiment defines the method for identifying data flow paths. First, the first computation graph is traversed, and all computation nodes that perform calculations using block format data (i.e., the outputs of mm_t2t and bmm_t2t) are marked. Then, starting from the output of each mm_t2t node, the data flow direction is traced, and downstream nodes that meet the following conditions are grouped into the same operation chain: the node is tileformat_to_linear, and its output is directly or after a few element-wise operations passed to the next pad_to_tileformat (i.e., there is a round-trip conversion); the node is an element-wise operation (such as silu, mul, add), and its input and output are both on the same data flow path; the node is a shape transformation operation (such as reshape, transpose). Finally, the boundaries of the chain are determined: the starting point of the chain is the position where data first enters the block format from the linear format (i.e., the output of linear_to_tileformat), and the ending point of the chain is the position where data is last converted from the block format back to the linear format (i.e., the input of tileformat_to_linear). It should be noted that the path identification method is not limited to a chain structure, but also includes any data flow path in a directed acyclic graph that meets the above conditions.

[0076] This embodiment achieves automatic identification of the operation chain through three steps: marking the block format computing nodes, tracing the chain data flow, and determining the chain boundary, thus providing accurate operation objects for subsequent block format penetration.

[0077] In some optional embodiments, when there is a shape transformation operation node in the data flow path, the cross-operation round-trip conversion elimination further includes: determining whether the shape transformation operation node meets the condition for direct execution in the block format; if it does, then by remapping the data access index relationship of the shape transformation operation node, the shape transformation is directly completed in the block format without reverting the data to the linear format.

[0078] Specifically, this embodiment defines the processing method for shape transformation operations within the chain. For nodes within the operator chain that require shape transformations such as reshape, head_split, and transpose, since the data is already in a block format (non-linear continuous format), if the block format is first converted back to linear format before performing shape transformations and then re-blocking, it will introduce additional memory bandwidth overhead from two full format conversions. This embodiment eliminates redundant format conversions within the chain by directly adjusting the address mapping parameters of data transport operations during the compilation stage, performing equivalent shape transformations in-place under the block format. Specifically, it first determines whether the shape transformation operation node meets the conditions for direct execution under the block format; if it does, the shape transformation is directly performed under the block format by remapping the data access index relationship of the shape transformation operation node, without reverting the data to the linear format. For operations that do not meet the conditions, a tileformat_to_linear and pad_to_tileformat→linear_to_tileformat are inserted before and after the node to revert to the linear format to complete the transformation before re-entering the block format chain, ensuring the correctness of the computational semantics.

[0079] This embodiment directly performs shape transformation in the block format, avoiding the overhead of two full format conversions caused by falling back to the linear format and then re-blocking the data, thus significantly reducing the memory bandwidth consumption within the chain.

[0080] In some optional embodiments, determining whether the shape transformation operation node meets the conditions for direct execution in the block format includes: determining whether the transformation boundary of the shape transformation operation node is aligned with the block boundary of the block format; determining whether the dimension involved in the shape transformation operation node is a dimension participating in the block format, so as to determine the adjustment method of inter-block index and intra-block index when performing the shape transformation in the block format; determining whether the shape transformation operation node introduces irregular data dependencies across block boundaries; if the transformation boundary is aligned with the block boundary and the shape transformation operation node does not introduce irregular data dependencies across block boundaries, then the shape transformation operation node is determined to meet the conditions for direct execution in the block format, and the shape transformation is performed according to the determined adjustment method of inter-block index and intra-block index.

[0081] Specifically, this embodiment defines a specific method for determining whether a shape transformation operation node meets the conditions for direct execution in a block format, including three aspects. First, whether the transformation boundary is aligned with the block boundary of the block format: If the splitting and merging points of the transformation operation in the processed dimension are both integer multiples of the block size T, then each block remains intact before and after the transformation, and will not be split or mixed with part of the data of adjacent blocks, which is a case that can be directly processed; if the splitting or merging points fall inside the block (not an integer multiple of T), then the integrity of the block will be destroyed, and it is necessary to restore it to a linear format before the operation can be completed. 2. Whether the dimensions involved in the transformation are those that participate in the block format: If the transformation only involves dimensions that do not participate in the block format (such as the batch dimension), then the data arrangement in that dimension is consistent with the linear format and can be processed directly; if the transformation involves row and column dimensions that participate in the block format (such as the two calculation dimensions of matrix multiplication), then it is necessary to make corresponding adjustments at both the inter-block index and intra-block index levels (such as transpose, which needs to complete both inter-block transpose and intra-block transpose) to obtain an equivalent transformation result. 3. Does the transformation introduce irregular data dependencies across block boundaries? Element-wise operations (such as `silu`, `mul`, `add`) do not change the relative positions of elements and do not introduce cross-block dependencies, naturally satisfying the conditions for direct processing. Operations such as `reshape` / `head_split`, which only reinterpret the dimensional organization and do not move the physical location of data, also do not introduce cross-block dependencies when the above boundary alignment and dimensional conditions are met. However, operations involving cross-block re-aggregation of data (such as unaligned `slice` / `split` / `cat`, or `squeeze` / `unsqueeze` / `expand` that change the block structure of the block-based dimensions will introduce irregular data dependencies across blocks, which cannot be completed solely through address mapping. If the transformation boundary is aligned with the block boundary and the shape transformation operation node does not introduce irregular data dependencies across block boundaries, then the conditions for direct execution under the block format are met, and the shape transformation is performed according to the determined inter-block index and intra-block index adjustment method.

[0082] This embodiment accurately identifies shape transformation operations that can be directly executed in the block format through three judgment conditions, avoiding the overly conservative handling of reverting all shape transformation operations to the linear format. While ensuring the correctness of computational semantics, it minimizes the overhead of format conversion.

[0083] In some optional embodiments, the shape transformation operation node is a tensor shape reshaping operation node. The step of directly completing the shape transformation in the block format by remapping the data access index relationship of the shape transformation operation node includes: recalculating the element index mapping relationship in the block format to make the physical storage addresses before and after the transformation equivalent, so as to perform zero-copy shape reinterpretation of the tensor shape reshaping operation node.

[0084] Specifically, this embodiment defines the specific processing method for tensor shape reshaping operations (reshape / head_split). The reshape operation is logically equivalent to reinterpreting the tensor dimensions, without changing the linear storage order of elements. In a block format, simply remapping the original block index to the block index under the target shape completes the equivalent transformation without moving the data. The following example uses head_merge (the inverse operation of head_split) to illustrate the address mapping formula in a block format. reshape and head_split follow the same index remapping principle.

[0085] Let the original tensor logic shape be [B, H, S, D], and the block size be [1, t]. h , t s , t d The target shape is [B, H·S, D] (taking head_split's inverse operation head_merge as an example, reshape is similar). The multidimensional index of the original split block is (b, i h i s i d ),in:

[0086]

[0087] Given the target shape, the two dimensions H and S are merged into H·S, and the corresponding target block index is:

[0088]

[0089] The number of elements in a single block is T=t h ·t s ·t d The source address (the block physical offset calculated according to the original shape) has the same physical offset as the destination address (the block index reinterpreted according to the destination shape), i.e., addr. src =addr dst ,

[0090]

[0091] Target address (by target shape) The Tile index is reinterpreted as follows:

[0092]

[0093] Therefore, `reshape / head_split` in block format is equivalent to zero-copy address reinterpretation. The compiler only needs to update the calculation method of the access index of the tensor for subsequent operators, without generating any data moving instructions. For the `head_split` operation (split [B, H·S, D] into [B, H, S, D]), the above mapping is inverted, and the principle is the same. It should be noted that the index mapping can be implemented at any level, such as in the data moving descriptor, the compiler intermediate representation, or the code generation stage.

[0094] This embodiment achieves zero-copy shape reinterpretation of the reshape / head_split operation by recalculating the element index mapping relationship under the block format, with an additional bandwidth overhead of 0.

[0095] In some optional embodiments, the shape transformation operation node is a tensor dimension transpose operation node. The step of directly completing the shape transformation in the block format by remapping the data access index relationship of the shape transformation operation node includes: modifying the source address step size parameter and the target address step size parameter of the data transport descriptor to complete the transpose corresponding to the tensor dimension transpose operation node during the data transport process.

[0096] Specifically, this embodiment defines the specific processing method for the tensor dimension transpose operation. The transpose operation changes the access order of elements in each dimension, which is equivalent to rearranging the block index order and locally transposing the elements within the block in the block format. This embodiment completes the transpose in the block format at the hardware transport layer by modifying the source address stride parameter and the target address stride parameter in the DMA descriptor (Direct Memory Access Descriptor, used to configure parameters such as source address, target address, and stride for hardware data transfer), without the need for additional computation kernel intervention.

[0097] matrix transpose K T Taking (dimension (S, D) → (D, S)) as an example, the block size is (t) s , t d Step 1: Transpose the block-level index; the original block index is (i s i d After transposition, the target block index is (i d i s The DMA target address is calculated using the transposed block index.

[0098]

[0099] Step 2: Transpose the stride of elements within the block. For each sub-block, set the original stride of the elements within the block to (stride). row =t d stride col =1), after transpose, the target step size is modified to (stride) row =1, stride col =t s ), that is, the source stride src =(t d 1) Change to the target step size stride dst =(1, t s )

[0100]

[0101] By writing the aforementioned step size parameters into the src_stride and dst_stride fields of the DMA descriptor, the DMA hardware automatically transposes the elements within the block during data transfer, without requiring additional CPU computation. It should be noted that the step size modification can be implemented at any level, including the data transfer descriptor, the compiler intermediate representation, or the code generation stage.

[0102] This embodiment modifies the step size parameter of the DMA descriptor to complete the transposition synchronously during the data transfer process, requiring only one DMA transfer, which saves about 50% of bandwidth compared to two full conversions.

[0103] In some optional embodiments, the shape transformation operation node is an element-wise operation node. The step of directly completing the shape transformation in the block format by remapping the data access index relationship of the shape transformation operation node includes: determining that the element-wise operation node does not need to remap the data access index relationship, keeping the block format label of the input data and output data of the element-wise operation node unchanged, and directly passing through for execution.

[0104] Specifically, this embodiment defines the specific processing method for element-wise operations (silu, mul, add, etc.). Element-wise operations depend only on the value of the element at the corresponding position, do not involve data dependencies across blocks, and do not require elements to be arranged contiguously in memory. The relative positions of elements within each block are completely consistent under the block format as under the linear format. Therefore, the computational logic of element-wise operations is independent of the physical storage arrangement of the data and can be executed directly under the block format without any address mapping adjustments or additional format conversion processing. The compiler's processing strategy for such nodes is as follows: determine that element-wise operation nodes do not require remapping of data access index relationships, keep the block format annotations of input and output tensors unchanged, directly pass them through to subsequent operators, and do not generate any bridging instructions.

[0105] This embodiment recognizes that element-by-element operations can be executed in a block format without format conversion, thus avoiding unnecessary format conversion and introducing no additional overhead.

[0106] In some optional embodiments, compiling the second computation graph into executable code for the target hardware includes at least one of the following: merging adjacent element-wise operations in the second computation graph into a single computation kernel; generating corresponding computation kernel code based on the target hardware; and generating call encapsulation code for orchestrating the execution order of each computation kernel.

[0107] Specifically, this embodiment defines the specific content of the backend compilation. The computation graph G4 optimized by steps S2 to S5 serves as the input for the backend compilation stage. The backend compilation process includes at least one of the following: operator fusion, which merges adjacent element-wise operations into a single computation kernel, reducing the number of computation kernel startups and memory read / write operations for intermediate results; code generation, which generates corresponding executable code based on the target hardware, such as the Triton computation kernel or hardware-specific instructions; and call wrapper code generation, which generates Python call wrapper code and arranges the execution order of each computation kernel.

[0108] This embodiment transforms the optimized computation graph into efficient executable code for the target hardware through operator fusion, code generation, and call encapsulation code generation, further reducing the number of computation kernel startups and memory read / write operations for intermediate results.

[0109] A second aspect of the present invention provides a graph compilation apparatus, comprising: a graph acquisition module for acquiring an initial computation graph corresponding to a model to be compiled, the model to be compiled being deployed on target hardware, the target hardware adopting a block format data layout; a subgraph replacement module for replacing target operator nodes in the initial computation graph with hardware-specific subgraphs, the hardware-specific subgraphs being extracted from the operator distribution logic of the target hardware through a graph capture mechanism, thereby explicitly introducing a data format conversion operation between linear and block formats in the initial computation graph to obtain a first computation graph; a redundancy elimination module for performing redundancy format conversion elimination on the first computation graph to obtain a second computation graph, the redundancy format conversion elimination including at least one of constant folding, common subexpression elimination, and cross-operation round-trip conversion elimination; and a compilation module for compiling the second computation graph into executable code for the target hardware.

[0110] Specifically, this embodiment provides a graph compilation device, including a graph acquisition module, a subgraph replacement module, a redundancy elimination module, and a compilation module. The graph acquisition module captures the Python code of the user model using the compiler's Dynamo mechanism to obtain an initial computation graph. The subgraph replacement module replaces the target operator nodes in the initial computation graph with hardware-specific subgraphs, which are extracted from the operator distribution logic of the target hardware through the graph acquisition mechanism. The redundancy elimination module performs redundancy format conversion elimination on the first computation graph, including at least one of constant folding, common subexpression elimination, and cross-operation round-trip conversion elimination. The compilation module compiles the second computation graph into executable code for the target hardware. It should be noted that the above functional module division is only a logical functional division; in actual implementation, the modules can be merged, split, or reorganized. For example, the graph acquisition module and the subgraph replacement module can be integrated into the same front-end processing unit, and the redundancy elimination module can be further subdivided into a constant folding submodule, a common subexpression elimination submodule, and a cross-operation round-trip conversion elimination submodule. Each module can be implemented in software (e.g., as part of a compiler program) or in a combination of software and hardware. In one possible implementation, the graph compilation device can be deployed on a cloud server, edge computing device, or terminal device, and provided as a software installation package or library file for developers to use.

[0111] The graph compilation device and the graph compilation method in this embodiment are based on the same inventive concept and can achieve the same technical effect, namely, by extracting meta-compiled subgraphs and eliminating redundant format conversions, the number of format conversions is significantly reduced, the model inference performance is improved, and the code maintenance cost is reduced.

[0112] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the graph compilation method as described in any of the first aspects above.

[0113] Specifically, computer-readable storage media may include, but are not limited to: semiconductor memory (such as solid-state drives (SSDs), universal serial bus flash drives (USB flash drives), magnetic media (such as hard disk drives, magnetic tapes), optical media (such as CD-ROMs, DVD-ROMs), etc.; the storage media may be volatile or non-volatile, and may be removable or non-removable. In a typical application scenario, the computer program is loaded into the runtime environment of the graph compiler and executed by the processor to perform graph compilation optimization on the user-submitted model code; the processor may be a general-purpose processor (such as a CPU) or a dedicated processor (such as an NPU, GPU, etc.); the computer program may be stored in the medium in the form of source code, intermediate code, or executable code. The computer program stored in the storage medium may contain all the instructions for implementing the graph compilation method, or may contain some instructions that cooperate with other program modules to implement the method. For example, the graph capture mechanism may reuse the Dynamo module of existing deep learning frameworks (such as PyTorch), while the method instructions of this application are mainly responsible for subgraph replacement, redundancy elimination, and compilation scheduling.

[0114] The computer-readable storage medium of this embodiment enables the graph compilation method to be distributed and deployed as a software product. Users can obtain all the beneficial effects achieved by this application by installing the program in the medium, namely, reducing the number of format conversions, improving model inference performance, and reducing compiler maintenance costs.

[0115] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the graph compilation method as described in any of the first aspects above.

[0116] Specifically, this embodiment provides an electronic device, including a memory (such as random access memory, read-only memory, flash memory, etc.), a processor (such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), etc.), and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the graph compilation method described in any of the first aspects above. The electronic device includes, but is not limited to, servers, workstations, personal computers, laptops, embedded devices, edge computing devices, mobile terminals, etc. In a typical application scenario, the user deploys the model to be compiled to the electronic device, and the graph compiler performs graph compilation optimization on the model code and generates executable code for the target hardware, thereby improving the inference performance of the model on the artificial intelligence chip.

[0117] The electronic device of this embodiment can reduce the number of format conversions, improve model inference performance, and reduce compiler maintenance costs by running the computer program stored therein, enabling the graph compilation method of the present invention to run in a real hardware environment.

[0118] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art, after considering the specification and practicing the technical solutions disclosed in this application, will readily conceive of other embodiments of this application. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary technical means in the art not disclosed in this application. The specification and embodiments are considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0119] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A graph compilation method, characterized in that, include: Obtain the initial computation graph corresponding to the model to be compiled, wherein the model to be compiled is deployed on the target hardware and the target hardware adopts a block format data layout; The target operator node in the initial computation graph is replaced with a hardware-specific subgraph, which is extracted from the operator distribution logic of the target hardware through a graph capture mechanism, so as to explicitly introduce a data format conversion operation between linear format and block format in the initial computation graph to obtain a first computation graph. The first computation graph is subjected to redundancy format transformation elimination to obtain the second computation graph. The redundancy format transformation elimination includes at least one of constant folding, common subexpression elimination and cross-operation round-trip transformation elimination. The second computation graph is compiled into executable code for the target hardware.

2. The method according to claim 1, characterized in that, The hardware-specific subgraph is extracted in the following way: Extract the processing flow function of the target operator from the operator distribution logic of the target hardware; The processing flow function is traced using the graph capture mechanism to generate a hardware-specific subgraph corresponding to the processing flow function.

3. The method according to claim 2, characterized in that, The step of tracing the processing flow function using the graph capture mechanism includes: Register a custom compilation backend, which is used to receive and save the graph object generated during the tracing process, and throw an exception after saving to short-circuit the actual execution of the processing flow function.

4. The method according to claim 2, characterized in that, The step of replacing the target operator node in the initial computation graph with a hardware-specific subgraph includes: Obtain the input edge of the target operator node; Map the input placeholder nodes of the hardware-specific subgraph to the input edges of the target operator node; The internal nodes of the hardware-specific subgraph are copied to the initial computation graph to replace the position of the target operator node, and the successor edges of the output nodes of the hardware-specific subgraph are redirected to the successor nodes of the target operator node; Delete the target operator node.

5. The method according to claim 2, characterized in that, Also includes: The generated hardware-specific subgraph is cached using a combination of operation type, shape, and data type of each input tensor as the cache key; When a target operator node that matches the cache key exists, the cached hardware-specific subgraph is reused to replace the target operator node.

6. The method according to claim 1, characterized in that, The constant folding includes: In response to the fact that the value type of the input placeholder node is a model parameter type and exists in the model parameter list, the input placeholder node is promoted to a constant attribute read node that points to the constant storage of the model module; Starting from the constant attribute reading node, trace the format conversion operation chain that includes the padding operation node and the linear format to block format conversion operation node in sequence; The format conversion operation chain is executed on the host machine of the compiler to obtain a pre-calculated result. The pre-calculated result is written into the constant storage. The constant attribute reading node that reads the pre-calculated result replaces the format conversion operation chain. The output edge of the replaced constant attribute reading node is connected to the successor node of the end point of the format conversion operation chain.

7. The method according to claim 1, characterized in that, The elimination of common subexpressions includes: Eliminate type conversion nodes in the first computation graph where the source data type and the target data type are the same; Perform common subexpression elimination on the view transformation nodes associated with multiple operator nodes that share the same input data, and merge view transformation nodes with the same input source and the same target shape; Based on the merging of the view transformation nodes, common subexpression elimination is performed on the fill nodes associated with each of the multiple operator nodes, and fill nodes with the same input source are merged. Based on the merging of the fill nodes, common subexpression elimination is performed on the linear format to block format conversion nodes associated with each of the multiple operator nodes, and linear format to block format conversion nodes with the same input source are merged.

8. The method according to claim 1, characterized in that, The cross-operation round-trip conversion elimination includes: Identify a data flow path in the first computation graph that involves a round-trip conversion from block format to linear format and back to block format, and where the subsequent operators after the round-trip conversion support block format data input. Delete the block format to linear format conversion node and the linear format to block format conversion node in the data flow path, and directly connect the block format data output of the entry predecessor node of the data flow path to the exit successor operator of the data flow path. At the entry point of the data flow path, a linear-to-block format conversion is performed once, and at the exit point of the data flow path, a block format-to-linear format conversion is performed once.

9. The method according to claim 8, characterized in that, The identification of a data flow path in the first computation graph that involves a round-trip transformation from block format to linear format and back to block format, and where subsequent operators after the round-trip transformation support block format data input, includes: Mark the computation nodes in the first computation graph that are computed using data in a block format; Starting from the output of the computing node, tracing along the data flow direction, downstream nodes that meet the following conditions are grouped into the same operation chain: the downstream node is a block format to linear format conversion node and its output is directly or after element-wise operation into the linear format to block format conversion node, or the downstream node is an element-wise operation node or a shape transformation operation node. The chain starts at the point where data first transitions from linear to block format and ends at the point where data last transitions back from block format to linear format.

10. The method according to claim 8, characterized in that, When the data flow path contains shape transformation operation nodes, the cross-operation round-trip transformation elimination also includes: Determine whether the shape transformation operation node meets the conditions for direct execution in the block format; if it does, then by remapping the data access index relationship of the shape transformation operation node, the shape transformation is directly completed in the block format without reverting the data to the linear format.

11. The method according to claim 10, characterized in that, The step of determining whether the shape transformation operation node meets the conditions for direct execution in a block format includes: Determine whether the transformation boundary of the shape transformation operation node is aligned with the block boundary of the block format; Determine whether the dimension involved in the shape transformation operation node is a dimension participating in the block format block division, so as to determine the adjustment method of inter-block index and intra-block index when performing the shape transformation under the block format; Determine whether the shape transformation operation node introduces irregular data dependencies across block boundaries; If the transformation boundary is aligned with the block boundary and the shape transformation operation node does not introduce irregular data dependencies across the block boundary, then the shape transformation operation node is determined to meet the conditions for direct execution in the block format, and the shape transformation is executed according to the determined adjustment method of inter-block index and intra-block index.

12. The method according to claim 10, characterized in that, The shape transformation operation node is a tensor shape reshaping operation node. The step of directly completing the shape transformation in the block format by remapping the data access index relationship of the shape transformation operation node includes: recalculating the element index mapping relationship in the block format to make the physical storage addresses before and after the transformation equivalent, so as to perform zero-copy shape reinterpretation of the tensor shape reshaping operation node.

13. The method according to claim 10, characterized in that, The shape transformation operation node is a tensor dimension transpose operation node. The step of directly completing the shape transformation in the block format by remapping the data access index relationship of the shape transformation operation node includes: modifying the source address step size parameter and the target address step size parameter of the data transport descriptor to complete the transpose corresponding to the tensor dimension transpose operation node during the data transport process.

14. The method according to claim 10, characterized in that, The shape transformation operation node is an element-wise operation node. The step of directly completing the shape transformation in the block format by remapping the data access index relationship of the shape transformation operation node includes: determining that the element-wise operation node does not need to remap the data access index relationship, keeping the block format label of the input data and output data of the element-wise operation node unchanged, and directly passing through for execution.

15. The method according to claim 1, characterized in that, The step of compiling the second computation graph into executable code for the target hardware includes at least one of the following: merging adjacent element-wise operations in the second computation graph into a single computation kernel; Generate corresponding computing kernel code based on the target hardware; generate call encapsulation code for orchestrating the execution order of each computing kernel.

16. A graph compilation apparatus, characterized in that, include: The graph acquisition module is used to acquire the initial computation graph corresponding to the model to be compiled. The model to be compiled is deployed on the target hardware, and the target hardware adopts a block format data layout. The subgraph replacement module is used to replace the target operator node in the initial computation graph with a hardware-specific subgraph. The hardware-specific subgraph is extracted from the operator distribution logic of the target hardware through a graph capture mechanism to explicitly introduce a data format conversion operation between linear format and block format in the initial computation graph to obtain a first computation graph. A redundancy elimination module is used to perform redundancy format conversion elimination on the first computation graph to obtain a second computation graph. The redundancy format conversion elimination includes at least one of constant folding, common subexpression elimination, and cross-operation round-trip conversion elimination. A compilation module is used to compile the second computation graph into executable code for the target hardware.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 15.

18. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 15.

Images