Deep neural network compilation method and related devices

Abstract

The application relates to a deep neural network compiling method and related equipment. The deep neural network compiling method comprises the following steps: creating a plurality of parallel sub-processes, distributing each intercepted subgraph to each sub-process, independently calling a backend compiler by each sub-process to execute a parallel compiling task to generate underlying target code corresponding to each subgraph, and placing a main process executing a native compiling process in a waiting state during the parallel compiling task; after the parallel compiling task executed by all the sub-processes is completed, storing the underlying target code corresponding to each subgraph in a native file cache; releasing the waiting state of the main process, submitting a compiling request to the backend compiler by the main process according to the native compiling process, and enabling the native compiling process to obtain and load the underlying target code corresponding to each subgraph from the native file cache. The application solves the problem that existing conventional technology cannot implement parallel compiling task acceleration processing on scattered subgraphs without reconstructing the underlying logic of a deep learning framework.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a deep neural network compilation method and related equipment. Background Technology

[0002] In the deployment and inference of modern large-scale neural networks (LLMs), deep learning compilers have become a core means of improving computational efficiency. Deep learning compilers include `torch.compile` or `Inductor` in the PyTorch framework. `torch.compile` is the Just-In-Time (JIT) compilation stack in the PyTorch framework. As a front-end entry point, it captures the user's PyTorch code into a computation graph and hands it over to the specified back-end compiler for optimization and acceleration. `Inductor` (i.e., `TorchInductor`) is the default back-end deep learning compiler for `torch.compile` in the PyTorch framework. Its role is to compile the captured operator network into code optimized for specific hardware, such as generating Triton kernels for GPUs and C++ code for CPUs. The operator network is the complete computation graph of the neural network model's forward computation process. In large-scale model deployment and inference scenarios, such neural network models mainly include deep neural networks based on the Transformer architecture. GPU stands for Graphics Processing Unit. CPU stands for Central Processing Unit.

[0003] However, in actual engineering code implementations, the computation graph is not simply thrown to the compiler as a whole. To achieve optimal performance, developers of large model frameworks typically write separate, optimized low-level implementations for key computational components in the system. These key components include special attention mechanisms, MoE dynamic routing, and dynamic bucketing mechanisms designed to handle different sequence lengths. Due to these externally imposed boundaries, specific hardware and software instruction optimization requirements, and the existence of custom fusion operators, a complete and massive model forward computation process is manually segmented and encapsulated by developers in the code implementation, resulting in hundreds or thousands of independent and unrelated deep fusion piecewise subgraphs.

[0004] Existing computation graph compilation mechanisms have the following significant drawbacks when performing multi-subgraph compilation tasks: 1. Single-threaded serial blocking: Current compilers typically use a queue to execute low-level hardware target code for each subgraph in a serial manner, which causes serious compilation delays and fails to take advantage of the parallelism of multiple subgraphs to accelerate compilation.

[0005] 2. Idle core computing resources: Due to the global lock limitations of the concurrency mechanism of the underlying language of deep learning frameworks (such as Python's GIL lock) and the singleton mutex write lock constraints of the compiler front-end, in multi-core (such as 64-core, 128-core) server environment, usually only a single core is working at full load, resulting in the massive computing power of the computing cluster being severely idle.

[0006] 3. Excessive cold start and warm-up time: Large-scale serial repetitive compilation can cause the startup and warm-up of large cloud models to take tens of minutes or even hours, which greatly restricts the business system's ability to quickly expand and elastically extend when facing traffic surges.

[0007] As can be seen from the above, existing conventional technologies are unable to accelerate the parallel compilation of scattered subgraphs without reconstructing the underlying logic of deep learning frameworks. Summary of the Invention

[0008] The deep neural network compilation method and related equipment provided in this application at least solve the problem that existing conventional technologies are unable to accelerate the parallel compilation of scatter subgraphs without reconstructing the underlying logic of the deep learning framework.

[0009] According to one aspect of this application, a deep neural network compilation method is provided, the deep neural network compilation method being applied to a deep learning framework, the deep neural network compilation method comprising: After the deep learning framework completes the capture and segmentation of the computation graph of the deep neural network, and before sending the segmented subgraphs to the backend compiler, the segmented subgraphs are intercepted. Multiple parallel subprocesses are created, and each subgraph is distributed to each subprocess. Each subprocess independently calls the backend compiler to execute parallel compilation tasks to generate the underlying target code corresponding to each subgraph. During the parallel compilation task, the main process executing the native compilation process is placed in a waiting state. After all the parallel compilation tasks executed by the subprocesses are completed, the underlying target code corresponding to each subgraph is stored in the native file cache of the deep learning framework; After all the underlying target code is stored in the native file cache, the waiting state of the main process is released, and the main process submits a compilation request to the backend compiler according to the native compilation process, so that the native compilation process can retrieve the underlying target code corresponding to each subgraph from the native file cache and load it.

[0010] Further, before distributing each of the subgraphs to each of the subprocesses, the method includes: Fine-grained features are extracted from each of the subgraphs, and these fine-grained features are used to characterize the compilation behavior of the subgraphs. Based on the fine-grained features and the global compilation configuration variables of the deep learning framework, feature hash fingerprints of each subgraph are generated, wherein the global compilation configuration variables include target device and compiler options; Multiple subgraphs with the same characteristic hash fingerprint are deduplicated to obtain a pool of tasks to be compiled, which is constructed from one or more subgraphs with different characteristic hash fingerprints. All subgraphs in the pool of tasks to be compiled are then distributed to the respective subprocesses.

[0011] Furthermore, the step of deduplicating multiple subgraphs having the same characteristic hash fingerprint includes: Global collision detection is performed on the feature hash fingerprints corresponding to all the subgraphs to obtain collision detection results, and the collision detection results include the subgraphs with the same feature hash fingerprint; The collision detection results are deduplicated using a set data structure. During the deduplication process, if the feature hash fingerprint to be inserted already exists in the set data structure, the subgraph corresponding to the feature hash fingerprint to be inserted is discarded.

[0012] Furthermore, the fine-grained features include network topology, logical dependencies between operators, constants involved in the computation, metadata of input and output tensors, and arithmetic precision rules.

[0013] Furthermore, before creating multiple parallel subprocesses, the method also includes: Detect the idle processor resources and system load status of the host node where the deep learning framework is located; The number of child processes to be created is determined based on the idle processor resources and the system load status, so as to create multiple child processes matching the number.

[0014] Furthermore, before distributing each of the subgraphs to each of the subprocesses, the method further includes: The global compilation context environment required by the backend compiler in the main process is cloned, and the cloned global compilation context environment is injected into each of the child processes. The global compilation context environment includes at least one of the following: Hardware device attributes, precision control parameters, system environment variables, and communication backend configuration.

[0015] Furthermore, before distributing each of the subgraphs to each of the subprocesses, the method further includes: Each subgraph is encapsulated by constructing placeholder nodes at the input and output boundaries of each subgraph. The placeholder nodes are used to simulate the tensors of the input and output boundaries.

[0016] Furthermore, the step of constructing placeholder nodes at the input and output boundaries of each of the subgraphs includes: Metadata of the input and output tensors is obtained from the extracted fine-grained features, and the metadata includes at least the tensor dimension and tensor step size; Based on the metadata of the input tensor, construct input placeholder nodes at the input boundaries of the subgraph; Based on the metadata of the output tensor, output placeholder nodes are constructed at the output boundaries of the subgraph.

[0017] Furthermore, when each of the child processes independently invokes the backend compiler to execute parallel compilation tasks, the method includes: Each of the subprocesses concurrently submits the assigned subgraph to the backend compiler, which then parses each subgraph and generates the corresponding underlying target code.

[0018] Further, the step of storing the underlying target code corresponding to each of the subgraphs into the native file cache of the deep learning framework includes: Each of the subprocesses extracts the generated low-level target code from the backend compiler and encapsulates the low-level target code into a compilation object adapted to the deep learning framework; Using the feature hash fingerprint as the cache index key, and following the native cache storage structure and native directory addressing protocol of the deep learning framework, the compiled objects of each subgraph are written in parallel to the native file cache.

[0019] Furthermore, the step of releasing the main process from its waiting state includes: After each of the subprocesses has completed its parallel compilation task, it clears the cache that stores the status of its respective task and submits an end flag to the main process. The main process cancels each of the child processes according to the end flag and resumes the execution of the native compilation process to release the waiting state.

[0020] Furthermore, when the native compilation process retrieves and loads the underlying target code corresponding to each subgraph from the native file cache, the method includes: The native compilation process retrieves the corresponding underlying target code from the native file cache based on the feature hash fingerprint. If the underlying target code is found, the code generation step is skipped, and the underlying target code is retrieved and loaded.

[0021] According to another aspect of this application, a deep neural network compilation apparatus is provided, the deep neural network compilation apparatus being used to execute the deep neural network compilation method described above.

[0022] According to another aspect of this application, a non-transitory machine-readable medium storing computer instructions for causing a computer to execute the deep neural network compilation method described above is provided.

[0023] The deep neural network compilation method provided in this application intercepts subgraphs and distributes them to multiple parallel child processes for compilation, reconstructing the original sequential queue execution into multi-process concurrent execution. This completely breaks the blocking limitation of single-threaded compilation in the framework and accelerates the compilation process. The original serial time consumption of "accumulated compilation time of all subgraphs" is reduced to "maximum time overhead of single subgraph compilation," fundamentally eliminating the excessively long warm-up wait during the startup of large-scale deep neural network inference services. This enables business systems to quickly start new nodes with extremely low latency (seconds) during peak traffic periods, ensuring high availability and agile response capabilities. Attached Figure Description

[0024] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other embodiments based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating a deep neural network compilation method provided in an embodiment of this application.

[0026] Figure 2 This is a flowchart of the process for deduplicating duplicate subgraphs provided in one embodiment of the application.

[0027] Figure 3This is a schematic diagram of the structure of a deep neural network compilation device provided in one embodiment of the application.

[0028] In the picture: 10. Subgraph Interceptor; 20. Concurrency Scheduling Engine; 30. Write-back Controller; 40. Deactivation Engine; 50. Context Deep Cloning Engine; 60. Hash Collision and Deduplication Module; 70. Sandbox Compilation Adapter. Detailed Implementation

[0029] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0030] To address the challenge of accelerating the parallel compilation of scattered subgraphs using existing conventional techniques without refactoring the underlying logic of deep learning frameworks, the first embodiment of this application provides a deep neural network compilation method. This method is applied to a deep learning framework, which may include one of the following: PyTorch, DyNet, or the dynamic graph mode PaddlePaddle.

[0031] Please see Figure 1 The deep neural network compilation method provided in this application includes the following steps: Step S11: After the deep learning framework completes the capture and segmentation of the computation graph of the deep neural network, and before sending the segmented subgraphs to the backend compiler, intercept the segmented subgraphs.

[0032] Step S12: Create multiple parallel subprocesses, distribute each subgraph to each subprocess, and each subprocess independently calls the backend compiler to execute parallel compilation tasks to generate the underlying target code corresponding to each subgraph. During the parallel compilation task, the main process executing the native compilation process is placed in a waiting state. Specifically, one subprocess can be responsible for compiling one subgraph, or one subprocess can be responsible for compiling multiple subgraphs. A one-to-one approach is preferred, but regardless of the approach, in multi-subgraph (also known as computational subgraph) scenarios, it can significantly shorten the service startup and warm-up time of deep neural networks compared to existing technologies. The waiting state is also called the suspended state. Suspension means blocking / pausing the execution flow of the main process, such as calling wait() or join(), so that it does not consume processor resources (such as CPU) time slices and does not issue compilation requests in advance, until all subprocesses have completed compilation.

[0033] Step S13: After all the parallel compilation tasks executed by the child processes are completed, store the underlying target code corresponding to each subgraph into the native file cache of the deep learning framework.

[0034] Step S14: After all the underlying target code is stored in the native file cache, the main process is released from its waiting state. The main process then submits a compilation request to the backend compiler according to the native compilation process, so that the native compilation process can retrieve the underlying target code corresponding to each sub-graph from the native file cache and load it.

[0035] Through steps S11 to S14, the deep neural network compilation method provided in this application can intercept subgraphs and distribute them to multiple parallel subprocesses for compilation, reconstructing the original serial queue execution into multi-process concurrent execution. This completely breaks the blocking limitation of single-threaded compilation in the framework and accelerates the compilation process. The original serial time consumption of "cumulative compilation time of all subgraphs" is reduced to "maximum time overhead of single subgraph compilation," fundamentally eliminating the extremely long warm-up wait during the startup process of large-scale deep neural network inference services. This allows business systems to quickly start new nodes with extremely low latency (seconds) during peak traffic periods, ensuring high availability and agile response capabilities.

[0036] like Figure 2 As shown, before distributing each subgraph to each subprocess, the method provided in this application includes the following steps: Step S21: Extract fine-grained features from each subgraph. These fine-grained features are used to characterize the compilation behavior of the subgraph.

[0037] Step S22: Generate feature hash fingerprints for each subgraph based on fine-grained features and global compilation configuration variables of the deep learning framework. Global compilation configuration variables include target device and compiler options.

[0038] Step S23: Perform deduplication on multiple subgraphs with the same hash fingerprint to obtain a pool of tasks to be compiled constructed from one or more subgraphs with different hash fingerprints, so as to distribute all subgraphs in the pool of tasks to be compiled to each subprocess.

[0039] Following steps S21 to S23, fine-grained features are extracted and fused with global compilation configuration variables to generate feature hash fingerprints. This binds the structural features of the subgraphs to the hardware and software environment, enabling the feature hash fingerprints to accurately distinguish subgraphs under different computational logics or compilation configurations. This avoids cache misses or compilation product mismatches due to insufficient granularity of the feature hash fingerprints. Furthermore, subgraphs with the same feature hash fingerprints are deduplicated, eliminating the wasted computational power caused by the repeated compilation of numerous structurally identical or logically equivalent subgraphs in large deep neural network models. This minimizes the number of subgraphs in the compilation task pool and, in conjunction with parallel compilation tasks, further improves compilation acceleration.

[0040] The target device in the global compiler configuration variable refers to the hardware platform that the generated low-level target code is intended for, such as NVIDIA GPUs' sm_80 (corresponding to A100) or sm_90 (corresponding to H100), AMD GPUs' gfx90a, or a specific model of the Ascend NPU. Compiler options are control parameters that the user passes to the backend compiler, such as the optimization level (optimization_level=4), enabling automatic tuning (max-autotune=True), and allowed computational precision types (enabled_precisions={torch.half}).

[0041] In the feature hash fingerprint generation process, global compiler configuration variables and fine-grained features of the subgraph are used as input to bind the compilation behavior of the subgraph to a specific hardware and software context. The target device determines which instruction set the compiler generates—GPUs of different architectures have different instruction sets, register counts, and memory access patterns. Compiler options determine the optimization strategy the compiler adopts—machine code generated under different optimization levels or precision settings may have significant differences in performance and numerical behavior. By incorporating these variables into the feature hash fingerprint calculation, the same subgraph will be assigned different feature hash fingerprints under different target devices or different compiler options, thereby obtaining independent cache entries. This avoids hardware instruction incompatibility or runtime errors caused by mismatch between the underlying target code and the current runtime environment, ensuring the correctness and security of cache reuse.

[0042] The steps for deduplicating multiple subgraphs with the same hash fingerprint characteristics include: Global collision detection is performed on the feature hash fingerprints corresponding to all subgraphs to obtain collision detection results. These results include subgraphs with the same feature hash fingerprint. Next, a set data structure is used to deduplicate the collision detection results. During this process, as all feature hash fingerprints are inserted into the set data structure one by one, if the feature hash fingerprint to be inserted already exists in the set data structure, the subgraph corresponding to that fingerprint is discarded. Discarded subgraphs will not be added to the task pool; otherwise, they will be added.

[0043] Therefore, through global collision detection, the feature hash fingerprints of all subgraphs are comprehensively compared to accurately identify subgraphs with the same feature hash fingerprint. Based on this, the inherent uniqueness of the set data structure is used for deduplication, retaining only one subgraph for each feature hash fingerprint in the compilation task pool. This mechanism ensures that when a large model has a large number of structurally identical or logically equivalent subgraphs (such as multiple identical Transformer layers, equivalent operators with different sequence lengths, etc.), duplicate compilation tasks are eliminated from the compilation source. The number of subgraphs to be compiled is reduced from "total number of subgraphs" to "number of unique fingerprints," minimizing the actual load of subsequent parallel compilation. Combined with the parallel compilation task mechanism, this maximizes the compression of the total compilation time.

[0044] Fine-grained features include network topology, logical dependencies between operators, constants involved in computation, metadata of input and output tensors, and arithmetic precision rules. Among these: Network topology refers to the connection relationships and arrangement order of operators in a subgraph, that is, the overall data flow path from input to output in the computation process of the subgraph. In feature hash fingerprint generation, it is used to distinguish subgraphs with different operator orders but the same type. For example, [Conv, BN, ReLU] and [Conv, ReLU, BN] have different logic, ensuring that the feature hash fingerprint reflects the overall structural differences of the subgraph.

[0045] Logical dependencies between operators refer to the data dependencies between operators within a subgraph, i.e., which operators' outputs are the inputs of which operators, and the resulting execution order. In feature hash fingerprint generation, these dependencies are used to capture the execution constraints between operators. Even if operators are of the same type and order, different dependencies can lead to differences in compilation optimization strategies, which need to be distinguished using feature hash fingerprints.

[0046] The constants involved in the computation refer to the fixed numerical data in the subgraph, such as the weights of convolutional layers, the scaling factors and biases of batch normalization layers, and the constant biases in tensor addition. In fingerprint generation, the specific values ​​of the constants will affect the compiler's optimization decisions such as constant folding, and therefore need to be included in the feature hash fingerprint calculation.

[0047] Metadata of input and output tensors: This refers to attribute information such as the dimension (Shape), stride, data type, and storage device of the subgraph input and output tensors. In fingerprint generation, the metadata of input and output tensors determines how the backend compiler allocates memory, optimizes memory access, and configures kernel startup parameters. It needs to be an important component of feature hash fingerprinting to ensure that tensors of different specifications obtain their respective adapted underlying target code (i.e., compilation artifacts).

[0048] Arithmetic precision rules refer to the numerical precision format used by the subgraph during computation, such as fp32, fp16, bf16, etc. In feature hash fingerprint generation, arithmetic precision rules directly affect the hardware instructions selected by the backend compiler (such as whether to enable Tensor Cores) and the differences in numerical behavior. They need to be included in the feature hash fingerprint to ensure that the underlying target code cached in the native file cache of the native compilation process is consistent with the precision rules.

[0049] Before creating multiple parallel subprocesses, the method provided in this application also includes the following steps: The system detects the idle processor resources and system load status of the host node hosting the deep learning framework. Based on these parameters, it determines the number of child processes to be created and creates multiple child processes matching the specified number. Processor resources include, for example, a multi-core CPU. System load status refers to the current workload of the host node's computing resources, including metrics such as processor resource utilization (e.g., CPU utilization), run queue length, context switching frequency, and I / O wait time.

[0050] Therefore, by dynamically detecting the idle processor resources and system load status of the host node, and adaptively determining the number of child processes to be created based on the detection results, the number of parallel child processes created is matched with the currently available computing resources. This avoids idle computing power due to too few child processes or additional overhead due to too many child processes, and further improves the compilation throughput to the theoretical maximum value under the current load conditions.

[0051] Before distributing each subgraph to each subprocess, the method provided in this application further includes the following steps: The global compilation context required by the backend compiler in the main process is cloned, and the cloned global compilation context is injected into each child process. The global compilation context includes at least one of the following: Hardware device attributes, precision control parameters, system environment variables, and communication backend configuration.

[0052] This application clones and injects the global compilation context environment from the main process into each child process, enabling each child process to have a compilation configuration completely identical to that of the main process in an independent "sandbox" environment. This ensures that the underlying target code generated independently by each child process maintains consistency in hardware instruction sets, precision strategies, system dependencies, and communication protocols, avoiding problems such as incompatibility between compilation artifacts and the native runtime environment, hardware instruction incompatibility, or runtime linking errors caused by child processes lacking context information from the main process. This "sandboxing" mechanism provides child processes with an independent and consistent compilation environment, guaranteeing the correctness of parallel compilation.

[0053] Hardware device attributes (such as GPU microarchitecture and memory capacity) are used to ensure that the compiled output matches the physical hardware. Precision control parameters (such as whether to allow precision shrinkage) are used to ensure consistent numerical behavior. System environment variables (such as CUDA path and LD_LIBRARY_PATH) are used to ensure that the compilation process depends correctly. Communication backend configurations (such as NCCL parameters) are used to ensure compilation compatibility in distributed scenarios. NCCL stands for NVIDIA Collective Communications Library.

[0054] Before distributing each subgraph to each subprocess, the method provided in this application further includes the following steps: Each subgraph is encapsulated by constructing placeholder nodes at its input and output boundaries. These placeholder nodes simulate the tensors of the input and output boundaries. The placement of these placeholder nodes allows the backend compiler to recognize each subgraph as a complete computational module and ensures that the encapsulated subgraphs pass the compiler's validity check.

[0055] As can be seen, this application can construct placeholder nodes for simulating tensors at the input and output boundaries of the subgraph, enabling independent subgraphs that would otherwise fail the legality check of the underlying backend compiler due to a lack of external real tensor dependencies to be recognized by the compiler as compliant and complete compilable units. This mechanism provides the compiler with the structural integrity required for the segmented isolated subgraphs without altering the internal computational logic of the subgraph, thus removing the prerequisite for the independent parallel compilation of each subgraph.

[0056] Here, "tensors simulating input and output boundaries" refers to placeholder nodes that replace the positions of actual tensors, declaring the input and output specifications (such as dimensions and stride) of the subgraph to the compiler. Since the compiler only checks whether the subgraph module has explicit input and output definitions during verification, and does not check whether the tensor actually exists, the encapsulated subgraph can pass verification and compile normally. After compilation, the placeholder nodes can be discarded, and the actual tensors are still used for inference.

[0057] The steps described above for constructing placeholder nodes at the input and output boundaries of each subgraph include: Metadata for the input and output tensors is obtained from the extracted fine-grained features. This metadata includes at least the tensor dimension and tensor stride. Based on the metadata of the input tensor, input placeholder nodes are constructed at the input boundaries of the subgraph. Based on the metadata of the output tensor, output placeholder nodes are constructed at the output boundaries of the subgraph.

[0058] In this context, the tensor dimension (Shape) determines the tensor shape claimed by the placeholder node in the topology, and the tensor stride determines the memory arrangement claimed by the placeholder node. Together, they constitute the core parameters upon which the backend compiler relies for memory allocation and memory access optimization: the dimension determines the data size, and the stride determines the data's jump access pattern in memory. Without either, the backend compiler cannot accurately assess the memory access behavior of the subgraph, potentially leading to errors where the generated compiled artifact does not match the actual tensor's memory layout at runtime. Therefore, metadata including at least these two parameters is sufficient to ensure that the placeholder node structurally passes the backend compiler's validity check, while also providing necessary memory access information for subsequent compilation optimization.

[0059] Meanwhile, since the placeholder nodes are constructed based on previously extracted fine-grained features rather than being acquired temporarily, repeated parsing of the subgraph or additional data collection is avoided. This approach ensures the consistency of data sources and allows placeholder nodes to realistically replace the actual tensor boundaries without affecting the optimization decisions of the backend compiler.

[0060] When each child process independently calls the backend compiler to execute parallel compilation tasks, the method provided in this application includes the following steps: Each child process concurrently submits its assigned subgraph to the backend compiler, enabling multiple compilation tasks to be distributed to the backend compiler simultaneously. This avoids the queuing latency caused by serial submissions and extends the backend compiler's service capabilities from single-task serial processing to multi-task concurrent processing. The backend compiler then parses each subgraph and generates the corresponding low-level target code, completing the final conversion from intermediate representation to hardware instructions. This provides loadable target code files for subsequent storage of the low-level target code in the native file cache.

[0061] Therefore, this application increases the concurrency of the "submit-compile" path to the same level as the number of child processes, so that the overall compilation throughput is no longer limited by the serial bottleneck of a single submission queue, and maximizes the concurrent service capabilities of the backend compiler.

[0062] The backend compiler in this application includes at least one of the following: the Triton compiler and the XLA compiler.

[0063] The steps of storing the underlying target code corresponding to each subgraph into the native file cache of the deep learning framework include: Each subprocess extracts the generated low-level target code from the backend compiler and encapsulates it into a compiled object adapted to the deep learning framework. Then, using the feature hash fingerprint as the cache index key, and following the native cache storage structure and native directory addressing protocol of the deep learning framework, the compiled objects of each subgraph are written in parallel to the native file cache.

[0064] This application extracts the generated low-level target code from the backend compiler through each subprocess and encapsulates it into compiled objects. This ensures that the compiled binary products have a unified format before being stored in the cache, facilitating subsequent loading. Then, using the feature hash fingerprint as the cache index key, the compiled objects are written in parallel to the native file cache according to the native cache storage structure and native directory addressing protocol of the deep learning framework. This ensures that the pre-compiled products (low-level target code) are stored in the cache location retrieved by the framework by default (implementing disk persistence), guaranteeing that the subsequent native process can access the corresponding low-level target code according to its existing logic during verification. Simultaneously, "parallel writing" works in conjunction with parallel compilation, enabling concurrent capabilities in the disk persistence stage after the low-level target code generation, avoiding serial writing becoming a new performance bottleneck, and ensuring end-to-end parallel efficiency from compilation to writing.

[0065] The steps to release the main process from a waiting state include: After each child process completes its parallel compilation task, it clears the buffer storing its task state and submits an end flag to the main process. Based on the end flag, the main process cancels each child process and resumes the native compilation flow to release the waiting state. The normal execution flow (inference or startup process) is then restored.

[0066] When the native compilation process retrieves and loads the underlying target code corresponding to each subgraph from the native file cache, the method provided in this application includes the following steps: The native compilation process retrieves the corresponding underlying target code from the native file cache based on the feature hash fingerprint. If the underlying target code is found, the code generation step is skipped, and the underlying target code is retrieved and loaded.

[0067] The native compilation process, based on the feature hash fingerprint, finds the underlying target code in the native file cache, skips the time-consuming code generation step, and directly retrieves and loads the underlying target code. This transforms the request, which originally required a complete compilation chain of "parsing → optimization → code generation," into a single cache read operation. Since the compilation phases performed by multiple parallel subprocesses have already completed the placement and disking of the underlying target code in the native file cache according to the feature hash fingerprint, the retrieval by the native compilation process is guaranteed to be successful. This allows the framework to "spontaneously" skip the compilation cycle, reducing compilation time from minutes to milliseconds.

[0068] In other words, this application, by decoupling the compilation tasks of the computation graph and combining multi-subprocess asynchronous pre-compilation with a file cache hit-loading mechanism, can significantly shorten service startup and warm-up time in scenarios with multiple computation subgraphs without disruptively reconstructing the underlying logic of the original deep learning framework. It also overcomes the single-threaded compilation limitations of deep learning frameworks, fully activating and releasing the computing resources of idle CPUs and other multi-core physical processors on host nodes. Furthermore, it ensures that the pre-compilation logic strictly adheres to the original deep learning framework, maintaining high tolerance and non-intrusive characteristics towards mainstream deep learning underlying engines. Finally, it provides a highly scalable base cache communication architecture, enabling collaborative compilation across distributed heterogeneous computing clusters on multiple nodes.

[0069] The method provided in this application is applicable to scenarios where computational graphs are frequently segmented, such as large language models (LLM), computational architectures that introduce custom fusion operators, dynamic bucketing, and expert hybrid model (MoE) routing and distribution.

[0070] The second embodiment of this application also provides a deep neural network compilation apparatus for executing a deep neural network compilation method. The flow of the deep neural network compilation method can be found in the first embodiment of this application, and will not be repeated here.

[0071] In some implementations, such as Figure 3 As shown, the deep neural network compilation device in this application can be modularly designed. In this case, the deep neural network compilation device may include: a subgraph interceptor 10, a concurrent scheduling engine 20, a write-back controller 30, a deactivation engine 40, a context deep cloning engine 50, a hash collision and deduplication module 60, a sandbox compilation adapter 70, etc.

[0072] Subgraph interceptor 10 is used to intercept multiple subgraphs obtained from the segmentation.

[0073] The concurrent scheduling engine 20 is connected to the subgraph interceptor 10. The concurrent scheduling engine 20 is used to: create multiple parallel subprocesses, distribute each subgraph to each subprocess, and allow each subprocess to independently call the backend compiler to execute parallel compilation tasks to generate the underlying target code corresponding to each subgraph. During the parallel compilation task, the main process executing the native compilation process is placed into a waiting state. The concurrent scheduling engine 20 also probes idle processor resources and system load status before creating multiple parallel subprocesses, and determines the number of subprocesses to be created based on the idle processor resources and system load status.

[0074] The write-back controller 30 is connected to the concurrent scheduling engine 20. The write-back controller 30 is used to store the underlying target code corresponding to each subgraph into the native file cache of the deep learning framework after all the parallel compilation tasks executed by the child processes are completed.

[0075] The connection between engine 40 and write-back controller 30 is disconnected. The purpose of disconnecting engine 40 is to release the main process from the waiting state after all the underlying target code has been stored in the native file cache, so that the main process can submit a compilation request to the backend compiler according to the native compilation process, so that the native compilation process can obtain the underlying target code corresponding to each subgraph from the native file cache and load it.

[0076] The context deep cloning engine 50 is connected between the subgraph interceptor 10 and the concurrent scheduling engine 20. The context deep cloning engine 50 is used to clone the global compilation context environment required by the backend compiler in the main process and inject the cloned global compilation context environment into each child process.

[0077] The hash collision and deduplication module 60 is connected between the subgraph interceptor 10 and the concurrent scheduling engine 20. It is used to perform deduplication processing on multiple subgraphs with the same hash fingerprint characteristics. The deduplication processing is implemented by performing global collision detection and using a set data structure.

[0078] The sandbox compilation adapter 70 is connected between the subgraph interceptor 10 and the concurrent scheduling engine 20, and is used to encapsulate each subgraph by constructing placeholder nodes at the input and output boundaries of each subgraph.

[0079] The third embodiment of this application provides an application example to further illustrate the deep neural network compilation method provided in this application, so as to make the method provided in this application clearer and more understandable.

[0080] This deep neural network compilation method is suitable for application scenarios with high-frequency dynamic control flow and computation graph slicing characteristics, such as large language model inference deployment, hybrid expert models, and complex end-to-end systems.

[0081] In this application embodiment, the overall technical approach of the deep neural network compilation method includes: After the computation graph is captured and segmented, but before the actual compilation task is sent to the backend compiler, the resulting subgraphs are intercepted. Multiple parallel subprocesses are created using the multi-core computing power of the host node (e.g., a multi-core CPU). Each subgraph segment is distributed to an independent subprocess for code compilation, while the main process is temporarily suspended. The generated underlying target code is stored in the native file cache according to the retrieval logic of the native deep learning framework. After all parallel compilation tasks are completed, the main process (or the main thread of the main process) is unsuspended, and compilation is requested according to the native compilation process. At this point, the native compilation process triggers a cache hit during the verification of feature hash fingerprints, thus skipping the code generation step and directly loading the underlying target code, thereby accelerating the compilation process.

[0082] Key technical features of deep neural network compilation methods include: (I) Multidimensional graph segmentation, interception, feature extraction, and deduplication (acquisition and preprocessing stage) The core operational logic and key technical points of this stage are as follows: 1.1 Customized Subgraph Interception and Extraction: Addressing the architectural partitioning designs introduced in large-scale model deployments to pursue maximum computing power, such as custom fusion operators (Custom Fused Kernels), dynamic bucketing, and complex routing logic (e.g., MoE), this mechanism actively intercepts these independent subgraph modules after the deep learning framework completes the initial computation graph capture (e.g., acquiring the decoupled and encapsulated distributed FxGraph by the developer). Before these subgraphs queue up to enter the underlying serial backend compiler, fine-grained features of each subgraph are extracted. These fine-grained features include the underlying network topology, logical dependencies between operators, constants involved in the computation, metadata of input and output tensors (e.g., Shape, Stride), and arithmetic precision rules (e.g., fp16, bf16, etc.).

[0083] 1.2 Multidimensional Feature Hash Fingerprint Generation: Combining the fine-grained features extracted from all subgraphs above with the underlying global compilation configuration variables of the current hardware and software environment (such as the target device and compiler options), a unique "feature hash fingerprint (Cache Key / Hash Signature)" is derived and bound to each isolated subgraph structure using a hash algorithm. This feature hash fingerprint can accurately and exhaustively characterize all runtime context dependencies required by the subgraph when generating the underlying target code. A hash algorithm, also called a hash function, is an algorithm that compresses input data of arbitrary length into a fixed-length output (i.e., a "hash value" or "hash code") through mathematical operations.

[0084] 1.3 Equivalent Subgraph Hash Collision and Task Deduplication: Given that large model networks often exhibit deep, repetitive stacking (e.g., multiple Transformer Blocks with identical structures), or frequently invoke logically equivalent operators in bucketing strategies with different sequence lengths, these highly similar subgraphs generate consistent feature hash fingerprints. This mechanism performs global collision detection based on the generated feature hash fingerprints and uses a set data structure to deduplicate subgraphs, discarding redundant related subgraph compilation requests and extracting and retaining only the unique "baseline subgraph" sequence with the feature hash fingerprint to construct the final pool of tasks to be compiled. This fundamentally eliminates the wasted computing power and time delays caused by issuing duplicate operator compilation instructions to the underlying backend compiler.

[0085] (II) Sandbox Isolation and Concurrent Compilation Scheduling Based on Environment Cloning (Parallel Scheduling Phase) The core operational logic and key technical points of this stage are as follows: 2.1 Adaptive probe detection and process pool development: Dynamically detect the idle CPU physical cores and system load status on the host node, and adaptively establish a concurrent task pool (process pool) consisting of multiple sub-processes that match the computing power based on the detection results, that is, a pool of tasks to be compiled, to maximize the activation and utilization of the idle computing resources of the host node.

[0086] 2.2 Deep Cloning and Mirror Isolation of Compilation Context: The global compilation context environment required by the backend compiler in the main process (such as GPU device attributes, precision control parameters, system environment variables, and communication backend configurations) is fully cloned and losslessly injected into each independent parallel worker subprocess (sandbox). This ensures that the compiled files (low-level target code) generated in the independent isolation environment are highly consistent with the native system state, avoiding hardware instruction mismatches or runtime compatibility issues during subsequent calls.

[0087] 2.3 Placeholder Node Wrapping and Subgraph Compliance Encapsulation: For decoupled and separated independent subgraphs (which typically cannot directly pass the legality verification of the underlying backend compiler due to the lack of external real tensor dependencies), a "placeholder node" wrapping strategy is introduced. Based on tensor metadata such as dimension (Shape) and stride extracted in the previous fine-grained features, equivalent topological wrapping is constructed for the input and output boundaries of the subgraph, enabling the backend compiler to recognize each subgraph as a compliant and complete computation module.

[0088] 2.4 Parallel Distribution of Backend Library to Generate Target Instructions: Each subprocess concurrently dispatches the subgraph encapsulated in 2.3 to the backend compiler of the deep learning framework (such as the Triton compiler, XLA compiler, etc.), triggering the low-level parsing and hardware-optimized compilation sequence code generation of the backend compiler, ultimately producing low-level machine instructions / assembly code (i.e., low-level target code) adapted to the current accelerated hardware (such as GPU). The Triton compiler is a "high-level tool" for developers to efficiently write custom operators. XLA (Accelerated Linear Algebra) is a framework-oriented "automated pipeline" for automatically optimizing the entire model computation graph.

[0089] (III) Non-intrusive write-back based on the native caching mechanism of deep learning framework (relocation disk writing stage) The core operational logic and key technical points of this stage are as follows: 3.1 Extraction of underlying target code and encapsulation of computing library: Each parallel and independent subprocess group extracts the generated underlying target code (such as dynamic link library .so or GPU machine code file) from the backend compiler and encapsulates the underlying target code into computing library objects (i.e., compilation objects) that are deeply optimized for local acceleration hardware.

[0090] 3.2 Cache Posting to Disk Following Native Addressing Protocols: The "feature hash digital fingerprint" generated in the first stage is extracted as the cache key. Strictly adhering to the target deep learning framework's native cache storage structure and native directory addressing protocol (such as AOTAutograd or the underlying kernel's default file caching mechanism), the compiled and encapsulated underlying target code is written across processes in parallel to the designated native file cache storage area, completing the persistent storage of the pre-compiled result (underlying target code). This process does not alter the original deep learning framework pattern at all, ensuring the absolute safety of subsequent retrieval logic.

[0091] (iv) On-demand loading and cache hit-based ultra-fast startup (ultra-fast recovery and execution phase) The core operational logic and key technical points of this stage are as follows: 4.1 Releasing System Resources and Resuming Main Process Suspension: After all pre-compilation tasks of parallel subgraphs in the worker pool have been processed, the cache area has been cleared, and the end marker has been submitted, the concurrent child processes are unregistered, the main process is fully released from suspension, and the normal execution flow (inference or startup process) of the main program is restored.

[0092] 4.2 Framework native compilation interception and same-origin comparison detection: When the main process submits just-in-time (JIT) compilation requests to the backend compiler one by one according to the native compilation process of the native deep learning framework, the native compilation process will first calculate the feature hash fingerprint of the current subgraph, and then automatically enter the comparison and retrieval process in the native file cache.

[0093] 4.3 Cache Accurate Hit and Direct Loading of Target Code: Since the underlying target code has been located and written to disk according to the same-origin feature hash fingerprint during the pre-compilation stage, the probe of the underlying engine in the native compilation process will trigger a 100% cache hit. This mechanism allows the underlying framework of the main process to spontaneously skip the extremely time-consuming code generation cycle and directly load the underlying target code through the feature hash fingerprint index pointer. This achieves a performance leap from "heavy single-core serial compilation" to "direct disk read / write loading," significantly shortening the startup and warm-up time of the entire cluster service.

[0094] As can be seen, the above key technical features reveal how this application embodiment addresses the massive number of independent subgraphs generated in modern large-scale models due to the artificial introduction of advanced features such as dynamic bucketing and customized operators. Its entire flow chain is strictly divided into four nested technical sub-stages: First, in the acquisition and preprocessing stage, the system performs offline interception at the front end, actively extracting fine-grained features such as tensor metadata, and generating a rigorous feature hash fingerprint based on these fine-grained features and global compilation configuration variables to perform same-origin verification and deduplication. Second, in the parallel scheduling stage, the system horizontally clones multiple independent child processes based on the computing power of the host nodes, cleverly employing a placeholder node wrapping strategy within these child processes to successfully "deceive" the backend compiler into completing the legality verification in an isolated environment. Following this is the relocation and disk write-back stage: each concurrent pool of tasks waiting to be compiled performs a silent cross-process write-back of the generated underlying target code, strictly adhering to the established protocol of native file caching (such as native AOTAutograd or FxGraph Cache). Finally, in the high-speed execution phase: as the main process is released from suspension and control is returned, the native compilation process engine triggers the comparison of the original security feature hash fingerprint during regular calls, thereby forming a 100% cache hit. This fundamentally eliminates the blocking dependency of the heavy operator generation (Codegen) cycle on a single thread, realizing cross-dimensional release of computing power and performance leap acceleration.

[0095] Compared with the prior art, the embodiments of this application have the following outstanding advantages: 1) Overcoming the bottleneck of single-threaded blocking in the underlying architecture and achieving efficient concurrent release of multi-core computing power: This application addresses the singleton pattern constraints and thread lock issues within large model frameworks. Through innovative subprocess decoupling and subgraph distribution mechanisms, it completely breaks down the serial execution barrier of traditional JIT (Just-In-Time) queues. It fully activates and utilizes the idle multi-core CPU computing power of host nodes, reconstructing highly serial, time-consuming graph compilation tasks into flattened parallel preprocessing subtasks, greatly improving the utilization rate of physical computing power in giant computing clusters.

[0096] 2) Significantly reduce system cold start time, enabling extreme elastic scaling in cloud-native scenarios: Through parallel compilation task processing, this application reduces the "serial cumulative compilation time" of hundreds or thousands of highly customized subgraphs to the "maximum time overhead of single subgraph compilation" at theoretical physical extreme dimensionality reduction. This fundamentally solves the pain points of extremely long read bars and warm-up during the startup process of large model inference services, enabling online businesses to quickly launch new computing nodes with extremely low latency when dealing with sudden traffic surges, ensuring high availability and agile response capabilities of business systems.

[0097] 3) Maintaining absolute non-intrusion on the native framework, with extremely high compatibility and maintenance advantages in the underlying ecosystem: Through the offline interception of upper-layer subgraphs and the silent write-back mechanism of lower-layer file caching, this solution achieves "zero reconstruction and zero modification" of the underlying deep learning engine source code throughout the entire workflow. This "black-box" plug-in intervention not only eliminates the version compatibility and maintenance debt (Technical Debt) caused by the high-frequency iteration and upgrade of artificial intelligence AI frameworks, but also establishes strong versatility, enabling low-cost migration and adaptation to any other heterogeneous compilation platform with a hash caching system.

[0098] 4) Constructing a highly scalable cached communication architecture to lay the foundation for data center-level cross-node collaborative compilation: This application elegantly decouples and converges the complex graph compilation call chains between systems into lightweight fingerprint hash calculations and standard file I / O access operations. This architecture is naturally compatible with network components such as high-performance distributed key-value (KV) repositories, enabling the subgraph compilation results of a single node to achieve second-level cross-domain sharing and lossless reuse within the entire high-computing power cluster, thereby effectively avoiding the macroscopic computing power waste of repeatedly executing homomorphic code generation on massive nodes.

[0099] The fourth embodiment of this application provides a non-transitory machine-readable medium storing computer instructions for causing a computer to execute a deep neural network compilation method. For details of the deep neural network compilation method, please refer to the first to third embodiments of this application; further details will not be repeated here.

[0100] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0101] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, or infrared system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, compact optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0102] It should be noted that the term "comprising" and its variations used in the embodiments of this invention are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "a plurality" mentioned in the embodiments of this invention are illustrative and not restrictive, and those skilled in the art should understand that unless explicitly indicated otherwise in the context, they should be understood as "one or more".

[0103] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0104] The steps described in the method embodiments provided by the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0105] The term "embodiment" in this specification refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily imply the same embodiment, nor does it imply independence or alternativeity from other embodiments. The various embodiments in this specification are described in a related manner, with reference to each other for similar or identical parts. In particular, for apparatus, device, and system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and relevant details are referred to in the description of the method embodiments.

[0106] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A deep neural network compilation method, characterized in that, The deep neural network compilation method is applied to a deep learning framework, and the deep neural network compilation method includes: After the deep learning framework completes the capture and segmentation of the computation graph of the deep neural network, and before sending the segmented subgraphs to the backend compiler, the segmented subgraphs are intercepted. Multiple parallel subprocesses are created, and each subgraph is distributed to each subprocess. Each subprocess independently calls the backend compiler to execute parallel compilation tasks to generate the underlying target code corresponding to each subgraph. During the parallel compilation task, the main process executing the native compilation process is placed in a waiting state. After all the parallel compilation tasks executed by the subprocesses are completed, the underlying target code corresponding to each subgraph is stored in the native file cache of the deep learning framework; After all the underlying target code is stored in the native file cache, the waiting state of the main process is released, and the main process submits a compilation request to the backend compiler according to the native compilation process, so that the native compilation process can retrieve the underlying target code corresponding to each subgraph from the native file cache and load it.

2. The deep neural network compilation method according to claim 1, characterized in that, Before distributing each of the subgraphs to each of the subprocesses, the method includes: Fine-grained features are extracted from each of the subgraphs, and these fine-grained features are used to characterize the compilation behavior of the subgraphs. Based on the fine-grained features and the global compilation configuration variables of the deep learning framework, feature hash fingerprints of each subgraph are generated, wherein the global compilation configuration variables include target device and compiler options; Multiple subgraphs with the same characteristic hash fingerprint are deduplicated to obtain a pool of tasks to be compiled, which is constructed from one or more subgraphs with different characteristic hash fingerprints. All subgraphs in the pool of tasks to be compiled are then distributed to the respective subprocesses.

3. The deep neural network compilation method according to claim 2, characterized in that, The step of deduplicating multiple subgraphs having the same characteristic hash fingerprint includes: Global collision detection is performed on the feature hash fingerprints corresponding to all the subgraphs to obtain collision detection results, and the collision detection results include the subgraphs with the same feature hash fingerprint; The collision detection results are deduplicated using a set data structure. During the deduplication process, if the feature hash fingerprint to be inserted already exists in the set data structure, the subgraph corresponding to the feature hash fingerprint to be inserted is discarded.

4. The deep neural network compilation method according to claim 2, characterized in that, The fine-grained features include network topology, logical dependencies between operators, constants involved in the computation, metadata of input and output tensors, and arithmetic precision rules.

5. The deep neural network compilation method according to claim 1, characterized in that, Before creating multiple parallel subprocesses, the method further includes: Detect the idle processor resources and system load status of the host node where the deep learning framework is located; The number of child processes to be created is determined based on the idle processor resources and the system load status, so as to create multiple child processes matching the number.

6. The deep neural network compilation method according to claim 1, characterized in that, Before distributing each of the subgraphs to each of the subprocesses, the method further includes: The global compilation context environment required by the backend compiler in the main process is cloned, and the cloned global compilation context environment is injected into each of the child processes. The global compilation context environment includes at least one of the following: Hardware device attributes, precision control parameters, system environment variables, and communication backend configuration.

7. The deep neural network compilation method according to claim 2, characterized in that, Before distributing each of the subgraphs to each of the subprocesses, the method further includes: Each subgraph is encapsulated by constructing placeholder nodes at the input and output boundaries of each subgraph. The placeholder nodes are used to simulate the tensors of the input and output boundaries.

8. The deep neural network compilation method according to claim 7, characterized in that, The steps of constructing placeholder nodes at the input and output boundaries of each of the subgraphs include: Metadata of the input and output tensors is obtained from the extracted fine-grained features, wherein the metadata includes at least the tensor dimension and tensor step size; Based on the metadata of the input tensor, construct input placeholder nodes at the input boundaries of the subgraph; Based on the metadata of the output tensor, output placeholder nodes are constructed at the output boundaries of the subgraph.

9. The deep neural network compilation method according to any one of claims 1 to 8, characterized in that, When each of the child processes independently calls the backend compiler to execute parallel compilation tasks, the method includes: Each of the subprocesses concurrently submits the assigned subgraph to the backend compiler, which then parses each subgraph and generates the corresponding underlying target code.

10. The deep neural network compilation method according to claim 2, characterized in that, The step of storing the underlying target code corresponding to each of the subgraphs into the native file cache of the deep learning framework includes: Each of the subprocesses extracts the generated low-level target code from the backend compiler and encapsulates the low-level target code into a compilation object adapted to the deep learning framework; Using the feature hash fingerprint as the cache index key, and following the native cache storage structure and native directory addressing protocol of the deep learning framework, the compiled objects of each subgraph are written in parallel to the native file cache.

11. The deep neural network compilation method according to claim 1, characterized in that, The steps to release the main process from its waiting state include: After each of the subprocesses has completed its parallel compilation task, it clears the cache that stores the status of its respective task and submits an end flag to the main process. The main process cancels each of the child processes according to the end flag and resumes the execution of the native compilation process to release the waiting state.

12. The deep neural network compilation method according to claim 10, characterized in that, When the native compilation process retrieves and loads the underlying target code corresponding to each subgraph from the native file cache, the method includes: The native compilation process retrieves the corresponding underlying target code from the native file cache based on the feature hash fingerprint. If the underlying target code is found, the code generation step is skipped, and the underlying target code is retrieved and loaded.

13. A deep neural network compilation device, characterized in that, The deep neural network compilation device is used to execute the deep neural network compilation method according to any one of claims 1 to 12.

14. A non-transitory machine-readable medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the deep neural network compilation method according to any one of claims 1 to 12.

Images