Model compiling method and device, computer equipment and storage medium
By parsing and rewriting the ONNX model, subgraph fragments suitable for deep learning accelerators and graphics processors are generated, solving the problems of low execution coverage and compilation efficiency of deep learning accelerators, and achieving more efficient hardware resource utilization and compilation efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING XIAOMA YIYI TECH CO LTD
- Filing Date
- 2025-11-21
- Publication Date
- 2026-04-21
AI Technical Summary
Among existing model compilation methods, deep learning accelerators have low execution coverage and compilation efficiency, making it difficult to effectively utilize hardware resources.
By parsing and classifying the ONNX model, we identify the operator types that can be directly mapped to deep learning accelerators and those that cannot be directly mapped. We then use a rewriting strategy to rewrite the operators that cannot be directly mapped and generate execution subgraph fragments and residual subgraph fragments based on operator capabilities and constraint libraries. These fragments are then compiled on deep learning accelerators and graphics processing units, respectively.
It improves the execution coverage of deep learning accelerators, reduces cross-device data transfer and kernel switching overhead, lowers overall latency and jitter, achieves data parallelism and model parallelism, and improves compilation efficiency and hardware resource utilization.
Smart Images
Figure CN121900762A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a model compilation method, apparatus, computer device, and storage medium. Background Technology
[0002] Deep learning computing frameworks can be used to perform model inference, employing appropriate data structures to represent neural network models, which typically include multiple operators. With the continuous development of deep learning technology, the number of operators in deep learning is also increasing daily; some complex neural network models may include hundreds of operators.
[0003] However, current model compilation methods suffer from problems such as low execution coverage or low compilation efficiency of deep learning accelerators. Summary of the Invention
[0004] Therefore, it is necessary to provide a model compilation method, apparatus, computer device, and storage medium that can improve the execution coverage and compilation efficiency of deep learning accelerators in response to the above-mentioned technical problems.
[0005] Firstly, a model compilation method is provided, the method comprising: Obtain the target ONNX model; the target ONNX model is a pre-trained ONNX model that has been quantized and calibrated. After parsing the target ONNX model, model feature data is obtained. Based on the operator capability and constraint library, each operator type is classified to obtain the corresponding first target operator type and the corresponding second target operator type. The model feature data includes each operator type. The first target operator type is used to characterize that the corresponding operator can be directly mapped to a deep learning accelerator. The second target operator type is used to characterize that the corresponding operator cannot be directly mapped to a deep learning accelerator. After rewriting each second objective operator type according to the rewriting strategy, the corresponding rewritten second objective operator type is obtained. After verifying and constraining each rewritten second objective operator type, the corresponding check result is obtained. The check result includes check success or check failure. Based on the operators corresponding to each first target operator type and each third target operator type in the target ONNX model, execution subgraph slices for each deep learning accelerator are generated, and each execution subgraph slice is sent to the corresponding deep learning accelerator for compilation. The third target operator type refers to the second target operator type after rewriting processing if the check result is successful. Based on the operator corresponding to the fourth target operator type in the target ONNX model, the execution residual subgraph fragments of each graphics processor are generated, and each execution residual subgraph fragment is sent to the corresponding graphics processor for compilation processing; the fourth target operator type refers to the second target operator type after rewriting processing when the check result is a check failure.
[0006] In one embodiment, the rewriting strategy includes one or more of the following: equivalent substitution rewriting operation, sequence fusion rewriting operation, and shape and alignment correction operation.
[0007] In one embodiment, each execution subgraph slice is sent to the corresponding deep learning accelerator for compilation, including: determining the slicing method of all execution subgraph slices based on the spatial feature parameters, channel feature parameters, and batch dimension feature parameters of the target ONNX model; determining the slice size of all execution subgraph slices based on the tensor core operation granularity and on-chip cache capacity constraints of each deep learning accelerator; slicing each execution subgraph slice according to the slicing method and slice size to obtain the corresponding sliced execution subgraph slices; and sending each sliced execution subgraph slice to the corresponding deep learning accelerator for compilation.
[0008] In one embodiment, the execution subgraphs after each slice are sent to the corresponding deep learning accelerators for compilation, including: determining the parallel computing method based on the computational cost of the target ONNX model and the expected parallel benefits; the parallel computing method includes data parallelism, model parallelism, or pipelined parallelism; and sending the execution subgraphs after each slice to the corresponding deep learning accelerators for compilation according to the parallel computing method.
[0009] In one embodiment, the execution subgraph fragments after each slice are distributed to the corresponding deep learning accelerators for compilation processing according to a parallel computing method. This includes: determining the deep learning accelerator to which the corresponding execution subgraph fragment belongs based on the data dependencies, buffer usage, and core utilization of each execution subgraph fragment; allocating each execution subgraph fragment to the pending data queue of the corresponding deep learning accelerator; and generating corresponding processing control instructions based on each execution subgraph fragment and the corresponding deep learning accelerator. The processing control instructions are used to instruct the corresponding deep learning accelerator to compile the corresponding execution subgraph fragment according to a parallel computing method and output the corresponding fragment compilation result.
[0010] In one embodiment, the method further includes: obtaining the fragmented compilation results corresponding to each processing control instruction; merging the fragmented compilation results to obtain the compilation result of the target ONNX model.
[0011] In one embodiment, the method further includes: When executing the step of sending the sliced execution subgraphs to the corresponding deep learning accelerators for compilation processing according to the parallel computing method, a compilation check is performed to obtain the first compilation check result. The execution subgraphs after each slice are sent to the shadow graphics processor for compilation processing according to the parallel computing method. During the step of sending the sliced execution subgraphs to the shadow graphics processor for compilation processing according to the parallel computing method, a compilation check is performed to obtain the second compilation check result. The first offline difference index of the execution subgraph after slicing is calculated based on the first compilation detection results and the corresponding second compilation detection results; In response to the existence of a first target execution subgraph fragment, the uncompiled first target execution subgraph fragment is sent to the graphics processor for compilation; the first target execution subgraph fragment is the execution subgraph fragment after slicing where the first offline difference index is greater than the first offline difference index threshold.
[0012] In one embodiment, before sending each execution subgraph slice to the corresponding deep learning accelerator for compilation, the method further includes: Randomly select from each execution subgraph to obtain the corresponding representative execution subgraph; Each representative execution subgraph is sent to the verification deep learning accelerator for compilation detection, and the corresponding third compilation detection result is obtained. Each representative execution subgraph is sent to the verification graphics processor for compilation detection, and the corresponding fourth compilation detection result is obtained; The second offline difference index of the corresponding representative execution subgraph is calculated based on the third compilation detection result and the corresponding fourth compilation detection result of each representative execution subgraph. In response to the existence of a second target execution subgraph, the second target execution subgraph is converted into an execution residual subgraph fragment; the second target execution subgraph is the representative execution subgraph where the second offline difference index is greater than the offline difference index threshold.
[0013] Secondly, a model compilation device is provided, the device including a model acquisition module, an operator classification module, a type rewriting module, an execution subgraph fragmentation generation module, and an execution residual subgraph fragmentation generation module.
[0014] The module comprises several sub-modules: a model acquisition module for acquiring the target ONNX model (a pre-trained ONNX model that has been quantized and calibrated); an operator classification module for parsing the target ONNX model to obtain model feature data, classifying each operator type according to an operator capability and constraint library to obtain the corresponding first target operator type and second target operator type; the model feature data includes each operator type; the first target operator type represents that the corresponding operator can be directly mapped to a deep learning accelerator; the second target operator type represents that the corresponding operator cannot be directly mapped to a deep learning accelerator; and a type rewriting module for rewriting each second target operator type according to a rewriting strategy to obtain the corresponding rewritten second target operator type, and performing verification and constraint checks on each rewritten second target operator type. The processing yields corresponding inspection results, including success or failure. The execution subgraph generation module generates execution subgraphs for each deep learning accelerator based on the operators corresponding to the first target operator type and the third target operator type in the target ONNX model. Each execution subgraph is then sent to the corresponding deep learning accelerator for compilation. The third target operator type refers to the rewritten second target operator type after a successful inspection. The execution residual subgraph generation module generates execution residual subgraphs for each graphics processor based on the operators corresponding to the fourth target operator type in the target ONNX model. Each execution residual subgraph is then sent to the corresponding graphics processor for compilation. The fourth target operator type refers to the rewritten second target operator type after a failed inspection.
[0015] Thirdly, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described in the above method embodiments.
[0016] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the methods described in the above method embodiments.
[0017] The aforementioned model compilation method, apparatus, computer equipment, and storage medium are used to obtain a target ONNX model. The target ONNX model is a pre-trained ONNX model that has undergone quantization and calibration. Then, the target ONNX model is parsed to obtain model feature data. Based on the operator capability and constraint library, each operator type is classified to obtain the corresponding first target operator type and the corresponding second target operator type. The model feature data includes each operator type. The first target operator type represents that the corresponding operator can be directly mapped to a deep learning accelerator; the second target operator type represents that the corresponding operator cannot be directly mapped to a deep learning accelerator. Next, each second target operator type is rewritten according to a rewriting strategy to obtain the corresponding rewritten second target operator type. Each rewritten second target operator type is then verified and constraint-checked to obtain the corresponding check result. The check result includes check success or check failure. Finally, based on the target ONNX model... The first target operator type in the target ONNX model and the third target operator type in the target ONNX model generate execution subgraph slices for each deep learning accelerator. Each execution subgraph slice is then sent to the corresponding deep learning accelerator for compilation. The third target operator type refers to the second target operator type after rewriting if the check result is successful. Finally, the execution residual subgraph slices for each graphics processor are generated based on the operator corresponding to the fourth target operator type in the target ONNX model. Each execution subgraph slice is then sent to the corresponding graphics processor for compilation. The fourth target operator type refers to the second target operator type after rewriting if the check result is unsuccessful. This allows more subgraph slices to be executed on the DLA, reducing cross-device data migration and kernel switching overhead, lowering overall latency and jitter, improving the execution coverage of the deep learning accelerator, achieving data parallelism and model parallelism, keeping the core in an effective working state for a long time, and reducing bubbles and blockages caused by dependencies. Attached Figure Description
[0018] Figure 1 This is a diagram illustrating the application environment of the model compilation method in one embodiment; Figure 2 This is a schematic diagram of the first process of a model compilation method in one embodiment; Figure 3 This is a schematic diagram of the process in one embodiment where each execution subgraph is distributed to the corresponding deep learning accelerator for compilation. Figure 4 This is a schematic diagram of the process in one embodiment of sending the sliced execution subgraphs to the corresponding deep learning accelerators for compilation. Figure 5This is a schematic diagram of the process in one embodiment where the sliced execution subgraphs are distributed to the corresponding deep learning accelerators for compilation processing according to a parallel computing method. Figure 6 This is a schematic diagram of the second process of the model compilation method in one embodiment; Figure 7 This is a structural block diagram of a model compilation device in one embodiment; Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0020] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be more thorough and complete.
[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.
[0022] It is understood that the terms "first," "second," etc., used herein may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of this application, a first resistor may be referred to as a second resistor, and similarly, a second resistor may be referred to as a first resistor. Both the first resistor and the second resistor are resistors, but they are not the same resistor.
[0023] It is understood that the term "connection" in the following embodiments should be understood as "electrical connection," "communication connection," etc., if the connected circuits, modules, units, etc., have electrical signal or data transmission with each other.
[0024] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” or “having,” etc., specify the presence of the stated feature, whole, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof.
[0025] The model compilation method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. (Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices; server 104 can be a standalone server or a server cluster consisting of multiple servers.)
[0026] In one embodiment, such as Figure 2 As shown, a model compilation method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 205.
[0027] Step 201: Obtain the target ONNX model.
[0028] The target ONNX model is a pre-trained ONNX model that has been quantized and calibrated. Specifically, server 104 acquires the target ONNX model.
[0029] In a specific example, ONNX (Open Neural Network Exchange) is an open-source machine learning model representation format designed to solve compatibility issues between deep learning frameworks. By defining standardized computation graphs and operator sets, it enables seamless model migration across different frameworks, platforms, and hardware. ONNX's core goal is to simplify the model training and deployment process, avoiding redundant development due to framework differences. The above is merely a specific example; in practical applications, settings should be flexibly configured according to user needs, and no restrictions are imposed here.
[0030] Step 202: After parsing the target ONNX model, the model feature data is obtained. After classifying each operator type according to the operator capability and constraint library, the corresponding first target operator type and the corresponding second target operator type are obtained.
[0031] The model feature data includes various operator types; the first target operator type represents the operator that can be directly mapped to a Deep Learning Accelerator (DLA); the second target operator type represents the operator that cannot be directly mapped to a Deep Learning Accelerator. Specifically, server 104 parses the target ONNX model to obtain model feature data, and classifies each operator type according to the operator capability and constraint library to obtain the corresponding first target operator type and the corresponding second target operator type.
[0032] In a specific example, the model feature data also includes each topology, each tensor shape, each weight, and each constant. The above is just a specific example, and in actual applications, it can be flexibly set according to user needs. There are no restrictions here.
[0033] Step 203: After rewriting each second objective operator type according to the rewriting strategy, the corresponding rewritten second objective operator type is obtained. After verifying and constraining each rewritten second objective operator type, the corresponding check result is obtained.
[0034] The inspection results include whether the inspection was successful or failed. Specifically, server 104 rewrites each second target operator type according to the rewriting strategy to obtain the corresponding rewritten second target operator type, and performs verification and constraint checks on each rewritten second target operator type to obtain the corresponding inspection result.
[0035] In one embodiment, the rewriting strategy includes one or more of the following: equivalent substitution rewriting operation, sequence fusion rewriting operation, and shape and alignment correction operation.
[0036] In a specific example, the equivalent substitution rewrite operation is used to replace operators that cannot be directly mapped to deep learning accelerators with combinations of operators that can be directly mapped to them. For example, it converts a padded Deconv operator (not supported by DLA) into a padded Deconv + slice operator, which can be directly and completely integrated into the DLA computation graph. The sequence fusion rewrite operation is used to fuse or rearrange sequences such as Conv + BN + Scale into DLA-executable sequences (obeying the Q / DQ boundaries of the ONNX graph) without changing the numerical path. The shape and alignment correction operation is used to ensure that the blocks meet the matrix / vector granularity and on-chip storage constraints of DLA through pad / stride / slice tiling. The above are just specific examples; in actual applications, they can be flexibly set according to user needs and are not limited here.
[0037] Step 204: Generate execution subgraph slices for each deep learning accelerator based on the operators corresponding to each first target operator type and each third target operator type in the target ONNX model, and send each execution subgraph slice to the corresponding deep learning accelerator for compilation.
[0038] The third target operator type refers to the second target operator type after the rewrite process, where the check result is successful. Specifically, server 104 generates execution subgraph fragments for each deep learning accelerator based on the operators corresponding to the first target operator types and the operators corresponding to the third target operator types in the target ONNX model. Each execution subgraph fragment is then distributed to the corresponding deep learning accelerator for compilation, allowing more subgraph fragments to be executed on the DLA, reducing cross-device data migration and kernel switching overhead, lowering overall latency and jitter, and improving the execution coverage of the deep learning accelerator.
[0039] In one embodiment, such as Figure 3 As shown, each execution subgraph is distributed to the corresponding deep learning accelerator for compilation, including steps 301 to 304.
[0040] Step 301: Determine the slicing method for all execution subgraphs based on the spatial feature parameters, channel feature parameters, and batch dimension feature parameters of the target ONNX model; Step 302: Determine the slice size of all execution subgraph slices based on the tensor core operation granularity and on-chip cache capacity constraints of each deep learning accelerator; Step 303: Slice each execution subgraph segment according to the slicing method and slice size to obtain the corresponding sliced execution subgraph segments; Step 304: The execution subgraphs after each slice are sent to the corresponding deep learning accelerators for compilation.
[0041] Specifically, server 104 determines the slicing method for all execution subgraph fragments based on the spatial feature parameters, channel feature parameters, and batch dimension feature parameters of the target ONNX model. Then, it determines the slice size of all execution subgraph fragments based on the tensor core operation granularity and on-chip cache capacity constraints of each deep learning accelerator. Next, it slices each execution subgraph fragment according to the slicing method and slice size to obtain the corresponding sliced execution subgraph fragments. Finally, it sends each sliced execution subgraph fragment to the corresponding deep learning accelerator for compilation processing, realizing data parallelism and model parallelism, ensuring that the core is in an effective working state for a long time and reducing bubbles and blockages caused by dependencies.
[0042] In one embodiment, such as Figure 4As shown, the execution subgraphs after each slice are sent to the corresponding deep learning accelerators for compilation, including steps 401 to 402.
[0043] Step 401: Determine the parallel computing method based on the computational cost of the target ONNX model and the expected benefits of parallel computing; Step 402: The execution subgraphs after each slice are distributed to the corresponding deep learning accelerators for compilation processing according to the parallel computing method.
[0044] The parallel computing methods include data parallelism, model parallelism, or pipeline parallelism. Specifically, server 104 determines the parallel computing method based on the computational load of the target ONNX model and the expected benefits of parallelism; then, according to the parallel computing method, the execution subgraphs after each slice are distributed to the corresponding deep learning accelerators for compilation, which improves the efficiency and convenience of model compilation.
[0045] In one embodiment, such as Figure 5 As shown, the execution subgraphs after each slice are distributed to the corresponding deep learning accelerators for compilation processing according to the parallel computing method, including steps 501 to 502.
[0046] Step 501: Determine the deep learning accelerator to which the corresponding execution subgraph fragment belongs based on the data dependency, buffer usage, and core usage of each sliced execution subgraph fragment. Step 502: Allocate the execution subgraph slices after each slice to the data queue to be processed of the corresponding deep learning accelerator, and generate corresponding processing control instructions based on the execution subgraph slices after each slice and the corresponding deep learning accelerator.
[0047] The processing control instructions are used to instruct the corresponding deep learning accelerator to compile the corresponding sliced execution subgraph fragments using parallel computing and output the corresponding fragment compilation results. Server 104 determines the deep learning accelerator to which each sliced execution subgraph fragment belongs based on its data dependencies, buffer usage, and core utilization. Then, it allocates each sliced execution subgraph fragment to the pending data queue of its corresponding deep learning accelerator and generates corresponding processing control instructions based on the sliced execution subgraph fragments and their respective deep learning accelerators. This achieves data parallelism and model parallelism, ensuring the core remains in an effective working state for an extended period and reducing dependency bubbles and blockages.
[0048] In this embodiment, the deep learning accelerator to which the corresponding execution subgraph fragment belongs is determined based on the data dependency, buffer usage, and core occupancy of each sliced execution subgraph fragment. Then, each sliced execution subgraph fragment is assigned to the pending data queue of the corresponding deep learning accelerator, and corresponding processing control instructions are generated based on each sliced execution subgraph fragment and the corresponding deep learning accelerator. This achieves data parallelism and model parallelism, ensuring that the core remains in an effective working state for a long time and reducing bubbles and blockages caused by dependencies.
[0049] In this embodiment, the parallel computing method is determined based on the computational cost of the target ONNX model and the expected benefits of parallel computing. Then, the execution subgraphs after each slice are distributed to the corresponding deep learning accelerators for compilation processing according to the parallel computing method, which improves the efficiency and convenience of model compilation.
[0050] In this embodiment, the slicing method of all execution subgraph fragments is determined based on the spatial feature parameters, channel feature parameters, and batch dimension feature parameters of the target ONNX model. Then, the slice size of all execution subgraph fragments is determined based on the tensor core operation granularity and on-chip cache capacity constraints of each deep learning accelerator. Next, each execution subgraph fragment is sliced according to the slicing method and slice size to obtain the corresponding sliced execution subgraph fragments. Finally, each sliced execution subgraph fragment is sent to the corresponding deep learning accelerator for compilation processing, realizing data parallelism and model parallelism, ensuring that the core is in an effective working state for a long time and reducing bubbles and blockages caused by dependencies.
[0051] Step 205: Generate execution residual subgraph fragments for each graphics processor based on the operator corresponding to the fourth target operator type in the target ONNX model, and send each execution residual subgraph fragment to the corresponding graphics processor for compilation processing.
[0052] The fourth target operator type refers to the second target operator type after rewriting processing when the check result is a failure. Server 104 generates execution residual subgraph fragments for each graphics processor based on the operator corresponding to the fourth target operator type in the target ONNX model, and sends each execution residual subgraph fragment to the corresponding graphics processing unit (GPU) for compilation processing, thereby improving the execution efficiency of the execution residual subgraph fragments and thus improving the overall compilation efficiency of the model.
[0053] The above model compilation method obtains the target ONNX model; the target ONNX model is a pre-trained ONNX model that has been quantized and calibrated; then, the target ONNX model is parsed to obtain model feature data, and each operator type is classified according to the operator capability and constraint library to obtain the corresponding first target operator type and the corresponding second target operator type; the model feature data includes each operator type; the first target operator type is used to characterize that the corresponding operator can be directly mapped to a deep learning accelerator; the second target operator type is used to characterize that the corresponding operator cannot be directly mapped to a deep learning accelerator; next, each second target operator type is rewritten according to the rewriting strategy to obtain the corresponding rewritten second target operator type, and each rewritten second target operator type is verified and constraint checked to obtain the corresponding check result; the check result includes check success or check failure; then, according to each first target operator type in the target ONNX model The corresponding operators and the operators corresponding to the third target operator types in the target ONNX model generate execution subgraph fragments for each deep learning accelerator. Each execution subgraph fragment is then sent to the corresponding deep learning accelerator for compilation. The third target operator type refers to the second target operator type after rewriting processing if the check result is successful. Finally, based on the operators corresponding to the fourth target operator type in the target ONNX model, execution residual subgraph fragments for each graphics processor are generated. Each execution subgraph fragment is then sent to the corresponding graphics processor for compilation. The fourth target operator type refers to the second target operator type after rewriting processing if the check result is unsuccessful. This allows more subgraph fragments to be executed on the DLA, reducing cross-device data migration and kernel switching overhead, lowering overall latency and jitter, improving the execution coverage of the deep learning accelerator, achieving data parallelism and model parallelism, keeping the core in an effective working state for a long time, reducing dependency bubbles and blockages, and improving compilation efficiency.
[0054] In one embodiment, such as Figure 6 As shown, the method further includes steps 601 to 602.
[0055] Step 601: Obtain the slice compilation results corresponding to each processing control instruction; Step 602: Merge the compilation results of each segment to obtain the compilation result of the target ONNX model.
[0056] Specifically, server 104 obtains the fragmented compilation results corresponding to each processing control instruction; then, it merges the fragmented compilation results to obtain the compilation result of the target ONNX model, ensuring the consistency of the values.
[0057] In this embodiment, the fragmented compilation results corresponding to each processing control instruction are obtained; then, the fragmented compilation results are merged to obtain the compilation result of the target ONNX model, ensuring the consistency of the values.
[0058] In one embodiment, the method further includes: When executing the step of sending the sliced execution subgraphs to the corresponding deep learning accelerators for compilation processing according to the parallel computing method, a compilation check is performed to obtain the first compilation check result. The execution subgraphs after each slice are sent to the shadow graphics processor for compilation processing according to the parallel computing method. During the step of sending the sliced execution subgraphs to the shadow graphics processor for compilation processing according to the parallel computing method, a compilation check is performed to obtain the second compilation check result. The first offline difference index of the execution subgraph after slicing is calculated based on the first compilation detection results and the corresponding second compilation detection results; In response to the existence of a first target execution subgraph fragment, the uncompiled first target execution subgraph fragment is sent to the graphics processor for compilation.
[0059] Specifically, the first target execution subgraph fragment is the execution subgraph fragment after slicing where the first offline difference index is greater than the first offline difference index threshold. Specifically, when server 104 performs the step of sending each sliced execution subgraph fragment to the corresponding deep learning accelerator for compilation processing according to a parallel computing method, it performs compilation detection to obtain a first compilation detection result. Then, it sends each sliced execution subgraph fragment to the shadow graphics processor for compilation processing according to a parallel computing method. Next, when performing the step of sending each sliced execution subgraph fragment to the shadow graphics processor for compilation processing according to a parallel computing method, it performs compilation detection to obtain a second compilation detection result. Then, based on each first compilation detection result and the corresponding second compilation detection result, it calculates the first offline difference index of the corresponding sliced execution subgraph fragment. Finally, in response to the existence of the first target execution subgraph fragment, it sends the uncompiled first target execution subgraph fragment to the graphics processor for compilation processing, realizing the dynamic adjustment of the first target execution subgraph fragment to be processed by runtime DLA to the execution residual subgraph fragment of the GPU, improving the efficiency and accuracy of model compilation.
[0060] In this embodiment, during the step of sending the sliced execution subgraph fragments to the corresponding deep learning accelerators for compilation processing according to the parallel computing method, compilation detection is performed to obtain a first compilation detection result. Then, the sliced execution subgraph fragments are sent to the shadow graphics processor for compilation processing according to the parallel computing method. Next, during the step of sending the sliced execution subgraph fragments to the shadow graphics processor for compilation processing according to the parallel computing method, compilation detection is performed to obtain a second compilation detection result. Then, based on the first compilation detection result and the corresponding second compilation detection result, a first offline difference index of the corresponding sliced execution subgraph fragment is calculated. Finally, in response to the existence of a first target execution subgraph fragment, the uncompiled first target execution subgraph fragment is sent to the graphics processor for compilation processing. The first target execution subgraph fragment is the sliced execution subgraph fragment whose first offline difference index is greater than the first offline difference index threshold. This realizes the dynamic adjustment of the first target execution subgraph fragment to be processed by the runtime DLA to the execution residual subgraph fragment of the GPU, improving the efficiency and accuracy of model compilation.
[0061] In one embodiment, before sending each execution subgraph slice to the corresponding deep learning accelerator for compilation, the method further includes: Randomly select from each execution subgraph to obtain the corresponding representative execution subgraph; Each representative execution subgraph is sent to the verification deep learning accelerator for compilation detection, and the corresponding third compilation detection result is obtained. Each representative execution subgraph is sent to the verification graphics processor for compilation detection, and the corresponding fourth compilation detection result is obtained; The second offline difference index of the corresponding representative execution subgraph is calculated based on the third compilation detection result and the corresponding fourth compilation detection result of each representative execution subgraph. In response to the existence of a second target execution subgraph, the second target execution subgraph is converted into an execution residual subgraph fragment.
[0062] The second target execution subgraph is a representative execution subgraph whose second offline difference index is greater than the offline difference index threshold. Specifically, server 104 randomly selects representative execution subgraphs from each execution subgraph; then, it sends each representative execution subgraph to the verification deep learning accelerator for compilation detection, obtaining the corresponding third compilation detection result; next, it sends each representative execution subgraph to the verification graphics processor for compilation detection, obtaining the corresponding fourth compilation detection result; then, it calculates the second offline difference index of the corresponding representative execution subgraph based on the third compilation detection result and the corresponding fourth compilation detection result; finally, in response to the existence of the second target execution subgraph, it converts the second target execution subgraph into execution residual subgraph fragments, realizing the adjustment of the second target execution subgraph to be processed by DLA offline to the execution residual subgraph fragments of the GPU, improving the efficiency and accuracy of model compilation.
[0063] In a specific example, the first offline difference metric may include, but is not limited to, one or more of MSE, Cosine similarity, relative error, and Top-k consistency. The above is merely a specific example; in actual applications, it can be flexibly set according to user needs, and no restrictions are imposed here.
[0064] In this embodiment, representative execution subgraphs are randomly selected from each execution subgraph. Then, each representative execution subgraph is sent to a validation deep learning accelerator for compilation detection, yielding a third compilation detection result. Next, each representative execution subgraph is sent to a validation graphics processor for compilation detection, yielding a fourth compilation detection result. Then, based on the third and fourth compilation detection results of each representative execution subgraph, a second offline difference index for the corresponding representative execution subgraph is calculated. Finally, in response to the existence of a second target execution subgraph, the second target execution subgraph is converted into execution residual subgraph fragments. This achieves the adjustment of the second target execution subgraph to be processed by the DLA offline to the execution residual subgraph fragments of the GPU, improving the efficiency and accuracy of model compilation.
[0065] In a specific example, the method further includes: Monitor the compilation and processing time of the execution subgraph fragments after each slice; In response to the existence of a first timeout execution subgraph slice, the uncompiled first timeout execution subgraph slice is adjusted from the pending data queue of the deep learning accelerator to which it belongs to the pending data queue of another deep learning accelerator; wherein, the first timeout execution subgraph slice is the execution subgraph slice after the compilation processing time is greater than a first time threshold; In response to the existence of a second timeout execution subgraph slice, the uncompiled second timeout execution subgraph slice is moved from the pending data queue of its deep learning accelerator to the graphics processor for compilation. The second timeout execution subgraph slice is the execution subgraph slice whose compilation time exceeds a second time threshold, where the second time threshold is greater than a first time threshold. Through these steps, the slice allocation strategy is adjusted, reducing blocking and idling, and maintaining high energy efficiency. This is merely a specific example; in practical applications, it can be flexibly configured according to user needs, and no restrictions are imposed here.
[0066] It should be understood that, although Figure 2-6 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2-6 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0067] Secondly, such as Figure 7 As shown, a model compilation device is provided, the device including a model acquisition module 710, an operator classification module 720, a type rewriting module 730, an execution subgraph segmentation generation module 740, and an execution residual subgraph segmentation generation module 750.
[0068] The model acquisition module 710 is used to acquire the target ONNX model; the target ONNX model is a pre-trained ONNX model that has been quantized and calibrated. The operator classification module 720 is used to parse the target ONNX model to obtain model feature data, and classify each operator type according to the operator capability and constraint library to obtain the corresponding first target operator type and the corresponding second target operator type. The model feature data includes each operator type; the first target operator type is used to represent that the corresponding operator can be directly mapped to a deep learning accelerator; the second target operator type is used to represent that the corresponding operator cannot be directly mapped to a deep learning accelerator. The type rewriting module 730 is used to rewrite each second target operator type according to the rewriting strategy to obtain the corresponding rewritten second target operator type, and to verify and constrain each rewritten second target operator type. After the inspection and processing, the corresponding inspection results are obtained; the inspection results include inspection success or inspection failure; the execution subgraph fragment generation module 740 is used to generate execution subgraph fragments for each deep learning accelerator according to the operators corresponding to each first target operator type and each third target operator type in the target ONNX model, and sends each execution subgraph fragment to the corresponding deep learning accelerator for compilation processing; the third target operator type refers to the second target operator type after rewriting processing if the inspection result is successful; the execution residual subgraph fragment generation module 750 is used to generate execution residual subgraph fragments for each graphics processor according to the operators corresponding to the fourth target operator type in the target ONNX model, and sends each residual execution subgraph fragment to the corresponding graphics processor for compilation processing; the fourth target operator type refers to the second target operator type after rewriting processing if the inspection result is failed.
[0069] In one embodiment, the rewriting strategy includes one or more of the following: equivalent substitution rewriting operation, sequence fusion rewriting operation, and shape and alignment correction operation.
[0070] In one embodiment, the subgraph fragmentation generation module 730 includes a slicing unit and a compilation unit.
[0071] The slicing unit is used to determine the slicing method of all execution subgraphs based on the spatial feature parameters, channel feature parameters, and batch dimension feature parameters of the target ONNX model; the slicing unit is used to determine the slice size of all execution subgraphs based on the tensor core operation granularity and on-chip cache capacity constraints of each deep learning accelerator; the slicing unit is used to slice each execution subgraph according to the slicing method and slice size to obtain the corresponding sliced execution subgraphs; and the compilation unit is used to send each sliced execution subgraph to the corresponding deep learning accelerator for compilation processing.
[0072] In one embodiment, the compilation unit includes a compilation subunit.
[0073] The compilation subunit is used to determine the parallel computing method based on the computational cost and expected parallel benefits of the target ONNX model. The parallel computing method includes data parallelism, model parallelism, or pipelined parallelism. The compilation subunit is used to distribute the execution subgraphs after each slice to the corresponding deep learning accelerator for compilation processing according to the parallel computing method.
[0074] In one embodiment, the compilation subunit includes a compilation processor.
[0075] The compilation processor is used to determine the deep learning accelerator to which the corresponding execution subgraph fragment belongs based on the data dependencies, buffer usage, and core utilization of each sliced execution subgraph fragment. The compilation processor is used to allocate each sliced execution subgraph fragment to the pending data queue of the corresponding deep learning accelerator, and generate corresponding processing control instructions based on each sliced execution subgraph fragment and the corresponding deep learning accelerator. The processing control instructions are used to instruct the corresponding deep learning accelerator to compile the corresponding sliced execution subgraph fragment according to the parallel computing method and output the corresponding fragment compilation result.
[0076] In one embodiment, the apparatus further includes a result merging module.
[0077] The result merging module is used to obtain the segmented compilation results corresponding to each processing control instruction; the result merging module is used to merge the segmented compilation results to obtain the compilation result of the target ONNX model.
[0078] In one embodiment, the apparatus further includes a compilation detection module. The compilation detection module performs compilation detection during the step of sending the sliced execution subgraph fragments to the corresponding deep learning accelerators for compilation processing according to a parallel computing method, obtaining a first compilation detection result; the compilation detection module sends the sliced execution subgraph fragments to the shadow graphics processor for compilation processing according to a parallel computing method; the compilation detection module performs compilation detection during the step of sending the sliced execution subgraph fragments to the shadow graphics processor for compilation processing according to a parallel computing method, obtaining a second compilation detection result; the compilation detection module calculates a first offline difference index for the corresponding sliced execution subgraph fragment based on each first compilation detection result and the corresponding second compilation detection result; the compilation detection module, in response to the existence of a first target execution subgraph fragment, sends the uncompiled first target execution subgraph fragment to the graphics processor for compilation processing; the first target execution subgraph fragment is the sliced execution subgraph fragment whose first offline difference index is greater than a first offline difference index threshold.
[0079] In one embodiment, the compilation detection module is further configured to randomly select representative execution subgraphs based on each execution subgraph; the compilation detection module is further configured to send each representative execution subgraph to a verification deep learning accelerator for compilation detection to obtain a corresponding third compilation detection result; the compilation detection module is further configured to send each representative execution subgraph to a verification graphics processor for compilation detection to obtain a corresponding fourth compilation detection result; the compilation detection module is further configured to calculate a second offline difference index for the corresponding representative execution subgraph based on the third compilation detection result and the corresponding fourth compilation detection result of each representative execution subgraph; the compilation detection module is further configured to convert the second target execution subgraph into an execution residual subgraph fragment in response to the existence of a second target execution subgraph; the second target execution subgraph is a representative execution subgraph whose second offline difference index is greater than the offline difference index threshold.
[0080] Specific limitations regarding the model compilation device can be found in the limitations of the model compilation method above, and will not be repeated here. Each module in the aforementioned model compilation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.
[0081] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data for the target ONNX model. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a model compilation method.
[0082] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0083] Thirdly, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described in the above method embodiments.
[0084] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the methods described in the above method embodiments.
[0085] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0086] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0087] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A model compilation method, the method comprising: Obtain the target ONNX model; the target ONNX model is a pre-trained ONNX model that has been quantized and calibrated. After parsing the target ONNX model, model feature data is obtained. After classifying each operator type according to the operator capability and constraint library, the corresponding first target operator type and the corresponding second target operator type are obtained. Model feature data includes various operator types; The first target operator type is used to characterize whether the corresponding operator can be directly mapped to a deep learning accelerator; The second target operator type is used to characterize that the corresponding operator cannot be directly mapped to the deep learning accelerator; After rewriting each of the second target operator types according to the rewriting strategy, the corresponding rewritten second target operator type is obtained. After verifying and constraining each of the rewritten second target operator types, the corresponding check result is obtained. The check result includes check success or check failure. Based on the operators corresponding to each first target operator type in the target ONNX model and the operators corresponding to each third target operator type in the target ONNX model, execution subgraph slices for each deep learning accelerator are generated, and each execution subgraph slice is sent to the corresponding deep learning accelerator for compilation processing; the third target operator type refers to the second target operator type after rewriting processing where the check result is successful. Based on the operator corresponding to the fourth target operator type in the target ONNX model, an execution residual subgraph fragment is generated for each graphics processor, and each execution residual subgraph fragment is sent to the corresponding graphics processor for compilation processing; the fourth target operator type refers to the second target operator type after rewriting processing when the check result is the check failure.
2. The method according to claim 1, characterized in that, The rewriting strategy includes one or more of the following: equivalent substitution rewriting operation, sequence fusion rewriting operation, and shape and alignment correction operation.
3. The method according to claim 1, characterized in that, The step of sending each of the execution subgraphs to the corresponding deep learning accelerators for compilation includes: The slicing method for all execution subgraphs is determined based on the spatial feature parameters, channel feature parameters, and batch dimension feature parameters of the target ONNX model; The slice size of all execution subgraph slices is determined based on the tensor core operation granularity and on-chip cache capacity constraints of each deep learning accelerator; Each execution subgraph segment is sliced according to the slicing method and the slicing size to obtain the corresponding sliced execution subgraph segment; The execution subgraphs after each slice are sent to the corresponding deep learning accelerators for compilation.
4. The method according to claim 3, characterized in that, The step of sending the sliced execution subgraphs to the corresponding deep learning accelerators for compilation includes: The parallel computing method is determined based on the computational cost and expected parallel benefits of the target ONNX model; the parallel computing method includes data parallelism, model parallelism, or pipelined parallelism. According to the parallel computing method, each sliced execution subgraph is sent to the corresponding deep learning accelerator for compilation.
5. The method according to claim 4, characterized in that, The step of sending the sliced execution subgraphs to the corresponding deep learning accelerators for compilation processing according to the parallel computing method includes: The deep learning accelerator to which the corresponding execution subgraph fragment belongs is determined based on the data dependencies, buffer usage, and core occupancy of each sliced execution subgraph fragment. Each of the sliced execution subgraphs is assigned to the pending data queue of the corresponding deep learning accelerator, and a corresponding processing control instruction is generated based on each sliced execution subgraph and the corresponding deep learning accelerator. The processing control instruction is used to instruct the corresponding deep learning accelerator to compile the corresponding sliced execution subgraph according to the parallel computing method and output the corresponding slice compilation result.
6. The method according to claim 5, characterized in that, The method further includes: Obtain the segmented compilation results corresponding to each of the aforementioned processing control instructions; The compilation results of each segment are merged to obtain the compilation result of the target ONNX model.
7. The method according to claim 4, characterized in that, The method further includes: During the step of sending the sliced execution subgraphs to the corresponding deep learning accelerators for compilation processing according to the parallel computing method, a compilation check is performed to obtain a first compilation check result. According to the parallel computing method, each sliced execution subgraph is sent to the shadow graphics processor for compilation. During the step of sending the sliced execution subgraphs to the shadow graphics processor for compilation processing according to the parallel computing method, a compilation check is performed to obtain a second compilation check result. The first offline difference index of the execution subgraph segment after the slice is calculated based on each of the first compilation detection results and the corresponding second compilation detection results. In response to the existence of a first target execution subgraph fragment, the uncompiled first target execution subgraph fragment is sent to the graphics processor for compilation; the first target execution subgraph fragment is the execution subgraph fragment after the slice in which the first offline difference index is greater than the first offline difference index threshold.
8. The method according to claim 1, characterized in that, Before the step of sending each of the execution subgraph slices to the corresponding deep learning accelerator for compilation processing, the method further includes: Randomly select from each of the aforementioned execution subgraphs to obtain the corresponding representative execution subgraph; Each of the representative execution subgraphs is sent to the verification deep learning accelerator for compilation detection, and the corresponding third compilation detection result is obtained. Each of the representative execution subgraphs is sent to the verification graphics processor for compilation detection, and the corresponding fourth compilation detection result is obtained. The second offline difference index of the corresponding representative execution subgraph is calculated based on the third compilation detection result and the corresponding fourth compilation detection result of each representative execution subgraph. In response to the existence of a second target execution subgraph, the second target execution subgraph is converted into the execution residual subgraph fragment; the second target execution subgraph is the representative execution subgraph where the second offline difference index is greater than the offline difference index threshold.
9. A model compilation device, characterized in that, The device includes: The model acquisition module is used to acquire the target ONNX model; the target ONNX model is a trained ONNX model that has been quantized and calibrated. The operator classification module is used to parse and process the target ONNX model to obtain model feature data, and classify each operator type according to the operator capability and constraint library to obtain the corresponding first target operator type and the corresponding second target operator type; the model feature data includes each operator type; the first target operator type is used to characterize that the corresponding operator can be directly mapped to the deep learning accelerator; the second target operator type is used to characterize that the corresponding operator cannot be directly mapped to the deep learning accelerator; The type rewriting module is used to rewrite each of the second target operator types according to the rewriting strategy to obtain the corresponding rewritten second target operator types, and to perform verification and constraint checks on each of the rewritten second target operator types to obtain the corresponding check results; the check results include check success or check failure. The execution subgraph generation module is used to generate execution subgraphs for each deep learning accelerator based on the operators corresponding to each first target operator type in the target ONNX model and the operators corresponding to each third target operator type in the target ONNX model, and to send each execution subgraph to the corresponding deep learning accelerator for compilation processing; the third target operator type refers to the second target operator type after rewriting processing where the check result is successful. The execution residual subgraph fragment generation module is used to generate execution residual subgraph fragments for each graphics processor according to the operator corresponding to the fourth target operator type in the target ONNX model, and to send each of the residual execution subgraph fragments to the corresponding graphics processor for compilation processing; the fourth target operator type refers to the second target operator type after rewriting processing when the check result is the check failure.
10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.