Model processing method and device

Abstract

The invention provides a model processing method and device, and relates to the technical field of artificial intelligence. The model processing method comprises the following steps: abstracting a target neural network model by constructing a directed acyclic graph, and taking the abstracted target neural network model as a high-order IR of the model; optimizing on the basis of the high-order IR, including operator fusion, reordering, sub-graph division and node multiplexing, so as to obtain a graph optimization result as a low-order IR of the model; and performing optimization on the basis of the low-order IR, including on-chip operator fusion, instruction side-by-side and hardware instruction mapping, so as to obtain an optimization code of the target neural network model. According to the method, the target neural network model is subjected to hierarchical processing, corresponding compilation optimization is carried out, the usability of an upper-layer application and the extreme performance of bottom-layer hardware are balanced, the calculation potential of a data flow architecture is fully released, meanwhile, a user does not need to deeply care about hardware details, and model programming with high efficiency and high hardware utilization rate is achieved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method and apparatus for processing a model. Background Technology

[0002] Currently, with the rapid expansion of applications, many scenarios can adopt data stream processing, such as natural language processing and recommendation engines, and the performance and efficiency challenges of traditional instruction set architectures have become very apparent.

[0003] To address this problem and enable next-generation scientific and machine learning applications, dataflow architecture has become the mainstream technology for international AI chip startups. It offers several times the performance with the same number of transistors, or similar performance with fewer transistors, significantly improving the cost-effectiveness of computing power. Developing programming model systems based on dataflow architecture technology provides high-performance, low-latency, and cost-effective computing software support for computer vision, speech and semantics, and large-scale model applications.

[0004] However, on the software side, the complexity of the models and the numerous frameworks, coupled with the unfamiliarity of technical personnel in various application areas with dataflow architecture hardware processing and a lack of professional training in dataflow architecture computing development, result in low efficiency and an inability to fully realize the computing potential of dataflow architecture when it is translated into actual productivity. Summary of the Invention

[0005] Based on this, this application provides a model processing method and apparatus to achieve high-efficiency and high-hardware-utilization model programming.

[0006] According to one aspect of this application, a model processing method is proposed, comprising: generating a directed acyclic graph (DAG) based on a programming paradigm and a target neural network model, wherein the DAG includes operator nodes and tensor edges; identifying continuous operator nodes conforming to fusion rules based on the DAG, and fusing the continuous operator nodes into new operator nodes to obtain an operator fusion result; sorting the operator nodes in the operator fusion result, and performing subgraph partitioning based on the sorting result to obtain a subgraph partitioning result; reusing operator nodes that need to be repeatedly calculated based on the subgraph partitioning result to obtain a graph optimization result; extracting multi-input operator nodes based on the graph optimization result, and performing on-chip operator fusion annotation on the graph optimization result according to the inputs of the multi-input operator nodes; in the graph optimization result after on-chip operator fusion annotation, performing instruction-side-by-instruction annotation on operator node pairs that are in the same subgraph and have no data dependency; and mapping the graph optimization result after instruction-side-by-instruction annotation to hardware instructions according to a predefined unified operation primitive, thereby obtaining the optimized code of the target neural network model.

[0007] According to some embodiments, a directed acyclic graph is generated based on a programming paradigm and a target neural network model, including: identifying the computational operations of the target neural network model based on the programming paradigm, obtaining operators, and using the operators as operator nodes; representing the input and output data between the layers of the target neural network model as tensors, and adding tensor edges between operator nodes according to the tensors to obtain a directed acyclic graph.

[0008] According to some embodiments, the operator nodes in the operator fusion result are sorted, and subgraph partitioning is performed based on the sorting result to obtain the subgraph partitioning result, including: S131: classifying the operator nodes to obtain the classification result; S132: grouping the linearly connected operator nodes based on the classification result to obtain multiple operator node groups; S133: sorting the operator nodes according to the depth-first traversal strategy to obtain the depth sorting result; S134: obtaining the dependency relationship of the operator nodes according to the depth sorting result; S135: sorting the current operator nodes according to the breadth-first traversal strategy to obtain the breadth sorting result; S136: based on the breadth sorting result, selecting operators with the same breadth... In the node, extract the reference operator node and the operator node to be inserted, and insert the operator node to be inserted and the operator nodes that have dependencies on it into the corresponding positions of the target operator node that meets the preset conditions in the depth sorting result, and insert the subgraph partitioning mark. The preset conditions include: the operator node to be inserted and the target operator node are located in the same operator node group, and the reference operator node depends on the target operator node; S137: Repeat steps S135-S136 until all operator nodes with the same breadth are traversed to obtain the sorting result of the operator nodes; S138: Based on the sorting result, according to the subgraph partitioning mark, the directed acyclic graph is divided to obtain multiple subgraphs, which are used as the subgraph partitioning result.

[0009] According to some embodiments, based on the graph optimization results, multi-input operator nodes are extracted, and on-chip operator fusion annotation is performed on the graph optimization results according to the inputs of the multi-input operator nodes, including: S151: Extracting multi-input operator nodes from the graph optimization results as the current operator node; S152: Reversely searching the input branches of the current operator node and saving the operator nodes in the input branches to the operator list of the current operator node; S153: Repeating steps S151-S152 until the operator nodes in the current subgraph are traversed; S154: According to the maximum block cutting strategy, the operator list is traversed and cut in reverse order to obtain multiple fusion blocks; S155: Writing the information of the fusion blocks back to the operator nodes of the graph optimization results as on-chip operator fusion annotation.

[0010] According to some embodiments, step S151 includes: selecting the input operator node of the graph optimization result as the starting node, finding the branch based on the starting node as the first branch; obtaining the child node of the starting node based on the first branch as the target node; and if the target node is a multi-input operator node, using the target node as the current operator node.

[0011] According to some embodiments, based on predefined unified operation primitives, the graph optimization results after instruction side-by-side annotation are mapped to hardware instructions to obtain the optimization code of the target neural network model. This includes: based on the unified operation primitives, identifying specific subgraph patterns and / or operator node sequences from the graph optimization results after instruction side-by-side annotation to obtain multiple operation primitive sequences; and converting the multiple operation primitive sequences into corresponding hardware instructions to obtain the optimization code of the target neural network model.

[0012] According to some embodiments, unified operation primitives include: memory management operation primitives, computation operation primitives, scheduler operation primitives, and / or layout transformation operation primitives.

[0013] According to some embodiments, based on a directed acyclic graph, continuous operator nodes that meet the fusion rules are identified, and continuous operator nodes are fused into new operator nodes to obtain operator fusion results. This includes: selecting seed operator nodes from operators in the directed acyclic graph; using a greedy algorithm, based on the seed operator nodes, continuous operator nodes that meet the fusion rules are fused downwards, and continuous operator nodes are fused into new operator nodes to obtain operator fusion results.

[0014] According to one aspect of this application, a model processing apparatus includes: a computation graph unit for generating a directed acyclic graph (DAG) based on a target neural network model according to a programming paradigm, wherein the DAG includes operator nodes and tensor edges; an operator fusion unit for identifying consecutive operator nodes conforming to fusion rules based on the DAG, and fusing the consecutive operator nodes into new operator nodes to obtain an operator fusion result; a sorting and partitioning unit for sorting the operator nodes in the operator fusion result, and performing subgraph partitioning based on the sorting result to obtain a subgraph partitioning result; and a node reuse unit for reusing nodes based on the subgraph partitioning result. The system reuses operator nodes that require repeated computation to obtain graph optimization results; the on-chip fusion unit extracts multi-input operator nodes based on the graph optimization results and performs on-chip operator fusion annotation on the graph optimization results according to the inputs of the multi-input operator nodes; the instruction rearrangement unit performs side-by-side instruction annotation on operator node pairs that are in the same subgraph and have no data dependency in the graph optimization results after on-chip operator fusion annotation; and the instruction mapping unit maps the graph optimization results after instruction side-by-side annotation to hardware instructions according to predefined unified operation primitives, thereby obtaining the optimized code of the target neural network model.

[0015] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0016] According to one aspect of this application, a computer-readable medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described above.

[0017] Through the embodiments provided in this application, a directed acyclic graph (DAG) is constructed to abstract the target neural network model, serving as the model's higher-order instruction recursion (IR). Optimization is then performed on the higher-order IR, including operator fusion, reordering, subgraph partitioning, and node reuse, resulting in a graph optimization result, which serves as the model's lower-order IR. Further optimization is then performed on the lower-order IR, including on-chip operator fusion, instruction parallelization, and hardware instruction mapping, resulting in optimized code for the target neural network model. This application performs layered processing of the target neural network model and corresponding compilation optimizations, balancing the ease of use of upper-layer applications with the ultimate performance of lower-layer hardware. While fully releasing the computational potential of the dataflow architecture, users do not need to delve into hardware details, achieving highly efficient and hardware-utilization-efficient model programming. Attached Figure Description

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0020] Figure 1 A flowchart illustrating the model processing method provided in this application embodiment; Figure 2 A flowchart of operator sorting and subgraph partitioning for the model processing method provided in the embodiments of this application; Figure 3 A schematic diagram of operator sorting and subgraph partitioning in the processing method of the model provided in the embodiments of this application; Figure 4 A flowchart of on-chip operator fusion for the model processing method provided in the embodiments of this application; Figure 5A flowchart illustrating the process of extracting multi-input operator nodes from graph optimization results as the current operator node in the model processing method provided in this application embodiment; Figure 6 The processing method for the model provided in this application maps the graph optimization result after the instructions are labeled side by side to hardware instructions according to the predefined unified operation primitives, thereby obtaining a flowchart of the optimized code for the target neural network model; Figure 7 The flowchart of the model processing method provided in the embodiments of this application is based on a directed acyclic graph, identifies continuous operator nodes that conform to the fusion rules, and fuses the continuous operator nodes into new operator nodes to obtain the operator fusion result. Figure 8 A schematic diagram of the data path for the model processing method provided in the embodiments of this application; Figure 9 A block diagram of a processing apparatus for a model provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0023] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0024] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0025] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0026] For specific implementation details, please refer to the following examples.

[0027] Figure 1 A flowchart illustrating the model processing method provided in this application embodiment. Figure 1 As shown, the method includes steps S110-S170.

[0028] In step S110, based on the programming paradigm, a directed acyclic graph is generated according to the target neural network model, wherein the directed acyclic graph includes operator nodes and tensor edges.

[0029] The target neural network model is the neural network model that the user wants to programmatically process.

[0030] The neural network model is represented as a directed acyclic graph consisting of operator nodes and tensor edges, completely abstracting the underlying hardware details to provide a hardware-independent model description.

[0031] According to the example embodiment, step S110 is implemented using a programming model.

[0032] This application does not restrict the deep learning framework of the target neural network model and supports the import of models based on various deep learning frameworks (such as PyTorch and TensorFlow).

[0033] During the abstraction process, the programming paradigm and the abstraction of tensors are clearly defined.

[0034] It is understandable that tensors are a fundamental data structure for computation in dataflow architectures. Programming models need to provide abstract definitions and operation interfaces for tensors, while shielding them from the complex underlying data arrangement formats.

[0035] Each deep learning framework has a dominant, default programming paradigm (e.g., imperative, declarative, functional, etc.). Based on the programming paradigm corresponding to the deep learning framework of the target neural network model, the computational logic of the target neural network model is expressed as a (explicit or implicit) computation graph, thus obtaining a directed acyclic graph.

[0036] Furthermore, it can be understood that, based on the constructed directed acyclic graph, choosing the correct parallel paradigm decomposes a large task into multiple smaller tasks and executes these smaller tasks simultaneously to reduce computation time or handle larger-scale problems. The parallel paradigm is crucial for the model's performance, maintainability, and correctness.

[0037] In step S120, based on the directed acyclic graph, continuous operator nodes that conform to the fusion rules are identified, and the continuous operator nodes are fused into new operator nodes to obtain the operator fusion result.

[0038] Next, the directed acyclic graph (DAG) is compiled, specifically through steps S120-S170. Its main task is to convert the target neural network model from the deep learning framework into optimized code that can be efficiently executed on dataflow architecture hardware. Specifically, it translates the abstract model computation graph defined in high-level languages ​​such as Python into machine instructions that the underlying hardware can understand. During this translation process, various complex optimization techniques are employed to optimize the model's computation, memory, and communication to achieve the highest running efficiency and lowest resource consumption.

[0039] In some embodiments, a compiler is used to compile the directed acyclic graph.

[0040] During the compilation process, operators are first fused based on the directed acyclic graph.

[0041] Specifically, multiple consecutive small operators (such as Conv + ReLU + Eltwise) in a directed acyclic graph are merged into a large operator to reduce kernel startup overhead and storage of intermediate results. The result of operator merging is denoted as operator fusion result.

[0042] In step S130, the operator nodes in the operator fusion result are sorted, and the subgraph is partitioned based on the sorting result to obtain the subgraph partitioning result.

[0043] Understandably, based on hardware characteristics, a processor has two processing units: a tensor processing unit and a vector processing unit, with each operator node belonging to a different unit depending on the type of data being processed.

[0044] Building upon this, when the computation graph contains multiple branches and the operator nodes are executed in different units, the operators are sorted based on the obtained operator fusion results. This specific sorting allows each processing unit to fuse more layers of operators.

[0045] Based on the sorting of operator nodes, the subgraph is segmented. Operator nodes belonging to different processing units in the computation graph are segmented in sequence to achieve heterogeneous acceleration. This allows a large task to be decomposed into multiple smaller tasks that can be executed simultaneously, thereby shortening the computation time or handling larger-scale problems.

[0046] In step S140, based on the subgraph partitioning results, the operator nodes that need to be repeatedly calculated are reused to obtain the graph optimization results.

[0047] Based on the obtained graph optimization nodes, when there are multiple nodes in the computation graph that need to be calculated repeatedly, the duplicate nodes are replaced and reused to avoid repeated calculations.

[0048] According to the example implementation, based on the compiler's traversal of the subgraph partitioning results, operator nodes that perform the same calculation operations and have the same input are identified. The calculation result of one of the nodes is retained, and other duplicate nodes share this result, thereby avoiding duplicate calculations.

[0049] In step S150, based on the graph optimization results, multi-input operator nodes are extracted, and on-chip operator fusion annotation is performed on the graph optimization results according to the inputs of the multi-input operator nodes.

[0050] Based on the graph optimization results, on-chip operator fusion is performed.

[0051] Understandably, on-chip fusion refers to the process of combining multiple layers (operators) in a neural network into a single logical unit or module.

[0052] It's important to clarify that devices typically have large amounts of RAM, such as 16GB in a computer, but their speed is relatively slow; this is called Global Memory. In contrast, dedicated NPU chips for neural network inference have fast caches, but their space may only be a few MB; this is called a Cache. Given the current situation, it's impossible to load all layers of the network model into the cache. This means that each operator needs to interact with the Cache and the external Global Memory to load input data and store results. Therefore, the network needs to be divided into small layer groups. Generally, by default, load / store operations with the external Global Memory are only required when entering and exiting a layer group. This loads the necessary data and stores the results. Layer groups reduce load / store operations, and they also serve as the basic unit for subsequent tiling, scheduling, and other operations, reducing problem complexity.

[0053] On-chip operator fusion aims to combine multiple discrete operator nodes into a unified composite operation performed on the high-speed cache inside the computing chip, efficiently managing on-chip memory and thus avoiding frequent round trips between slow global memory and high-speed on-chip cache for intermediate results.

[0054] In practice, based on the dependencies between operator nodes, a safe fusion boundary is determined, and fusionable operator nodes are then fused.

[0055] In step S160, in the graph optimization result after on-chip operator fusion annotation, operator node pairs that are in the same subgraph and have no data dependency are annotated side by side with instructions.

[0056] Instruction rearrangement is performed based on the graph optimization results after on-chip operator fusion annotation.

[0057] Understandably, instruction reordering aims to make the computing resources of processing units, especially pipelines and multi-issue architectures, work as "full" as possible by adjusting the execution order of instructions.

[0058] The significance of instruction reordering lies in the fact that, during the execution of an operator, if there is extra space in the buffer, the relevant parameters for the next tensor computation, such as weights and biases, can be loaded in advance. This allows the current computation to be carried out in parallel with the data transfer between the current layer and the next layer.

[0059] To achieve parallel data loading, based on the graph optimization results after on-chip operator fusion annotation, operator nodes corresponding to instructions without dependencies in the unified processing unit are extracted and annotated to enable parallel loading, denoted as instruction side-by-side annotation.

[0060] According to the example implementation, an instruction dependency graph is constructed using a compiler based on the graph optimization results after on-chip operator fusion annotation. This identifies data dependencies between instructions (such as read-after-write RAW, write-after-write WAW, and read-after-write WAR), and annotates operator nodes related to instructions that can be executed in parallel. During actual execution, instructions on the critical path are prioritized for scheduling, and instructions of the same type (such as multiple memory loads) are grouped together as much as possible to improve parallelism.

[0061] In step S170, the graph optimization result after the instructions are labeled side by side is mapped to hardware instructions according to the predefined unified operation primitives, thereby obtaining the optimized code of the target neural network model.

[0062] Based on predefined unified operation primitives, we find hardware declarations that match the computational patterns of the side-by-side labeled graph optimization results, and downgrade the computational patterns to the corresponding hardware instruction operations, thus obtaining the optimized code for the target neural network model.

[0063] It is understandable that dataflow architecture compiler architectures typically contain two core layers: higher-order IR (graph IR) and lower-order IR. To perform corresponding optimizations, in this embodiment, the compiler executes steps S120-140 for higher-order IR optimization and steps S150-S170 for lower-order IR optimization.

[0064] This application abstracts the target neural network model by constructing a directed acyclic graph (DAG), serving as the model's higher-order instruction recursion (IR). Optimization is then performed on this higher-order IR, including operator fusion, reordering, subgraph partitioning, and node reuse, resulting in a graph-optimized result, which serves as the model's lower-order IR. Further optimization is then performed on this lower-order IR, including on-chip operator fusion, instruction parallelization, and hardware instruction mapping, yielding the optimized code for the target neural network model. This application employs a layered approach to the target neural network model and performs corresponding compilation optimizations, balancing the ease of use of upper-layer applications with the ultimate performance of the underlying hardware. This fully unleashes the computational potential of the dataflow architecture while allowing users to program the model efficiently and with high hardware utilization without needing to delve into hardware details.

[0065] According to some embodiments, in step S110, a directed acyclic graph is generated based on the programming paradigm and the target neural network model, which can be specifically implemented through steps S111-S112.

[0066] In step S111, based on the programming paradigm, the computational operations of the target neural network model are identified to obtain operators, and the operators are used as operator nodes.

[0067] Based on the programming paradigm, determine whether the generated computation graph is static or dynamic.

[0068] This application uses the static diagram paradigm as an example for illustration.

[0069] Each layer or basic operation (such as convolution, matrix multiplication, and activation function) in the target neural network model is abstracted into an independent computational unit, i.e., an operator. This operator is treated as a node in the computation graph and denoted as an operator node, which encapsulates the specific computational logic.

[0070] In step S112, the input and output data between each layer of the target neural network model are represented as tensors, and tensor edges are added between operator nodes according to the tensors to obtain a directed acyclic graph.

[0071] The input and output data between operators (such as feature maps, weights, biases, etc.) are uniformly abstracted into tensors, which serve as the data carriers flowing within the computation graph. Based on the input-output relationships of the operators, directed edges connect the operator nodes, forming tensor edges in the computation graph. The direction of the edges represents the flow direction of the tensors, i.e., the data dependencies. This results in a line-based, non-transferable graph.

[0072] In this embodiment, the target neural network model is abstracted as a directed acyclic graph, and subsequent compilation is performed on this basis.

[0073] According to some embodiments, refer to Figure 2 In step S130, the operator nodes in the operator fusion result are sorted, and the subgraph is partitioned based on the sorting result to obtain the subgraph partitioning result. Specifically, this can be achieved through steps S131-S138.

[0074] In step S131, the operator nodes are classified to obtain the classification results.

[0075] The operator nodes are classified according to different processing units to obtain the classification results.

[0076] Furthermore, for ease of processing, different coloring processes are applied to the operators of different processing units in the classification results, such as... Figure 3 As shown.

[0077] In step S132, based on the classification results, the linearly connected operator nodes are grouped to obtain multiple operator node groups.

[0078] Grouping identical operator processing units into an operator group yields multiple operator node groups.

[0079] In step S133, the operator nodes are sorted according to the depth-first traversal strategy to obtain the depth sorting result.

[0080] For each operator node, sort it according to depth-first traversal.

[0081] Understandably, depth-first traversal will explore the computation graph along the dependency paths in the operator fusion result, thereby identifying operator chains with linear dependencies, which are denoted as depth-sorting results.

[0082] According to the example embodiment, for example Figure 3 The operator nodes 1 to 9 shown are sorted by depth-first traversal, resulting in the following order: 1->2, 3->4->5->6->7, 8->9, which is the depth sorting result.

[0083] In step S134, the dependencies of the operator nodes are obtained based on the depth sorting results.

[0084] In step S135, the current operator nodes are sorted according to the breadth-first traversal strategy to obtain the breadth-first sorting result.

[0085] For each operator node, sort it by breadth-first traversal to obtain the breadth-sorted result.

[0086] Understandably, breadth-first traversal helps identify operators at the same "level" in the computation graph that can be executed in parallel or have dependencies.

[0087] In step S136, based on the breadth-ordering result, reference operator nodes and operator nodes to be inserted are extracted from operator nodes with the same breadth. The operator nodes to be inserted and the operator nodes that have dependencies on them are inserted into the corresponding positions of the target operator nodes that meet the preset conditions in the depth-ordering result. Subgraph splitting marks are then inserted. The preset conditions include: the operator nodes to be inserted and the target operator nodes are located in the same operator node group, and the reference operator nodes depend on the target operator nodes.

[0088] Based on the breadth-ordering results, operators with the same breadth and operators with dependencies are inserted after operators of the same type with dependencies, and subgraph splitting markers are inserted at the same time.

[0089] According to the example implementation, two operator nodes with the same breadth are extracted, denoted as the reference operator node and the operator node to be inserted. In actual operation, the operator node to be inserted and the operator nodes that have dependencies on it are inserted after the operator node (denoted as the target operator node) that belongs to the same operator node group as the operator node to be inserted and is depended on by the reference operator node.

[0090] In step S137, steps S135-S136 are repeated until all operator nodes with the same breadth are traversed to obtain the sorting result of the operator nodes.

[0091] Re-sort the breadth, traverse the operator nodes after the insertion operation to find operators with the same breadth, and repeat the insertion operation if an operator with the same breadth exists.

[0092] According to the example embodiment, such as Figure 3 As shown, the sorting result is as follows: 1->2->3->6->4->7->8->5->9.

[0093] In step S138, based on the sorting result, the directed acyclic graph is divided according to the subgraph partitioning marker to obtain multiple subgraphs, which are used as the subgraph partitioning result.

[0094] Subgraph partitioning is performed based on the inserted subgraph partitioning markers to obtain the subgraph partitioning results.

[0095] According to the example embodiment, such as Figure 3 As shown, the subgraph partitioning results are as follows: group 0: 1; group 1: 2, 3, 6; group 2: 4, 7, 8; group 3: 5, 9.

[0096] This application embodiment obtains dependencies through depth sorting, rearranges operators through breadth sorting, and partitions subgraphs based on operator rearrangement to achieve heterogeneous acceleration.

[0097] According to some embodiments, refer to Figure 4 In step S150, based on the graph optimization results, multi-input operator nodes are extracted, and on-chip operator fusion annotation is performed on the graph optimization results according to the input of the multi-input operator nodes. Specifically, this can be achieved through steps S151-S155.

[0098] In step S151, multi-input operator nodes are extracted from the graph optimization results as the current operator nodes.

[0099] Since tensor processing units using layer groups involve redundant computations of operators such as Conv (convolution) and Pool (pooling), the efficiency gains are limited. Therefore, this application embodiment only considers on-chip fusion of vector computation units.

[0100] Extract the multi-input operator node and record it as the current operator node.

[0101] In step S152, the input branch of the current operator node is searched in reverse, and the operator nodes in the input branch are saved to the operator list of the current operator node.

[0102] Based on the current operator node, search for the input branch in the reverse direction of the data flow, collect all the predecessor nodes of the current node, and save them to the operator list of the current operator node.

[0103] In this embodiment, the implicit data flow dependencies of operator nodes (connected by tensor edges) are transformed into node attributes in the form of an operator list, which facilitates subsequent traversal and decision-making.

[0104] In step S153, steps S151-S152 are repeated until the operator nodes in the current subgraph are traversed.

[0105] In step S154, the operator list is traversed and cut in reverse order according to the maximum block cutting strategy to obtain multiple fusion blocks.

[0106] The list of operators is traversed in reverse order and cut to obtain multiple fusion blocks.

[0107] It is understandable that, due to the limited capacity of on-chip memory (such as shared memory of GPU), it is impossible to merge all operators indefinitely. The total amount of intermediate data must not exceed this limit. Therefore, this application performs reverse traversal of the maximum number of blocks of multi-input operator nodes in the operator list.

[0108] According to the example embodiment, the group blocking module is used for slicing.

[0109] According to the example embodiment, operator 3 obtains a maximum block cutting method based on the input of operator 4 / 7, and cuts operator 1 / 2 in reverse order.

[0110] In step S155, the information of the fusion block is written back to the operator node of the graph optimization result as an on-chip operator fusion annotation.

[0111] The information of the multiple merged blocks after slicing is written back to the corresponding operator, and is recorded as the on-chip operator fusion annotation of the corresponding operator.

[0112] According to some embodiments, refer to Figure 5 In step S151, multi-input operator nodes are extracted from the graph optimization results as the current operator nodes, which can be implemented through steps S510-S530.

[0113] In step S510, the input operator node of the graph optimization result is selected as the starting node, and the branch based on the starting node is found as the first branch.

[0114] Using the input node of the subgraph resulting from the graph optimization as the starting node, find all branches.

[0115] In step S520, based on the first branch, the child nodes of the starting node are obtained as the target node.

[0116] Get the user (child node) of the starting node and use it as the target node.

[0117] In step S530, if the target node is a multi-input operator node, the target node is used as the current operator node.

[0118] Determine if the target node is a multi-input operator node. If so, use it as the current operator node and continue with subsequent steps.

[0119] If not, obtain the child nodes of the target node, and repeat the judgment in step S530 until all child nodes are traversed.

[0120] In this embodiment, the search starts from the subgraph input and traverses all nodes to ensure that all nodes that depend on this input can be accessed, thus avoiding omissions.

[0121] According to some embodiments, refer to Figure 6 In step S170, the graph optimization result after the instructions are labeled side by side is mapped to hardware instructions according to the predefined unified operation primitives, thereby obtaining the optimization code of the target neural network model. This can be achieved through steps S610-S620.

[0122] In step S610, based on the unified operation primitives, specific subgraph patterns and / or operator node sequences are identified from the graph optimization results after the instructions are labeled side by side, resulting in multiple operation primitive sequences.

[0123] Unified operation primitives predefine specific subgraph patterns and / or operator node sequences corresponding to operation primitive sequences. Based on unified operation primitives, the graph optimization results after side-by-side instruction annotation are identified, resulting in multiple operation primitive sequences.

[0124] In step S620, multiple operation primitive sequences are converted into corresponding hardware instructions, thereby obtaining the optimized code of the target neural network model.

[0125] Based on unified operation primitives, the computation modes corresponding to multiple operation primitive sequences are downgraded to corresponding hardware instruction operations.

[0126] The embodiments of this application use predefined unified operation primitives to map hardware instructions, bridging the semantic gap between high-level algorithm descriptions and low-level hardware instructions. This automates and intelligently generates hardware code, achieving near-limit inference performance on diverse hardware without requiring extensive manual intervention.

[0127] According to some embodiments, unified operation primitives include: memory management operation primitives, computation operation primitives, scheduler operation primitives, and / or layout transformation operation primitives.

[0128] Memory management is implemented based on memory management primitives. Memory management includes: 1) Memory type abstraction: Expose different types of memory (such as buffer, global memory) to the upper layer in a unified manner.

[0129] 2) Memory reuse: Analyze the operator lifecycle and allow operators with non-overlapping lifecycles to share the same physical memory, greatly reducing peak memory usage.

[0130] Based on scheduler-class operation primitives, the scheduler can automatically analyze the data dependencies between operations to ensure the correctness of the execution order. It also enables parallel computation by different computing units, accelerating inference.

[0131] The instruction mapper maps computation operation primitives. Specifically, it defines a set of unified primitives, finds hardware declarations that match the computation mode, and downgrades the computation mode to the corresponding hardware instruction operation.

[0132] Layout transformation is performed based on layout transformation primitives to achieve data reading format conversion.

[0133] According to the example implementation, the unified operation primitives are shown in Table 1.

[0134] Table 1. Unified Operation Primitives

[0135] According to some embodiments, refer to Figure 7 In step S120, based on the directed acyclic graph, continuous operator nodes that conform to the fusion rules are identified, and the continuous operator nodes are fused into new operator nodes to obtain the operator fusion result. This can be achieved through steps S710-S720.

[0136] In step S710, a seed operator node is selected from the operators in the directed acyclic graph.

[0137] From a directed acyclic graph, the seed operator node can be selected arbitrarily.

[0138] In step S720, a greedy algorithm is used to merge consecutive operator nodes that satisfy the fusion rules based on the seed operator node, and the consecutive operator nodes are merged into new operator nodes to obtain the operator fusion result.

[0139] Operator nodes are fused using computation graph-based data flow and pre-defined fusion rules.

[0140] This application uses downward fusion as the fusion rule for its embodiments.

[0141] According to the example implementation, a greedy algorithm is used to fuse as many operators downwards as possible.

[0142] The merged continuous operator nodes are updated with new operator nodes, and the merged directed acyclic graph is denoted as the operator fusion result.

[0143] The data flow of the computation graph includes hardware-supported datapath data flow, such as... Figure 8 As shown.

[0144] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0145] Figure 9 A block diagram of a processing apparatus according to an exemplary embodiment is shown.

[0146] Figure 9 The apparatus shown can perform the processing method of the model according to the embodiments of this application.

[0147] like Figure 9 As shown, the model processing device may include: See Figure 9 Referring to the preceding description, the computation graph unit 910 is used to generate a directed acyclic graph (DAG) based on the programming paradigm and the target neural network model, wherein the DAG includes operator nodes and tensor edges.

[0148] The operator fusion unit 920 is used to identify continuous operator nodes that conform to the fusion rules based on the directed acyclic graph, and to fuse the continuous operator nodes into new operator nodes to obtain the operator fusion result.

[0149] The sorting and partitioning unit 930 is used to sort the operator nodes in the operator fusion result and perform subgraph partitioning based on the sorting result to obtain the subgraph partitioning result.

[0150] The node reuse unit 940 is used to reuse operator nodes that need to be repeatedly calculated based on the subgraph partitioning results, so as to obtain the graph optimization results.

[0151] The on-chip fusion unit 950 is used to extract multi-input operator nodes based on graph optimization results, and to perform on-chip operator fusion annotation on the graph optimization results according to the input of the multi-input operator nodes.

[0152] The instruction rearrangement unit 960 is used to perform instruction side-by-side annotation on operator node pairs that are in the same subgraph and have no data dependency in the graph optimization result after on-chip operator fusion annotation.

[0153] The instruction mapping unit 970 is used to map the graph optimization result after the instructions are labeled side by side into hardware instructions according to the predefined unified operation primitives, so as to obtain the optimized code of the target neural network model.

[0154] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0155] This application discloses an electronic device, including: a processor; and a memory storing a computer program, which, when executed by the processor, causes the processor to execute the above-described instruction generation method.

[0156] For example, refer to Figure 10 , Figure 10 The illustrated electronic device 1000 includes a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, for example, via a bus 1002. Optionally, the electronic device 1000 may also include a transceiver 1004. It should be noted that in practical applications, the transceiver 1004 is not limited to one type, and the structure of this electronic device 1000 does not constitute a limitation on the embodiments of this application.

[0157] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in this application. Processor 1001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0158] Bus 1002 may include a pathway for transmitting information between the aforementioned components. Bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0159] The memory 1003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other storage medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0160] The memory 1003 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 1001. The processor 1001 is used to execute the application code stored in the memory 1003 to implement the content shown in the foregoing method embodiments.

[0161] Figure 10 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0162] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to execute an instruction generation method.

[0163] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0164] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing a model, characterized in that, include: Based on the programming paradigm, a directed acyclic graph is generated according to the target neural network model, wherein the directed acyclic graph includes operator nodes and tensor edges; Based on the directed acyclic graph, continuous operator nodes that conform to the fusion rules are identified, and the continuous operator nodes are fused into new operator nodes to obtain the operator fusion result; The operator nodes in the operator fusion result are sorted, and the subgraph is partitioned based on the sorting result to obtain the subgraph partitioning result; Based on the subgraph partitioning results, the operator nodes that need to be repeatedly calculated are reused to obtain the graph optimization results; Based on the graph optimization results, multi-input operator nodes are extracted, and on-chip operator fusion annotation is performed on the graph optimization results according to the inputs of the multi-input operator nodes. In the graph optimization results after on-chip operator fusion annotation, operator node pairs that are in the same subgraph and have no data dependency are annotated side by side with instructions. Based on predefined unified operation primitives, the graph optimization results after the instructions are labeled side by side are mapped to hardware instructions, thereby obtaining the optimized code of the target neural network model.

2. The method according to claim 1, characterized in that, Based on the programming paradigm, a directed acyclic graph is generated according to the target neural network model, including: Based on the programming paradigm, the computational operations of the target neural network model are identified, operators are obtained, and the operators are used as operator nodes; The input and output data between the layers of the target neural network model are represented as tensors, and tensor edges are added between the operator nodes according to the tensors to obtain the directed acyclic graph.

3. The method according to claim 1, characterized in that, The operator nodes in the operator fusion result are sorted, and a subgraph is partitioned based on the sorting result to obtain the subgraph partitioning result, including: S131: Classify the operator nodes to obtain classification results; S132: Based on the classification results, the linearly connected operator nodes are grouped to obtain multiple operator node groups; S133: Sort the operator nodes according to the depth-first traversal strategy to obtain the depth sorting result; S134: Based on the depth sorting result, obtain the dependency relationship of the operator node; S135: Sort the current operator nodes according to the breadth-first traversal strategy to obtain the breadth-first sorting result; S136: Based on the breadth-ordering result, extract reference operator nodes and operator nodes to be inserted from operator nodes with the same breadth, and insert the operator node to be inserted and the operator nodes that have a dependency relationship with it into the corresponding positions of the target operator node in the depth-ordering result that meet the preset conditions, and insert subgraph segmentation markers, wherein the preset conditions include: the operator node to be inserted and the target operator node are located in the same operator node group, and the reference operator node depends on the target operator node; S137: Repeat steps S135-S136 until all operator nodes with the same breadth are traversed to obtain the sorting result of the operator nodes; S138: Based on the sorting result, the directed acyclic graph is divided according to the subgraph partitioning marker to obtain multiple subgraphs, which are used as the subgraph partitioning result.

4. The method according to claim 1, characterized in that, Based on the graph optimization results, multi-input operator nodes are extracted, and on-chip operator fusion annotation is performed on the graph optimization results according to the inputs of the multi-input operator nodes, including: S151: Extract multi-input operator nodes from the graph optimization results as the current operator nodes; S152: Reverse the search for the input branch of the current operator node, and save the operator nodes in the input branch to the operator list of the current operator node; S153: Repeat steps S151-S152 until the operator nodes in the current subgraph are traversed; S154: According to the maximum block cutting strategy, the operator list is traversed and cut in reverse order to obtain multiple fusion blocks; S155: Write the information of the fusion block back to the operator node of the graph optimization result as the on-chip operator fusion annotation.

5. The method according to claim 4, characterized in that, Step S151 includes: Select the input operator node of the graph optimization result as the starting node, and find the branch based on the starting node as the first branch; Based on the first branch, obtain the child nodes of the starting node as the target node; If the target node is a multi-input operator node, then the target node is used as the current operator node.

6. The method according to claim 1, characterized in that, Based on predefined unified operation primitives, the graph optimization results after instruction side-by-side annotation are mapped to hardware instructions, thereby obtaining the optimized code of the target neural network model, including: Based on the unified operation primitives, specific subgraph patterns and / or operator node sequences are identified from the graph optimization results after the instructions are labeled side by side, resulting in multiple operation primitive sequences; The multiple sequences of operational primitives are converted into corresponding hardware instructions, thereby obtaining the optimized code for the target neural network model.

7. The method according to claim 1 or 6, characterized in that, The unified operation primitives include: memory management operation primitives, computation operation primitives, scheduler operation primitives, and / or layout transformation operation primitives.

8. The method according to claim 1, characterized in that, Based on the directed acyclic graph, continuous operator nodes that conform to the fusion rules are identified, and these continuous operator nodes are fused into new operator nodes to obtain the operator fusion result, including: Select a seed operator node from the operators in the directed acyclic graph; Using a greedy algorithm, based on the seed operator node, consecutive operator nodes that satisfy the fusion rule are fused downwards, and the consecutive operator nodes are fused into a new operator node to obtain the operator fusion result.

9. A model processing device, characterized in that, include: The computation graph unit is used to generate a directed acyclic graph based on a programming paradigm and a target neural network model, wherein the directed acyclic graph includes operator nodes and tensor edges; The operator fusion unit is used to identify continuous operator nodes that conform to the fusion rules based on the directed acyclic graph, and to fuse the continuous operator nodes into new operator nodes to obtain the operator fusion result; The sorting and partitioning unit is used to sort the operator nodes in the operator fusion result and perform subgraph partitioning based on the sorting result to obtain the subgraph partitioning result; A node reuse unit is used to reuse operator nodes that need to be repeatedly calculated based on the subgraph partitioning result to obtain the graph optimization result. An on-chip fusion unit is used to extract multi-input operator nodes based on the graph optimization results, and to perform on-chip operator fusion annotation on the graph optimization results according to the inputs of the multi-input operator nodes; The instruction rearrangement unit is used to perform instruction side-by-side annotation on operator node pairs that are in the same subgraph and have no data dependency in the graph optimization result after on-chip operator fusion annotation; The instruction mapping unit is used to map the graph optimization result after the instructions are labeled side by side into hardware instructions according to the predefined unified operation primitives, so as to obtain the optimized code of the target neural network model.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-8.

Images