An automatic computation graph segmentation system for inference scenarios and its implementation method

Abstract

This invention relates to the field of embedded intelligent computing and discloses an automatic computation graph segmentation system and its implementation method for inference scenarios. Based on the NPU hardware support list and operator fusion fixed mode, an automatic segmentation model based on layer-attention and cross-attention mechanisms is adopted to automatically segment a whole computation graph into several subgraphs and output several different segmentation schemes. Based on the temporal arrangement rules of all subgraph vectors and topology corresponding to each segmentation scheme, the inference latency of each different segmentation scheme is predicted, which can formally describe the segmentation scheme and the corresponding predicted inference latency. By selecting the segmentation scheme with the largest inference latency and satisfying the typical model fusion rules, automatic computation graph segmentation is achieved, and the two indicators of inference latency and data access overhead are balanced.

Description

Technical Field

[0001] This invention relates to the field of embedded intelligent computing, and discloses an automatic computation graph segmentation system and its implementation method for inference scenarios. Background Technology

[0002] Embedded deep learning inference computation runs on embedded devices, providing a fundamental methodology for intelligent application computing. With the influx of NPU chips into the AI ​​hardware ecosystem, data structures centered on tensor data computation operators and computation graphs provide essential support for accelerated NPU computation. Improving NPU inference latency can be considered not only in the initial hardware design but also through software optimization. Whoever can better integrate hardware with computation graph inference latency will gain a competitive edge in the diverse development of NPUs.

[0003] When inferring classic deep learning models, the NPU structures the model file into a computation graph and then splits the graph. This splitting aims to reduce the additional memory access overhead caused by calling hardware implementations for each operator computation. A good computation graph splitting method achieves the dual goals of maximizing inference latency and minimizing data memory access overhead. However, good splitting methods are often carefully designed manually, which is time-consuming and labor-intensive, and due to the limited search space, the results may get stuck in local optima.

[0004] Current methods for automatic computation graph segmentation in inference scenarios do not adequately integrate the hardware characteristics of the NPU processor during the segmentation process, failing to achieve good hardware-adaptive segmentation. Furthermore, they cannot effectively balance inference latency and data access overhead when segmenting subgraphs. Summary of the Invention

[0005] The purpose of this invention is to provide an automatic computation graph segmentation system and its implementation method for inference scenarios, which can achieve automatic segmentation of the computation graph and balance the two indicators of inference latency and data access overhead.

[0006] To achieve the above-mentioned technical effects, the technical solution adopted by the present invention is as follows:

[0007] An automatic computation graph segmentation method for inference scenarios includes:

[0008] The file to be reasoned is formatted, and the formatted file is structured according to the rules of directed acyclic graphs to obtain the computation graph topology of the file to be reasoned.

[0009] The initial vectors of nodes and the initial vectors of the computation graph topology are obtained based on the graph attention neural network model.

[0010] Based on the NPU hardware support list and operator fusion fixed mode, an automatic splitting model based on hierarchical attention mechanism and cross attention mechanism is adopted to split the node initial vector and the computation graph initial vector into several different splitting schemes, and each splitting scheme contains several subgraphs.

[0011] Based on the time arrangement rules of all subgraph vectors and topology corresponding to each partitioning scheme, predict the inference latency of each different partitioning scheme;

[0012] The segmentation scheme that has the largest inference delay and satisfies the typical model fusion rules is selected. The typical model fusion rules include the convolution normalization operator fusion rule and the convolution activation fusion rule.

[0013] Furthermore, methods for splitting the node initialization vectors and computation graph initialization vectors into several different partitioning schemes, with each partitioning scheme containing several subgraphs, include:

[0014] Using node initial vectors and computation graph initial vectors as input, an automatic splitting model employing hierarchical attention and cross-attention mechanisms is used to obtain node vectors and computation graph vectors.

[0015] Based on the topology of the computation graph, computation graph vectors, and node vectors, subgraphs that are isomorphic and have a frequency of occurrence of not less than a preset threshold are segmented according to the principle of maximum matching of the same structure. If a situation is encountered that is not supported by the hardware, the subgraph is split directly and no further matching is performed.

[0016] Then, starting with the node with the highest weight in the attention mechanism, it radiates outwards and traverses until the set weight threshold is reached. After reaching this threshold, it stops traversing and splits the graph, turning the remaining single operator into a subgraph.

[0017] Furthermore, methods for obtaining node vectors and computation graph vectors using automatic splitting models employing hierarchical attention and cross-attention mechanisms, with node initial vectors and computation graph initial vectors as input, include:

[0018] To obtain the computation graph topology, select the initial vector of the two-hop topology node and perform cross attention with the NPU hardware support list to obtain a subgraph with the same structure.

[0019] Obtain the out-degree and in-degree vectors of each node in the computation graph topology, and perform cross-attention with the NPU hardware support list to obtain the node with the highest weight;

[0020] The cross-attention mechanism is used to match the computation graph vector with the operator fusion fixed pattern to obtain the node vector and the computation graph vector.

[0021] Furthermore, based on the temporal arrangement rules of all subgraph vectors and topology corresponding to each partitioning scheme, methods for predicting the inference latency of each different partitioning scheme include:

[0022] Using the single-point latency of each operator in the NPU hardware support list as prior knowledge, and taking the computation graph vector and the vector corresponding to each partitioning scheme as input, the latency of each subgraph of each partitioning scheme is predicted in the predictor. The latency of each subgraph is weighted and summed to obtain the final latency prediction. The computation graph vector, the vector corresponding to each subgraph, and the topological temporal arrangement information of each subgraph are weighted, summed, and normalized to obtain the confidence of each partitioning scheme. If the confidence is greater than or equal to the preset confidence lower limit threshold, the predicted inference latency meets the requirements.

[0023] Furthermore, the selected segmentation scheme is fed back to the automatic segmentation model, and the training is optimized to obtain an automatic segmentation model based on hierarchical attention mechanism and cross-attention mechanism.

[0024] To achieve the above technical effects, the present invention also provides an automatic computation graph segmentation system for inference scenarios, comprising:

[0025] File processing module: Formats the file to be reasoned, and performs structuring processing on the formatted file according to the rules of directed acyclic graphs to obtain the computation graph topology of the file to be reasoned.

[0026] Initial Vector Acquisition Module: Acquires the initial vectors of nodes in the computation graph topology and the initial vectors of the computation graph based on the graph attention neural network model;

[0027] Subgraph segmentation module: Based on the NPU hardware support list and operator fusion fixed mode, an automatic splitting model based on hierarchical attention mechanism and cross attention mechanism is adopted to split the node initial vector and computation graph initial vector into several different splitting schemes, and each splitting scheme contains several subgraphs;

[0028] Prediction module: Based on the temporal arrangement rules of all subgraph vectors and topology corresponding to each partitioning scheme, predict the inference latency of each different partitioning scheme;

[0029] Output module: Selects the segmentation scheme with the largest inference latency that satisfies the typical model fusion rules, which include convolution normalization operator fusion rules and convolution activation fusion rules.

[0030] Furthermore, the subgraph segmentation module obtains the initial vectors of two-hop topology nodes by acquiring the computation graph topology structure and performing cross-attention with the NPU hardware support list to obtain subgraphs with the same structure; it obtains the out-degree and in-degree vectors of each node in the computation graph topology structure and performs cross-attention with the NPU hardware support list to obtain the node with the highest weight; it uses the cross-attention mechanism to match the computation graph vectors with the operator fusion fixed pattern to obtain node vectors and computation graph vectors; based on the computation graph topology structure, computation graph vectors, and node vectors, it segments the isomorphic subgraphs with a frequency not less than a preset threshold, based on the principle of maximum matching of the same structure; if a hardware-unsupported situation is encountered, it is directly split and no further matching is performed; then, starting from the node with the highest weight in the attention mechanism, it radiates outwards and traverses until the set weight threshold is reached, then stops traversing and segments, and the remaining single operator becomes a subgraph.

[0031] Furthermore, the prediction module uses the single-point latency of each operator in the NPU hardware support list under the NPU as prior knowledge, and takes the computation graph vector and the vector corresponding to each partitioning scheme as input to predict the latency of each subgraph of each partitioning scheme in the predictor; after weighted summation of the latency of each subgraph, the final latency prediction is obtained; after weighted summation and normalization of the computation graph vector, the vector corresponding to each subgraph, and the topological time arrangement information of each subgraph, the confidence of each partitioning scheme is obtained. If the confidence is greater than or equal to the preset confidence lower limit threshold, the predicted inference latency meets the requirements.

[0032] Compared with existing technologies, the beneficial effects of this invention are as follows: Based on the NPU hardware support list and operator fusion fixed mode, this invention adopts an automatic splitting model based on layer-attention and cross-attention mechanisms to automatically split a whole computation graph into several subgraphs and output several different splitting schemes. Based on the temporal arrangement rules of all subgraph vectors and topology corresponding to each splitting scheme, the inference latency of each different splitting scheme is predicted, which can formally describe the splitting scheme and the corresponding predicted inference latency. By selecting the splitting scheme with the largest inference latency and satisfying the typical model fusion rules, automatic computation graph splitting is achieved. During the training process of the automatic computation graph splitting model, a loss function balancing inference latency and data access overhead is designed, and the weights of these two indicators are learned to achieve a balance between inference latency and data access overhead, thereby facilitating the further selection of the scheme with the fastest overall inference process of the computation graph. Attached Figure Description

[0033] Figure 1 This is a flowchart of the automatic computation graph segmentation method for reasoning scenarios in Example 1 or 2.

[0034] Figure 2 This is a block diagram of the automatic computation graph segmentation system for reasoning scenarios in Example 1;

[0035] Figure 3 This is a schematic diagram of the computational graph decomposition model in Example 2;

[0036] Figure 4 This is a schematic diagram of the automatic splitting model based on hierarchical attention mechanism and cross attention mechanism in Example 2;

[0037] The module consists of: 1. File processing module; 2. Initial vector acquisition module; 3. Subgraph segmentation module; 4. Prediction module; and 5. Output module. Detailed Implementation

[0038] The present invention will now be described in further detail with reference to the embodiments and accompanying drawings. However, this should not be construed as limiting the scope of the above-described subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0039] Example 1

[0040] See Figure 1 and Figure 2 An automatic computation graph segmentation method for inference scenarios includes:

[0041] The file to be reasoned is formatted, and the formatted file is structured according to the rules of directed acyclic graphs to obtain the computation graph topology of the file to be reasoned.

[0042] The initial vectors of nodes and the initial vectors of the computation graph topology are obtained based on the graph attention neural network model.

[0043] Based on the NPU hardware support list and operator fusion fixed mode, an automatic splitting model based on layer-attention and cross-attention is adopted to split the node initial vector and the computation graph initial vector into several different splitting schemes, and each splitting scheme contains several subgraphs.

[0044] Based on the time arrangement rules of all subgraph vectors and topology corresponding to each partitioning scheme, predict the inference latency of each different partitioning scheme;

[0045] The segmentation scheme that has the largest inference delay and satisfies the typical model fusion rules is selected. The typical model fusion rules include the convolution normalization operator fusion rule and the convolution activation fusion rule.

[0046] In this embodiment, based on the NPU hardware support list and operator fusion fixed mode, an automatic splitting model based on layer-attention and cross-attention mechanisms is adopted to automatically split a whole computation graph into several subgraphs and output several different splitting schemes. Based on the temporal arrangement rules of all subgraph vectors and topology corresponding to each splitting scheme, the inference latency of each different splitting scheme is predicted, which can formally describe the splitting scheme and the corresponding predicted inference latency. By selecting the splitting scheme with the largest inference latency and satisfying the typical model fusion rules, the automatic splitting of the computation graph is achieved. In addition, during the training process of the automatic splitting model of the computation graph, a loss function balancing inference latency and data access overhead is designed, and the weights of these two indicators are learned to achieve the effect of balancing inference latency and data access overhead, thereby facilitating the further selection of the scheme with the fastest overall inference process of the computation graph.

[0047] Based on the same inventive concept, this embodiment also provides an automatic computation graph segmentation system for reasoning scenarios, including:

[0048] File processing module 1: Formats the file to be reasoned, and performs structuring on the formatted file according to the rules of directed acyclic graphs to obtain the computation graph topology of the file to be reasoned.

[0049] Initial Vector Acquisition Module 2: Obtains the initial vectors of nodes in the computation graph topology and the initial vectors of the computation graph based on the graph attention neural network model;

[0050] Subgraph Segmentation Module 3: Based on the NPU hardware support list and operator fusion fixed mode, an automatic splitting model based on layer-attention and cross-attention is used to split the node initial vector and computation graph initial vector into several different splitting schemes, and each splitting scheme contains several subgraphs;

[0051] Prediction Module 4: Based on the temporal arrangement rules of all subgraph vectors and topology corresponding to each partitioning scheme, predict the inference latency of each different partitioning scheme;

[0052] Output module 5: Selects the segmentation scheme with the largest inference delay that satisfies the typical model fusion rules, which include the convolution normalization operator fusion rule and the convolution activation fusion rule.

[0053] In this embodiment, the subgraph segmentation module 3 obtains the initial vector of the two-hop topology node by acquiring the computation graph topology structure and performing cross-attention with the NPU hardware support list to obtain a subgraph with the same structure; it obtains the out-degree vector and in-degree vector of each node in the computation graph topology structure and performs cross-attention with the NPU hardware support list to obtain the node with the highest weight; it uses the cross-attention mechanism to match the computation graph vector with the operator fusion fixed pattern to obtain the node vector and the computation graph vector; based on the topology structure of the computation graph, the computation graph vector, and the node vector, it segments the isomorphic subgraphs with a frequency of not less than a preset threshold, based on the principle of maximum matching of the same structure; if the hardware does not support the subgraph, it directly splits the subgraph and stops matching; then, starting from the node with the highest weight in the attention mechanism, it traverses outwards until the set weight threshold is reached, then stops traversing and segments the subgraph, and the remaining single operator becomes a subgraph.

[0054] In this embodiment, the prediction module 4 uses the single-point latency of each operator in the NPU hardware support list under the NPU as prior knowledge, and takes the computation graph vector and the vector corresponding to each partitioning scheme as input to predict the latency of each subgraph of each partitioning scheme in the predictor; after weighted summation of the latency of each subgraph, the final latency prediction is obtained; after weighted summation and normalization of the computation graph vector, the vector corresponding to each subgraph and the topological time arrangement information of each subgraph, the confidence of each partitioning scheme is obtained. If the confidence is greater than or equal to the preset confidence lower limit threshold, the predicted inference latency meets the requirements.

[0055] Example 2

[0056] See Figures 1-4 An automatic computation graph segmentation method for inference scenarios includes:

[0057] Step 1: Format the file to be reasoned, and perform structuring on the formatted file according to the rules of directed acyclic graphs to obtain the computation graph topology of the file to be reasoned.

[0058] In this embodiment, the model file M_file containing the inference data is obtained. This file needs to be in ONNX format; if other model formats exist, they also need to be converted to ONNX format. After the file is read in, it needs to be structured according to the rules of directed acyclic graphs to obtain the computation graph topology of the file to be inferred. There are many open-source structuring methods available, which will not be elaborated here.

[0059] After the model file is structured, it becomes a complete computation graph. The computation graph consists of computation nodes and edge relationships, and is a directed acyclic graph. The description format of the computation graph includes a list of nodes and a set of relationships between nodes. Each node is a structure containing information about the computation, such as its ID, predecessor, successor, input dimension, and output dimension. Edge relationships include source and destination nodes, as well as edge weights.

[0060] Node set: op_listi = {op1, op2, op3, ..., opi};

[0061] Edge relationship list: relationship_listj = [(op1,op2,w1),(op2,op3,w2),(op2,op4,w3)......(opa,opb,wl)]

[0062] The computation graph description format is: Graphu = {op_listi, relationship_listj}.

[0063] Step 2: Obtain the initial vectors of the nodes and the initial vectors of the computation graph topology based on the graph attention neural network model;

[0064] In this embodiment, the computation graph is first encoded using a graph attention neural network model to obtain an initial computation graph embedding; then, a new attention mechanism is introduced to learn the attention of different nodes considering the hardware-supported characteristics to obtain the node embedding.

[0065] Step 3: Based on the NPU hardware support list and operator fusion fixed mode, use an automatic splitting model based on layer-attention and cross-attention to split the node initial vector and computation graph initial vector into several different splitting schemes, and each splitting scheme contains several subgraphs.

[0066] In this embodiment, a schematic diagram of the automatic splitting model based on layer-attention and cross-attention mechanisms is shown below. Figure 3 As shown, layer-attention consists of three layers. The first two layers perform cross-attention with the hardware operator list, and the third layer performs cross-attention with the existing operator fusion template. The specific segmentation process is as follows:

[0067] (1) Cross-attention is performed between the two-hop topology encoding of the computation graph and the hardware operator list. That is, with each node as the center, the hardware operator list is considered to encode the edge relationship and node representation within the two hops. Its purpose is to find subgraphs with the same structure.

[0068] (2) The out-degree and in-degree encoding of each node in the computation graph are cross-attentioned with the list of hardware operators. That is, with each node as the center, the list of hardware operators is considered and its out-degree and in-degree are encoded. The purpose is to find the nodes with a large amount of computation.

[0069] (3) Cross-encoding between the computation graph and the existing pattern matching graph, that is, using the cross-attention mechanism to match the existing operator fusion pattern into the computation graph encoding;

[0070] After obtaining the embeddings of the computation graph based on layer-attention and cross-attention mechanisms, the entire graph is segmented according to three pieces of information: the original computation graph's topology, the computation graph's embeddings, and the node embeddings, following the following segmentation rules:

[0071] A. Frequent subgraph mining: This method uses the principle of maximum matching of identical structures to segment isomorphic subgraphs that occur at a frequency not less than a preset threshold. If the hardware does not support the segmentation, the subgraph is split directly without further matching. In this embodiment, the preset threshold is set to 3, meaning that subgraphs that occur 3 times or more are segmented.

[0072] B. After the first step of segmentation is completed, find the graph with high computational cost and split it. That is, start with the node with high weight in the attention mechanism and traverse outwards. Stop traversing after reaching a certain set weight threshold.

[0073] C. After completing the first and second steps, match the commonly used fixed combinations based on the operator patterns summarized by predecessors.

[0074] D. After completing the first three steps, transform the remaining operators into a subgraph.

[0075] The description format of the subgraph is: Sub-Graph m ={op_list m relationship_list m};

[0076] The partitioning scheme is described in the following format: Partition n=[Sub-Graph1,Sub-Graph2,{Sub-Graph3,Sub-Graph4}......Sub-Graph m ];

[0077] The output description format of the automatic partitioning algorithm for the computational graph is: Result = {Partition1, Partition2, ..., Partition...} n}

[0078] Step 4: Based on the time arrangement rules of all subgraph vectors and topology corresponding to each partitioning scheme, predict the inference latency of each different partitioning scheme;

[0079] Using the single-point latency of each operator in the NPU hardware support list as prior knowledge, and taking the computation graph vector and the vector corresponding to each partitioning scheme as input, the latency of each subgraph of each partitioning scheme is predicted in the predictor. The latency of each subgraph is weighted and summed to obtain the final latency prediction. The computation graph vector, the vector corresponding to each subgraph, and the topological temporal arrangement information of each subgraph are weighted, summed, and normalized to obtain the confidence of each partitioning scheme. If the confidence is greater than or equal to the preset confidence lower limit threshold, the predicted inference latency meets the requirements.

[0080] Step 5: Select the segmentation scheme with the largest inference delay that satisfies the typical model fusion rules, which include the convolution normalization operator fusion rule and the convolution activation fusion rule.

[0081] Step 6: Feed the selected segmentation scheme back to the automatic splitting model, optimize the training to obtain an automatic splitting model based on layer-attention and cross-attention mechanisms, which can actively optimize the automatic splitting algorithm of the computation graph and form an optimization closed loop.

[0082] In the process of optimizing the training model for automatic computation graph splitting, it is necessary to design a loss function for computation graph splitting to guide the automatic subgraph splitting model to generate a candidate set of subgraphs that simultaneously considers performance latency and data access.

[0083] The computational graph splitting loss function is a weighted sum of performance latency and the number of data copies. Its formula is as follows:

[0084] Loss all =αLoss predict +βLoss number

[0085] Loss numberThe number of subgraphs after splitting, i.e., the default number of data copies is approximately equal to the number of subgraphs; Loss predict The prediction inference delay is applied to all generated subgraphs. α and β need to be learned, and the loss... all The overall loss function for calculating graph segmentation.

[0086] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automatically segmenting computational graphs for inference scenarios, characterized in that, include: The file to be reasoned is formatted, and the formatted file is structured according to the rules of directed acyclic graphs to obtain the computation graph topology of the file to be reasoned. The initial vectors of nodes and the initial vectors of the computational graph topology are obtained based on the graph attention neural network model. Based on the NPU hardware support list and operator fusion fixed mode, an automatic splitting model based on hierarchical attention mechanism and cross attention mechanism is adopted to split the node initial vector and the computation graph initial vector into several different splitting schemes, and each splitting scheme contains several subgraphs. Based on the time arrangement rules of all subgraph vectors and topology corresponding to each partitioning scheme, predict the inference latency of each different partitioning scheme; The segmentation scheme with the largest inference delay and that satisfies the typical model fusion rules is selected. The typical model fusion rules include the convolution normalization operator fusion rule and the convolution activation fusion rule. in: Methods for splitting node initialization vectors and computation graph initialization vectors into several different partitioning schemes, with each partitioning scheme containing several subgraphs, include: Using node initial vectors and computation graph initial vectors as input, an automatic splitting model employing hierarchical attention and cross-attention mechanisms is used to obtain node vectors and computation graph vectors. Based on the topology of the computation graph, computation graph vectors, and node vectors, subgraphs that are isomorphic and have a frequency of occurrence of not less than a preset threshold are segmented according to the principle of maximum matching of the same structure. If a situation is encountered that is not supported by the hardware, the subgraph is split directly and no further matching is performed. Then, starting with the node with the highest weight in the attention mechanism, it radiates outwards and traverses until the set weight threshold is reached. After reaching this threshold, it stops traversing and splits the graph, turning the remaining single operator into a subgraph. Methods for automatically splitting nodes and computation graphs using hierarchical attention and cross-attention mechanisms as input include: To obtain the computation graph topology, select the initial vector of the two-hop topology node and perform cross attention with the NPU hardware support list to obtain a subgraph with the same structure. Obtain the out-degree and in-degree vectors of each node in the computation graph topology, and perform cross-attention with the NPU hardware support list to obtain the node with the highest weight; The cross-attention mechanism is used to match the computation graph vector with the operator fusion fixed pattern to obtain the node vector and the computation graph vector.

2. The automatic computation graph segmentation method for inference scenarios according to claim 1, characterized in that, Based on the temporal arrangement rules of all subgraph vectors and topology corresponding to each partitioning scheme, methods for predicting the inference latency of each different partitioning scheme include: Using the single-point latency of each operator in the NPU hardware support list as prior knowledge, and taking the computation graph vector and the vector corresponding to each partitioning scheme as input, the latency of each subgraph of each partitioning scheme is predicted in the predictor. The latency of each subgraph is weighted and summed to obtain the final latency prediction. The computation graph vector, the vector corresponding to each subgraph, and the topological temporal arrangement information of each subgraph are weighted, summed, and normalized to obtain the confidence of each partitioning scheme. If the confidence is greater than or equal to the preset confidence lower limit threshold, the predicted inference latency meets the requirements.

3. The automatic computation graph segmentation method for inference scenarios according to claim 1, characterized in that, The selected segmentation scheme is fed back to the automatic splitting model, and the training is optimized to obtain an automatic splitting model based on hierarchical attention mechanism and cross attention mechanism.

4. An automatic computation graph segmentation system for reasoning scenarios, characterized in that, include: File processing module: Formats the file to be reasoned, and performs structuring processing on the formatted file according to the rules of directed acyclic graphs to obtain the computation graph topology of the file to be reasoned. Initial Vector Acquisition Module: Acquires the initial vectors of nodes in the computation graph topology and the initial vectors of the computation graph based on the graph attention neural network model; Subgraph Segmentation Module: Based on the NPU hardware support list and operator fusion fixed mode, an automatic splitting model based on hierarchical attention and cross-attention mechanisms is used to split the node initial vector and computation graph initial vector into several different splitting schemes, with each splitting scheme containing several subgraphs. Specifically, the initial vectors of nodes with two-hop topology structures are selected by obtaining the computation graph topology structure and cross-attention is performed with the NPU hardware support list to obtain subgraphs with the same structure. The out-degree and in-degree vectors of each node in the computation graph topology structure are obtained and cross-attention is performed with the NPU hardware support list to obtain the node with the highest weight. The cross-attention mechanism is used to match the computation graph vectors with the operator fusion fixed mode to obtain node vectors and computation graph vectors. Based on the computation graph topology structure, computation graph vectors, and node vectors, subgraphs with isomorphic structures and a frequency not less than a preset threshold are split according to the principle of maximum matching of the same structure. If a hardware-unsupported situation is encountered, the subgraph is split directly without further matching. Then, starting from the node with the highest weight in the attention mechanism, the traversal radiates outwards until the set weight threshold is reached, at which point the traversal stops and the subgraph is split, and the remaining single operator becomes a subgraph. Prediction module: Based on the temporal arrangement rules of all subgraph vectors and topology corresponding to each partitioning scheme, predict the inference latency of each different partitioning scheme; Output module: Selects the segmentation scheme with the largest inference latency that satisfies the typical model fusion rules, which include convolution normalization operator fusion rules and convolution activation fusion rules.

5. The automatic computation graph segmentation system for reasoning scenarios according to claim 4, characterized in that, The prediction module uses the single-point latency of each operator in the NPU hardware support list under the NPU as prior knowledge, and takes the computation graph vector and the vector corresponding to each partitioning scheme as input to predict the latency of each subgraph of each partitioning scheme in the predictor. After weighted summation of the latency of each subgraph, the final latency prediction is obtained. After weighted summation and normalization of the computation graph vector, the vector corresponding to each subgraph and the topological time arrangement information of each subgraph, the confidence of each partitioning scheme is obtained. If the confidence is greater than or equal to the preset confidence lower limit threshold, the predicted inference latency meets the requirements.

Images