Neural network model deployment method, system, device and storage medium

Abstract

The present application discloses a neural network model deployment method, system, device and storage medium, which belongs to the field of deep learning technology. The method includes: based on the loaded neural network model, generating a computational graph of the neural network model; dividing the computational graph into multiple termination sets, each termination set including one or more nodes in the computational graph; for each termination set, based on the hardware resources of the heterogeneous platform used to process the neural network model, generating the optimal segmentation strategy corresponding to the termination set through heuristic search; based on the optimal segmentation strategy corresponding to each termination set, generating the optimal allocation strategy of the neural network model, so that the heterogeneous platform can deploy the neural network model based on the optimal allocation strategy. The present application aims to improve the execution efficiency of heterogeneous platforms when executing neural network models, and solve the technical problem of low execution efficiency when executing neural network models on existing heterogeneous platforms.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a neural network model deployment method, system, device and storage medium. Background Art

[0002] Deep learning is widely used in various fields, such as image recognition, search technology, and speech recognition. The underlying technology of deep learning is a model composed of multiple neural network layers, namely a neural network model.

[0003] These neural network layers typically rely on computationally intensive matrix operations. To meet the time constraints of these neural network models, their operators are typically distributed across heterogeneous computing units on the heterogeneous platforms used to process them. In real-world production environments, developers manually allocate operators for different neural network models and platforms, resulting in suboptimal allocation strategies and, in turn, low execution efficiency when executing neural network models on heterogeneous platforms. Summary of the Invention

[0004] In view of this, the present application provides a neural network model deployment method, system, device and storage medium, aiming to improve the execution efficiency of heterogeneous platforms when executing neural network models, and solve the technical problem of low execution efficiency when executing neural network models on existing heterogeneous platforms.

[0005] The present application provides a neural network model deployment method, which includes the following steps:

[0006] Based on the loaded neural network model, generating a computational graph of the neural network model;

[0007] Splitting the computation graph into a plurality of termination sets, each termination set including one or more nodes in the computation graph;

[0008] For each termination set, based on the hardware resources of the heterogeneous platform used to process the neural network model, an optimal segmentation strategy corresponding to the termination set is generated through heuristic search;

[0009] According to the optimal segmentation strategy corresponding to each termination set, an optimal allocation strategy of the neural network model is generated, so that the heterogeneous platform can deploy the neural network model based on the optimal allocation strategy.

[0010] In a possible implementation of the present application, the computation graph includes a plurality of nodes and directed edges, wherein each node represents a corresponding operator, each directed edge connects a corresponding first node to a corresponding second node, and the input of the operator represented by the corresponding second node is the output of the operator represented by the corresponding first node; the step of dividing the computation graph into a plurality of termination sets, each termination set including one or more nodes in the computation graph, includes:

[0011] The computation graph is bisected to generate a termination set and a remaining subgraph; wherein the termination set and the remaining subgraph are disjoint, and the inputs of the operators represented by the nodes in the termination set are the outputs of the operators represented by the nodes in the remaining subgraph;

[0012] Determining whether the remaining subgraphs meet a preset stop segmentation condition;

[0013] If the remaining subgraph does not meet the preset stop segmentation condition, the remaining subgraph is split into two parts to generate a termination set and a new remaining subgraph, and the process returns to the step of determining whether the remaining subgraph meets the preset stop segmentation condition and subsequent steps;

[0014] If the remaining subgraph meets the preset stop splitting condition, the binary splitting of the remaining subgraph is stopped, and the termination sets obtained by multiple binary splits and the remaining subgraph are used as multiple termination sets of the computation graph.

[0015] In a possible implementation of the present application, the multiple termination sets are sorted in reverse order according to their segmentation generation time, and the step of generating, for each termination set, an optimal segmentation strategy corresponding to the termination set through a heuristic search based on hardware resources of a heterogeneous platform for processing the neural network model includes:

[0016] For each termination set, merge the optimal splitting strategies corresponding to all termination sets sorted before the termination set to generate an initial splitting strategy;

[0017] Using the initial segmentation strategy as the current segmentation strategy;

[0018] Splitting the termination set to generate a splitting strategy corresponding to the termination set, and merging the splitting strategy corresponding to the termination set with the initial splitting strategy to generate a new splitting strategy;

[0019] Comparing the new segmentation strategy with the running time of the current segmentation strategy based on an objective function to determine whether to update the current segmentation strategy to the new segmentation strategy; wherein the objective function is the running time of the neural network model in the heterogeneous platform based on the segmentation strategy and the hardware resources of the heterogeneous platform for processing the neural network model;

[0020] Obtaining the current number of iterations, and determining whether the current number of iterations reaches a preset number;

[0021] If not, returning to the step of segmenting the termination set, generating a segmentation strategy corresponding to the termination set, merging the segmentation strategy corresponding to the termination set with the initial segmentation strategy, and generating a new segmentation strategy and subsequent steps until the current number of iterations reaches a preset number;

[0022] If so, the optimal segmentation strategy corresponding to the termination set is determined based on the current segmentation strategy and the initial segmentation strategy.

[0023] In a possible implementation of the present application, the step of segmenting the termination set, generating a segmentation strategy corresponding to the termination set, and merging the segmentation strategy corresponding to the termination set with the initial segmentation strategy to generate a new segmentation strategy includes:

[0024] Recursively partitioning the termination set to generate a plurality of termination set partitioning subgraphs;

[0025] Determining whether the number of the plurality of termination set-cut subgraphs reaches the number of heterogeneous computing units in the heterogeneous platform;

[0026] If so, generate a segmentation strategy corresponding to the termination set, and merge the segmentation strategy corresponding to the termination set with the initial segmentation strategy to generate a new segmentation strategy;

[0027] If not, return to execute the step of recursively partitioning the termination set to generate multiple termination set partition subgraphs and subsequent steps.

[0028] In a possible implementation of the present application, the step of comparing the new segmentation strategy with the running time of the current segmentation strategy obtained based on the objective function to determine whether to update the current segmentation strategy to the new segmentation strategy includes:

[0029] Calculating the running time of the neural network model in the heterogeneous platform based on the new segmentation strategy and the running time of the neural network model in the heterogeneous platform based on the current segmentation strategy;

[0030] Compare the running time corresponding to the new segmentation strategy with the running time corresponding to the current segmentation strategy;

[0031] If the running time corresponding to the new segmentation strategy is less than the running time corresponding to the current segmentation strategy, then the current segmentation strategy is updated to the segmentation strategy;

[0032] If the running time corresponding to the new segmentation strategy is greater than or equal to the running time corresponding to the current segmentation strategy, the current segmentation strategy is updated to the new segmentation strategy with probability exp(-(t(p_new)-t(p_best)) / t(p_best)), where t(p_new) is the running time corresponding to the new segmentation strategy, and t(p_best) is the running time corresponding to the current segmentation strategy.

[0033] In a possible implementation of the present application, the step of calculating the running time of the neural network model based on the new segmentation strategy in the heterogeneous platform, and the step of calculating the running time of the neural network model based on the current segmentation strategy in the heterogeneous platform, includes:

[0034] Exporting the model of the termination set cut molecular graph in the splitting strategy to obtain multiple termination set cut molecular graph models;

[0035] Allocating the terminated set-cut molecular graph model to the corresponding heterogeneous computing unit based on the hardware resources corresponding to the heterogeneous platform and the model parameters of the terminated set-cut molecular graph model;

[0036] Determining a scheduling order of a plurality of the termination set segmentation subgraph models according to the dependency relationship and parallel relationship between the termination set segmentation subgraphs in the segmentation strategy;

[0037] Control the heterogeneous computing unit assigned with the termination set segmentation subgraph model to perform operations based on the scheduling order to obtain the running time of the neural network model in the heterogeneous platform based on the segmentation strategy; wherein the segmentation strategy includes the new segmentation strategy and the current segmentation strategy.

[0038] In a possible implementation of the present application, the hardware resources include a heterogeneous computing unit topology.

[0039] In a possible implementation of the present application, the step of generating a computation graph of the neural network model based on the loaded neural network model includes:

[0040] Parse the loaded neural network model into a universal format to obtain a neural network model in the universal format;

[0041] Based on the neural network model in the general format, a computational graph of the neural network model is generated.

[0042] The present application also provides a neural network model deployment system, the system comprising:

[0043] A computational graph acquisition module, configured to generate a computational graph of the neural network model based on the loaded neural network model;

[0044] A computation graph segmentation module, configured to segment the computation graph into a plurality of termination sets, each termination set including one or more nodes in the computation graph;

[0045] An optimal segmentation strategy generation module is used to generate an optimal segmentation strategy corresponding to each termination set through heuristic search based on the hardware resources of the heterogeneous platform used to process the neural network model;

[0046] The optimal allocation strategy determination module is used to generate the optimal allocation strategy of the neural network model according to the optimal segmentation strategy corresponding to each termination set, so that the heterogeneous platform can deploy the neural network model based on the optimal allocation strategy.

[0047] In one possible implementation of the present application, the computation graph includes a plurality of nodes and directed edges, wherein each node represents a corresponding operator, each directed edge connects a corresponding first node to a corresponding second node, and the input of the operator represented by the corresponding second node is the output of the operator represented by the corresponding first node; the computation graph splitting module is specifically configured to:

[0048] Performing a binary split on the computation graph to generate a termination set and a remaining subgraph; wherein the termination set does not intersect with the remaining subgraph, and the input of the operator represented by the node in the termination set is the output of the operator represented by the node in the remaining subgraph; judging whether the remaining subgraph satisfies a preset stop splitting condition; if the remaining subgraph does not satisfy the preset stop splitting condition, performing a binary split on the remaining subgraph to generate a termination set and a new remaining subgraph, and returning to execute the step of judging whether the remaining subgraph satisfies the preset stop splitting condition and subsequent steps; if the remaining subgraph satisfies the preset stop splitting condition, stopping the binary splitting of the remaining subgraph, and using the termination sets obtained by multiple binary splits and the remaining subgraph as multiple termination sets of the computation graph;

[0049] And / or the multiple termination sets are sorted in reverse order according to their segmentation generation time, and the optimal segmentation strategy generation module is specifically used to:

[0050] For each termination set, the optimal segmentation strategies corresponding to all termination sets sorted before the termination set are merged to generate an initial segmentation strategy; the initial segmentation strategy is used as the current segmentation strategy; the termination set is segmented to generate a segmentation strategy corresponding to the termination set, and the segmentation strategy corresponding to the termination set is merged with the initial segmentation strategy to generate a new segmentation strategy; the new segmentation strategy is compared with the running time of the current segmentation strategy based on the objective function to determine whether to update the current segmentation strategy to the new segmentation strategy; wherein the objective function is the neural network model based on the The method comprises the steps of: determining a partitioning strategy and hardware resources of a heterogeneous platform for processing the neural network model, and determining whether the current number of iterations reaches a preset number; obtaining a current number of iterations, and determining whether the current number of iterations reaches a preset number; if not, returning to execute the steps of partitioning the termination set, generating a partitioning strategy corresponding to the termination set, and merging the partitioning strategy corresponding to the termination set with the initial partitioning strategy to generate a new partitioning strategy, and subsequent steps, until the current number of iterations reaches a preset number; if so, determining an optimal partitioning strategy corresponding to the termination set based on the current partitioning strategy and the initial partitioning strategy;

[0051] And / or the optimal segmentation strategy generation module is further specifically used to:

[0052] Recursively partitioning the termination set to generate a plurality of termination set partitioning subgraphs; determining whether the number of the plurality of termination set partitioning subgraphs reaches the number of heterogeneous computing units in the heterogeneous platform; if so, generating a partitioning strategy corresponding to the termination set, and merging the partitioning strategy corresponding to the termination set with the initial partitioning strategy to generate a new partitioning strategy; if not, returning to executing the step of recursively partitioning the termination set to generate a plurality of termination set partitioning subgraphs and subsequent steps;

[0053] And / or the optimal segmentation strategy generation module is further specifically used to: calculate the running time of the neural network model based on the new segmentation strategy in the heterogeneous platform, and the running time of the neural network model based on the current segmentation strategy in the heterogeneous platform; compare the running time corresponding to the new segmentation strategy with the running time corresponding to the current segmentation strategy; if the running time corresponding to the new segmentation strategy is less than the running time corresponding to the current segmentation strategy, update the current segmentation strategy to the segmentation strategy; if the running time corresponding to the new segmentation strategy is greater than or equal to the running time corresponding to the current segmentation strategy, update the current segmentation strategy to the new segmentation strategy with probability exp(-(t(p_new)-t(p_best)) / t(p_best)), where t(p_new) is the running time corresponding to the new segmentation strategy, and t(p_best) is the running time corresponding to the current segmentation strategy;

[0054] And / or the optimal splitting strategy generation module is further specifically used to: derive a model of the termination set-cut molecular graph in the splitting strategy to obtain a plurality of termination set-cut molecular graph models; based on the hardware resources corresponding to the heterogeneous platform and the model parameters of the termination set-cut molecular graph model, allocate the termination set-cut molecular graph model to the corresponding heterogeneous computing unit; determine the scheduling order of the plurality of termination set-cut molecular graph models according to the dependency and parallel relationship between the termination set-cut molecular graphs in the splitting strategy; control the heterogeneous computing unit to which the termination set-cut molecular graph model is allocated to perform operations based on the scheduling order to obtain the running time of the neural network model in the heterogeneous platform based on the splitting strategy; wherein, the splitting strategy includes the new splitting strategy and the current splitting strategy;

[0055] and / or the hardware resources include a heterogeneous computing unit topology;

[0056] And / or the computational graph acquisition module is specifically used to: parse the loaded neural network model into a general format to obtain a neural network model in a general format; and generate a computational graph of the neural network model based on the neural network model in the general format.

[0057] The present application also provides a neural network model deployment device, which includes: a memory, a processor, and a neural network model deployment program stored on the memory and executable on the processor, wherein the neural network model deployment program is configured to implement the steps of the neural network model deployment method described above.

[0058] The present application also provides a storage medium on which a neural network model deployment program is stored. When the neural network model deployment program is executed by a processor, the steps of the neural network model deployment method described above are implemented.

[0059] The present application provides a neural network model deployment method, system, device and storage medium. Compared with the prior art, which cannot obtain the optimal allocation strategy of the neural network model, resulting in low execution efficiency when the heterogeneous platform executes the neural network model, in the present application, based on the loaded neural network model, a computation graph of the neural network model is generated; the computation graph is divided into multiple termination sets, each termination set including one or more nodes in the computation graph; for each termination set, based on the hardware resources of the heterogeneous platform used to process the neural network model, an optimal splitting strategy corresponding to the termination set is generated through heuristic search; based on the optimal splitting strategy corresponding to each termination set, an optimal allocation strategy of the neural network model is generated, so that the heterogeneous platform can deploy the neural network model based on the optimal allocation strategy. Therefore, the present application will split the computation graph of the network model into multiple termination sets, and based on the hardware resources of the heterogeneous platform used to process the neural network model, an optimal splitting strategy corresponding to each termination set is generated through heuristic search, and finally, based on the optimal splitting strategy corresponding to each termination set, an optimal allocation strategy of the neural network model is generated, thereby improving the execution efficiency when the heterogeneous platform executes the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a flowchart of the first embodiment of the neural network model deployment method of the present application;

[0061] Figure 2 for Figure 1 Detailed flow chart of step S20;

[0062] Figure 3 for Figure 1 Detailed flow chart of step S30;

[0063] Figure 4 A schematic diagram of the neural network model deployment device structure for the hardware operating environment involved in the embodiment of the present application.

[0064] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0065] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0066] The present application embodiment provides a neural network model deployment method, such as Figure 1 As shown, in one embodiment of the neural network model deployment method of the present application, the method includes the following steps:

[0067] Step S10: generating a computation graph of the neural network model based on the loaded neural network model;

[0068] Step S20: dividing the computation graph into a plurality of termination sets, each termination set including one or more nodes in the computation graph;

[0069] Step S30: for each termination set, based on the hardware resources of the heterogeneous platform used to process the neural network model, generate an optimal segmentation strategy corresponding to the termination set through heuristic search;

[0070] Step S40: Generate an optimal allocation strategy for the neural network model according to the optimal segmentation strategy corresponding to each termination set, so that the heterogeneous platform can deploy the neural network model based on the optimal allocation strategy.

[0071] This embodiment aims to improve the execution efficiency of heterogeneous platforms when executing neural network models.

[0072] Specifically, this application divides the computational graph of the network model into multiple termination sets, and generates the optimal segmentation strategy corresponding to each termination set through heuristic search based on the hardware resources of the heterogeneous platform used to process the neural network model. Finally, based on the optimal segmentation strategy corresponding to each termination set, the optimal allocation strategy of the neural network model is generated, thereby improving the execution efficiency of the heterogeneous platform when executing the neural network model.

[0073] Furthermore, in the present application, the computation graph is bisected to generate a termination set and a remaining subgraph. When the remaining subgraph does not meet the preset stop-splitting conditions, the remaining subgraph is iteratively bisected to generate a termination set and a new remaining subgraph, thereby obtaining all termination sets of the computation graph. This is conducive to generating an optimal allocation strategy, avoiding generating a suboptimal allocation strategy for the neural network model, and thereby improving the execution efficiency of the neural network model when executing on heterogeneous platforms.

[0074] Furthermore, in the present application, if the running time corresponding to the new segmentation strategy is less than the running time corresponding to the current segmentation strategy, the current segmentation strategy is updated to the segmentation strategy; if the running time corresponding to the new segmentation strategy is greater than or equal to the running time corresponding to the current segmentation strategy, the current segmentation strategy is updated to the new segmentation strategy with probability exp(-(t(p_new)-t(p_best)) / t(p_best)), where t(p_new) is the running time corresponding to the new segmentation strategy.

[0075] t(p_best) is the running time corresponding to the current segmentation strategy. That is, in this application, if the running time of the new segmentation strategy 5 in the heterogeneous platform is greater than or equal to the running time of the current segmentation strategy in the heterogeneous platform,

[0076] When the optimal splitting strategy is obtained, there is a certain probability that the current splitting strategy will be updated to the new splitting strategy, which prevents falling into the local optimal solution and failing to obtain the global optimal solution when obtaining the optimal splitting strategy, thereby improving the execution efficiency of the neural network model when executing on heterogeneous platforms.

[0077] Furthermore, in the present application, the termination set is recursively split to generate multiple termination set split 0 subgraphs until the number of the multiple termination set split subgraphs reaches the number of heterogeneous computing units in the heterogeneous platform. Since the termination set split subgraphs generated after the termination set is recursively split are in a parallel relationship, the number of the multiple termination set split subgraphs reaches the number of heterogeneous computing units in the heterogeneous platform, which can maximize the utilization of the heterogeneous computing units in the heterogeneous platform at the same time, thereby improving the execution efficiency of the heterogeneous platform when executing the neural network model.

[0078] 5 Further, in this application, based on the hardware resources and termination set segmentation corresponding to the heterogeneous platform

[0079] After allocating the termination set-cut molecular graph model to the corresponding heterogeneous computing unit, the scheduling order of multiple termination set-cut molecular graph models is determined according to the dependency and parallel relationship between the termination set-cut molecular graphs in the splitting strategy, so that the termination set-cut molecular graph models with parallel relationships can be parallelized.

[0080] It accelerates the execution of neural network models on heterogeneous platforms and improves the execution efficiency of neural network models on heterogeneous platforms.

[0081] Furthermore, in the present application, when calculating the operation time of the partitioning strategy in the heterogeneous platform, the hardware resources corresponding to the heterogeneous platform include the heterogeneous computing unit topology structure, and the termination set-cut molecular graph model is terminated based on the hardware resources corresponding to the heterogeneous platform and the model parameters of the termination set-cut molecular graph model.

[0082] When allocating to the corresponding heterogeneous computing units, the connection relationship between the heterogeneous computing units is taken into consideration to reduce the communication overhead when the 5 heterogeneous platforms execute the neural network model.

[0083] Furthermore, in this application, the loaded neural network model is parsed into a universal format to obtain a neural network model in a universal format. The neural network model in the universal format supports exporting from different deep learning frameworks. Therefore, this neural network model deployment method is applicable to models trained by different deep learning frameworks and is universal.

[0084] 0The specific steps are as follows:

[0085] Step S10: Based on the loaded neural network model, generate a computation graph of the neural network model.

[0086] As an example, the loaded neural network model is a model trained and generated based on a deep learning framework, such as TensorFlow, PyTorch, and MXNet.

[0087] As an example, in the image field, the neural network model is the convolutional neural network model; in the fields of speech recognition and natural speech processing, the neural network model is the recurrent neural network model, the self-attention mechanism model, and the Transformer network model.

[0088] As an example, step S10 includes the following steps:

[0089] Step S11: Parse the loaded neural network model into a universal format to obtain a neural network model in the universal format;

[0090] Step S12: Based on the neural network model in the general format, generate a computational graph of the neural network model.

[0091] As an example, the general format of the neural network model may be the ONNX (Open Neural Network Exchange) format.

[0092] As an example, the specific implementation process of step S12 is: read the storage file corresponding to the neural network model in the format, traverse each operator in the storage file, extract the feature information (input, output, shape, function, etc.) of the traversed operator, and use the above feature information to construct a graph representation of the operator. The graph representation of all operators in the storage file constructs the computational graph of the neural network model.

[0093] The computational graph of the neural network model can be expressed as G = (V; E), where V represents the set of nodes in the computational graph (i.e., the set of graph representations of operators in the neural network model), and E represents the set of directed edges in the computational graph (i.e., the dependencies between all operators in the neural network model). Operators in the neural network model include computational, data processing, and control types. Examples are given for each type: computational, such as tensor addition, subtraction, multiplication, and division, matrix multiplication, convolution, activation, etc.; data processing, such as transposition, embedding, tensor concatenation, tensor splitting, etc.; and control, such as loops and jumps. The input and output of an operator are both tensors, and the number of tensors input or output by an operator is more than one, and the specific number is uncertain, such as operations like concat and split. In addition to the input and output tensors, each operator also has some additional parameters. For example, for convolution operations, there are parameters such as stride, padding, and dilation.

[0094] As an example, step S10 includes the following steps:

[0095] Step S101: simplifying the loaded neural network model;

[0096] Step S102: parsing the simplified neural network model into a universal format to obtain a neural network model in the universal format;

[0097] Step S103: Based on the neural network model in the general format, generate a computational graph of the neural network model.

[0098] As an example, the simplified processing performed on the loaded neural network model can be merging batch normalization layers of the neural network model.

[0099] In this application, the loaded neural network model is simplified, which can improve the execution efficiency of the neural network model while ensuring the correct equivalence of the neural network model.

[0100] Step S20: Divide the computation graph into multiple termination sets, each termination set including one or more nodes in the computation graph.

[0101] As an example, a computational graph includes multiple nodes and directed edges, wherein each node represents a corresponding operator, each directed edge connects a corresponding first node to a corresponding second node, and the input of the operator represented by the corresponding second node is the output of the operator represented by the corresponding first node.

[0102] It should be noted that the computation graph is divided into multiple termination sets. Since the nature of the termination set determines that it can ensure that when the nodes in the termination set are executed, the inputs of the operators they represent have been generated by the remaining subgraphs, so data dependency can be guaranteed.

[0103] like Figure 2 As shown, as an example, the computation graph is divided into a plurality of termination sets, each termination set including one or more nodes in the computation graph, including:

[0104] Step S21: split the computation graph into two parts to generate a termination set and a remaining subgraph; wherein the termination set does not intersect with the remaining subgraph, and the input of the operator represented by the node in the termination set is the output of the operator represented by the node in the remaining subgraph.

[0105] As an example, the step of performing a binary split on the computation graph to generate a termination set and a remaining subgraph includes:

[0106] Based on the depth-first traversal algorithm, the computation graph is bisected to generate a termination set and a remaining subgraph.

[0107] Step S22: Determine whether the remaining subgraphs meet a preset stop segmentation condition.

[0108] As an example, the preset stopping condition is that the remaining subgraph is empty, or there is a single node in the remaining subgraph, or there is no directed edge between the nodes in the remaining subgraph.

[0109] Step S23: If the remaining subgraph does not meet the preset stop segmentation condition, the remaining subgraph is split into two parts to generate a termination set and a new remaining subgraph, and the process returns to the step of determining whether the remaining subgraph meets the preset stop segmentation condition and subsequent steps.

[0110] As an example, the step of performing a binary split on the remaining subgraph to generate a termination set and a new remaining subgraph includes:

[0111] Based on the depth-first traversal algorithm, the remaining subgraph is bisected to generate a termination set and a new remaining subgraph.

[0112] Step S24: If the remaining subgraph meets the preset stop splitting condition, then stop binary splitting the remaining subgraph, and use the termination sets obtained by multiple binary splits and the remaining subgraph as multiple termination sets of the computation graph.

[0113] As an example, the computation graph V0 is split into two parts to generate the termination set S1 and the remaining subgraph V1; if the remaining subgraph V1 does not meet the preset stop splitting condition, the remaining subgraph V1 is split into two parts to generate the termination set S2 and the new remaining subgraph V2; if the remaining subgraph V2 does not meet the preset stop splitting condition, the remaining subgraph V2 is split into two parts to generate the termination set S3 and the new remaining subgraph V3; ...; if the remaining subgraph V n-1 If the preset stop segmentation condition is not met, the remaining subgraph V n-1 Perform binary split to generate the termination set S n and the new remaining subgraph V n ; If the remaining subgraph V n When the preset stop segmentation condition is met, the remaining subgraph V is stopped. n Perform binary splitting, and the termination sets S1, S2, S3, ..., S n , and the remaining subgraph V n As multiple termination sets of a computation graph.

[0114] Step S30: For each termination set, based on the hardware resources of the heterogeneous platform used to process the neural network model, generate the optimal segmentation strategy corresponding to the termination set through heuristic search.

[0115] As an example, the heterogeneous platform used to process the neural network model is a heterogeneous platform.

[0116] As an example, the multiple termination sets in step S20 are sorted in reverse order according to their segmentation generation time.

[0117] As an example, the specific implementation process of dividing the computation graph into multiple termination sets is as follows: perform binary splitting on the computation graph V0 to generate the termination set S1 and the remaining subgraph V1; if the remaining subgraph V1 does not meet the preset stop splitting condition, perform binary splitting on the remaining subgraph V1 to generate the termination set S2 and the new remaining subgraph V2; if the remaining subgraph V2 does not meet the preset stop splitting condition, perform binary splitting on the remaining subgraph V2 to generate the termination set S3 and the new remaining subgraph V3; ...; if the remaining subgraph V n-1 If the preset stop segmentation condition is not met, the remaining subgraph V n-1 Perform binary split to generate the termination set S n and the new remaining subgraph V n ; If the remaining subgraph V n When the preset stop segmentation condition is met, the remaining subgraph V is stopped. n Perform binary splitting, and the termination sets S1, S2, S3, ..., S n , and the remaining subgraph V n As multiple termination sets of the computation graph. On this basis, multiple termination sets are sorted in reverse order according to their segmentation generation time, and the order can be obtained as S n+1 、S n ,…,S3,S2,S1. Among them, the termination set S n+1 That is the remaining subgraph V n .

[0118] like Figure 3 As shown, as an example, for each termination set, based on the hardware resources of the heterogeneous platform for processing the neural network model, the step of generating the optimal segmentation strategy corresponding to the termination set through heuristic search includes:

[0119] Step S31: For each termination set, the optimal segmentation strategies corresponding to all termination sets that are sorted before the termination set are merged to generate an initial segmentation strategy.

[0120] As an example, for the termination set ranked first, since there is no termination set before it, there is no optimal splitting strategy corresponding to the termination set that can be merged. In other words, there is no initial splitting strategy for the termination set ranked first.

[0121] Step S32: Use the initial segmentation strategy as the current segmentation strategy.

[0122] As an example, for the termination set ranked first, since there is no initial segmentation strategy,

[0123] Therefore, there is no current segmentation strategy.

[0124] Step S33: Split the termination set to generate a splitting strategy corresponding to the termination set, and merge the splitting strategy corresponding to the termination set with the initial splitting strategy to generate a new splitting strategy.

[0125] As an example, for the termination set ranked first, since there is no current segmentation strategy, it can be understood that: for the termination set ranked first, the termination set is segmented, and the segmentation strategy corresponding to the termination set is generated. The segmentation strategy corresponding to the termination set is the new segmentation strategy.

[0126] As an example, for splitting the termination set, generating a splitting strategy corresponding to the termination set, and merging the splitting strategy corresponding to the termination set with the initial splitting strategy to generate a new splitting strategy, the steps include: 0 Step S331, recursively splitting the termination set to generate multiple termination set split subgraphs.

[0127] As an example, the step of recursively partitioning the termination set to generate a plurality of termination set partition subgraphs includes:

[0128] Based on a depth-first traversal algorithm, the termination set is recursively partitioned to generate a plurality of termination set partition subgraphs.

[0129] Step S332: Determine whether the number of the plurality of terminated set segmented graphs reaches the number of the heterogeneous platform.

[0130] The number of heterogeneous computing units in .

[0131] Step S333: If yes, generate a segmentation strategy corresponding to the termination set, and merge the segmentation strategy corresponding to the termination set with the initial segmentation strategy to generate a new segmentation strategy.

[0132] Step S334: If not, return to execute the step of recursively partitioning the termination set to generate multiple termination set partition subgraphs and subsequent steps.

[0133] Step S34: Compare the running time of the new segmentation strategy with that of the current segmentation strategy based on the objective function to determine whether to update the current segmentation strategy to the new segmentation strategy; wherein, the objective function is the running time of the neural network model in the heterogeneous platform based on the segmentation strategy and the hardware resources of the heterogeneous platform for processing the neural network model.

[0134] As an example, the step of comparing the new segmentation strategy with the running time of the current segmentation strategy obtained based on the objective function to determine whether to update the current segmentation strategy to the new segmentation strategy includes:

[0135] Step S341: Calculate the running time of the neural network model in the heterogeneous platform based on the new segmentation strategy, and the running time of the neural network model in the heterogeneous platform based on the current segmentation strategy.

[0136] As an example, the step of calculating the time consumed by running the neural network model based on the new segmentation strategy in the heterogeneous platform includes:

[0137] Step A1: exporting a model of the termination set-cut molecular graph in the new segmentation strategy to obtain a plurality of termination set-cut molecular graph models.

[0138] As an example, the plurality of terminated set-cut molecular graph models are all terminated set-cut molecular graph models in a universal format.

[0139] Step A2: allocating the terminated set-cut molecular graph model to the corresponding heterogeneous computing unit based on the hardware resources corresponding to the heterogeneous platform and the model parameters of the terminated set-cut molecular graph model;

[0140] As an example, when the heterogeneous platform is a heterogeneous platform, the computing unit is a heterogeneous heterogeneous computing unit.

[0141] As an example, hardware resources include the number of heterogeneous computing units, the computing power of each heterogeneous computing unit, the storage capacity of each heterogeneous computing unit, and the operator type applicable to each heterogeneous computing unit.

[0142] As an example, the model parameters include the computational complexity of the terminated set-cut molecular graph model, the operator type of the terminated set-cut molecular graph model, and the memory usage of the terminated set-cut molecular graph model.

[0143] As an example, when the number of heterogeneous computing units is the same as the number of termination set-cut molecular graph models, when the termination set-cut molecular graph model is allocated to the corresponding heterogeneous computing unit, the computing power of the heterogeneous computing unit must be greater than or equal to the computing amount of the termination set-cut molecular graph model, the operator type applicable to the heterogeneous computing unit must be the same as the operator type of the termination set-cut molecular graph model, and the storage capacity of the heterogeneous computing unit must be greater than or equal to the memory usage of the termination set-cut molecular graph model.

[0144] As an example, when the number of heterogeneous computing units is less than the number of termination set-cut molecular graph models, there is a heterogeneous computing unit that is assigned multiple termination set-cut molecular graph models. In the above case, the computing power of the heterogeneous computing unit is greater than the computing power of any of the multiple termination set-cut molecular graph models assigned to it, or greater than the computing power of the multiple termination set-cut molecular graph models assigned to it.

[0145] The total computational effort of the segmented graph model; the operator types applicable to the heterogeneous computing unit must match the operator types of the five assigned termination set segmented graph models; the storage capacity of the heterogeneous computing unit must be greater than the total memory usage of the multiple assigned termination set segmented graph models.

[0146] As an example, when the number of heterogeneous computing units is less than the number of termination set cut subgraph models, the termination set cut subgraph models with parallel relationships are allocated to different heterogeneous computing units according to the parallel relationships between the termination set cut subgraphs in the new splitting strategy.

[0147] As an example, hardware resources also include a heterogeneous computing unit topology.

[0148] Step A3: Determine the scheduling order of the plurality of termination set segmentation sub-graph models according to the dependency and parallel relationship between the termination set segmentation sub-graphs in the new segmentation strategy.

[0149] As an example, according to the parallel relationship between the termination set segmentation subgraphs in the new segmentation strategy,

[0150] The running order of the termination set cut molecular graph models with parallel relationships is set to parallel running; according to the dependency relationship between the termination set cut molecular graphs in the new 5-splitting strategy, the running order of the running models with dependency relationships is set to front and back running, thereby determining the scheduling order of multiple termination set cut molecular graph models corresponding to the new splitting strategy.

[0151] As an example, when recursively partitioning the termination set, the termination set partition subgraphs of the parallel relations can be identified and labeled. There are several different forms of parallel relations: bit-level, instruction-level, data, and task parallelism.

[0152] Step A4: Control the heterogeneous computing unit assigned with the termination set segmentation subgraph model to perform operations based on the scheduling order to obtain the running time of the neural network model in the heterogeneous platform based on the new segmentation strategy.

[0153] As a type of termination set cut molecular graph model 5, multiple termination set cut molecular graph models are of a common format, and the heterogeneous computing units assigned with the termination set cut molecular graph models are controlled to run based on the scheduling order.

[0154] When the termination set is cut, the molecular graph model is converted from a common format to a program that can be run on the heterogeneous computing unit. This conversion process is completed by the conversion tool of the heterogeneous platform itself, and the conversion tools of different heterogeneous platforms are different.

[0155] As an example, the control of heterogeneous computing units assigned with termination set segmentation graph models to run based on the scheduling order is completed in the runtime module. The runtime module provides an abstraction of heterogeneous platform resources, that is, the runtime module provides a registration interface for different heterogeneous computing units. By describing the hardware parameters corresponding to the heterogeneous computing units, the heterogeneous computing units can be registered into the runtime module. At the same time, the runtime module also provides resource initialization, call execution, and resource release interfaces that different heterogeneous computing units need to provide. In addition, the runtime module also supports the input of the topological structure of heterogeneous computing units. This information is used to consider the connection relationship between heterogeneous computing units when allocating the termination set segmentation graph in the new segmentation strategy to the corresponding heterogeneous computing units to reduce communication overhead.

[0156] As an example, the step of calculating the time consumed by running the neural network model in the heterogeneous platform based on the current segmentation strategy includes:

[0157] Step B1, exporting a model of the termination set-cut molecular graph in the current segmentation strategy to obtain multiple termination set-cut molecular graph models;

[0158] Step B2: allocating the terminated set-cut molecular graph to corresponding heterogeneous computing units based on the hardware resources corresponding to the heterogeneous platform and the model parameters of the terminated set-cut molecular graph model;

[0159] Step B3: determining the scheduling order of the plurality of termination set segmentation sub-graph models according to the dependency and parallel relationship between the termination set segmentation sub-graphs in the current segmentation strategy;

[0160] Step B4: Control the heterogeneous computing unit assigned with the termination set segmentation subgraph model to perform operations based on the scheduling order to obtain the running time of the neural network model in the heterogeneous platform based on the current segmentation strategy.

[0161] It should be noted that the specific process of calculating the running time of the neural network model based on the current segmentation strategy in the heterogeneous platform is the same as the specific process of calculating the running time of the neural network model based on the new segmentation strategy in the heterogeneous platform, and will not be repeated here.

[0162] As an example, in actual application, since the initial splitting strategy is a combination of the optimal splitting strategies corresponding to all termination sets sorted before the termination set, and the running time corresponding to the optimal splitting strategies corresponding to all termination sets sorted before the termination set is known when the optimal splitting strategy is determined. Therefore, when calculating the running time corresponding to the new splitting strategy, it is only necessary to calculate the running time corresponding to the splitting strategy corresponding to the termination set, and add the running time corresponding to the splitting strategy corresponding to the termination set to the running time corresponding to the optimal splitting strategy corresponding to all termination sets sorted before the termination set to obtain the running time corresponding to the new splitting strategy. Similarly, when the current splitting strategy is the initial splitting strategy, the running time corresponding to the current splitting strategy is the sum of the running times corresponding to the optimal splitting strategies corresponding to all termination sets sorted before the termination set.

[0163] Step S342: Compare the running time corresponding to the new segmentation strategy with the running time corresponding to the current segmentation strategy.

[0164] Step S343: If the running time corresponding to the new segmentation strategy is less than the running time corresponding to the current segmentation strategy, the current segmentation strategy is updated to the segmentation strategy.

[0165] Step S344: If the running time corresponding to the new segmentation strategy is greater than or equal to the running time corresponding to the current segmentation strategy, the current segmentation strategy is updated to the new segmentation strategy with probability exp(-(t(p_new)-t(p_best)) / t(p_best)), where t(p_new) is the running time corresponding to the new segmentation strategy, and t(p_best) is the running time corresponding to the current segmentation strategy.

[0166] Step S35: Obtain the current number of iterations, and determine whether the current number of iterations reaches a preset number.

[0167] As an example, the preset number Kmax is specified in advance by the user, which is not specifically limited in this embodiment.

[0168] Step S36: If not, return to the step of splitting the termination set, generating a splitting strategy corresponding to the termination set, merging the splitting strategy corresponding to the termination set with the initial splitting strategy, and generating a new splitting strategy and subsequent steps until the current number of iterations reaches the preset number.

[0169] Step S37: If yes, then determine the optimal segmentation strategy corresponding to the termination set based on the current segmentation strategy and the initial segmentation strategy.

[0170] As an example, the initial splitting strategy is obtained by merging the optimal splitting strategies corresponding to all termination sets sorted before the termination set. In the termination set splitting subgraph of the current splitting strategy, the termination set splitting subgraph corresponding to the initial splitting strategy is deleted, and the remaining termination set splitting subgraph is the optimal splitting strategy corresponding to the termination set.

[0171] Step S30: Generate an optimal allocation strategy for the neural network model according to the optimal segmentation strategy corresponding to each termination set, so that the heterogeneous platform can deploy the neural network model based on the optimal allocation strategy.

[0172] As an example, multiple termination sets are sorted in reverse order according to their segmentation generation time, and the optimal segmentation strategies corresponding to each termination set are sorted in reverse order to obtain the optimal allocation strategy of the neural network model.

[0173] Compared with the prior art, which cannot obtain the optimal allocation strategy of the neural network model, resulting in low execution efficiency when the heterogeneous platform executes the neural network model, in this embodiment, based on the loaded neural network model, a computation graph of the neural network model is generated; the computation graph is divided into multiple termination sets, each termination set including one or more nodes in the computation graph; for each termination set, based on the hardware resources of the heterogeneous platform used to process the neural network model, an optimal splitting strategy corresponding to the termination set is generated through heuristic search; based on the optimal splitting strategy corresponding to each termination set, an optimal allocation strategy for the neural network model is generated, so that the heterogeneous platform can deploy the neural network model based on the optimal allocation strategy. Therefore, in this embodiment, the computation graph of the network model is divided into multiple termination sets, and based on the hardware resources of the heterogeneous platform used to process the neural network model, an optimal splitting strategy corresponding to each termination set is generated through heuristic search, and finally, based on the optimal splitting strategy corresponding to each termination set, an optimal allocation strategy for the neural network model is generated, thereby improving the execution efficiency when the heterogeneous platform executes the neural network model.

[0174] Reference Figure 4 , Figure 4 A schematic diagram of the neural network model deployment device structure for the hardware operating environment involved in the embodiment of the present application.

[0175] like Figure 4As shown, the neural network model deployment device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 may be a high-speed random access memory (RAM) memory, or a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0176] Those skilled in the art will understand that Figure 4 The structure shown in does not constitute a limitation on the neural network model deployment device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0177] like Figure 4 As shown, the memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module and a neural network model deployment program.

[0178] exist Figure 4 In the neural network model deployment device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the neural network model deployment device of the present application can be set in the neural network model deployment device, and the neural network model deployment device calls the neural network model deployment program stored in the memory 1005 through the processor 1001 to implement the steps of the neural network model deployment method described in any one of the above items.

[0179] The specific implementation of the neural network model deployment device of the present application is basically the same as the various embodiments of the above-mentioned neural network model deployment method, and will not be repeated here.

[0180] This application also provides a neural network model deployment system, the system comprising:

[0181] A computational graph acquisition module, configured to generate a computational graph of the neural network model based on the loaded neural network model;

[0182] A computation graph segmentation module, configured to segment the computation graph into a plurality of termination sets, each termination set including one or more nodes in the computation graph;

[0183] An optimal segmentation strategy generation module is used to generate an optimal segmentation strategy corresponding to each termination set through heuristic search based on the hardware resources of the heterogeneous platform used to process the neural network model;

[0184] The optimal allocation strategy determination module is used to generate the optimal allocation strategy of the neural network model according to the optimal segmentation strategy corresponding to each termination set, so that the heterogeneous platform can deploy the neural network model based on the optimal allocation strategy.

[0185] In one possible implementation of the present application, the computation graph includes a plurality of nodes and directed edges, wherein each node represents a corresponding operator, each directed edge connects a corresponding first node to a corresponding second node, and the input of the operator represented by the corresponding second node is the output of the operator represented by the corresponding first node; the computation graph splitting module is specifically configured to:

[0186] Performing a binary split on the computation graph to generate a termination set and a remaining subgraph; wherein the termination set does not intersect with the remaining subgraph, and the input of the operator represented by the node in the termination set is the output of the operator represented by the node in the remaining subgraph; judging whether the remaining subgraph satisfies a preset stop splitting condition; if the remaining subgraph does not satisfy the preset stop splitting condition, performing a binary split on the remaining subgraph to generate a termination set and a new remaining subgraph, and returning to execute the step of judging whether the remaining subgraph satisfies the preset stop splitting condition and subsequent steps; if the remaining subgraph satisfies the preset stop splitting condition, stopping the binary splitting of the remaining subgraph, and using the termination sets obtained by multiple binary splits and the remaining subgraph as multiple termination sets of the computation graph;

[0187] And / or the multiple termination sets are sorted in reverse order according to their segmentation generation time, and the optimal segmentation strategy generation module is specifically used to:

[0188] For each termination set, the optimal segmentation strategies corresponding to all termination sets sorted before the termination set are merged to generate an initial segmentation strategy; the initial segmentation strategy is used as the current segmentation strategy; the termination set is segmented to generate a segmentation strategy corresponding to the termination set, and the segmentation strategy corresponding to the termination set is merged with the initial segmentation strategy to generate a new segmentation strategy; the new segmentation strategy is compared with the running time of the current segmentation strategy based on the objective function to determine whether to update the current segmentation strategy to the new segmentation strategy; wherein the objective function is the neural network model based on the The method comprises the steps of: determining a partitioning strategy and hardware resources of a heterogeneous platform for processing the neural network model, and determining whether the current number of iterations reaches a preset number; obtaining a current number of iterations, and determining whether the current number of iterations reaches a preset number; if not, returning to execute the steps of partitioning the termination set, generating a partitioning strategy corresponding to the termination set, and merging the partitioning strategy corresponding to the termination set with the initial partitioning strategy to generate a new partitioning strategy, and subsequent steps, until the current number of iterations reaches a preset number; if so, determining an optimal partitioning strategy corresponding to the termination set based on the current partitioning strategy and the initial partitioning strategy;

[0189] And / or the optimal segmentation strategy generation module is further specifically used to:

[0190] Recursively partitioning the termination set to generate a plurality of termination set partitioning subgraphs; determining whether the number of the plurality of termination set partitioning subgraphs reaches the number of heterogeneous computing units in the heterogeneous platform; if so, generating a partitioning strategy corresponding to the termination set, and merging the partitioning strategy corresponding to the termination set with the initial partitioning strategy to generate a new partitioning strategy; if not, returning to executing the step of recursively partitioning the termination set to generate a plurality of termination set partitioning subgraphs and subsequent steps;

[0191] And / or the optimal segmentation strategy generation module is further specifically used to: calculate the running time of the neural network model based on the new segmentation strategy in the heterogeneous platform, and the running time of the neural network model based on the current segmentation strategy in the heterogeneous platform; compare the running time corresponding to the new segmentation strategy with the running time corresponding to the current segmentation strategy; if the running time corresponding to the new segmentation strategy is less than the running time corresponding to the current segmentation strategy, update the current segmentation strategy to the segmentation strategy; if the running time corresponding to the new segmentation strategy is greater than or equal to the running time corresponding to the current segmentation strategy, update the current segmentation strategy to the new segmentation strategy with probability exp(-(t(p_new)-t(p_best)) / t(p_best)), where t(p_new) is the running time corresponding to the new segmentation strategy, and t(p_best) is the running time corresponding to the current segmentation strategy;

[0192] And / or the optimal splitting strategy generation module is further specifically used to: derive a model of the termination set-cut molecular graph in the splitting strategy to obtain a plurality of termination set-cut molecular graph models; based on the hardware resources corresponding to the heterogeneous platform and the model parameters of the termination set-cut molecular graph model, allocate the termination set-cut molecular graph model to the corresponding heterogeneous computing unit; determine the scheduling order of the plurality of termination set-cut molecular graph models according to the dependency and parallel relationship between the termination set-cut molecular graphs in the splitting strategy; control the heterogeneous computing unit to which the termination set-cut molecular graph model is allocated to perform operations based on the scheduling order to obtain the running time of the neural network model in the heterogeneous platform based on the splitting strategy; wherein, the splitting strategy includes the new splitting strategy and the current splitting strategy;

[0193] and / or the hardware resources include a heterogeneous computing unit topology;

[0194] And / or the computational graph acquisition module is specifically used to: parse the loaded neural network model into a general format to obtain a neural network model in a general format; and generate a computational graph of the neural network model based on the neural network model in the general format.

[0195] The specific implementation of the neural network model deployment device of the present application is basically the same as the various embodiments of the above-mentioned neural network model deployment method, and will not be repeated here.

[0196] An embodiment of the present application provides a storage medium, and the storage medium stores one or more programs, and the one or more programs can also be executed by one or more processors to implement the steps of any of the above-mentioned neural network model deployment methods.

[0197] The specific implementation of the storage medium of this application is basically the same as the embodiments of the above-mentioned neural network model deployment method, and will not be repeated here.

[0198] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned neural network model deployment method when executed by a processor.

[0199] The specific implementation of the computer program product of the present application is basically the same as the various embodiments of the above-mentioned neural network model deployment method, and will not be repeated here.

[0200] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0201] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0202] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal heterogeneous platform (which can be a mobile phone, computer, server, or network heterogeneous platform, etc.) to execute the methods described in each embodiment of the present application.

[0203] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A neural network model deployment method, characterized in that: The neural network model deployment method comprises the following steps: Based on the loaded neural network model, generating a computational graph of the neural network model; Splitting the computation graph into a plurality of termination sets, each termination set including one or more nodes in the computation graph; For each termination set, based on the hardware resources of the heterogeneous platform used to process the neural network model, an optimal segmentation strategy corresponding to the termination set is generated through heuristic search; Generating an optimal allocation strategy for the neural network model according to the optimal segmentation strategy corresponding to each termination set, so that the heterogeneous platform can deploy the neural network model based on the optimal allocation strategy; The computation graph includes a plurality of nodes and directed edges, wherein each node represents a corresponding operator, each directed edge connects a corresponding first node to a corresponding second node, and the input of the operator represented by the corresponding second node is the output of the operator represented by the corresponding first node; the step of dividing the computation graph into a plurality of termination sets, each termination set including one or more nodes in the computation graph, includes: The computation graph is bisected to generate a termination set and a remaining subgraph; wherein the termination set and the remaining subgraph are disjoint, and the inputs of the operators represented by the nodes in the termination set are the outputs of the operators represented by the nodes in the remaining subgraph; Determining whether the remaining subgraphs meet a preset stop segmentation condition; If the remaining subgraph does not meet the preset stop segmentation condition, the remaining subgraph is split into two parts to generate a termination set and a new remaining subgraph, and the process returns to the step of determining whether the remaining subgraph meets the preset stop segmentation condition and subsequent steps; If the remaining subgraph meets the preset stop splitting condition, the binary splitting of the remaining subgraph is stopped, and the termination sets obtained by multiple binary splits and the remaining subgraph are used as multiple termination sets of the computation graph.

2. The neural network model deployment method according to claim 1, wherein: The multiple termination sets are sorted in reverse order according to their segmentation generation time, and the step of generating, for each termination set, an optimal segmentation strategy corresponding to the termination set through heuristic search based on hardware resources of a heterogeneous platform for processing the neural network model comprises: For each termination set, merge the optimal splitting strategies corresponding to all termination sets sorted before the termination set to generate an initial splitting strategy; Using the initial segmentation strategy as the current segmentation strategy; Splitting the termination set to generate a splitting strategy corresponding to the termination set, and merging the splitting strategy corresponding to the termination set with the initial splitting strategy to generate a new splitting strategy; Comparing the new segmentation strategy with the running time of the current segmentation strategy based on an objective function to determine whether to update the current segmentation strategy to the new segmentation strategy; wherein the objective function is the running time of the neural network model in the heterogeneous platform based on the segmentation strategy and the hardware resources of the heterogeneous platform for processing the neural network model; Obtaining the current number of iterations, and determining whether the current number of iterations reaches a preset number; If not, returning to the step of segmenting the termination set, generating a segmentation strategy corresponding to the termination set, merging the segmentation strategy corresponding to the termination set with the initial segmentation strategy, and generating a new segmentation strategy and subsequent steps until the current number of iterations reaches a preset number; If so, the optimal segmentation strategy corresponding to the termination set is determined based on the current segmentation strategy and the initial segmentation strategy.

3. The neural network model deployment method according to claim 2, wherein: The step of segmenting the termination set, generating a segmentation strategy corresponding to the termination set, and merging the segmentation strategy corresponding to the termination set with the initial segmentation strategy to generate a new segmentation strategy includes: Recursively partitioning the termination set to generate a plurality of termination set partitioning subgraphs; Determining whether the number of the plurality of termination set-cut subgraphs reaches the number of heterogeneous computing units in the heterogeneous platform; If so, generate a segmentation strategy corresponding to the termination set, and merge the segmentation strategy corresponding to the termination set with the initial segmentation strategy to generate a new segmentation strategy; If not, return to execute the step of recursively partitioning the termination set to generate multiple termination set partition subgraphs and subsequent steps.

4. The neural network model deployment method according to claim 2, wherein: The step of comparing the new segmentation strategy with the running time of the current segmentation strategy obtained based on the objective function to determine whether to update the current segmentation strategy to the new segmentation strategy includes: Calculating the running time of the neural network model in the heterogeneous platform based on the new segmentation strategy and the running time of the neural network model in the heterogeneous platform based on the current segmentation strategy; Compare the running time corresponding to the new segmentation strategy with the running time corresponding to the current segmentation strategy; If the running time corresponding to the new segmentation strategy is less than the running time corresponding to the current segmentation strategy, the current segmentation strategy is updated to the segmentation strategy; If the running time corresponding to the new segmentation strategy is greater than or equal to the running time corresponding to the current segmentation strategy, then the probability The current segmentation strategy is updated to the new segmentation strategy, wherein: is the running time corresponding to the new segmentation strategy, The running time corresponding to the current segmentation strategy.

5. The neural network model deployment method according to claim 4, wherein: The step of calculating the running time of the neural network model based on the new segmentation strategy in the heterogeneous platform, and the step of calculating the running time of the neural network model based on the current segmentation strategy in the heterogeneous platform, includes: Exporting the model of the termination set cut molecular graph in the splitting strategy to obtain multiple termination set cut molecular graph models; Allocating the terminated set-cut molecular graph model to the corresponding heterogeneous computing unit based on the hardware resources corresponding to the heterogeneous platform and the model parameters of the terminated set-cut molecular graph model; Determining a scheduling order of a plurality of the termination set segmentation subgraph models according to the dependency relationship and parallel relationship between the termination set segmentation subgraphs in the segmentation strategy; Control the heterogeneous computing unit assigned with the termination set segmentation subgraph model to perform operations based on the scheduling order to obtain the running time of the neural network model in the heterogeneous platform based on the segmentation strategy; wherein the segmentation strategy includes the new segmentation strategy and the current segmentation strategy.

6. The neural network model deployment method according to claim 5, wherein: The hardware resources include a heterogeneous computing unit topology.

7. The neural network model deployment method according to claim 1, wherein: The step of generating a computation graph of the neural network model based on the loaded neural network model includes: Parse the loaded neural network model into a universal format to obtain a neural network model in the universal format; Based on the neural network model in the general format, a computational graph of the neural network model is generated.

8. A neural network model deployment system, characterized in that: The system comprises: A computational graph acquisition module, configured to generate a computational graph of the neural network model based on the loaded neural network model; A computation graph segmentation module, configured to segment the computation graph into a plurality of termination sets, each termination set including one or more nodes in the computation graph; An optimal segmentation strategy generation module is used to generate an optimal segmentation strategy corresponding to each termination set through heuristic search based on the hardware resources of the heterogeneous platform used to process the neural network model; An optimal allocation strategy determination module, configured to generate an optimal allocation strategy for the neural network model according to the optimal segmentation strategy corresponding to each termination set, so that the heterogeneous platform can deploy the neural network model based on the optimal allocation strategy; The computation graph includes a plurality of nodes and directed edges, wherein each node represents a corresponding operator, each directed edge connects a corresponding first node to a corresponding second node, and the input of the operator represented by the corresponding second node is the output of the operator represented by the corresponding first node; the computation graph splitting module is specifically used to: The computation graph is bisected to generate a termination set and a remaining subgraph; wherein the termination set does not intersect with the remaining subgraph, and the input of the operator represented by the node in the termination set is the output of the operator represented by the node in the remaining subgraph; whether the remaining subgraph satisfies a preset stop-splitting condition is determined; if the remaining subgraph does not satisfy the preset stop-splitting condition, the remaining subgraph is bisected to generate a termination set and a new remaining subgraph, and the step of determining whether the remaining subgraph satisfies the preset stop-splitting condition and subsequent steps are returned to execute; if the remaining subgraph satisfies the preset stop-splitting condition, the bisecting of the remaining subgraph is stopped, and the termination sets obtained by multiple bisecting and the remaining subgraph are used as multiple termination sets of the computation graph.

9. The neural network model deployment system according to claim 8, wherein: The multiple termination sets are sorted in reverse order according to their segmentation generation time, and the optimal segmentation strategy generation module is specifically used to: For each termination set, merge the optimal splitting strategies corresponding to all termination sets sorted before the termination set to generate an initial splitting strategy; Using the initial segmentation strategy as the current segmentation strategy; Splitting the termination set to generate a splitting strategy corresponding to the termination set, and merging the splitting strategy corresponding to the termination set with the initial splitting strategy to generate a new splitting strategy; Compare the running time of the new splitting strategy with that of the current splitting strategy based on the objective function to determine whether to update the current splitting strategy to the new splitting strategy; wherein the objective function is the running time of the neural network model in the heterogeneous platform based on the splitting strategy and the hardware resources of the heterogeneous platform for processing the neural network model; obtain the current number of iterations, and judge whether the current number of iterations reaches a preset number; if not, return to execute the steps of splitting the termination set, generating a splitting strategy corresponding to the termination set, and merging the splitting strategy corresponding to the termination set with the initial splitting strategy to generate a new splitting strategy and subsequent steps, until the current number of iterations reaches the preset number; if so, determine the optimal splitting strategy corresponding to the termination set based on the current splitting strategy and the initial splitting strategy; And / or the optimal segmentation strategy generation module is further specifically used to: Recursively partitioning the termination set to generate a plurality of termination set partitioning subgraphs; determining whether the number of the plurality of termination set partitioning subgraphs reaches the number of heterogeneous computing units in the heterogeneous platform; if so, generating a partitioning strategy corresponding to the termination set, and merging the partitioning strategy corresponding to the termination set with the initial partitioning strategy to generate a new partitioning strategy; if not, returning to executing the step of recursively partitioning the termination set to generate a plurality of termination set partitioning subgraphs and subsequent steps; And / or the optimal segmentation strategy generation module is further specifically used to: calculate the running time of the neural network model based on the new segmentation strategy in the heterogeneous platform, and the running time of the neural network model based on the current segmentation strategy in the heterogeneous platform; compare the running time corresponding to the new segmentation strategy with the running time corresponding to the current segmentation strategy; if the running time corresponding to the new segmentation strategy is less than the running time corresponding to the current segmentation strategy, then update the current segmentation strategy to the segmentation strategy; if the running time corresponding to the new segmentation strategy is greater than or equal to the running time corresponding to the current segmentation strategy, then use the probability The current segmentation strategy is updated to the new segmentation strategy, wherein: is the running time corresponding to the new segmentation strategy, The running time corresponding to the current segmentation strategy; And / or the optimal splitting strategy generation module is further specifically used to: derive a model of the termination set-cut molecular graph in the splitting strategy to obtain a plurality of termination set-cut molecular graph models; based on the hardware resources corresponding to the heterogeneous platform and the model parameters of the termination set-cut molecular graph model, allocate the termination set-cut molecular graph model to the corresponding heterogeneous computing unit; determine the scheduling order of the plurality of termination set-cut molecular graph models according to the dependency and parallel relationship between the termination set-cut molecular graphs in the splitting strategy; control the heterogeneous computing unit to which the termination set-cut molecular graph model is allocated to perform operations based on the scheduling order to obtain the running time of the neural network model in the heterogeneous platform based on the splitting strategy; wherein, the splitting strategy includes the new splitting strategy and the current splitting strategy; and / or the hardware resources include a heterogeneous computing unit topology; And / or the computational graph acquisition module is specifically used to: parse the loaded neural network model into a general format to obtain a neural network model in a general format; and generate a computational graph of the neural network model based on the neural network model in the general format.

10. A neural network model deployment device, characterized in that: The device includes: a memory, a processor, and a neural network model deployment program stored in the memory and executable on the processor, wherein the neural network model deployment program is configured to implement the steps of the neural network model deployment method according to any one of claims 1 to 7.

11. A storage medium, characterized in that: The storage medium stores a neural network model deployment program, which, when executed by a processor, implements the steps of the neural network model deployment method according to any one of claims 1 to 7.

Images