Neural network model adaptive division method for cloud edge collaborative reasoning

Abstract

The invention discloses a neural network model adaptive division method for cloud edge collaborative reasoning, and the method comprises the steps: constructing a directed acyclic graph of a to-be-divided model, and setting the weight of a connecting line between vertexes in the directed acyclic graph; obtaining a first division point from the directed acyclic graph of the to-be-divided model based on a weight of a connecting line between vertexes in the directed acyclic graph and a minimum cut algorithm; if a block structure exists in the nearest layer node behind the first division point, establishing a directed acyclic graph based on the block structure, and obtaining a second division point from the directed acyclic graph of the block structure based on a minimum cut algorithm; and selecting an optimal model layer division strategy in a cloud edge collaborative reasoning mode so as to minimize the end-to-end time delay. Compared with the prior art, the method has the advantages that the end-to-end delay of the model is reduced, and the reasoning speed of the AI model is increased. The accuracy of time delay prediction and the correctness of a subsequent division strategy are higher, and the method has better performance.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and specifically relates to a neural network model adaptive partitioning method for cloud-edge collaborative reasoning. Background Art

[0002] With the rapid development of IoT technology, smart applications such as smart wearable devices, smart assistants, and smart homes are becoming more and more popular due to their personalized and knowledge-based characteristics, but the latency requirements for tasks in smart scenarios are also more stringent. In the process of human-application interaction, flexible and efficient deep learning technology is needed as support. The most common method is to build a model with powerful analytical capabilities through DNN. In the traditional cloud computing model, due to the influence of network bandwidth and cloud load, under poor network conditions, a large amount of latency overhead is consumed during data transmission, which cannot meet the latency requirements of key DNN tasks. With the improvement of edge device performance, people have proposed edge computing mode, which writes calculations on edge devices to avoid latency overhead during data transmission. However, due to the high complexity of DNN networks, small, light-weight, and resource-constrained edge devices are not enough to complete the calculation process of the entire DNN model within the specified latency limit.

[0003] Therefore, in response to the above technical problems, it is necessary to provide an improved cloud-edge collaboration strategy for the neural network model.

[0004] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0005] The purpose of the present invention is to provide a neural network model adaptive partitioning method for cloud-edge collaborative reasoning, which can obtain the optimal neural network model layer partitioning strategy and minimize the end-to-end reasoning delay by changing the layer deployment method.

[0006] In order to achieve the above object, a technical solution provided by a specific embodiment of the present invention is as follows:

[0007] In a first aspect, the present invention provides a neural network model adaptive partitioning method for cloud-edge collaborative reasoning, which is applied to cloud-edge collaborative reasoning and includes:

[0008] Construct a directed acyclic graph of the model to be divided, and set the weights of the connections between the vertices in the directed acyclic graph based on the inference delay of each model layer on the edge device and the cloud device, the data communication delay between the model layers, and the inference delay of each model layer on the cloud device, wherein the vertices include the layer nodes, edge nodes, and cloud nodes of the model to be divided;

[0009] Based on the weights of the lines between the vertices in the directed acyclic graph and the minimum cut algorithm, obtaining a first partitioning point from the directed acyclic graph of the model to be partitioned;

[0010] If there is a block structure in the layer node closest to the first partition point, a directed acyclic graph is established based on the block structure, and a second partition point is obtained from the directed acyclic graph of the block structure based on a minimum cut algorithm;

[0011] The optimal model layer partitioning strategy is selected in the cloud-edge collaborative reasoning mode to minimize the end-to-end latency.

[0012] In one or more embodiments of the present invention, the method further comprises:

[0013] Collect model layer configuration information and the inference latency of the corresponding model layer on the edge device, the data communication latency between the adjacent next model layer, and the inference latency on the cloud device to form a training data set and a test data set;

[0014] The random forest model is trained by taking the model layer configuration information in the training data set as input parameters, and the inference delay of the corresponding model layer on the edge device, the data communication delay between the adjacent next model layer, and the inference delay on the cloud device as output parameters;

[0015] Based on the random forest model, the inference delay of each model layer on the edge device, the data communication delay between the model layers, and the inference delay of each model layer on the cloud device are obtained.

[0016] In one or more embodiments of the present invention, the acquisition model layer configuration information includes:

[0017] Collect the width of the input features corresponding to the convolution layer, the height of the input features, the size of the convolution kernel, the stride of the convolution layer, the number of channels of the input features and output features, and the number of floating-point operations;

[0018] Collect the width of the input feature corresponding to the DW convolution layer, the height of the input feature, the convolution kernel size, the stride of the convolution layer, the number of channels of the input feature, and the number of floating-point operations;

[0019] Collect the number of channels and floating-point operations corresponding to the input and output features of the fully connected layer.

[0020] In one or more embodiments of the present invention, the configuration information calculation formula includes:

[0021]

[0022] FLOPs3=2×(C in +1)×Cout

[0023] Among them HW out Indicates the height and width of the output feature, HW in represents the height and width of the input feature, K represents the convolution kernel size, P represents padding, S represents stride, and C in ,C out They represent the number of channels of input features and output features respectively, FLOPs1 is the number of floating-point operations of the convolutional layer, FLOPs2 is the number of floating-point operations of the DW convolutional layer, and FLOPs3 is the number of floating-point operations of the fully connected layer.

[0024] In one or more embodiments of the present invention, the step of constructing a directed acyclic graph of the model to be divided and setting the weights of the lines between vertices in the directed acyclic graph includes:

[0025] Construct edges from the edge node to each of the layer nodes, and set the weight to the inference latency of the corresponding layer on the cloud device;

[0026] Constructing edges from each of the layer nodes to the cloud node, and setting the weight to the inference latency of the corresponding layer on the edge device;

[0027] Add input nodes, construct edges from vertices with out-degree 1 to adjacent nodes in the next layer, and calculate the weights of the corresponding edges.

[0028] In one or more embodiments of the present invention, the weight calculation formula from the vertex with out-degree 1 to the adjacent next-layer node is:

[0029]

[0030] in, is the i-th model layer v i To the i+1th model layer v i+1 Data communication delay, For the model layer v i The optimal inference delay of the block structure, is the i-th model layer v i Inference latency on cloud devices.

[0031] In one or more embodiments of the present invention, the method further comprises:

[0032] Add a new node after the layer node with an out-degree greater than 1, and set the weight of the edge from the layer node with an out-degree greater than 1 to the new node;

[0033] Delete the edge from the layer node with out-degree greater than 1 to other layer nodes, and set the weight from the new node to the adjacent next layer node to inf.

[0034] In one or more embodiments of the present invention, the weight calculation formula of the edge from the layer node with out-degree greater than 1 to the new node is:

[0035]

[0036] in, is the data communication delay between the layer node with out-degree greater than 1 and its predecessor node, is the i-th model layer v i Inference latency on cloud devices, For the model layer v i The optimal inference delay of the block structure, is the i-th model layer v i To the i+1th model layer v i+1 Data communication delay, is the inference delay of the successor node of the layer node with out-degree greater than 1 on the cloud device, is a layer node v with out-degree greater than 1 i The weight of the edge to the new node.

[0037] In one or more embodiments of the present invention, the method further comprises:

[0038] Set the weight of the edge from the last layer node to the cloud device to inf;

[0039] Set the weight of the edge from the input node to the edge node to inf.

[0040] In one or more embodiments of the present invention, the step of obtaining the end-to-end delay for dividing the model layer based on each division point, and dividing the model layer based on the division point corresponding to the minimum end-to-end delay includes:

[0041] If the end-to-end delay when all layer nodes of the model to be divided are deployed on the edge device is greater than the end-to-end delay when the model layer is divided based on the second division point, the model is divided based on the second division point;

[0042] If the end-to-end delay when all layer nodes of the model to be divided are deployed on the edge device is less than or equal to the end-to-end delay when the model layer is divided based on the second division point, all layer nodes of the model to be divided are deployed on the edge device.

[0043] Compared with the prior art, the adaptive partitioning method of the neural network model for cloud-edge collaborative reasoning provided by the present invention targets the DNN model with block structure in the field of deep learning, reduces the end-to-end delay of the model, and improves the reasoning speed of the AI ​​model. Specifically, in the technical solution of the present invention, by dynamically setting the type of configuration information based on the attributes of each model layer, and then collecting the DNN hierarchical feature data set through relevant methods and using the random forest algorithm to build a hierarchical predictor, the accuracy of the delay prediction and the correctness of the subsequent partitioning strategy are higher. On the other hand, the present invention incorporates the block structure of the model layer in the neural network model into the solution of the optimal partitioning point, not only treating each layer of the model as a non-iterative single-layer structure, but also can find the optimal partitioning strategy hidden in the DNN block structure under the cloud-edge collaborative mode, reduce the end-to-end reasoning delay of the DNN model, improve the operating efficiency of AI tasks, and have better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 is a schematic diagram of the value range of each level configuration in one embodiment of the present invention;

[0046] Figure 2 It is a flowchart of a method for adaptive partitioning of a neural network model for cloud-edge collaborative reasoning in one embodiment of the present invention;

[0047] Figure 3 It is a structural block diagram of a neural network model adaptive partitioning system for cloud-edge collaborative reasoning in one embodiment of the present invention;

[0048] Figure 4 is a structural block diagram of an electronic device in one embodiment of the present invention;

[0049] Figure 5 It is an example of a directed acyclic graph of a model to be divided in one embodiment of the present invention. DETAILED DESCRIPTION

[0050] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation modes.

[0051] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.

[0052] In the existing technical solutions, there is a method of accelerating DNN reasoning by dividing the model layer during cloud-edge collaboration and then deploying the parts before and after the division point on the cloud-edge device for inference. Specifically, the method includes: establishing a model predictor by means of linear regression, treating each layer of the DNN model as a division point, so that the optimal division strategy can be selected in a way that is convenient for each division point.

[0053] However, due to the complex structure of the neural network model, the existing method's predictor is established through linear regression, and its prediction performance for latency is low, and it cannot reliably select the optimal partitioning strategy. In addition, the existing solution does not consider the data collection and data set construction on specific devices in detail, which makes its work in the deployment phase too complicated. Furthermore, with the rapid development of technologies related to neural network models, DNN models have become more and more complex. In order to more conveniently build deep networks, the same structure is often abstracted into a DNN block. The existing technology regards the model layer as a non-iterative single-layer structure to reason about the partition points, without considering the hidden partition points in the DNN block. With the continuous abstraction of the DNN structure, there are partition points inside the DNN block, which may further reduce the end-to-end inference latency.

[0054] The inventor of the present invention discovered the main shortcomings of the prior art, and based on the shortcomings of the prior art, proposed a new technical implementation idea. For complex DNN models with block structures, an adaptive partitioning framework that can be analyzed directly from the model level is designed. In the deployment stage, a model predictor composed of fine-grained kernel predictors is proposed, the specified layer type and configuration range are selected, and the prediction model is established using the random forest algorithm to ensure the correctness of the optimal strategy; in the partitioning stage, based on the model structure and bandwidth resources, the hidden partitioning points in the DNN block are analyzed, and a recursive DAG construction algorithm is proposed; in the collaborative reasoning stage, the optimal partitioning strategy is selected based on the constructed directed acyclic graph structure, and the DNN model is deployed on the edge device and the cloud device for collaborative reasoning.

[0055] Please refer to Figure 2 FIG. 1 is a flow chart of a method for adaptively partitioning a neural network model for cloud-edge collaborative reasoning in one embodiment of the present invention. The method for adaptively partitioning a neural network model for cloud-edge collaborative reasoning specifically includes the following steps:

[0056] S201: construct a directed acyclic graph of the model to be divided, and set the weights of the connections between the vertices in the directed acyclic graph based on the inference delay of each model layer on the edge device and the cloud device, the data communication delay between the model layers, and the inference delay of each model layer on the cloud device;

[0057] First of all, it should be noted that the neural network model is usually composed of multiple neural network layers, wherein each neuron layer receives the output of the previous layer as input, and calculates the output through a series of nonlinear transformations and weight adjustments. DNN is trained by a back-propagation algorithm, that is, by calculating the error between the predicted output and the actual output, and using the gradient descent method to update the weights and bias values ​​in the network until the network reaches a predetermined performance level. In an embodiment of the present invention, the model layer refers to the neuron layer in the neural network model, wherein the model layer may include but is not limited to a convolutional layer, a DW convolutional layer, a pooling layer, a fully connected layer, etc. The embodiment of the present invention is not limited to this.

[0058] It should also be noted that the above-mentioned directed acyclic graph refers to a directed graph that does not contain directed cycles, specifically composed of multiple vertices and directed edges, wherein the directed edge connects two different vertices, and the graph does not contain a cycle of repeated vertices and edges. It can be understood that the vertices include layer nodes, edge nodes, and cloud nodes of the model to be divided.

[0059] In an exemplary embodiment of the present invention, it is necessary to set the delay between different directed edges, then the inference delay of each model layer of the model to be divided on the edge device and the cloud device, the data communication delay between each model layer, and the inference delay of each model layer on the cloud device must be obtained. The present invention uses a training layer predictor to calculate the relevant delay data of each model layer in the model to be divided. The predictor is used to predict the inference delay of layer configurations and devices based on various core types. Furthermore, the machine learning algorithm used by the above-mentioned layer predictor is not limited by the embodiment of the present invention. It may include but is not limited to: random forest algorithm, decision tree algorithm, Bayesian classification algorithm, logistic regression algorithm, etc.

[0060] Preferably, in an exemplary embodiment of the present invention, the method for adaptive partitioning of neural network models for cloud-edge collaborative reasoning also includes: collecting model layer configuration information and the inference delay of the corresponding model layer on the edge device, the data communication delay between the adjacent next model layer, and the inference delay on the cloud device to form a training data set and a test data set; using the model layer configuration information in the training data set as input parameters, and the inference delay of the corresponding model layer on the edge device, the data communication delay between the adjacent next model layer, and the inference delay on the cloud device as output parameters to train the random forest model.

[0061] Random forests are composed of many independent decision trees (ranging from dozens to hundreds), which form the final prediction by summarizing the prediction results of all decision trees. The final result is obtained by voting or weighted average calculation of the predictions of all trees. By integrating multiple decision tree models, the overfitting problem can be effectively handled and the generalization ability of the model can be improved. Compared with other learning models, the realization of its functions is more in line with the application scenario of the present invention.

[0062] In the above embodiment, it should be noted that in the method for adaptive partitioning of neural network models for cloud-edge collaborative reasoning provided by the present invention, for different types of model layers, different configuration information is set based on their own properties to obtain better prediction results. Specifically, the acquisition of model layer configuration information includes: acquiring the width of the input features corresponding to the convolution layer, the height of the input features, the size of the convolution kernel, the stride of the convolution layer, the number of channels of the input features and the output features, and the number of floating-point operations; acquiring the width of the input features corresponding to the DW convolution layer, the height of the input features, the size of the convolution kernel, the stride of the convolution layer, the number of channels of the input features, and the number of floating-point operations; acquiring the number of channels of the input features and the output features corresponding to the fully connected layer, and the number of floating-point operations.

[0063] Furthermore, the step of constructing a data set is crucial. Appropriate data samples can improve the performance of the predictor. Therefore, when sampling data, commonly used configurations should be selected for sampling. For different configurations, the most common ranges are selected for sampling. Since the most common sampling ranges are taken into account, the prediction performance of the predictor can be improved as much as possible while reducing the deployment time. For example, in a specific embodiment of the present invention, if Figure 1 The figure shows the value range of each level configuration.

[0064] Among them, since most of the inference delay of the entire model is contributed by the convolutional layer and the fully connected layer, we introduced FLOPs to better predict inference. The FLOPs (floating point operations) refers to floating point operations, which are used to represent floating point operations and are understood as the amount of calculation. The inference of the convolutional layer usually uses inputs with the same height H and width W. Then, given the HW of the input feature, we can infer the HW properties of the output feature through the following formula:

[0065]

[0066] Among them HW out Indicates the height and width of the output feature, HW inrepresents the height and width of the input feature, K represents the kernel size, P represents padding, and S represents stride. In a neural network model, the height and width of the output feature are usually the same as those of the input feature, so no special distinction is made in the formula; if the height and width of the feature are inconsistent, the width and height of the input feature must be respectively substituted to calculate the width and height of the output feature.

[0067] Based on the above formula, the calculation of floating-point operations FLOPs in the ordinary convolution process can be obtained as follows:

[0068]

[0069] In the neural network model, when the convolution layer uses the MobileNet structure, it is called a DW convolution layer. Unlike the above-mentioned ordinary convolution layer, the FLOPs calculation formula of the DW convolution layer is as follows:

[0070]

[0071] In addition, the floating point operation number FLOPs of the fully connected layer is as follows:

[0072] FLOPs3=2×(C in +1)×C out

[0073] It should be noted that in the above four sets of formulas, HW out Indicates the height and width of the output feature, HW in represents the height and width of the input feature, K represents the convolution kernel size, P represents padding, S represents stride, and C in ,C out They represent the number of channels of input features and output features respectively, FLOPs1 is the number of floating-point operations of the convolutional layer, FLOPs2 is the number of floating-point operations of the DW convolutional layer, and FLOPs3 is the number of floating-point operations of the fully connected layer.

[0074] It should be noted that if Figure 5 The figure shows a directed acyclic graph of a model to be divided in a specific embodiment, and the dotted line part is a presumed division path. In the figure, v1-v8 are the model layers of the model to be divided, e is the edge device, and c is the cloud device. Based on the division strategy shown by the dotted line in the figure, it means that v1-v4 is deployed on the edge device for training, and v5-v8 is deployed on the cloud device for training. Further, v i to v i+1 The weight of the edge is related to the size of the layer v i To layer v i+1 The delay is related to For layer v i The weight of the edge to the cloud device node is proportional to the size of the layer v i The latency to the cloud device is related; For layer v i The weight of the edge to the edge device node is proportional to the size of the layer v i The latency to the edge device is related.

[0075] It is understandable that the ultimate goal of the solution is to minimize the end-to-end delay of the divided model. Therefore, if the model layer for inference on the edge device is defined as V R , which is the model layer that can be reached from the edge node e; the model layer for inference on the cloud device is defined as V NR , that is, the model layer that the edge node e cannot reach; the model split point is represented by P.

[0076] The cloud-edge collaborative approach to reasoning places part of the neural network model on edge devices for execution, with intermediate data transmitted over wireless networks. The other part of the model layer is executed on cloud devices, with the end-to-end delay of reasoning determined by T inf Indicates that T inf The edge inference latency of the model consists of three parts:

[0077]

[0078] The model’s cloud-based inference latency:

[0079]

[0080] Data communication delay required during cloud-edge collaboration:

[0081]

[0082] Furthermore, the required end-to-end delay is:

[0083] T inf =T E +T T +T C

[0084] In the above specific embodiment, in order to more quickly confirm the end-to-end delay under different partitioning strategies and then determine whether the current partitioning strategy is optimal, the weights of the directed edges in the directed acyclic graph that intersect with the dotted line representing the partitioning strategy and have the edge node / cloud node as the vertex can be summed and regarded as the above T T +T C .

[0085] It can be understood that in the above embodiment, the directed edge that intersects with the dotted line and has the cloud node as the vertex is connected to the layer that is divided to the edge node for deployment at the other end. Therefore, in this embodiment, in order to establish a partitioning strategy based on the directed acyclic graph of the model to be partitioned, the directed acyclic graph of the model to be partitioned is constructed, and the weights of the lines between the vertices in the directed acyclic graph are set, including: constructing an edge from the edge node to each of the layer nodes, and setting the weight to the inference delay of the corresponding layer on the cloud device; constructing an edge from each of the layer nodes to the cloud node, and setting the weight to the inference delay of the corresponding layer on the edge device.

[0086] Furthermore, the out-degree of nodes in different layers can be any positive integer. In an exemplary embodiment of the present invention, the weight of the edge from the vertex with out-degree 1 to the adjacent node in the next layer is set to Then we have:

[0087]

[0088] in, is the i-th model layer v i To the i+1th model layer v i+1 Data communication delay, For the model layer v i The optimal inference delay of the block structure, is the i-th model layer v i Inference latency on cloud devices.

[0089] It should be noted that when the next layer node adjacent to the layer node with out-degree tail 1 is a single-layer structure, it is obviously not necessary to further analyze the block structure, so the weight of the connection between the two can be directly set to the data communication delay between the two. When it is perceived that the next layer node adjacent to the next layer is not a single-layer structure, the weight of the directed edge between the two is set to the model layer v i The difference between the optimal inference delay of the block structure and the inference delay of the next model layer on the cloud device. During the calculation process, the inference delay of the next model layer on the cloud device is offset by the weight between the two, leaving only the optimal inference delay of the block structure, which is used to mark the next adjacent module here as a block structure. This facilitates the further calculation of the second partition point after the first partition point is confirmed.

[0090] On the other hand, for nodes with an out-degree greater than 1, since the path delays formed by the node and its successor nodes must be equal, and any node among the successor nodes receives the information sent by the node with an out-degree greater than 1, other successor nodes can use it. Therefore, for nodes with an out-degree greater than 1, only one transmission is required in actual situations. Furthermore, in the process of generating the partitioning strategy, the successor nodes must be uniformly deployed on cloud devices or edge devices, that is, the dotted line representing the partitioning strategy must also pass through the paths formed by the node with an out-degree greater than 1 and its successor nodes. In the subsequent delay calculation process, it is easy to cause multiple calculations of the same path delay.

[0091] Therefore, in an exemplary embodiment of the present invention, the adaptive partitioning method of the neural network model for cloud-edge collaborative reasoning also includes: adding a new node after the layer node with an out-degree greater than 1, and setting the weight of the edge from the layer node with an out-degree greater than 1 to the new node; deleting the edge from the layer node with an out-degree greater than 1 to other layer nodes, and setting the weight of the new node to the adjacent next layer node to inf. At this time, the new node replaces the original node with an out-degree of 1, and the weight of the directed edge between the new node and the aforementioned successor node is set to inf, so the partitioning strategy will not be divided here, avoiding the problem of repeated calculation of link delay.

[0092] Furthermore, the weight calculation formula of the edge from the layer node with out-degree greater than 1 to the new node is:

[0093]

[0094] in, is the data communication delay between the layer node with out-degree greater than 1 and its predecessor node, is the i-th model layer v i Inference latency on cloud devices, For the model layer v i The optimal inference delay of the block structure, is the i-th model layer v i To the i+1th model layer v i+1 Data communication delay, is the inference delay of the successor node of the layer node with out-degree greater than 1 on the cloud device, is a layer node v with out-degree greater than 1 i The weight of the edge to the new node.

[0095] It should also be noted that, considering that when determining the first partition point, there is a situation where the entire neural network model is deployed on a cloud device / edge device. Therefore, in the adaptive partitioning method of the neural network model for cloud-edge collaborative reasoning provided by the present invention, it is also necessary to add an edge from the edge node e to the newly added input node v0, with the weight set to inf, and add an edge from v0 to v1, with the weight set to inf indicates that the value is infinite. Set the value from the last layer v n The weight of the edge to cloud device c is inf. This step can ensure the correctness of the partitioning strategy and prevent the problem of not being able to confirm the first partitioning point, which makes the solution unable to proceed. In other words, when partitioning the neural network model block, it is necessary to avoid the submodules bypassing the data communication delay, resulting in the final partitioning strategy losing consideration of data communication.

[0096] S202: Based on the weights of the lines between the vertices in the directed acyclic graph of the model to be divided and a minimum cut algorithm, obtaining a first division point from the directed acyclic graph of the model to be divided;

[0097] It should be noted that the minimum cut maximum flow algorithm refers to the binary segmentation of the input point cloud in a directed graph, that is, dividing it into two groups: foreground points and background points. When the topological relationship between the points and the weights of the lines connecting the points are given, the minimum cut algorithm can be used to find the minimum cut. This minimum cut is the set of weights and minimum edges. The disconnection of these edges can just separate the foreground and background. That is, the minimum cut corresponds to the minimization of energy.

[0098] S203: If a block structure exists in the layer node closest to the first partition point, a directed acyclic graph is established based on the block structure, and a second partition point is obtained from the directed acyclic graph of the block structure based on a minimum cut algorithm;

[0099] It should be noted that the block structure in the model layer is not considered in the directed acyclic graph of the model to be divided, so the first division point is not the optimal division point. Based on the properties of the minimum cut algorithm, the minimum cut at the division point is the optimal division point when all model layers are regarded as non-iterative layers, but if the next model layer adjacent to the first division point is an iterative block structure, it is necessary to further obtain the optimal division point in the above block structure.

[0100] Specifically, a recursive method can be used to construct a directed acyclic graph for a block structure. That is, when the adjacent layers after the first division point contain a block structure, the block structure is treated as an independent submodule, and based on the above-mentioned directed acyclic graph construction strategy, a directed acyclic graph corresponding to the submodule is generated, and the weights of each directed edge in the graph are set. The submodule is further divided based on the minimum cut algorithm. If there is still a block structure after the division point in the above-mentioned submodule, refer to the above-mentioned method and recursively divide the block structure until the layer after the division point in the submodule is a non-iterative single-layer structure. At this time, the final division point is the second division point.

[0101] S204: Select the optimal model layer partitioning strategy in the cloud-edge collaborative reasoning mode to minimize the end-to-end delay.

[0102] It should be noted that in the adaptive partitioning method of the neural network model for cloud-edge collaborative reasoning provided by the present invention, the partitioning strategy obtained by the directed acyclic graph constructed above does not represent the completion of the model partitioning task. In model construction, in order to ensure that the first partitioning point can be obtained based on the directed acyclic graph of the model, the weight of the last layer for the cloud device is set to inf to avoid data loss and communication delay in recursive construction. But in fact, there is a situation where all model layers need to be deployed on edge devices to achieve minimum end-to-end delay.

[0103] Therefore, after partitioning the model through the above scheme, it is necessary to compare its end-to-end latency with the inference latency when the entire model is deployed on edge devices. DAG and P DAG They represent the optimal end-to-end delay and optimal partitioning strategy obtained through the above DAG, T opt and P opt The optimal end-to-end delay and optimal partitioning strategy of the representation model can be described by the following formula.

[0104]

[0105] Therefore, we can derive the optimal partitioning strategy in the cloud-edge collaborative reasoning mode, that is, the model layer partitioning strategy with the smallest end-to-end latency.

[0106] Please refer to Figure 3 As shown, based on the same inventive concept as the aforementioned method for adaptive partitioning of neural network models for cloud-edge collaborative reasoning, an embodiment of the present invention provides an adaptive partitioning system 300 for neural network models for cloud-edge collaborative reasoning, which includes: a construction module 301, a first partitioning module 302, a second partitioning module 303 and a selection module 304.

[0107] Specifically, the construction module 301 is used to construct a directed acyclic graph of the model to be divided, and set the weights of the connections between the vertices in the directed acyclic graph based on the inference delay of each model layer on the edge device and the cloud device, the data communication delay between the model layers, and the inference delay of each model layer on the cloud device, wherein the vertices include the layer nodes, edge nodes, and cloud nodes of the model to be divided;

[0108] The first partitioning module 302 is used to obtain a first partitioning point from the directed acyclic graph of the model to be partitioned based on the weights of the lines between the vertices in the directed acyclic graph of the model to be partitioned and a minimum cut algorithm;

[0109] The second partitioning module 303 is used for establishing a directed acyclic graph based on the block structure if there is a block structure in the layer node closest to the first partitioning point, and obtaining a second partitioning point from the directed acyclic graph of the block structure based on a minimum cut algorithm;

[0110] The selection module 304 is used to select the optimal model layer partitioning strategy in the cloud-edge collaborative reasoning mode to minimize the end-to-end delay.

[0111] Please refer to Figure 4 As shown, an embodiment of the present invention further provides an electronic device 400, which includes at least one processor 401, a memory 402 (e.g., a non-volatile memory), a memory 403, and a communication interface 404, and the at least one processor 401, the memory 402, the memory 403, and the communication interface 404 are connected together via a bus 405. At least one processor 401 is used to call at least one program instruction stored or encoded in the memory 402, so that the at least one processor 401 performs various operations and functions of the method for adaptive partitioning of a neural network model for cloud-edge collaborative reasoning described in various embodiments of this specification.

[0112] In an embodiment of the present specification, the electronic device 400 may include, but is not limited to, a personal computer, a server computer, a workstation, a desktop computer, a laptop computer, a notebook computer, a mobile electronic device, a smart phone, a tablet computer, a cellular phone, a personal digital assistant (PDA), a handheld device, a messaging device, a wearable electronic device, a consumer electronic device, and the like.

[0113] An embodiment of the present invention also provides a computer-readable medium, which carries computer execution instructions. When the computer execution instructions are executed by a processor, they can be used to implement various operations and functions of the adaptive partitioning method of the neural network model for cloud-edge collaborative reasoning described in the various embodiments of this specification.

[0114] The computer-readable medium in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0115] In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0116] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatuses, systems, and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0118] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A neural network model adaptive partitioning method for cloud-edge collaborative reasoning, applied to cloud-edge collaborative reasoning, characterized in that: include: Construct a directed acyclic graph of the model to be divided, and set the weights of the connections between the vertices in the directed acyclic graph based on the inference delay of each model layer on the edge device and the cloud device, the data communication delay between the model layers, and the inference delay of each model layer on the cloud device, wherein the vertices include the layer nodes, edge nodes, and cloud nodes of the model to be divided; Based on the weights of the lines between the vertices in the directed acyclic graph of the model to be divided and a minimum cut algorithm, obtaining a first dividing point from the directed acyclic graph of the model to be divided; If there is a block structure in the layer node closest to the first partition point, a directed acyclic graph is established based on the block structure, and a second partition point is obtained from the directed acyclic graph of the block structure based on a minimum cut algorithm; The optimal model layer partitioning strategy is selected in the cloud-edge collaborative reasoning mode to minimize the end-to-end latency.

2. The method for adaptive partitioning of neural network models for cloud-edge collaborative reasoning according to claim 1 is characterized in that: The method further comprises: Collect model layer configuration information and the inference latency of the corresponding model layer on the edge device, the data communication latency between the adjacent next model layer, and the inference latency on the cloud device to form a training data set and a test data set; The random forest model is trained by taking the model layer configuration information in the training data set as input parameters, and the inference delay of the corresponding model layer on the edge device, the data communication delay between the adjacent next model layer, and the inference delay on the cloud device as output parameters; Based on the random forest model, the inference delay of each model layer on the edge device, the data communication delay between the model layers, and the inference delay of each model layer on the cloud device are obtained.

3. The method for adaptive partitioning of neural network models for cloud-edge collaborative reasoning according to claim 2 is characterized in that: The acquisition model layer configuration information includes: Collect the width of the input features corresponding to the convolution layer, the height of the input features, the size of the convolution kernel, the stride of the convolution layer, the number of channels of the input features and output features, and the number of floating-point operations; Collect the width of the input feature corresponding to the DW convolution layer, the height of the input feature, the convolution kernel size, the stride of the convolution layer, the number of channels of the input feature, and the number of floating-point operations; Collect the number of channels and floating-point operations corresponding to the input and output features of the fully connected layer.

4. The method for adaptive partitioning of neural network models for cloud-edge collaborative reasoning according to claim 3 is characterized in that: The configuration information calculation formula includes: FLOPs3=2×(C in +1)×C out Among them, HW out Indicates the height and width of the output feature, HW in represents the height and width of the input feature, K represents the convolution kernel size, P represents padding, S represents stride, and C in ,C out They represent the number of channels of input features and output features respectively, FLOP1 is the number of floating-point operations of the convolutional layer, FLOPs2 is the number of floating-point operations of the DW convolutional layer, and FLOPs3 is the number of floating-point operations of the fully connected layer.

5. The method for adaptive partitioning of neural network models for cloud-edge collaborative reasoning according to claim 1 is characterized in that: The step of constructing a directed acyclic graph of the model to be divided and setting the weights of the lines between the vertices in the directed acyclic graph includes: Construct edges from the edge node to each of the layer nodes, and set the weight to the inference latency of the corresponding layer on the cloud device; Constructing edges from each of the layer nodes to the cloud node, and setting the weight to the inference latency of the corresponding layer on the edge device; Add input nodes, construct edges from vertices with out-degree 1 to adjacent nodes in the next layer, and calculate the weights of the corresponding edges.

6. The method for adaptive partitioning of a neural network model for cloud-edge collaborative reasoning according to claim 5 is characterized in that: The weight calculation formula from the vertex with out-degree 1 to the adjacent next-layer node is: in, is the i-th model layer v i To the i+1th model layer v i+1 Data communication delay, For the model layer v i The optimal inference delay of the block structure, is the i-th model layer v i Inference latency on cloud devices.

7. The method for adaptive partitioning of neural network models for cloud-edge collaborative reasoning according to claim 5, characterized in that: The method further comprises: Add a new node after the layer node with an out-degree greater than 1, and set the weight of the edge from the layer node with an out-degree greater than 1 to the new node; Delete the edge from the layer node with out-degree greater than 1 to other layer nodes, and set the weight from the new node to the adjacent next layer node to inf.

8. The method for adaptive partitioning of neural network models for cloud-edge collaborative reasoning according to claim 5, characterized in that: The weight calculation formula of the edge from the layer node with out-degree greater than 1 to the new node is: in, is the data communication delay between the layer node with out-degree greater than 1 and its predecessor node, is the i-th model layer v i Inference latency on cloud devices, For the model layer v i The optimal inference delay of the block structure, is the i-th model layer v i To the i+1th model layer v i+1 Data communication delay, is the inference delay of the successor node of the layer node with out-degree greater than 1 on the cloud device, is a layer node v with out-degree greater than 1 i The weight of the edge to the new node.

9. The method for adaptive partitioning of neural network models for cloud-edge collaborative reasoning according to claim 5, characterized in that: The method further comprises: Set the weight of the edge from the last layer node to the cloud device to inf; Set the weight of the edge from the input node to the edge node to inf.

10. The method for adaptive partitioning of neural network models for cloud-edge collaborative reasoning according to claim 1, characterized in that: The step of obtaining the end-to-end delay for dividing the model layer based on each division point, and dividing the model layer based on the division point corresponding to the minimum end-to-end delay includes: If the end-to-end delay when all layer nodes of the model to be divided are deployed on the edge device is greater than the end-to-end delay when the model layer is divided based on the second division point, the model is divided based on the second division point; If the end-to-end delay when all layer nodes of the model to be divided are deployed on the edge device is less than or equal to the end-to-end delay when the model layer is divided based on the second division point, all layer nodes of the model to be divided are deployed on the edge device.

Images