A neural network edge-cloud collaborative computing split deployment method

Abstract

The present invention provides a neural network edge-cloud collaborative computing split deployment method, including: establishing a delay prediction model, each delay prediction model is used to predict the computing delay of a basic neural network layer under a computing resource; determining the optimal split point of the neural network to be deployed, that is, the split point corresponding to the minimum overall delay; using the optimal split point to split the trained neural network to be deployed into a first and a second sub-neural network, and deploying the two on the edge device and the cloud server respectively for collaborative computing. The method of the present invention divides the neural network into two sub-neural networks and deploys them on the edge device and the cloud device respectively. Compared with pure edge computing, it can reduce computing sharing and improve computing efficiency, and compared with cloud computing, it can reduce the network transmission burden, thereby reducing the neural network application delay and improving the response speed; at the same time, the model is not compressed, so there is no loss of accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and edge computing, and specifically relates to a neural network edge-cloud collaborative computing split deployment method, which can be used for neural network application deployment under high load and low latency requirements. Background Art

[0002] With the rise of artificial intelligence and the popularity of neural network applications, more and more devices can support neural network applications. Various types of applications such as image classification, speech recognition, and natural language processing are becoming more and more popular in various types of user terminal devices. Faced with a large number of user requests and an increasingly heavy network transmission burden, how to increase the response speed of neural network applications and improve request processing capabilities are technical difficulties that need to be solved at present.

[0003] Neural network technology is an effective way to implement artificial intelligence applications. Current artificial intelligence applications often use a series of neural network layers to build models, so this model is also called a deep neural network. Each layer of a deep neural network is composed of neurons that can produce nonlinear outputs based on the input and output data of the neurons. The neurons in the input layer receive data and propagate it to the middle layer (also called the hidden layer). Then, the neurons in the middle layer generate a weighted sum of the input data and output the weighted sum using a specific activation function, and then propagate the output to the output layer. The final result is displayed in the output layer. Deep neural networks have more complex and abstract layers than general machine learning models, and are able to learn advanced features to achieve high-precision task reasoning. There are three common structures of deep neural networks: fully connected layers (FNN), convolutional neural network layers (CNN), and recurrent neural network layers (RNN).

[0004] Edge computing is an emerging research field that aims to push cloud services from the core of the network to network edge devices closer to the requester, such as network base stations, routers, etc. Its core purpose is to provide computing services to the requester while being as close to the requesting terminal as possible. Compared with cloud computing, edge computing can bring lower transmission delay and bandwidth consumption, but at the same time, the computing resources of edge computing are relatively limited. In order to better utilize the computing resources in the network and improve the response speed of neural networks. Now some scholars have begun to introduce edge computing in the process of neural network calculation.

[0005] In existing neural network application scenarios, neural network applications generally require relatively high computing power, and neural network models are often deployed on cloud servers. Users at the edge send the request data to the cloud through the network for inference, and then return the inference results to the user. This method is highly dependent on the network conditions between the requester and the cloud. If the amount of data on the requester is too large, or the network bandwidth is limited, the response speed of the request will be affected by the communication delay, which will ultimately lead to a significant increase in the data transmission delay and a long response time for the request, thereby reducing the response speed of the neural network application. Therefore, when the network bandwidth is limited, the network transmission delay overhead accounts for a higher proportion than the delay caused by cloud computing.

[0006] In order to reduce the overhead of network transmission delay, some studies have attempted to deploy neural networks directly on the edge, that is, network base stations, edge routers and other devices. However, the computing power of these edge devices is limited. If the computing resource requirements of the neural network model are too large or more complex, the edge devices will cause higher computing delays. To solve this problem, existing technologies are committed to compressing the model, reducing the computing resource requirements of the model through early termination, pruning, and reducing the calculation accuracy, but this technical means often leads to a decrease in the prediction accuracy of the model. Summary of the invention

[0007] The purpose of the present invention is to provide a neural network edge-cloud collaborative computing split deployment method to reduce the neural network application delay, improve the response speed, and ensure that there is no loss of accuracy.

[0008] In order to achieve the above object, the present invention provides a neural network edge-cloud collaborative computing segmentation deployment method, comprising:

[0009] S1: Determine the layer type of the basic neural network layer of the neural network to be deployed, and establish a delay prediction model f(x, S) for each computing resource and each basic neural network layer. Each delay prediction model is only used to predict the computing delay of a basic neural network layer under one computing resource;

[0010] S2: Execute a neural network segmentation algorithm to determine the optimal segmentation point of the neural network to be deployed, the optimal segmentation point refers to the segmentation point K corresponding to the minimum value of each overall delay T(K) when the segmentation point K is 1 to l-1, and the segmentation point is used to segment the neural network to be deployed into a first sub-neural network and a second sub-neural network; l is the number of neural network layers in the neural network to be deployed;

[0011] S3: Use the optimal split point to split the trained neural network to be deployed into a first sub-neural network and a second sub-neural network, deploy the first sub-neural network on the edge device, and deploy the second sub-neural network on the cloud server for collaborative computing.

[0012] The input parameters of the delay prediction model f(x,S) are x and S, where x is the feature vector corresponding to the neural network layer, the feature vector has one or more feature variables, and S is the type of computing resource. The type of computing resource S includes the edge type S cloud and Cloud Type S cloud .

[0013] In step S1, for all types of feature variables corresponding to each basic neural network layer, a delay prediction model is established and trained in the form of a regression tree.

[0014] The step S1 comprises: performing the following steps for each computing resource and each basic neural network layer:

[0015] S10: Obtain all characteristic variables of a basic neural network layer and samples of the computational delay of the neural network layer under a computing resource to form a sample data set D of a delay prediction model;

[0016] S11: For one of the characteristic variables x i , set the current split point s so that the sample data set D of the delay prediction model is within the feature variable x i The dimension is divided into two sub-sample spaces, namely the first sub-sample space and the second sub-sample space;

[0017] S12: Determine the output prediction value y of the sample corresponding to the current segmentation point s predict and prediction error e;

[0018] S13: continuously update the current segmentation point s and repeat the above step S12 to determine the segmentation point s that minimizes the value of the prediction error e as the optimal segmentation point.

[0019] In step S11, the first sub-sample space R1(s)={x|x i ≤s}, the second subsample space R2(s)={x|x i >s}, x i is the i-th characteristic variable;

[0020] In step S12, the predicted output value y of the delay prediction model predict for;

[0021]

[0022] In the formula, x k is the characteristic variable of the kth sample, R1 is the first sub-sample space, R2 is the second sub-sample space, c1 is the predicted output value of the delay prediction model in the first sub-sample space, c2 is the predicted output value of the delay prediction model in the second sub-sample space; N1 is the number of samples in the first sub-sample space, and N2 is the number of samples in the second sub-sample space;

[0023] The prediction error e corresponding to the current segmentation point s is:

[0024]

[0025] Among them, y k is the computation delay of the kth sample.

[0026] The feature vector corresponding to the basic neural network layer has multiple feature variables;

[0027] The step S1 further comprises:

[0028] S14: Repeat the above steps S11 to S13 for another characteristic variable until the optimal splitting points corresponding to all characteristic variables are determined;

[0029] S15: Determine the prediction output value of the delay prediction model in each sample area according to each sample area divided by the optimal segmentation point.

[0030] In step S15, the predicted output value of the delay prediction model f(x, S) is:

[0031]

[0032] in,

[0033] Among them, x is the feature vector corresponding to the neural network layer, R m is the mth sample area, m is the number of the sample area, n represents the total number of types of feature variables, c m is the predicted output value of the delay prediction model in the mth sample area.

[0034] In step S2, the overall delay T(K) when the segmentation point is K is determined according to the first sub-neural network and the second sub-neural network segmented when the segmentation point is K, and the calculation delay T of the first sub-neural network at the edge is determined. e (K) and the computing delay T of the second sub-neural network in the cloud c (K), and combined with the amount of data D output by the Kth neural network layer K And the network bandwidth B between the edge and the cloud is determined.

[0035] In step S2, the neural network segmentation algorithm is based on a global search of prefix and array;

[0036] The step S2 comprises:

[0037] S21: For each neural network layer of the neural network to be deployed, determine the computational delay f(x(l p ),S edge ) and the computational latency of the neural network layer in the cloud f(x(l p ),S cloud );l p is the type of the basic neural network layer corresponding to the pth neural network layer; p is the layer number of the neural network layer, p is 1…l;

[0038] S22: for each neural network layer of the neural network to be deployed, respectively determine the prefix sum array prefixEdge[p] of the neural network layer at the edge and the prefix sum array prefixCloud[p] of the neural network layer at the cloud;

[0039] S23: For each segmentation point K, determine the computational delay T of the edge end when the segmentation point is K. e (K), the computing delay of the cloud when the split point is K is T c (K) and the transmission delay when the split point is K, and sum these three to get the overall delay T(K) when the split point is K;

[0040] S24: Traverse the segmentation points K in the range of 1 to l-1, and take the segmentation point K corresponding to the minimum value in the overall delay T(K) as the optimal segmentation point K best .

[0041] prefixEdge[p]=prefixEdge[p-1]+f(x(l p ),S edge );

[0042] prefixCloud[p]=prefixCloud[p-1]+f(x(l p ),S cloud );

[0043] T e (K)=prefixEdge[K]-prefixEdge[0];

[0044] T c (K)=prefixEdge[l]-prefixEdge[K];

[0045] The transmission delay when the split point is K is Among them, D K is the amount of data output by the Kth neural network layer, and B is the network bandwidth between the edge and the cloud.

[0046] The neural network edge-cloud collaborative computing split deployment method of the present invention proposes neural network split computing, which divides the neural network into two sub-neural networks, deploys them on edge devices and cloud devices respectively, and coordinates the computing methods of the edge and cloud. The edge device only calculates the split sub-model, which can reduce the computing burden and improve computing efficiency compared to pure edge computing, and can reduce the network transmission burden compared to cloud computing, thereby reducing the neural network application delay and improving the response speed; at the same time, the neural network is not compressed, so there is no loss of accuracy. As a result, the neural network application delay is reduced, the response speed is improved, and the accuracy is not lost, and the low latency requirement of the neural network application is achieved in the scenario of limited network bandwidth.

[0047] In addition, the neural network edge-cloud collaborative computing split deployment method of the present invention obtains the running time of the neural network by establishing a delay prediction model, and the neural network split algorithm is based on the delay prediction model and prefix and array design, thereby improving the running speed of the neural network edge-cloud collaborative computing split deployment method of the present invention itself. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the overall process of the neural network edge-cloud collaborative computing split deployment method of the present invention.

[0049] Figure 2 It is a flow chart of step S3 of the neural network edge-cloud collaborative computing split deployment method of the present invention.

[0050] Figure 3 It is a schematic diagram of the segmentation point selection of LeNet5, AlexNet and VGG networks by the neural network edge-cloud collaborative computing segmentation deployment method of the present invention under the condition that the network bandwidth is 500KB.

[0051] Figure 4 This is a performance comparison chart of step S2 of the neural network edge-cloud collaborative computing segmentation deployment method of the present invention based on global search and prefix and global search.

[0052] Figure 5 This is a comparison chart of the task response time results under the three computing modes of edge-cloud collaborative computing, cloud computing, and edge computing. DETAILED DESCRIPTION

[0053] The present invention is further described below in conjunction with specific examples. It should be understood that the following examples are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0054] The present invention proposes a neural network edge-cloud collaborative computing split deployment method, which is suitable for neural network application scenarios with low latency requirements and high response speed requirements.

[0055] like Figure 1 As shown, the neural network edge-cloud collaborative computing segmentation deployment method of the present invention includes:

[0056] Step S1: Determine the layer type of the basic neural network layer of the neural network to be deployed, and establish a delay prediction model f(x, S) for each computing resource and each basic neural network layer. Each delay prediction model is only used to predict the computing delay of a basic neural network layer under one computing resource;

[0057] The neural network to be deployed is a convolutional neural network, such as LeNet5, AlexNet, and VGG. However, the delay prediction model does not care about the specific neural network, but only cares about what types of basic neural network layers the neural network to be predicted is composed of. Therefore, in theory, the neural network to be deployed can be arbitrary, and any neural network is composed of multiple basic neural network layers. According to the model established for each basic neural network layer of the neural network layer to be deployed in step S1, in the subsequent steps, as long as the calculation delays predicted by the delay prediction models of all the basic neural network layers of the neural network to be deployed are added, the total calculation delay of the neural network to be deployed can be obtained.

[0058] For any neural network, the tasks to be processed are also different. In this embodiment, the neural network to be deployed is a convolutional neural network, such as LeNet5, AlexNet, and VGG. These neural networks to be deployed input image-type data and output classification results. Commonly used basic neural network layers can be FNN, CNN or RNN, but in actual modeling, the basic neural network layer can also be ReLU or dropout, etc.

[0059] The input parameters of the delay prediction model f(x,S) are x and S, where x is the feature vector corresponding to the neural network layer and S is the type of computing resource. In this embodiment, the type of computing resource S includes the edge type S cloud and Cloud Type S cloud .

[0060] The purpose of step S1 is to obtain the predicted value of the calculation delay y through the vector x composed of all types of feature variables corresponding to the type of basic neural network layer, that is, to establish a delay prediction model f(x, S) so that the output of the delay prediction model f(x, S) is as close as possible to the actual calculation delay of the basic neural network layer.

[0061] Different types of basic neural network layers have different factors that affect the computation time, which are called characteristic variables. The characteristic variables of some commonly used basic neural network layers are shown in Table 1 below.

[0062] Table 1 Basic types of neural network layers and their corresponding feature variables

[0063]

[0064] As shown in Table 1, the feature variables are determined according to the type of the corresponding basic neural network layer, and different types of neural network layers correspond to different multiple feature variables.

[0065] All characteristic variables corresponding to the basic neural network layer type are expressed in the form of vectors, which are recorded as the characteristic vector x corresponding to the neural network layer = {x 1 ,x 2 ,..,x i ,..,x n}, where x i is the ith feature variable, and n is the total number of feature variables corresponding to the basic neural network layer type. For example, the only factor that affects the activation layer calculation delay is the data size input to the layer, so the input data size is the feature variable corresponding to the activation layer, and there is only this one feature variable, so n = 1; similarly, the convolution layer is affected by 6 factors, so the feature variables corresponding to the convolution layer are 6, n = 6.

[0066] In order to establish the delay prediction model f(x,S), we need to collect samples of the same basic neural network layer and its calculation delay as the sample data set of the delay prediction model. Assuming that we have collected N samples of the same basic neural network layer under one computing resource (i.e., N basic neural network layers with different parameters of feature variables), we can get the corresponding sample data set D of the delay prediction model, which is:

[0067]

[0068] in, represents the kth sample, is the i-th characteristic variable corresponding to the k-th sample, The subscript of represents the sample ordinal number, and the superscript represents the ordinal number of the variable; k is the calculation delay of the kth sample, that is, the output of the model; N represents the total number of samples in the sample data set; n represents the total number of types of feature variables.

[0069] To simplify the explanation, assume that the feature vector of the basic neural network layer has only one feature variable, then the kth sample is represented as (x k ,y k ), x k is the characteristic variable of the kth sample, y k is the computation delay of the kth sample.

[0070] In step S1, for all types of feature variables corresponding to each basic neural network layer, a time delay prediction model is established and trained in the form of a regression tree, wherein the regression tree is a machine learning model.

[0071] Specifically, step S1 includes: performing the following steps for each computing resource and each basic neural network layer:

[0072] Step S10: Obtain all characteristic variables of a basic neural network layer and samples of the computational delay of the neural network layer under a computing resource to form a sample data set D of a delay prediction model;

[0073] That is to say, the samples collected at the edge are used as the delay prediction model f(x,S edge ) is used as a sample dataset D for training, and the samples collected in the cloud are used as the latency prediction model f(x,S cloud ) for training.

[0074] As mentioned above, the sample data set D of the delay prediction model is:

[0075]

[0076] in, represents the kth sample, is the i-th characteristic variable corresponding to the k-th sample, The subscript of represents the sample ordinal number, and the superscript represents the ordinal number of the variable; k is the calculation delay of the kth sample, that is, the output of the delay prediction model; N represents the total number of samples in the sample data set; n represents the total number of types of feature variables.

[0077] Step S11: For one of the characteristic variables x i , set the current split point s so that the sample data set D of the delay prediction model is within the feature variable xi The dimension is divided into two sub-sample spaces, namely the first sub-sample space R1(s) (also expressed as R1) and the second sub-sample space R2(s) (also expressed as R2);

[0078] R1(s)={x|x i ≤s},R2(s)={x|x i >s}, x i is the i-th characteristic variable.

[0079] The range of the ordinal number i of the characteristic variable is 1-n, where n represents the total number of characteristic variable types; the range of the split point s is between, is the i-th characteristic variable corresponding to the k-th sample.

[0080] Therefore, the 1st sample to the Nth sample of the sample data set D of the delay prediction model will be divided into two sub-sample spaces due to the split point s, which are respectively recorded as the first sub-sample space R1(s) and the second sub-sample space R2(s). The number of samples in these two sub-sample spaces R1(s) and R2(s) can be recorded as N1 and N2 respectively. It is easy to see that N1+N2=N.

[0081] Step S12: Determine the output prediction value y of the sample corresponding to the current segmentation point s predict and prediction error e;

[0082] Through sample training, we can get a mapping y predict ←f(x,S). In this embodiment, the mean of the true output values ​​y of all samples in each sub-sample space is used as the predicted output value y of the delay prediction model. predict .

[0083] That is to say, the predicted output value y of the delay prediction model predict for;

[0084]

[0085]

[0086] In the formula, x k is the characteristic variable of the kth sample, R1 is the first sub-sample space, R2 is the second sub-sample space, c1 is the predicted output value of the delay prediction model in the first sub-sample space, c2 is the predicted output value of the delay prediction model in the second sub-sample space; N1 is the number of samples in the first sub-sample space, and N2 is the number of samples in the second sub-sample space.

[0087] Since we hope that when the feature variable is x, the corresponding sample's actual output value y and the predicted output value y of the delay prediction model predict The prediction error is as small as possible, and the prediction error e needs to be evaluated to obtain the minimum prediction error. Different machine learning models have different methods for evaluating prediction errors. The prediction error e in this embodiment is calculated using the error evaluation method corresponding to the regression tree model.

[0088] Therefore, the prediction error e corresponding to the current segmentation point s is:

[0089]

[0090] Among them, R1 is the first sub-sample space, R2 is the second sub-sample space, x k is the characteristic variable of the kth sample, y k is the calculated delay of the kth sample, c1 is the predicted output value of the delay prediction model in the first sub-sample space, and c2 is the predicted output value of the delay prediction model in the second sub-sample space.

[0091] Step S13: continuously update the current segmentation point s and repeat the above step S12 to determine the segmentation point s that minimizes the value of the prediction error e as the optimal segmentation point.

[0092] At this time, if the feature vector corresponding to the basic neural network layer has only one feature variable, the above-mentioned steps S11-S13 only need to be performed once to select a segmentation point for one feature variable, and the sample space is finally divided into 2 sample areas (i.e., 2 sub-sample spaces).

[0093] It should be noted that in this embodiment and in other embodiments, if the feature vector corresponding to the basic neural network layer has multiple feature variables, the above operation will be performed n times in total. That is, step S1 also needs to include the following steps:

[0094] Step S14: Repeat the above steps S11 to S13 for another characteristic variable until the optimal splitting points corresponding to all characteristic variables are determined.

[0095] Because each sample has n characteristic variables, each characteristic variable will be divided into 2 sub-sample spaces by the split point. Therefore, the sample space is finally divided into 2n sample areas by the optimal split point, where n represents the total number of types of characteristic variables.

[0096] Step S15: According to each sample area divided by the optimal segmentation point, the predicted output value of the delay prediction model in each sample area is determined, thereby completing the establishment of the delay prediction model (ie, the decision tree).

[0097] Therefore, as long as the feature vector of a certain basic neural network layer is given, the predicted value of its calculated delay is the predicted value of the delay prediction model.

[0098] Among them, the predicted output value of the delay prediction model f(x,S) is:

[0099]

[0100] in,

[0101] Among them, x is the feature vector corresponding to the neural network layer, R m is the mth sample area, m is the number of the sample area, there are 2n sample areas in total, so m takes a value of 1-2n, n represents the total number of types of feature variables, c m is the predicted output value of the delay prediction model in the mth sample area. m Take the mean of the true output values ​​y of all samples in the mth sample area.

[0102] At this point, the establishment of the binary tree-type delay prediction model (ie, decision tree) is completed. In step S1, the delay prediction model we established is used for delay prediction, and the delay prediction model f(x) is the predicted value of the computational delay of the neural network layer.

[0103] As described in step S1, each delay prediction model is only used to predict the computing delay of a basic neural network layer under a computing resource. Therefore, for the same basic neural network layer, its delay prediction model includes the delay prediction model f(x, S edge ) ) corresponding to the computational delay of the neural network layer when it is computed at the edge, and the latency prediction model f(x,S cloud ) (corresponding to the computational latency of this neural network layer when it is computed in the cloud).

[0104] If the neural network to be deployed has 5 neural network layers, it is necessary to determine the type of basic neural network layer corresponding to each of the 5 neural network layers. Then, it is necessary to perform the above steps S10 to S14 twice for each basic neural network layer (once corresponding to the edge and once corresponding to the cloud).

[0105] Step S2: Execute the neural network segmentation algorithm to determine the optimal segmentation point of the neural network to be deployed. The optimal segmentation point refers to the segmentation point K corresponding to the minimum value of each overall delay T(K) when the segmentation point K is 1 to l-1. The segmentation point is used to divide the neural network to be deployed into two sub-neural networks, namely the first sub-neural network and the second sub-neural network. l is the number of neural network layers in the neural network to be deployed.

[0106] The split point here refers to the split point of a specific neural network, and has nothing to do with the split point and optimal split point mentioned above. For example, if the neural network to be deployed has 100 layers and the split point is the 50th layer, then the first sub-neural network (i.e. the first 50 layers) is deployed on the edge, and the second sub-neural network (i.e. the last 50 layers) is deployed on the cloud. The first sub-neural network before the split point is used to be deployed on the edge, and the second sub-neural network after the split point is used to be deployed on the cloud.

[0107] The neural network to be deployed is denoted as L, then the input information of the model segmentation algorithm is the neural network to be deployed L = {l1,l2,…l p ,…,l l}, l p Represents the type of basic neural network layer corresponding to the pth neural network layer of the neural network to be deployed. The neural network L to be deployed has a total of l neural network layers, where p represents the layer number of the neural network layer, p = 1 ~ l. Each basic neural network layer of the neural network L has only one corresponding feature vector (the feature vector contains multiple feature variables), that is, the type of basic neural network layer corresponding to the pth neural network layer is l p and the eigenvector x(l p ) There is a one-to-one correspondence between the two. p represents the layer number of the neural network layer. The layer type l of each neural network layer p Corresponding to a feature vector x(l p ), the eigenvector x(l p ) is used as input to the delay prediction model.

[0108] A specific example is as follows: Assume that the type of the basic neural network layer l1 corresponding to the first neural network layer is an activation layer, corresponding to one feature variable, then its feature vector x(1) = (x 1 ); Assuming that the type of basic neural network layer l2 corresponding to the second neural network layer is a convolutional layer, corresponding to 6 feature variables, then its feature vector x(2) = (x 1 ,x 2 ,…x 6 ).

[0109] In step S2, the overall delay T(K) when the segmentation point is K is determined according to the first sub-neural network and the second sub-neural network segmented when the segmentation point is K, and the calculation delay T of the first sub-neural network at the edge is determined. e (K) and the computing delay T of the second sub-neural network in the cloud c (K), and combined with the amount of data D output by the Kth neural network layer K And the network bandwidth B between the edge and the cloud is determined.

[0110] For a neural network model with l layers, there can be a total of l-1 split points. When the split point is K, the two sub-neural networks are divided into edge terminal neural network and and cloud-based neural networks Among them, edge terminal neural network and cloud-based neural networks They are:

[0111]

[0112] The delay prediction model at the edge is recorded as f(x,S edge ), and the latency prediction model in the cloud is recorded as f(x,S cloud ), then, when the split point is K, the computation delay at the edge is T e (K) and the computing latency T in the cloud c (K) can be expressed as:

[0113]

[0114] Among them, l p is the type of the basic neural network layer corresponding to the pth neural network layer, x(l p ) represents the type of basic neural network layer corresponding to the pth neural network layer l p The corresponding feature vector, p is the layer number of the basic neural network layer, and K is the segmentation point.

[0115] At the same time, the amount of data output by the Kth neural network layer is recorded as D K , the network bandwidth between the edge and the cloud is B, and the overall delay T(K) when the split point is K can be obtained. The overall delay T(K) when the split point is K is:

[0116]

[0117] Among them, D K is the amount of data output by the Kth neural network layer, B is the network bandwidth between the edge and the cloud, and T e (K) is the computation delay at the edge when the segmentation point is K, T c (K) is the computing latency of the cloud when the segmentation point is K.

[0118] The segmentation point K ranges from 1 to l-1 because for a neural network model with l layers, there can be a total of l-1 segmentation points.

[0119] In other words, the determination of the optimal split point is an optimization problem:

[0120] argmin{(K,T(K)):K=1,2,…,l-1}.

[0121] This optimization problem can be solved by global search, that is, by traversing all the overall delays T(K) when the segmentation point K is 1 to l-1 to determine the minimum value of the overall delays T(K) when the segmentation point K is 1 to l-1, and then selecting the segmentation point K corresponding to the minimum value of the overall delays T(K).

[0122] However, because the sub-neural network needs to be summed repeatedly, there will be additional time overhead, which ultimately makes the search time complexity o(n 2 ). Therefore, in order to further optimize the processing speed of the algorithm, in this embodiment, step S22 introduces the prefix and array for optimization based on the global search, and finally reduces the time complexity to o(n 2 ).

[0123] Therefore, in step S2, the neural network segmentation algorithm finally designed is based on a global search of the prefix sum array.

[0124] The step S2 specifically includes:

[0125] Step S21: for each neural network layer of the neural network to be deployed, determine the computational delay f(x(l p ),S edge ) and the computational latency of the neural network layer in the cloud f(x(l p ),S cloud );l p is the type of the basic neural network layer corresponding to the pth neural network layer; p is the layer number of the neural network layer, p is 1…l;

[0126] Step S22: for each neural network layer of the neural network to be deployed, respectively determine the prefix sum array prefixEdge[p] of the neural network layer at the edge and the prefix sum array prefixCloud[p] of the neural network layer at the cloud;

[0127] Where prefixEdge[p]=prefixEdge[p-1]+f(x(l p ),S edge ),

[0128] prefixCloud[p]=prefixCloud[p-1]+f(x(l p ),S cloud ).

[0129] prefixEdge and prefixCloud represent the prefix and array of edge and cloud respectively.

[0130] The prefix sum array is defined to optimize the summation time in the algorithm. The specific example is:

[0131] Assuming there are 4 layers of neural network, their calculation delays obtained by calculating f(x) are:

[0132] T = [2, 3, 1, 4]

[0133] Then its prefix sum array is defined by the following recursive formula

[0134] prefixT[i]=T[i]+prefixT[i-1]

[0135] According to the recursive formula, we can get:

[0136] prefixT = [0, 2, 5, 6, 9]

[0137] Its advantage is that it converts the sum into difference, which is an optimization method and has no special meaning in itself.

[0138] Step S23: For each segmentation point K, determine the computational delay T of the edge end when the segmentation point is K. e (K), the computing delay of the cloud when the split point is K is T c (K) and the transmission delay when the split point is K, and sum these three to get the overall delay T(K) when the split point is K;

[0139] Among them, the computation delay at the edge when the split point is K is T e (K)(i.e. T edge )for:

[0140] T e (K)=prefixEdge[K]-prefixEdge[0];

[0141] The computing delay T in the cloud when the split point is K c (K)(i.e. T cloud )for:

[0142] T c (K)=prefixEdge[l]-prefixEdge[K];

[0143] The transmission delay when the split point is K (i.e. T comm )for Among them, D K is the amount of data output by the Kth neural network layer, and B is the network bandwidth between the edge and the cloud.

[0144] Step S24: Traverse the segmentation points K in the range of 1 to l-1, and take the segmentation point K corresponding to the minimum value in the overall delay T(K) as the optimal segmentation point K best .

[0145] That is, the input parameters of the neural network segmentation algorithm include:

[0146] {l p |p=1…l}:l p is the type of basic neural network layer corresponding to the pth neural network layer;

[0147] {D K |K=1…N}:D K is the amount of data output by the Kth neural network layer;

[0148] f(x,S): delay prediction model;

[0149] B: Network bandwidth between the edge and the cloud; and

[0150] T best : The current optimal overall delay.

[0151] The output parameters of the neural network segmentation algorithm include the optimal segmentation point K best .

[0152] Some of the code is as follows:

[0153] 1.forpin range 1…l:

[0154] 2.prefixEdge[p]=prefixEdge[p-1]+f(x(l p ),S edge )

[0155] 3.prefixCloud[p]=prefixCloud[p-1]+f(x(l p ),S cloud )

[0156] 4.for K in range 1…l-1:

[0157] 5.T edge =prefixEdge[K]-prefixEdge[0]

[0158] 6.T cloud =prefixEdge[l]-prefixEdge[K]

[0159] 7.

[0160] 8.if T edge +T colud +T comm <T best :

[0161] 9.T best =T edge +T colud +T comm

[0162] 10.K best =K

[0163] 11.return K best

[0164] Step S3: Use the optimal segmentation point to segment the trained neural network to be deployed into a first sub-neural network and a second sub-neural network, deploy the first sub-neural network on the edge device, and deploy the second sub-neural network on the cloud server for collaborative computing.

[0165] Among them, the sub-neural network before the split point is the first sub-neural network, which is used to be deployed on the edge; the sub-neural network after the split point is the second sub-neural network, which is used to be deployed on the cloud.

[0166] The overall collaborative process of step S3 can be found in Figure 2 For the segmented and deployed neural network model, the user initiates a request to the edge device and transmits the request data. The request data is first calculated on the edge device for the first sub-neural network, and then the intermediate calculation results are sent to the cloud server for the second sub-neural network calculation, thereby realizing edge-cloud collaborative computing in turn.

[0167] In practice, edge devices can be base stations, routers, edge gateways, and other edge devices close to users. When a user's terminal device sends a request, the request data first passes through the edge device, which then performs preliminary processing on the request data through the sub-network and transmits the preliminary processing results to the cloud for subsequent calculations.

[0168] In summary, the edge cloud collaborative computing process can be summarized as follows:

[0169] 1. For the neural network to be deployed, obtain the model's hierarchical information; based on the model's hierarchical information, establish a latency prediction model. If the prediction model has been established before, proceed directly to step 2.

[0170] 2. According to the time delay prediction model, the optimal segmentation point is obtained through the neural network segmentation algorithm, and the optimal segmentation point is used to segment the neural network into a first sub-neural network and a second sub-neural network at the segmentation point.

[0171] 3. The first sub-neural network is deployed on the edge device requested by the user business, and the second sub-neural network is deployed on the cloud server, and finally collaborative computing is performed.

[0172] Experimental Results

[0173] We used LeNet5, AlexNet and VGG networks to verify the implementation of the neural network edge-cloud collaborative computing segmentation deployment method of the present invention.

[0174] First, for the layer types of each neural network layer of the neural network we used, we selected feature variables according to the following table to establish the delay prediction model. The feature variable selection of the layer type can be seen in the following table:

[0175]

[0176] Subsequently, the delay prediction model is used to determine the neural network split point. The determination of the split point depends on the network bandwidth of the edge and cloud. When the network bandwidth is poor, the split point tends to be split at the later layers of the neural network, so that most of the neural network layers can be placed on the edge for calculation to reduce network overhead. When the network bandwidth is good, the split point tends to be split at the front layer to maximize the use of computing resources. Figure 3 The schematic diagram of segmentation point selection for LeNet5, AlexNet and VGG networks by the neural network edge-cloud collaborative computing segmentation deployment method of the present invention is shown under the condition of a network bandwidth of 500KB. It can be seen that the segmentation point is located on the neural network layer with a small output data volume and as close to the front as possible, which is in line with expectations.

[0177] Figure 4 This is a performance improvement comparison chart based on global search and prefix and global search in step S2 of the neural network edge-cloud collaborative computing segmentation deployment method of the present invention. The left subgraph is the change in algorithm complexity in theory as the problem scale increases, and the right subgraph is the time consumed by the algorithm in practice as the number of neural network layers processed increases.

[0178] It can be seen that the global search algorithm after prefix and array optimization can bring better processing efficiency, and the actual improvement is consistent with the theoretical improvement.

[0179] After determining the split point, the neural network before the split point is deployed on the edge device, and the neural network after the split point is deployed on the cloud. In order to visually observe the improvement of edge-cloud collaborative computing compared with cloud computing and edge computing as the network bandwidth changes, we set the bandwidth between the edge and the cloud to 0.1-10MB, and counted the task response time under the three computing modes. The results can be seen in Figure 5 .

[0180] It can be seen that when the network bandwidth is poor, cloud computing is limited by communication delays, resulting in a slow response speed, so it is far inferior to edge computing and edge cloud collaborative computing. Edge computing will cause large computing delays when the neural network is complex due to the limited computing power of edge devices. This is particularly obvious in the VGG network. Figure 5 From the results in , we can see that edge-cloud collaborative computing is a computing model that can take into account the advantages of both edge computing and cloud computing, and can bring faster response speed to neural network applications when network bandwidth is limited.

[0181] The neural network edge-cloud collaborative computing split deployment method of the present invention is a compromise solution that can take into account both computing delay and communication delay. At the computing level, part of the neural network layer can be offloaded to the cloud server for computing, thereby reducing the computing load on the edge and speeding up the computing speed. At the communication level, by selecting appropriate neural network splitting points, the amount of data that needs to be transmitted in the network is reduced, thereby reducing the burden on the network. In this way, the neural network can be flexibly split according to the network conditions and the computing power of the edge device, and the neural network can be divided into two, and deployed on the edge and cloud for computing respectively. Compared with traditional edge computing and cloud computing, this computing mode can bring better response speed, and the proposed split deployment algorithm is also more effective.

[0182] The above is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. The above embodiments of the present invention can also be modified in various ways. All simple, equivalent changes and modifications made according to the claims and the description of the present invention fall within the scope of protection of the claims of the present invention. The contents not described in detail in the present invention are all conventional technical contents.

Claims

1. A neural network edge-cloud collaborative computing split deployment method, characterized in that: include: Step S1: Determine the layer type of the basic neural network layer of the neural network to be deployed, and establish a delay prediction model f(x, S) for each computing resource and each basic neural network layer. Each delay prediction model is only used to predict the computing delay of a basic neural network layer under one computing resource; Step S2: executing a neural network segmentation algorithm to determine an optimal segmentation point of the neural network to be deployed, wherein the optimal segmentation point refers to a segmentation point K corresponding to the minimum value of each overall delay T(K) when the segmentation point K is 1 to l-1, and the segmentation point is used to segment the neural network to be deployed into a first sub-neural network and a second sub-neural network, wherein l is the number of neural network layers in the neural network to be deployed; Step S3: using the optimal segmentation point to segment the trained neural network to be deployed into a first sub-neural network and a second sub-neural network, deploying the first sub-neural network on an edge device, and deploying the second sub-neural network on a cloud server for collaborative computing; The input parameters of the delay prediction model f(x,S) are x and S, where x is the feature vector corresponding to the neural network layer, the feature vector has one or more feature variables, and S is the type of computing resource. The type of computing resource S includes the edge type S cloud and Cloud Type S cloud ; In the step S2, the neural network segmentation algorithm is based on a global search of prefix and array; The step S2 comprises: Step S21: for each neural network layer of the neural network to be deployed, determine the computational delay f(x(l p ),S edge ) and the computational latency of the neural network layer in the cloud f(x(l p ),S cloud );l p is the type of the basic neural network layer corresponding to the pth neural network layer; p is the layer number of the neural network layer, p is 1…l; Step S22: for each neural network layer of the neural network to be deployed, respectively determine the prefix sum array prefixEdge[p] of the neural network layer at the edge and the prefix sum array prefixCloud[p] of the neural network layer at the cloud; Step S23: For each segmentation point K, determine the computational delay T of the edge end when the segmentation point is K. e (K), the computing delay of the cloud when the split point is K is T c (K) and the transmission delay when the split point is K, and sum these three to get the overall delay T(K) when the split point is K; Step S24: Traverse the segmentation points K in the range of 1 to l-1, and take the segmentation point K corresponding to the minimum value in the overall delay T(K) as the optimal segmentation point K best .

2. The neural network edge-cloud collaborative computing split deployment method according to claim 1 is characterized in that: In step S1, for all types of feature variables corresponding to each basic neural network layer, a delay prediction model is established and trained in the form of a regression tree.

3. The neural network edge-cloud collaborative computing split deployment method according to claim 2 is characterized in that: The step S1 comprises: performing the following steps for each computing resource and each basic neural network layer: Step S10: Obtain all characteristic variables of a basic neural network layer and samples of the computational delay of the neural network layer under a computing resource to form a sample data set D of a delay prediction model; Step S11: For one of the characteristic variables x i , set the current split point s so that the sample data set D of the delay prediction model is within the feature variable x i The dimension is divided into two sub-sample spaces, namely the first sub-sample space and the second sub-sample space; Step S12: Determine the output prediction value y of the sample corresponding to the current segmentation point s predict and prediction error e; Step S13: continuously update the current segmentation point s and repeat the above step S12 to determine the segmentation point s that minimizes the value of the prediction error e as the optimal segmentation point.

4. The neural network edge-cloud collaborative computing split deployment method according to claim 3 is characterized in that: In step S11, the first sub-sample space R1(s)={x|x i ≤s}, the second subsample space R2(s)={x|x i >s}, x i is the i-th characteristic variable; In step S12, the predicted output value y of the delay prediction model predict for; In the formula, x k is the characteristic variable of the kth sample, R1 is the first sub-sample space, R2 is the second sub-sample space, c1 is the predicted output value of the delay prediction model in the first sub-sample space, c2 is the predicted output value of the delay prediction model in the second sub-sample space; N1 is the number of samples in the first sub-sample space, and N2 is the number of samples in the second sub-sample space; The prediction error e corresponding to the current segmentation point s is: Among them, y k is the computation delay of the kth sample.

5. The neural network edge-cloud collaborative computing split deployment method according to claim 3 is characterized in that: The feature vector corresponding to the basic neural network layer has multiple feature variables; The step S1 further comprises: Step S14: Repeat the above steps S11 to S13 for another characteristic variable until the optimal splitting points corresponding to all characteristic variables are determined; Step S15: Determine the predicted output value of the delay prediction model in each sample area according to each sample area divided by the optimal segmentation point.

6. The neural network edge-cloud collaborative computing split deployment method according to claim 3 is characterized in that: In step S15, the predicted output value of the delay prediction model f(x, S) is: in, Among them, x is the feature vector corresponding to the neural network layer, R m is the mth sample area, m is the number of the sample area, n represents the total number of types of feature variables, c m is the predicted output value of the delay prediction model in the mth sample area.

7. The neural network edge-cloud collaborative computing split deployment method according to claim 1, characterized in that: In step S2, the overall delay T(K) when the segmentation point is K is determined according to the first sub-neural network and the second sub-neural network segmented when the segmentation point is K, and the calculation delay T of the first sub-neural network at the edge is determined. e (K) and the computing delay T of the second sub-neural network in the cloud c (K), and combined with the amount of data D output by the Kth neural network layer K And the network bandwidth B between the edge and the cloud is determined.

8. The neural network edge-cloud collaborative computing split deployment method according to claim 1, characterized in that: prefixEdge[p]=prefixEdge[p-1]+f(x(l p ),S edge ); prefixCloud[p]=prefixCloud[p-1]+f(x(l p ),S cloud ); T e (K)=prefixEdge[K]-prefixEdge[0]; T c (K)=prefixEdge[l]-prefixEdge[K]; The transmission delay when the split point is K is Among them, D K is the amount of data output by the Kth neural network layer, and B is the network bandwidth between the edge and the cloud.

Images