Edge-end collaborative deep learning computing acceleration system and method

Abstract

The application relates to a kind of edge-end collaborative deep learning computing acceleration system and method, the system includes at least one terminal device (1) and at least one edge server (2), the terminal device (1) is configured to: in the case of entering the service range of at least one described edge server (2), determine the strategy of inter-layer partition and / or intra-layer partition of deep learning model based on first configuration information and second configuration information of the edge server (2) itself;The edge server (2) is configured to: in response to inference request information, execute the strategy of inter-layer partition and / or intra-layer partition of the deep learning model to carry out collaborative inference.The application predicts DNN model execution time based on workload random forest method, and the prediction result is more accurate.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to an edge-to-edge collaborative deep learning computing acceleration system and method. Background Technology

[0002] As machine learning applications continue to grow in scale, more and more image analysis problems are being solved using deep learning models (DNNs), such as ResNet-50-based workpiece surface recognition, Faster R-CNN-based skin analysis, 3D scanning based on 3D-R2N2, and 3D reconstruction based on Pix2VoX. On the other hand, due to the development of mobile and embedded devices, the deployment of these deep learning-based applications on end devices is becoming increasingly common. However, although the computing resources in embedded and mobile devices are becoming increasingly powerful and energy-efficient, pushing computationally intensive DNN applications to end devices remains a significant challenge in terms of the computing and storage performance of those devices.

[0003] Some studies address the issue of accelerating DNN inference by offloading the entire DNN to a remote cloud server. When the terminal device needs to use the DNN for inference, it sends the collected raw data to the cloud, which then handles the entire inference process. However, this approach requires the complete transmission of private data collected by the terminal device to the cloud, potentially leading to data privacy leaks. Furthermore, the large volume of raw data results in significant transmission latency when sending it from the terminal device to the remote cloud server.

[0004] With the development of 5G, edge server nodes and fog nodes are becoming increasingly common, such as servers deployed in base stations and home gateways. These edge server nodes provide available computing and storage resources for terminal devices within their service range. Therefore, based on this, some studies have proposed accelerating the inference process by offloading DNN partitions through collaboration between terminal devices and edge servers. First, these methods ignore the inaccuracy of predicting DNN layer execution time using regression models trained with binary equations when multiple terminal devices compete for edge server resources. Second, because of the competition for edge node resources among multiple terminal devices, using a single edge node as the offloading target can actually increase the DNN inference acceleration time.

[0005] For example, Chinese patent CN110309914A discloses a method for accelerating deep learning model inference based on collaboration between edge servers and mobile devices. It proposes combining model segmentation and model simplification, accurately estimating the runtime latency of the deep learning model's network layers on edge servers and mobile devices through training and using regression models. This allows for the search for exit points and segmentation points that meet latency requirements. Compared to traditional methods based on cloud data centers and direct deployment on devices, this method not only achieves efficient and low-latency inference of deep learning models on mobile devices but also provides a deep learning model inference scheme that meets latency requirements while balancing latency and accuracy. However, this invention has the following drawback:

[0006] First, the simple two-partition approach is mainly used for models where the computational load increases or decreases systematically according to the layer order, such as VGG and AlexNet. These models typically involve multiple convolutional layers followed by fully connected layers, resulting in a higher computational load in the first half and a lower load in the second half. The split point is often only needed in the later convolutional layers or the earlier fully connected layers. However, for many models composed of blocks, such as the NiN model with four blocks where all fully connected layers are replaced by 1×1 convolutional layers, or the Inception model with InceptionA, InceptionB, and InceptionC blocks, each consisting of multiple parallel convolutional layers, the computational load is unevenly distributed according to the layer order. A simple two-partition approach cannot effectively balance the computational load between edge servers and terminal devices. Especially on heavily loaded edge servers, multiple partitions allow for more granular utilization of node resources.

[0007] Second, the execution time of the DNN layer predicted by the regression model trained by the binary equation is inaccurate.

[0008] Third, while simple iterative algorithms can be used to find a split point, in the case of multiple partitions, due to the complex layer configuration and large set of split point locations, iterative algorithms cannot find the optimal set of split point locations.

[0009] For example, CN115034390B discloses a method for accelerating deep learning model inference based on cloud-edge-device collaboration, specifically involving a layered offloading method for deep learning models. This method theoretically models the computational latency, data transmission latency, data propagation latency, and the generation latency of the layered offloading strategy throughout the deep learning model inference process. It then determines the optimal layered offloading strategy for the deep learning model by minimizing the computational task response latency. Compared to deep learning model execution frameworks dominated by physical terminals and cloud computing centers, this method combines edge computing with cloud computing and layered offloads the deep learning model to different edge computing nodes, minimizing computational task response latency while maintaining computational accuracy. However, this invention has the following drawback:

[0010] First, because the DNN layer segmentation technique is not used, the computational load of the DNN layer cannot be segmented in a fine-grained manner, thus failing to fully utilize the resources of high-load edge servers. Second, because reinforcement learning techniques are not used, the DNN cannot be segmented quickly and accurately based on the load of the edge servers in complex scenarios with multiple edge servers.

[0011] To address the problems caused by the aforementioned deep learning model inference methods, this invention aims to provide a better way to accelerate edge-to-edge collaborative deep learning computation.

[0012] Furthermore, on the one hand, there are differences in understanding among those skilled in the art; on the other hand, the applicant studied a large number of documents and patents when making this invention, but due to space limitations, not all details and contents were listed in detail. However, this does not mean that the present invention does not possess the features of these prior art. On the contrary, the present invention already possesses all the features of the prior art, and the applicant reserves the right to add relevant prior art to the background art. Summary of the Invention

[0013] Existing technologies that use edge servers and terminal devices to collaboratively offload DNN partitions to accelerate the inference process often overlook the following issues:

[0014] First, when multiple terminal devices compete for edge servers, the regression model trained by binary equations is inaccurate in predicting the partitioning time of the DNN.

[0015] Second, when multiple terminal devices compete for resources on the edge server, the push acceleration time of the DNN will actually increase when using the same edge server as the offloading target.

[0016] To address the shortcomings of existing technologies, this invention provides an edge-to-edge collaborative deep learning computing acceleration system, comprising at least one terminal device and at least one edge server. The terminal device is configured to: upon entering the service range of at least one of the edge servers, determine a strategy for inter-layer partitioning and / or intra-layer partitioning of a deep learning model based on its own first configuration information and the second configuration information of the edge server. The edge server is configured to: in response to an inference request, execute the strategy for inter-layer partitioning and / or intra-layer partitioning of the deep learning model to perform collaborative inference.

[0017] Preferably, the terminal device is configured to: predict the inference execution time of each layer of the deep learning model based on a pre-trained random forest model; minimize the total inference time between the terminal device and the edge server based on the set of split point locations between layers determined by the ILP algorithm; and minimize the inference time between the edge servers based on the set of split point locations within the decision layer determined by the reinforcement learning algorithm and the intra-layer partitioning strategy.

[0018] Based on the random forest and DDPG models, the two together determine the optimal split points that can minimize the inference latency after DNN partitioning and / or intra-layer splitting.

[0019] This invention predicts the inference execution time of each layer of a deep neural network (DNN) using a pre-trained random forest model. Then, it uses the ILP algorithm to find a set of inter-layer splitting points. After splitting, some partitions remain on the terminal device, while others are offloaded to edge servers. If the edge server load is high, causing the inference latency of a partition to not meet user time requirements, a further pre-trained DDPG model is used to split the partitions based on feature map height and the number of neurons in the fully connected layers, finding a set of intra-layer splitting points in the deep learning model. The split partitions are then distributed in parallel to multiple high-load edge servers. Therefore, this invention can select a matching collaborative strategy based on the actual situation of the terminal device and the edge server, minimizing the total inference time of the collaborative strategy and thus eliminating the problem of increased total inference time due to excessive edge server load.

[0020] Preferably, the inter-layer partitioning strategy of the deep learning model includes at least: dividing the neural network layers of the deep learning model into at least two partitions according to the inter-layer granularity.

[0021] In existing technologies, neural networks are typically divided into two partitions, which fails to fully utilize the computing resources of terminal devices and edge servers. This invention divides the neural network into three or more partitions, maximizing the collaborative advantages of terminal devices and edge servers.

[0022] Preferably, the strategy for intra-layer partitioning of the deep learning model includes at least: collecting execution data of at least one neural network layer from several terminal devices and edge servers during the execution of the deep learning model dataset, and training a random forest model of the execution time of the neural network layer.

[0023] Preferably, the inter-layer splitting point location decision method includes at least the following: predicting the execution time of the neural network layers of the deep learning model on the terminal device and the edge server based on the random forest model; determining the transmission time of the intermediate feature vector between the terminal device and the edge server based on the output data of each neural network layer of the deep learning model and the communication bandwidth data of the edge server; determining the total time of partition unloading based on the sum of the inference execution time and the transmission time of the intermediate feature vector; and determining the optimal set of inter-layer splitting point locations based on the ILP algorithm to find the optimal solution that minimizes the total time. This calculation ensures that the most appropriate inter-layer splitting point can be calculated for the total time of collaborative inference between the terminal device and the edge server.

[0024] Preferably, the strategy for intra-layer partitioning of the deep learning model includes at least: dividing the model according to the internal structure of at least one layer of the neural network of the deep learning model, such that at least two partitions after being divided are deployed in parallel on the corresponding edge servers.

[0025] Preferably, the method of partitioning the deep learning model within layers further includes at least: partitioning convolutional layers based on a feature map height dimension grid, and partitioning fully connected layers based on the number of neurons.

[0026] This invention also provides a method for accelerating edge-to-edge collaborative deep learning computation, the method comprising at least:

[0027] When entering the service range of at least one edge server, the terminal device determines the strategy for inter-layer partitioning and / or intra-layer partitioning of the deep learning model based on its own first configuration information and the second configuration information of the edge server; in response to the inference request information, the edge server executes the strategy for inter-layer partitioning and / or intra-layer partitioning of the deep learning model to perform collaborative inference.

[0028] Preferably, the method further includes: predicting the inference execution time of the DNN layer based on a pre-trained random forest model; minimizing the total inference time between the terminal device and the edge server based on the set of split point locations between the decision layers using the ILP algorithm; and minimizing the inference time between the edge servers based on the set of split point locations within the decision layers using the reinforcement learning algorithm DDPG and the intra-layer partitioning strategy.

[0029] The advantages of the method of the present invention include at least the following: First, the DNN model execution time is predicted by using a workload-based random forest method, resulting in more accurate predictions; Second, the DNN model can be divided into multiple partitions by layer using an integer linear programming method, instead of just two partitions; Third, the partitions on the edge server are further divided by layer using DDPG, avoiding the problem of increased inference time due to server resource overload when only one edge server is used.

[0030] The present invention also provides a terminal device for collaborative deep learning computation with an edge server. The terminal device is configured to: upon entering the service range of at least one of the edge servers, determine a strategy for inter-layer partitioning and / or intra-layer partitioning of a deep learning model based on its own first configuration information and the second configuration information of the edge server; wherein, based on a pre-trained random forest model, the inference execution time of the DNN layer is predicted; based on the ILP algorithm, the set of inter-layer split point locations is determined to minimize the total inference time between the terminal device and the edge server; based on the reinforcement learning algorithm DDPG and the intra-layer partitioning strategy, the set of intra-layer split point locations is determined to minimize the inference time between the edge servers.

[0031] The terminal device of the present invention can further divide the partitions on the edge server according to the layer based on DDPG, thereby avoiding the problem of increased inference time caused by server resource overload when only one edge server is used.

[0032] The DNN layer segmentation method includes at least the following: segmentation of convolutional layers based on feature map height dimension grids, and segmentation of fully connected layers based on the number of neurons.

[0033] Preferably, the pre-trained DDPG model includes at least the following methods: establishing three elements representing the intra-layer segmentation of the deep learning model based on the Markov chain problem, pre-training the DDPG model based on the three elements; the DDPG model calculates the total inference time between the edge servers based on the execution data according to the DNN intra-layer segmentation technique, and determines the optimal intra-layer segmentation point position when the total inference time between the edge servers is minimized.

[0034] This calculation minimizes the total time for collaborative inference among multiple edge servers under high load conditions, and allows for the calculation of the most appropriate set of intra-layer splitting points. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of a preferred embodiment of the present invention.

[0036] Figure 2 This is a schematic diagram of collaborative reasoning according to a preferred embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the structure of an edge-to-edge collaborative deep learning computing acceleration system according to a preferred embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of a height-dimensional grid partitioning of a convolutional layer feature map provided by the present invention;

[0039] Figure 5 This is a schematic diagram of the number of neurons in a fully connected layer provided by the present invention;

[0040] Figure 6 This is a flowchart of the steps of a preferred embodiment of the edge-to-edge collaborative deep learning computation acceleration method provided by the present invention.

[0041] List of reference numerals

[0042] 1: Terminal equipment; 2: Edge server; 3: Wireless communication equipment; 4: Intermediate data. Detailed Implementation

[0043] The following is a detailed explanation with reference to the accompanying drawings.

[0044] Existing technologies that use edge servers and terminal devices to collaboratively offload DNN partitions to accelerate the inference process often overlook the following issues:

[0045] First, when multiple terminal devices compete for edge servers, the regression model trained by binary equations is inaccurate in predicting the partitioning time of the DNN.

[0046] Second, when multiple terminal devices compete for resources on the edge server, the push acceleration time of the DNN will actually increase when using the same edge server as the offloading target.

[0047] Therefore, the present invention aims to improve the above problems and provide an edge-to-edge collaborative deep learning computing acceleration system and method that can reasonably configure edge servers and avoid the inference time from increasing due to edge servers competing for resources.

[0048] This invention also aims to provide a terminal device capable of collaborative inference with an edge server, and a method for executing the terminal device thereof. Furthermore, this invention aims to provide an edge server suitable for the edge-to-edge collaborative deep learning computation acceleration method of this invention.

[0049] The following is an explanation of some of the terms and concepts in this invention.

[0050] Edge servers: These provide users with a channel to access the network and the ability to communicate with other server devices. Typically, an edge server is a group of servers performing a single function, such as a firewall server, cache server, load balancer, or DNS server.

[0051] Terminal devices refer to input / output devices connected to a computer system, typically located far from the computer. Smart terminal devices are preferred. Smart terminal devices can run deep learning models and collaborate with edge servers to complete the inference process. Examples of smart terminal devices include intelligent devices. Intelligent devices are, for example, communicative devices equipped with components such as chips and processors. Examples of intelligent devices include computers, tablets, mobile phones, smart glasses, smartwatches, and smart bracelets—terminal devices with computing capabilities.

[0052] Deep learning models are algorithms based on artificial neural networks that learn representations of data. They typically consist of an input layer, intermediate layers, and an output layer. Intermediate layers include convolutional layers, pooling layers, noise layers, recurrent layers, and activation layers. The output layer is usually a fully connected (dense) layer, which controls the dimensionality to achieve classification / regression tasks.

[0053] Random Forest Model: Random forest is a commonly used machine learning algorithm. It's an ensemble algorithm composed of decision trees, with no correlation between the different decision trees. When a new input sample enters the forest model, each decision tree in the forest makes a judgment and classification independently. Each decision tree produces its own classification result, and the random forest uses the classification result with the most frequent classifications as the final result. Random forests can be used for high-dimensional data without dimensionality reduction or feature selection; they are less prone to overfitting; training is relatively fast, and they are easily parallelized.

[0054] The Markov Decision Process (MDP) is a stochastic process in the state space that transitions from one state to another. The future state distribution depends only on the present and is independent of the past. Specifically, if a problem is divided into stages, the state in stage k can only be obtained from the state in stage k+1 through the state transition equation, and is independent of other states. The formula is: P[St+1|St] = P[St+1|S1,...,St]. The states in reinforcement learning problems also conform to the Markov property, meaning that performing an action at in the current state St transitions to the next state St+1 without considering previous states St-1,...,S1.

[0055] Reinforcement learning refers to how an agent maximizes its reward in a complex and uncertain environment. It guides actions by perceiving the state of the environment and responding to actions, thereby maximizing the return. A common reinforcement learning model is the standard Markov Decision Process (MDP). Commonly used reinforcement learning algorithms include: table-based Q-Learning algorithms without neural network involvement; value-based Deep Q Network (DQN) algorithms; policy-based Policy Gradient (PG) algorithms; and Actor Critic algorithms combining value-based and policy-based approaches (such as DDPG and A3C).

[0056] Reinforcement learning algorithm (DDPG): A deep deterministic policy gradient algorithm proposed to solve continuous action control problems. The most important word in DDPG is "deterministic," which means that the output of a continuous action is a specific value. When actions are discrete, the policy function outputs the probability of each action based on maximizing long-term reward; however, when actions are continuous, under the goal of maximizing long-term reward, the output can only be a specific numerical value representing a specific action, thus becoming a deterministic policy. In reinforcement learning algorithms, Q-Learning, DQN, and PG are used for decision-making problems involving discrete actions. DDPG, on the other hand, is a value-based and policy-based Actor Critic algorithm used in Markov chain decision-making processes involving continuous actions. In this invention, due to the large number of DNN layers, each with different configuration information, the set of split points within each layer is large, and the number of decision actions is numerous, making it impossible to use algorithms based on discrete action problems. Therefore, the DDPG algorithm based on continuous actions is adopted.

[0057] ILP (Integer Linear Programming) algorithm refers to a linear programming problem that requires some or all decision variables to take integer values. In this invention, the split point position between DNN layers is an integer value, so the set of split point variables can be determined by the ILP method.

[0058] Interlayer granularity: refers to each layer in a DNN being an independent unit, such as a convolutional layer or a fully connected layer in a DNN. Interlayer segmentation is the segmentation of the DNN at the interlayer granularity.

[0059] Intra-layer granularity: This refers to the structure within each layer of a DNN being considered a single unit. For example, one or more neurons in a fully connected layer of a DNN can be viewed as a unit, or features in a convolutional layer can be considered as a single unit. Figure 1 One or more rows can be considered as a unit. Intra-layer segmentation is the segmentation of the DNN at the intra-layer granularity.

[0060] This invention provides an edge-to-edge collaborative deep learning computation acceleration system, such as... Figure 2 and Figure 3 As shown, it includes at least one terminal device 1 and at least one edge server 2.

[0061] Terminal device 1 is configured to: upon entering the service range of at least one edge server 2, determine the strategy for inter-layer partitioning and / or intra-layer partitioning of the deep learning model based on its own first configuration information and the second configuration information of the edge server 2.

[0062] Edge server 2 is configured to: in response to inference request information, execute the strategy of inter-layer partitioning and / or intra-layer partitioning of the deep learning model for collaborative inference.

[0063] Specifically, terminal device 1 can determine whether to use an inter-layer partitioning strategy, an intra-layer partitioning strategy, or both strategies for collaborative reasoning based on the resource competition between multiple terminal devices and edge server 2.

[0064] Under normal circumstances, when the connection between terminal devices and edge servers does not lead to resource contention, the estimated collaborative inference time between terminal devices and edge servers is normal and predictable. However, when multiple terminal devices connect to a small number of edge servers, resulting in more than two terminal devices connecting to the same edge server and creating contention, the collaborative inference time between terminal devices and edge servers will increase, exceeding the predicted time. Therefore, under resource contention, simply using inter-layer partitioning will lead to a decrease in the inference efficiency of the edge servers.

[0065] Preferably, the terminal device 1 is configured to perform inter-layer segmentation or intra-layer segmentation.

[0066] The inter-layer segmentation method includes: predicting the execution time of each layer of the deep learning model on the terminal device and edge server based on a pre-trained random forest model. Then, using the Integer Linear Programming (ILP) algorithm, a set of inter-layer segmentation points is found to minimize the total inference time between the terminal device and the edge server.

[0067] The intra-layer segmentation method includes: based on a pre-trained reinforcement learning DDPG model, segmentation is performed according to feature map height and the number of neurons in the fully connected layer to find the set of intra-layer segmentation points in the deep learning model, minimizing the total inference time between the high-load edge servers. Specifically, the offline training steps include a first step of offline training the random forest model and a second step of offline training the reinforcement learning DDPG model.

[0068] S1: Offline training steps.

[0069] S1.1: Train a random forest model to estimate the execution time of neural network layers on edge servers and terminal devices.

[0070] Specifically: Collect execution data from each layer of a deep learning model from several terminal devices and edge servers to train a random forest model.

[0071] Preferably, a random forest prediction model is established for each layer type. A random forest model is established for convolutional layers (CL), pooling layers (PL), activation layers (AL), and fully connected layers (CL).

[0072] Specifically: A deep learning model dataset is executed on several terminal devices and edge servers, and execution time data for each layer of the neural network in this dataset is collected. The collected execution time data of different types of layers and layer hyperparameter configuration information are used as input data to train various random forest prediction models.

[0073] Preferably, the dataset for the deep learning model comes from a neural network set generated by Neural Architecture Search (NAS). Neural Architecture Search (NAS) is a technique for automatically designing neural networks, which can automatically design high-performance network structures based on a sample set using algorithms. Therefore, NAS can generate various sets of variant neural network models based on the provided model architecture, but with different hyperparameters. For example, using VGG16 as input data for NAS, NAS can generate various sets of variant neural network models based on the VGG16 structure, but with different hyperparameters (such as kernel size, stride, padding mode, etc.). This set of variant models contains a large number of convolutional layers, pooling layers, activation layers, and fully connected layers with different hyperparameter configurations. Therefore, executing this set of variant models on terminal devices and edge servers can obtain a large dataset of execution times for different types of layers with different hyperparameter configurations, which is sufficient to train random forest prediction models of various types of layers.

[0074] Preferably, the method for collecting execution time data of deep learning models includes: collecting the execution time of each type of layer by adjusting the CPU / GPU utilization of terminal devices and edge servers.

[0075] Specifically: 12 representative DNN models (such as AlexNet, VGG, DenseNet, ResNet, etc.) were collected offline from ImageNet2012. A dataset with 500 variant models was generated using NAS. This variant model dataset was run on terminal devices and edge servers, while varying GPU / CPU utilization, and the execution time of each type of layer was collected as input datasets for random forest models of different layer types. Using this dataset, random forest models were trained for each layer type (CL, FL, PL, and AL). The trained random forest models can predict the inference execution time of DNN layers under different loads on edge servers and terminal devices. The random forest model only needs to be trained once and can then be used continuously to predict the actual execution time of different DNN layers on terminal devices and edge servers.

[0076] S1.2: Train the reinforcement learning DDPG model to determine the location of the split point within the decision layer.

[0077] Specifically, a three-element representation for achieving intra-layer partitioning in multiple edge server scenarios is established based on the Markov chain problem, and the DDPG model is trained based on the three elements and execution time data.

[0078] Specifically, the problem of the set of splitting points within the decision layer is first described as a Markov chain problem (MDP): An action represents splitting the convolutional layer feature map using a high-dimensional grid, and a fully connected layer is split based on the number of neurons. A state represents the height and width of the convolutional layer feature map, the number of neurons in the fully connected layer, the workload of the edge server (GPU / CPU utilization), and the total inference time on the edge server when taking an action. A reward represents the time gain from the current state when taking an action.

[0079] Furthermore, based on the Markov chain problem, three elements are established: state, action, and reward. Therefore, State = {number of model layers, number of neurons, feature map height, feature map width, number of edge servers, edge server execution time}, action = {feature map height split point, number of neurons split point}, reward = ... t represents the time limit accepted by the user.

[0080] Furthermore, based on the established MDP three elements, a DDPG decision algorithm is built: In reinforcement learning, the classic DDPG algorithm contains two components: actor and critic, where actor and critic are two neural networks. The actor is used to execute specific actions (corresponding to the action in the MDP three elements); the critic evaluates the reward generated by the action (corresponding to the reward in the MDP three elements), thereby determining whether to take the action. The actor and critic are designed as four fully connected layers. A dataset of 500 variant models is run on three edge servers, while the CPU / GPU utilization of the edge servers is varied, to train DDPG until convergence. The variant model dataset is generated in step S1.1.

[0081] During the online optimization phase, the trained DDPG algorithm automatically determines the most suitable set of intra-layer splitting points for convolutional and fully connected layers on heavily loaded edge servers, thus completing the intra-layer splitting. The DDPG algorithm only needs to be trained once and can be used continuously to determine the set of intra-layer splitting point locations between edge servers.

[0082] S2: The online optimization steps specifically include:

[0083] An integer linear programming mathematical model is established based on the sum of the inference execution time predicted by the random forest model and the transmission time of the intermediate feature vectors. The result obtained by the integer linear programming mathematical model is the set of optimal inter-layer split point locations.

[0084] Based on the height dimension grid segmentation of the feature map of the convolutional layer and the number of neurons in the fully connected layer, the set of segmentation point locations within the decision layer is determined according to the DDPG algorithm.

[0085] S2.1: Based on the sum of the inference execution time predicted by the random forest model and the transmission time of the intermediate feature vector, establish an integer linear programming mathematical model. The result obtained by the integer linear programming mathematical model is the set of optimal inter-layer split point locations.

[0086] Specifically, the execution time of each layer of the random forest model predicting the deep learning model (DNN) on the terminal device is set to... The time required for the random forest model to perform inference on the edge server is set to [time value missing]. Set the size of the output data for each layer of the DNN used to collect the deep learning model to O. Set the bandwidth for communication between the terminal device and the edge server to B. Based on the bandwidth B and the output data size O for each layer, the transmission time of the intermediate feature vector between the terminal device and the edge server can be calculated as follows: .

[0087] Total time for partition unload strategy It is the sum of inference execution time and intermediate feature vector transmission time:

[0088] .

[0089] In the above formula, This indicates the number of layers in a DNN. ， ， ， This represents different layers of the DNN. The binary variables e and s represent the layer executed on the user terminal device and the layer executed on the edge server, respectively. For example, =1 indicates that the m-th to g-th layers of the DNN are executed on the terminal device; =0 indicates that layers m through g of the DNN are not executed on the terminal device. 。

[0090] For the convex optimization problem min The optimal interlayer splitting point can be obtained by solving the ILP algorithm. After splitting according to the splitting point, W end Deployed on terminal devices, W server Deployed on an edge server.

[0091] The present invention uses a pre-trained random forest model in the offline phase to predict the execution time of the layers of a deep learning model DNN, and uses an ILP algorithm to decide how to partition the deep learning model DNN between the terminal device and the edge server according to the inter-layer granularity. The present invention uses a pre-trained reinforcement learning method in the offline phase to decide how to partition the DNN between the edge servers according to the intra-layer granularity.

[0092] S2.2: Based on the convolutional layer feature map height dimension grid splitting and fully connected layer neuron number splitting techniques, determine the set of intra-layer splitting points according to the DDPG algorithm.

[0093] Preferably, as Figure 4 shown, the computational overhead of the convolutional layer comes from the convolution operation of the feature map, so the convolutional layer is split according to the height dimension grid of the feature map.

[0094] Specifically, the computational overhead of the convolutional layer comes from the convolution operation of the feature map, so the splitting scheme uses the height dimension grid splitting of the feature map. For example, in the scenario of three high-load edge servers, the intra-layer splitting operation example of the convolutional layer is as Figure 4 shown. For an 8×8 convolutional layer, according to the DDPG algorithm provided in the present invention, the trained DDPG based on state = {model layer number = 1, feature map height = 8, feature map width = 8, number of edge servers = 4, edge server load (GPU / CPU occupancy rate) = ( , , , ), edge server execution time = T, convolutional layer partition = [(h1)], the actor takes the first-round action = {feature map height splitting points (1, 3, 4)}, after splitting it is partition 1 = [h1], partition 2 = [h2, h3], partition 3 = [h4], partition 4 = [h5, h6, h7, h8]. The execution time T1 of this partition result is evaluated by the critic, and the reward judges that T1 > t, T1 < T, so the reward is 0.

[0095] Then the DDPG model is updated

[0096] state = {model layer number = 1, feature map height = 8, feature map width = 8, number of edge servers = 4, edge server load (GPU / CPU occupancy rate) = ( , , , ), Edge server execution time = T1, Convolution layer partition = [(h1), (h2, h3), (h4), (h5, h6, h7, h8)]}。

[0097] The actor performs the second round of action = {Feature map height splitting points (2, 3, 4)}. After splitting, it is partition 1 = [h1, h2], partition 2 = [h3], partition 3 = [h4], partition 4 = [h5, h6, h7, h8]. The execution time T2 of this partition result is evaluated by the critic, and the reward determines that T2 < t, T2 < T1, so the reward is 0 + 1 / T2 = 1 / T2.

[0098] Then the DDPG model updates the state = {Number of model layers = 1, Feature map height = 8, Feature map width = 8, Number of edge servers = 4, Edge server load (GPU\CPU occupancy rate) = ( , , , ), Edge server execution time = T2, Convolution layer partition = [(h1, h2), (h3), (h4), (h5, h6, h7, h8)]}。

[0099] The actor performs the third round of action = {Feature map height splitting points (2, 4, 6)}, after splitting, it is partition 1 = [h1, h2], partition 2 = [h3, h4], partition 3 = [h5, h6], partition 4 = [h7, h8]. The execution time T3 of this partition result is evaluated by the critic, and the reward determines that T3 < t, T3 > T2, so the reward is 1 / T3.

[0100] The DDPG model trained through the offline steps will continuously try to take actions in the direction of maximizing the cumulative reward value. Therefore, after multiple rounds of action operations, until the cumulative reward value remains stable, it is the convergence state. At this time, the partition result is the optimal partition selection. For example Figure 4 in, the final result after convergence is to split the convolution layer of the output layer into partition 1 = [h1, h2], partition 2 = [h3, h4], partition 3 = [h5, h6] and partition 4 = [h7, h8] according to the height H, and these four partitions are respectively placed on four edge servers.

[0101] Preferably, the computational overhead of the fully connected layer comes from the addition and multiplication operations between neurons, so the method of splitting the number of neurons is adopted.

[0102] Specifically, in the scenario of two high-load edge servers, an example of in-layer splitting of the fully connected layer is Figure 5As shown, the input layer contains two neurons x1 and x2, and the output layer neuron contains four b1, b2, b3, and b4. According to the DDPG algorithm provided by the present invention, the trained DDPG algorithm is based on state = {number of model layers = 1, input neurons = 2, output neurons = 4, number of edge servers = 2, edge server load (GPU / CPU occupancy rate) = ( , ), edge server execution time = T, fully connected layer partition = [(b1, b2, b3, b4)]}, the actor takes the first round of action = {output neuron cut point}, after splitting, partition 1 = [b1], partition 2 = [b2, b3, b4], the execution time of this partition result is evaluated by the critic, and the reward judges T1 > t, T1 < T, so the reward is 0. Then DDPG updates state = {number of model layers = 1, input neurons = 2, output neurons = 4, number of edge servers = 2, edge server load (GPU / CPU occupancy rate) = ( , ), edge server execution time = T1, fully connected layer partition = [(b1), (b2, b3, b4)]}, the actor takes the second round of action = {output neuron cut point}, after splitting, partition 1 = [b1, b2], partition 2 = [b3, b4], the execution time of this partition result is evaluated by the critic, and the reward judges T2 < t, T2 < T1, so the reward is 1 / T2. The DDPG trained in the offline step will continuously try to take actions in the direction of the maximum cumulative reward value. Therefore, after multiple rounds of action operations, until the cumulative reward value remains stable, it is the convergence state, and the partition result at this time is the optimal partition selection. For example Figure 5 In, the converged result is to split the output layer neurons into two partitions: partition 1 = [b1, b2] and partition 2 = [b3, b4], and these two partitions are respectively placed on edge service A and edge server B. The input layer neurons x1, x2 and their replicas are respectively placed on edge servers A and B. The complete output vector can be obtained after merging the output layer results. The in-layer splitting of the present invention mainly considers the fully connected layer and the convolutional layer, because for other layers, such as the pooling layer and the activation layer, the storage and calculation amount are very small compared to the fully connected layer and the convolutional layer. Therefore, the calculation overhead of the pooling layer and the activation layer itself is not enough to offset the overhead of the splitting operation. In addition, the in-layer splitting of the convolutional layer and the fully connected layer in the present invention is not limited to two or three partitions, and the number of partitions is determined by the trained DDPG algorithm according to the number, load and execution time of edge servers in the actual scenario.

[0103] Preferably, the strategy for partitioning the deep learning model between layers includes at least: dividing the neural network of the deep learning model into at least three partitions according to the interlayer granularity.

[0104] Specifically, this invention uses the ILP algorithm to find multiple splitting points in a neural network at the inter-layer granularity, thereby dividing it into multiple partitions. For example, a neural network model can be divided into three partitions: W1, W2, and W3. W1 and W3 are deployed on terminal devices, while W2 is offloaded to an edge server. Clearly, this multi-partition inter-layer partitioning scheme maximizes the advantages of collaboration between terminal devices and edge servers.

[0105] The method for intra-layer segmentation of deep learning models in this invention is as follows.

[0106] Since edge servers may be devices such as routers with limited computing resources, or the resources of edge servers may be overloaded due to competition from multiple clients, the acceleration of inference for deep learning models (DNNs) by a single edge server is limited. Therefore, multiple nearby edge servers need to work together to perform DNN inference tasks.

[0107] Intra-layer partitioning is a more fine-grained partitioning method than inter-layer partitioning. Inter-layer partitioning partitions the data according to the execution order of the layers in a deep learning model (DNN), which is a serial partitioning scheme. Intra-layer partitioning partitions the internal structure of a specific layer in a DNN model, and the partitioned partitions can be deployed in parallel on multiple edge servers. The computational overhead in a DNN model mainly comes from convolutional layers (CL) and fully connected layers (FL), therefore intra-layer partitioning primarily targets convolutional layers (CL) and fully connected layers (FL).

[0108] Preferably, the strategy for intra-layer partitioning of the deep learning model includes at least: dividing the model according to the internal structure of at least one layer of the neural network of the deep learning model, such that at least two partitions after being divided are deployed in parallel on the corresponding edge server 2.

[0109] Preferably, the strategy for intra-layer partitioning of the deep learning model includes at least: partitioning convolutional layers based on a feature map height-dimensional grid, and partitioning fully connected layers based on the number of neurons.

[0110] Specifically, the computational cost of the convolutional layer (CL) comes from the convolution operation of the feature map; therefore, the segmentation scheme uses a feature map height-dimensional grid. The computational cost of the fully connected layer comes from the addition and multiplication operations between neurons; therefore, the segmentation is based on the number of neurons. This intra-layer segmentation scheme can fully utilize the resources of multiple edge servers.

[0111] For example, this instance includes three high-load edge servers and one terminal device. The deep learning model DNN is illustrated using the first 5 layers of AlexNet. The edge servers are equipped with CPU and GPU hardware resources. The terminal device is connected to the edge servers, and the edge servers themselves are connected via Wi-Fi.

[0112] Based on an offline-trained random forest model, the GPU resources of edge servers and terminal devices, along with the five layers of a DNN, are used as input. The random forest model outputs the execution time of each layer on the terminal device and the edge server. The inter-layer partitioning problem is described as a convex optimization problem. The ILP algorithm is used to find two partitioning points, located between conv_1 and conv_2, and between conv_4 and conv_5, respectively. Figure 1 As shown.

[0113] The deep learning model (DNN) is divided into three partitions according to the inter-layer granularity: partition W1, partition W2, and partition W3. Partition W1 contains the conv_1 layer, partition W2 contains conv_2, conv_3, and conv_4 layers, and partition W3 contains the conv_5 layer. Partitions W1 and W3 are deployed on terminal devices, while partition W2 is offloaded to an edge server.

[0114] The edge server performs intra-layer partitioning on the aforementioned unloaded partition W2. The DDPG model trained offline divides the convolutional layers conv_2, conv_3, and conv_4 into three partitions based on the feature map height, according to the number of layers, neurons, feature map height, feature map width, and bandwidth, CPU, and GPU information between the edge servers. Figure 1 As shown, the split partitions are deployed on three edge servers.

[0115] Collaborative reasoning stage. For example... Figure 2 As shown, when the terminal device issues an inference request, it performs the inference task according to the partitions of the deep learning model DNN divided during the online optimization phase. The terminal device inputs the original data into partition W1. Partition W1 outputs intermediate feature vectors and sends them to three edge servers for execution. During execution, the three edge servers communicate intermediate feature data, and the inference results are merged and sent to the terminal device. The terminal device's partition W3 receives the data and finally generates the inference result.

[0116] This invention also provides a method for accelerating edge-to-edge collaborative deep learning computation, specifically implemented as follows: Figure 6 As shown.

[0117] S11: Begin.

[0118] S12: The terminal device enters the service range of the edge server.

[0119] Edge servers and terminal devices register on the QingCloud platform or the Huawei Cloud IoTEdge platform. When a terminal device enters the service range of an edge server, QingCloud or IoTEdge provides interconnection services between the two. QingCloud and IoTEdge are platforms that provide services for the Internet of Things and edge nodes. They primarily use the MQTT protocol to achieve interconnection between registered edge servers and terminal devices. However, the methods for interconnecting terminal devices and edge servers are not limited to these two solutions.

[0120] S13: Use the terminal device to obtain the second configuration information of the nearby edge server.

[0121] After the terminal device connects to the edge server, the edge server sends the CPU utilization rate (or GPU utilization rate if there is a GPU) to the terminal device, so that the terminal device can obtain the load information of the edge server.

[0122] S14: Execute the ILP algorithm.

[0123] S15: Unload the deep learning model DNN by partitioning it between layers.

[0124] S16: The edge server receives at least one partition that has been unloaded by the terminal device.

[0125] S17: The edge server determines the resource contention situation, that is, whether the execution inference time meets the user's needs.

[0126] S18: If resource contention is not severe, that is, if the inference time meets the user's needs, start responding to and executing the user's inference request.

[0127] S19: In cases of severe resource contention, i.e., when the inference time does not meet user requirements, execute the DDPG algorithm.

[0128] S20: Parallel partitioning of multiple edge servers within the same layer.

[0129] S21: After the edge servers merge the output results, at least one edge server returns the deep model inference results to the terminal device.

[0130] It should be noted that the specific embodiments described above are exemplary. Those skilled in the art can devise various solutions inspired by the disclosure of this invention, and these solutions all fall within the scope of this invention and its protection. Those skilled in the art should understand that this specification and its accompanying drawings are illustrative and not intended to limit the scope of the claims. The scope of protection of this invention is defined by the claims and their equivalents. This specification contains multiple inventive concepts; terms such as "preferredly," "according to a preferred embodiment," or "optionally" indicate that the corresponding paragraph discloses an independent concept. The applicant reserves the right to file divisional applications based on each inventive concept.

Claims

1. An edge-to-device collaborative deep learning computing acceleration system, comprising at least one terminal device and at least one edge server, characterized in that, The terminal device is configured as follows: When entering the service range of at least one edge server, the strategy for inter-layer partitioning and intra-layer partitioning of the deep learning model is determined based on its own first configuration information and the second configuration information of the edge server. Among them, the inference execution time of each layer of the deep learning model is predicted based on the pre-trained random forest model, and the total inference time between the terminal device and the edge server is minimized based on the set of split point positions between the decision layers of the ILP algorithm. Based on reinforcement learning algorithms and intra-layer partitioning strategies, the set of split point locations within the decision layer is used to minimize the inference time between edge servers. The edge server is configured as follows: After the terminal device connects to the edge server, the edge server sends the CPU utilization rate to the terminal device, so that the terminal device can obtain the load status of the edge server and execute the ILP algorithm; and perform inter-layer partitioning and offloading of the deep learning model DNN. In response to the inference request information from the terminal device, the edge server receives at least one partition offloaded by the terminal device, determines whether the execution inference time meets the user's requirements, and executes the inter-layer partitioning and intra-layer partitioning strategy of the deep learning model for collaborative inference; if the execution inference time meets the user's requirements, it begins to respond to and execute the user's inference request; if the execution inference time does not meet the user's requirements, it executes the reinforcement learning algorithm; multiple edge servers perform parallel intra-layer partitioning. After the edge servers merge the output results, at least one edge server returns the deep model inference results to the terminal device; The strategy for intra-layer partitioning of deep learning models includes: dividing the model according to the internal structure of at least one layer of the neural network, such that at least two partitions are deployed in parallel on the corresponding edge servers.

2. The edge-to-edge collaborative deep learning computation acceleration system according to claim 1, characterized in that, The inter-layer partitioning strategy of the deep learning model includes at least the following: Divide the neural network layers of the deep learning model into at least two partitions according to the inter-layer granularity.

3. The edge-to-edge collaborative deep learning computation acceleration system according to claim 1 or 2, characterized in that, The intra-layer partitioning strategy of the deep learning model includes at least the following: Collect execution data of at least one neural network layer from several terminal devices (1) and edge servers (2) during the execution of a dataset of a deep learning model, and train a random forest model of the execution time of the neural network layer.

4. The edge-to-edge collaborative deep learning computation acceleration system according to claim 3, characterized in that, The method for determining the location of the interlayer split point includes at least the following: The execution time of neural network layers of a deep learning model based on the random forest model is predicted on terminal devices (1) and edge servers (2). The transmission time of the intermediate feature vector between the terminal device (1) and the edge server (2) is determined based on the output data of each neural network layer of the deep learning model and the communication bandwidth data of the edge server (2). The total time for partition unloading is determined by summing the inference execution time and the transmission time of intermediate feature vectors. The optimal solution with the minimum total time is obtained based on the ILP algorithm, and the set of optimal interlayer splitting points is determined.

5. The edge-to-edge collaborative deep learning computation acceleration system according to claim 4, characterized in that, The intra-layer partitioning method of the deep learning model includes at least the following: Convolutional layers are partitioned based on a high-dimensional grid of feature maps. Fully connected layers are segmented based on the number of neurons.

6. A method for accelerating edge-to-edge collaborative deep learning computation, characterized in that, The methods include at least: When entering the service range of at least one edge server, the terminal device determines the inter-layer partitioning and intra-layer partitioning strategies of the deep learning model based on its own first configuration information and the second configuration information of the edge server. Specifically, the inference execution time of the DNN layer is predicted based on the pre-trained random forest model, and the set of inter-layer splitting point locations is decided based on the ILP algorithm to minimize the total inference time between the terminal device and the edge server. Based on the reinforcement learning algorithm and the intra-layer partitioning strategy, the set of intra-layer splitting point locations is decided to minimize the inference time between the edge servers. After the terminal device connects to the edge server, the edge server sends the CPU utilization rate to the terminal device, so that the terminal device can obtain the load status of the edge server and execute the ILP algorithm; and perform inter-layer partitioning and offloading of the deep learning model DNN. In response to the inference request information from the terminal device, the edge server receives at least one partition offloaded by the terminal device and determines whether the execution inference time meets the user's requirements. The edge server executes the inter-layer partitioning and intra-layer partitioning strategy of the deep learning model for collaborative inference. If the execution inference time meets the user's requirements, it starts responding to and executing the user's inference request. If the execution inference time does not meet the user's requirements, it executes the reinforcement learning algorithm. Multiple edge servers perform parallel intra-layer partitioning. After the edge servers merge the output results, at least one edge server returns the deep model inference results to the terminal device; The strategy for intra-layer partitioning of deep learning models includes: dividing the model according to the internal structure of at least one layer of the neural network, such that at least two partitions are deployed in parallel on the corresponding edge servers.

7. A terminal device for collaborative deep learning computation with an edge server, characterized in that, The terminal device is configured as follows: When entering the service range of at least one edge server, the strategy for inter-layer partitioning and intra-layer partitioning of the deep learning model is determined based on its own first configuration information and the edge server's second configuration information; wherein, The inference execution time of the DNN layer is predicted based on the pre-trained random forest model, and the total inference time between the terminal device and the edge server is minimized based on the set of split point locations between the decision layers using the ILP algorithm. Based on reinforcement learning algorithms and intra-layer partitioning strategies, the set of split point locations within the decision layer is determined to minimize the inference time between edge servers. After the terminal device connects to the edge server, the edge server sends the CPU utilization rate to the terminal device, so that the terminal device can obtain the load status of the edge server and execute the ILP algorithm; and perform inter-layer partitioning and offloading of the deep learning model DNN. The edge server receives at least one partition offloaded by the terminal device and determines whether the execution inference time meets the user's requirements. If the execution inference time meets the user's requirements, it starts responding to and executing the user's inference request. If the execution inference time does not meet the user's requirements, it executes the reinforcement learning algorithm. Multiple edge servers perform parallel intra-layer partitioning. After the edge servers merge the output results, at least one edge server returns the deep model inference results to the terminal device; The strategy for intra-layer partitioning of deep learning models includes: dividing the model according to the internal structure of at least one layer of the neural network, such that at least two partitions are deployed in parallel on the corresponding edge servers.

Images