Cooperative computing strategy of artificial intelligence model reasoning task under edge-end architecture

Abstract

The invention discloses a cooperative computing strategy for an artificial intelligence model reasoning task under an edge-end architecture, solves the problem of limited computing power of terminal equipment, and improves the efficiency through cooperative computing with an edge-side server. The method comprises the following steps: firstly, determining an edge-end framework, and then establishing a system model, a communication model and a calculation model according to the edge-end framework to establish a joint optimization model. And according to the joint optimization model, in combination with the advantages of a deep reinforcement learning algorithm, an optimal reasoning task division strategy is solved by adopting a multi-agent depth deterministic strategy gradient (MADDPG) algorithm, and an optimal task scheduling strategy is solved by adopting a DQN algorithm. The MADDPG algorithm runs on an edge device side and is responsible for searching an optimal division point of a reasoning task, and the DQN algorithm runs on an edge server side and is responsible for coordinating a server cluster state and co-scheduling the reasoning task. The two reinforcement learning algorithms are respectively applied to the task division part and the task scheduling part, so that the overall overhead is reduced while the task is ensured to be completed in time.

Description

Technical Field

[0001] The present invention belongs to the field of Internet of Things and mobile edge computing, and specifically designs a multi-objective optimization strategy for collaborative computing of artificial intelligence model reasoning tasks based on reinforcement learning under an edge architecture. Background Art

[0002] In recent years, the number of terminal devices connected to the Internet has exploded. IoT devices and mobile terminals are widely present in all scenes of modern life. They are responsible for running various applications and require powerful and intensive service computing capabilities. On the other hand, modern artificial intelligence technology has also made breakthrough progress. Its applications in image recognition, natural language processing, etc. have gradually matured. Artificial intelligence models have gradually combined with complex real-world scenarios, resulting in a series of intelligent applications running on terminal devices. They can be seen everywhere in our production and life. Typical intelligent applications include real-time intelligent monitoring, intelligent dialogue assistants, and face recognition. A significant feature of intelligent applications is that they require a large amount of computing and memory resources, which makes it difficult for a single IoT device with limited resources to meet the popularization of intelligent applications. This also makes the requirements for computing resources more prominent. However, the existing computing architecture is limited by the laws of physics, and the computing power of a single device is difficult to increase exponentially in a short period of time. How to efficiently coordinate computing between terminal devices and servers has become an urgent research direction.

[0003] As an extension of cloud computing, edge computing can effectively reduce the communication delay between IoT devices and servers by being close to the physical location of IoT devices. At the same time, by effectively utilizing edge server resources with stronger computing power, a single IoT device can partially offload the AI ​​application computing tasks that are difficult to process locally in real time to the edge server for collaborative computing, thereby achieving lower application response time, lower end-side energy consumption, and more efficient computing resource utilization. Containers are a lightweight virtualization technology deployed on edge server clusters that can encapsulate applications and their operating environments together to achieve convenient cross-platform deployment. Compared with traditional virtual machines, containers share the kernel of the host operating system, so they start faster and have less resource overhead. Containers avoid dependency conflicts and ensure a consistent operating environment by isolating different applications and their dependencies, which is very useful for development, testing, and production environments. Using container technology is a useful tool for implementing collaborative computing in the field of edge computing. AI reasoning tasks with complex dependencies are packaged into containers and sent to edge servers for computing and execution as a whole task, and finally the results of the calculations are returned to the requester.

[0004] Edge computing makes up for the shortcomings of insufficient computing power of user terminal devices. At the same time, since edge servers are usually deployed in computer rooms near base stations and are closer to users, they can provide faster request responses than cloud computing, thereby reducing the latency of computing requests. Therefore, by combining edge computing with edge devices to provide collaborative computing of artificial intelligence reasoning tasks, service latency and energy consumption are jointly optimized to provide users with better application experience quality. Summary of the invention

[0005] In view of the difficulty of collaborative computing between edge devices and edge servers and the inefficient resource allocation under the edge architecture, this paper proposes a multi-objective optimization strategy for collaborative computing of edge-side artificial intelligence applications. After the artificial intelligence application on the edge device generates multiple reasoning tasks, it can reduce processing delays and improve the endurance of IoT devices by collaborative computing with the edge server, while efficiently utilizing edge server resources.

[0006] The present invention provides a multi-agent reinforcement learning model running on a terminal device, which is used to determine which layer of the model the current reasoning task starts to be divided from, and by selecting the optimal division point, the completion time of the reasoning task and the terminal energy consumption are optimized. The reinforcement learning model is deployed on the edge server to coordinate the resource status of the edge server cluster, package the reasoning tasks sent by the terminal device into containers for scheduling and execution within the cluster, and improve the user service quality and optimize the energy cost by jointly optimizing the task execution time and the edge server energy consumption. In order to achieve this goal, a many-to-many edge architecture is first determined, and then modeling is performed according to the architecture. The model includes a system model, a communication model, and a calculation model, and then a joint optimization model is established based on the established model. According to the joint optimization model, combined with the advantages of the deep reinforcement learning algorithm, the edge device adopts the multi-agent deep deterministic policy gradient (MADDPG) algorithm to obtain the optimal model division point, and the edge server adopts the deep Q value network (DQN) to schedule the reasoning task. Among them, the use of reinforcement learning algorithms requires building a deep reinforcement learning model based on the system model, calculation model and joint optimization model. The model includes four parts: state unit, action unit, reward and punishment system and Q value function. The reinforcement learning algorithm is used to optimize the execution and scheduling of reasoning, obtain the optimal partitioning strategy, and reduce the overall overhead of the system execution task.

[0007] The method of the present invention and its implementation principle are described in detail below.

[0008] Step 1. Edge architecture system model

[0009] The system model based on the edge architecture is as follows Figure 1As shown in Figure 1, the model is divided into an edge terminal layer and an edge server layer. The edge terminal runs artificial intelligence applications, and the edge server runs applications responsible for executing the remaining neural network execution. There are a total of N devices in the edge terminal layer, marked as N = {1, 2, ..., N}, and each terminal device has corresponding performance information {cal n ,com n ,ram n ,bat n}, cal n Represents the computing power available to the nth terminal device, in FLOPS, i.e. floating point operations per second; n Represents the allocatable communication bandwidth of the nth terminal device, in MB / s; RAM n Represents the available memory size of the nth terminal device, in MB; bat n Represents the available energy of the nth terminal device, in joules.

[0010] Edge terminal device applications generate several neural network reasoning tasks Indicates the i-th task generated by the n-th device; Indicates the input data size of the calculation model required for this inference task, in MB; Represents the computing resources required to execute the remaining neural network part of the task, in FLOPS; Represents the maximum tolerable delay of the reasoning task; Represents the task from the neural network Starting from the first layer, the inference task is sent to the edge server for calculation.

[0011] There are a total of M edge servers in the edge server layer, marked as M = {1, 2, ..., M}, and each edge server has its status information S m ={cal m ,load m ,com m ,ram m ,Model m},S m Represents the state information set of the mth edge server, cal m Represents the computing power of this edge server, load m Represents the load of this edge server, com m Represents the available communication bandwidth, ram m Represents available memory resources, Model m Indicates whether there are resources for the neural network model. If an inference task is received from an edge terminal but there is no corresponding neural network model, it is necessary to go to the central server to pull the model parameters.

[0012] Step 2. Communication Model

[0013] In the communication model, the total communication delay is divided into four parts. The first part is the propagation delay, which is defined as a constant D prop , represents the delay of the signal in the channel propagation process, which is determined by the physical properties of the propagation carrier and this part of the delay cannot be avoided; the second part is the transmission delay, defined as D trans , which represents the time required for the sender to send the data to be transmitted according to the channel rate. This part of the delay is related to the transmission rate. The third part is the queuing delay, which is defined as D queue , which represents the time that the transmitted data packet waits in the queue in the router for sending; the fourth part is the processing delay, which is defined as D proc , which represents the delay of the routing device in processing forwarding data packets during the sending and receiving process.

[0014] In this paper, in the edge architecture, task data is transmitted between computing nodes in the form of communication links. When the AI ​​application on the terminal device starts to generate reasoning tasks, the reinforcement learning algorithm is used to determine how to divide the tasks. The reasoning tasks are divided into two parts. The first part is executed locally by the terminal device, and the second part is executed by a powerful edge server.

[0015] Let H n represents the channel gain between the terminal device and its adjacent edge server, let represents the transmission power of the nth terminal device, σ 2 represents the noise power consumption, B represents the wireless channel bandwidth allocated to the terminal device by the edge server, then the transmission rate of the terminal device is defined as The transmission delay is calculated as represents the transmission delay of the i-th task; the transmission power consumption of edge terminal n is calculated as

[0016] Edge server side channel queuing delay Channel processing delay represents the transmission power of the mth edge server and the total transmission rate of the edge server Transmission power consumption of edge server m

[0017] Finally, the total transmission delay of task i is calculated as

[0018]

[0019] The total transmission power consumption is calculated as

[0020]

[0021] Step 3. Calculate the model

[0022] In this article, each artificial intelligence model inference task generated by the application on the edge device will first be divided into tasks to determine which layer computing tasks are executed locally, and the remaining neural network model layer computing tasks are sent to the edge server.

[0023] When the inference task on the edge device decides to perform inference locally, the execution time of each task i locally on the edge terminal n is defined as It is obtained by dividing the floating-point computing power required by each task by the floating-point computing power of the local device; the local computing energy consumption is defined as in Represents the average computing power of the terminal device. When the inference task is sent to the low-load edge server for inference execution, the execution time of task i generated by terminal n on edge server m is defined as Server computing energy consumption in Represents the average computing power of the edge server that executes the task. Since the edge server receives computing tasks requested by multiple terminal devices, as the number of requests increases, a queue waiting time delay will be generated. Define the queue waiting time before each task is executed.

[0024] Because each task must be read from the external memory before being transferred into the memory by the central processor, a reading delay and a transmission delay of the memory bus will be generated. This delay is determined by the device status of the edge device or edge server and is recorded as a constant T proc .

[0025] The total computation delay is the sum of local processing delay, edge server processing delay, queue waiting time and read transmission delay; the total computation delay is calculated as

[0026]

[0027] The total computing power consumption is the sum of the local computing power consumption and the edge server computing power consumption. Since the energy consumed by queuing and reading the hard disk to transfer to the memory is extremely small compared to the power consumption, it is ignored in the calculation model; the total computing power consumption is calculated as

[0028]

[0029] Step 4. Joint optimization model of latency and energy consumption

[0030] According to the computational model and the communication model, the total delay of task i is calculated as

[0031]

[0032] The total power consumption is calculated as

[0033]

[0034] The goal of this invention is to jointly optimize the completion time and energy consumption of edge device reasoning tasks. The weights λ1 and λ2 are the weighted parameters of the computing time and energy consumption required for the edge terminal device decision, respectively. The current task executed on the terminal device is the i-th task. The i-1 tasks before task i have been completed, and their average execution completion time can be obtained. The average execution delay of the completed tasks is calculated as in, represents the execution delay of the rth task among the i-1 tasks, Represents the average execution delay of the first i-1 completed tasks. When there are no completed tasks on the device, the average delay can be defined as

[0035] Then define the weight parameter Among them, 0≤λ1,λ2≤1,λ1+λ2=1.

[0036] The total cost consumed by task i to complete the calculation is:

[0037]

[0038] Therefore, the joint optimization model of step 4 is:

[0039]

[0040] Step 5. Based on the system model, computational model and joint optimization model described above, a multi-agent reinforcement learning model with task division is established.

[0041] The MADDPG task division model includes:

[0042] (1) Status Unit

[0043] MADDPG model state unit: In each time slot, the state space of each agent should contain the current resource usage of the device and the inference task characteristics. Specifically, the resource usage includes the CPU load of each terminal n. n , memory usage ram n 、Remaining battery n , bandwidth utilization com n ; Reasoning task characteristics include the model calculation amount of each task i Data size Delay Requirements The status unit is used to reflect the task size and device status of the system on the terminal device at each moment, that is,

[0044] (2) Action unit

[0045] The action of MADDPG is to select the partition point of the model, that is, to decide which layer of the deep learning model is used as the boundary, execute the first half locally and send the rest to the edge server for execution. Action representation: Since deep learning models are usually composed of multiple layers, the action space is defined as a discrete range of layers, representing the number of model layers. Assuming there is a deep learning model with N layers, the action space is represented as a t =[L0,L1,L2,…,L n ], where L j represents the agent's choice of partitioning from layer j in each time slot, L0 represents all execution by the edge server, L n Represents all calculations performed locally.

[0046] (3) Q-value function

[0047] In the process of neural network training, we will also use the Q value function to update the parameters of the neural network and select action a t The corresponding Q value function Q(s t ,a t )The formula is:

[0048] Q(s t ,a t )=r t +γmaxQ(s t+1 ,a n ) (9)

[0049] Among them, γ is a parameter used as a discount factor to measure the relative importance of future rewards. Its value range is between 0 and 1, and it should be as large as possible while ensuring the convergence of the algorithm. When γ is close to 0, it means that the agent attaches more importance to short-term rewards. When γ is close to 1, the agent attaches more importance to long-term cumulative rewards. During the training process, the initial value is generally set to 0.5, which means that short-term and long-term rewards are equally important. If the device needs to achieve the maximum throughput of tasks in the long term, the discount factor should be increased to ensure the maximum throughput of long-term tasks; if the device needs to meet the tasks with lower latency requirements as much as possible in the short term, the discount factor should be reduced to ensure the maximum delivery of tasks in the short term. t To select action a t The corresponding reward and punishment value.

[0050] Enter the current state s at the input of the neural network t+1 , the neural network will output all action units a tThe corresponding Q value function Q(s t+1 ,a n ), thereby using the maximum Q value function to update the existing Q value function Q(s t ,a t ).

[0051] The goal of the MADDPG algorithm is to find a suitable partitioning strategy a t , reducing the system's overhead of performing all tasks, so a t The choice is defined as:

[0052] a t = argmax(Q(s t ,a n )) (10)

[0053] a t The choice is to select the action a with the maximum value in the Q value function n , and get t In the state, select action a t The corresponding reward and punishment value r t .

[0054] (4) Reward and Penalty Value

[0055] The reward function of the MADDPG agent needs to comprehensively consider the impact of the partitioning strategy on the task execution time and device energy consumption. The reward function is designed based on the inference speed, communication overhead and resource consumption, taking into account both the inference delay and the device energy consumption. If the inference time is short and the power consumption is low, a high reward is given; if the communication overhead is large, the delay is high, or the device resources are over-consumed, the reward is reduced. The system optimization goal is to minimize the overhead, while the goal of deep reinforcement learning is to maximize the reward and punishment value. Therefore, the reward and punishment value we set should be negatively correlated with the optimization objective function. The reward form is defined as:

[0056]

[0057] in It is the total time from the start to the completion of a task, including local computing time, data transmission time, and execution time on the server. is the total energy consumed by executing the task. λ1 and λ2 are the weight parameters of time and energy consumption, 0≤λ1,λ2≤1,λ1+λ2=1. By setting the reward and punishment value function, the reward value R of the partitioning strategy of each task is guaranteed to be the largest, that is, the delay and power consumption of each task are minimized, and finally the total cost of all tasks is Cost. total Minimum.

[0058] Step 6. Based on the system model, computational model, and joint optimization model described above, a reinforcement learning model for edge-side task scheduling is established.

[0059] The DQN task scheduling model includes:

[0060] (1) Status Unit

[0061] DQN model state unit: In each time slot, the state space includes the computing power, current load, memory usage, network bandwidth and other characteristics of each server in the edge server cluster, which is used to uniformly schedule the reasoning task flow.

[0062] (2) Action unit

[0063] The action space of the DQN model defines the choices that the scheduler can make in each time slot. For the task scheduling problem, the action space is represented by the task assignment action: assigning the task to a certain edge server. Assuming there are M servers in the cluster, the action space is a t =[A1,A2,…,A M ], where A m Indicates that the task is assigned to the mth server.

[0064] (3) Q-value function

[0065] Through the DQN network learning and updating the Q value, the system gradually adjusts the scheduling strategy to minimize execution time and energy consumption. The Q value function is approximated by a neural network, and the following update formula is used during the training process

[0066] Q(s t ,a t )←Q(s t ,a t )+α(r t +γmaxQ(s t+1 ,a n )-Q(s t ,a t )) (12)

[0067] Where α is the learning rate, which is used to indicate the impact of the new value on the updated value, and the range is 10 -6 to 10 0 If the learning rate is too small, it will be too slow to converge. If it is too large, it will not be able to find the optimal solution, thus falling into the local optimum. The general setting value is 0.001. If the learning time is very long and it cannot converge to a solution during training, the learning rate value will be increased by ten times. If the training result converges but is not the optimal solution, the learning rate will be reduced to one tenth of the original value.

[0068] r t To select action a tThe corresponding reward and punishment value; γ is the discount factor, which ranges from 0 to 1. The meaning of this formula is that when selecting action a t The next state after t+1 Under this condition, the Q value function maxQ(s) with the maximum reward value is used. t+1 ,a n ) to partially update the existing Q-value function Q(s t ,a t ).

[0069] (4) Reward and Penalty Value

[0070] The reward function of DQN aims at high task execution efficiency, high resource utilization and low energy consumption of the edge server cluster system. The reward function usually needs to consider task execution time, server resource utilization and energy consumption control. The shorter the task completion time, the higher the reward; if the task completion time exceeds the preset deadline, a penalty is given. Improve the resource utilization of the server as much as possible, for example, keep all servers balanced to avoid overloading some servers while other servers are idle. Reduce power consumption by reasonably scheduling task execution. The lower the energy consumption, the higher the reward. Therefore, the reward function is designed as:

[0071]

[0072] definition represents the impact of task delay; the average computing power of all servers on the edge side is defined as in Represents the average computing power of the mth server in the edge cluster, M is the total number of servers in the edge cluster, and the meaning of the formula is the average computing power of the total M servers in the edge cluster. Definition Represents the impact of server power. The coefficients of the three parameters in the reward function are normalized according to λ3 and λ4.

[0073] In the reward function, is the execution time of the task on the edge side, λ5 is the weight value of time, calculated as is the energy consumption generated by server computing, λ6 is the weight of energy consumption, calculated as L edge is the load balancing degree of the server, λ7 is the load balancing weight, calculated as The weight relationship and range between these three parts are 0≤λ5,λ6,λ7≤1,λ5+λ6+λ7=1.

[0074] Beneficial Effects

[0075] Compared with the prior art, the present invention has the following advantages:

[0076] 1. In order to fully consider the computing resources of edge mobile terminal devices and edge servers, the present invention designs an edge-end collaboration model to improve the performance of the overall architecture through the collaboration between edge devices and edge servers.

[0077] 2. The present invention takes into account the limited resources of edge servers, and through the setting of queuing delay in the model, establishes an optimization model based on task division and scheduling with the goal of jointly optimizing minimum delay and minimum energy consumption.

[0078] 3. In the present invention, since the optimization problem is a non-convex problem, an optimization algorithm based on deep reinforcement learning is proposed, and better results are obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a schematic diagram of the edge collaboration model;

[0080] Figure 2 For collaborative computing method steps;

[0081] Figure 3 This is the overall system cost diagram. Specific implementation methods

[0083] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0084] The present invention designs a collaborative computing strategy for multi-objective optimization scheduling using a deep reinforcement learning method under an edge-end architecture. Under the edge-end collaborative architecture, the terminal device transmits the artificial intelligence tasks generated by the application to the edge server for execution. The terminal device performs the calculations of the first several layers of artificial intelligence, and sends the more complex and arduous remaining computing layer tasks to the edge server with more powerful computing resources; the edge server packages the computing tasks into containers and uses a deep reinforcement learning model to schedule and execute the tasks, thereby realizing pipeline-level task parallelism, and the edge server returns the final calculation results to the terminal device after obtaining them.

[0085] The specific steps are as follows:

[0086] Step 1: According to the edge collaboration architecture, first establish the system model, communication model, initialize parameters, and obtain specific information of the terminal device and edge server.

[0087] In this embodiment, the entire system model is composed of multiple mobile terminal devices and multiple edge servers from bottom to top, where the edge server cluster is scheduled by a general proxy server with relatively strong performance. The edge server is deployed next to the base station. There are N terminal devices within the coverage of each base station. The terminal device set is N, that is, N = {1, 2, ..., N}. Assume that each terminal will generate a task to be executed, and each task is It is represented by the task computation amount, data size, maximum tolerance time and partition point. There are M edge servers in the edge server layer, marked as M = {1, 2, ..., M}. Each edge server m has its status information S m ={cal m ,load m ,com m ,ram m ,Model m}, cal m Represents the computing power of the edge server, load m Represents the load of the edge server, com m Represents the available communication bandwidth, ram m Represents available memory resources, Model m Indicates whether there are resources for the neural model. If an inference task is received from an edge terminal but there is no corresponding neural network model, it is necessary to go to the central server to pull the model parameters.

[0088] The transmission rate between the terminal device and the nearby edge server is defined as Among them, H n represents the channel gain between the terminal device and its adjacent edge server, let represents the transmission power of terminal device n, σ 2 represents the noise power consumption, and B represents the wireless channel bandwidth allocated by the edge server to the terminal device.

[0089] The transmission delay of task i on the nth edge device is calculated as The transmission power consumption on the terminal device is calculated as Queuing delay on edge server side Processing Delays Total transfer rate on edge servers Transmission power consumption on edge servers

[0090] The total transmission delay of task i is calculated as The total transmission power consumption is calculated as

[0091] Step 2: Develop a computing model based on the edge architecture model.

[0092] In this embodiment, the local execution time of each task is defined as It is obtained by dividing the floating-point computing power required by each task by the floating-point computing power of the local device. The local computing power consumption is defined as in Represents the average computing power of the terminal device. When the inference task is sent to the edge server for inference execution, the execution time on the edge server is defined as Server computing energy consumption in Represents the average computing power of the edge server. Since the edge server receives computing tasks from multiple terminal devices, as the number of requests increases, a queue waiting time delay will be generated. Define the queue waiting time before each task is executed

[0093] Because each task must be read from the external memory before being transferred into the memory by the central processor, a reading delay and a transmission delay of the memory bus will be generated. This delay is determined by the device status of the edge device or edge server and is recorded as a constant T proc .

[0094] The total computational delay of task i is Represents the sum of local processing delay, edge server processing delay, queue waiting time, and read transmission delay.

[0095] The total computing power consumption is Represents the sum of local computing energy consumption and edge server computing energy consumption. Since the energy consumed by queuing and reading the hard disk to transfer to the memory is extremely small compared to the power consumption, it is ignored in the computing model.

[0096] Step 3: Establish a joint optimization model for latency and energy consumption.

[0097] In this embodiment, the goal of this invention is to jointly optimize the completion time and energy consumption of the terminal device reasoning task. The total delay for the completion of reasoning task i is The total power consumption is

[0098] The goal of this invention is to jointly optimize the completion time and energy consumption of edge device reasoning tasks. The weights λ1 and λ2 are the weighted parameters of the computing time and energy consumption required for the edge terminal device decision, respectively. The current task executed on the terminal device is the i-th task. The i-1 tasks before task i have been completed, and their average execution completion time can be obtained. The average execution delay of the completed tasks is calculated as in, represents the execution delay of the rth task among the i-1 tasks, Represents the average execution delay of the first i-1 completed tasks. When there are no completed tasks on the device, the average delay can be defined as Defining weight parameters Among them, 0≤λ1,λ2≤1,λ1+λ2=1. Then the total cost consumed by task i to complete the calculation is: The joint optimization model is:

[0099] Step 4: Use the MADDPG algorithm to control task division.

[0100] S1: Randomly initialize the Actor network and Critic network of each agent, and the device state space s t , the iteration count k is initially set to 1, and the training is exited when k is greater than or equal to the maximum number of iterations K. The experience replay pool size is initialized to 20,000, and K is initialized to 1,000 times. If convergence still fails after this training is completed, the maximum number of iterations can be increased. If the training results converge ahead of time, the maximum number of iterations can be reduced to obtain training results faster;

[0101] S2: The time step t in iteration k is initialized to 1;

[0102] S3: Input the current state s at the input of the neural network t , the neural network will output all action units a t The corresponding Q value function, select the action a with the maximum value in the Q value function n , which is the division point of the current task, and at the same time obtain t In the state, select action a t The reward and punishment value r t , and transition to the next state s t+1 , the next state s t+1 Medium Cost total Update to Cost total +Cost n , Cos n is the cost of executing the current task. Thus, we can obtain the training data (s t ,a t ,r t ,s t+1 );

[0103] S4:put (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool;

[0104] S5: The neural network is passed through the formula Q(s t ,a t )=r t +γmaxQ(s t+1 ,a n)Update the Q-value function, where γ is set to 0.5 to ensure the balance between long-term and short-term expectations, and continuously iterate the Q-value function to maximize the reward and punishment value r t .

[0105] S6: Continuously and randomly draw training data from the experience replay pool and keep learning to maximize the reward and punishment value r t as much as possible.

[0106] S7: Determine whether t < T is satisfied, where T is the total number of time steps in one iteration k. If so, t = t + 1 and enter S3; if not, enter S8;

[0107] S8: Determine whether k < K is satisfied, where K is the maximum number of iterations. If so, k = k + 1 and enter S2; if not, the iteration ends, and the overall system cost Cost total in the state unit tends to be stable, the entire learning process ends, and the actions a of all tasks in the action unit at the last round of iteration are obtained t , that is, the optimal partitioning positions of all tasks. At the same time, Cost in the state unit is obtained total , which is the total system cost when all tasks are executed according to the algorithm decision.

[0108] Step 5: Use the DQN algorithm to schedule task execution.

[0109] S1: Randomly initialize the parameters of the Q-network of each agent, the device state space s t , initially set the iteration count k to 1, exit the training when k is greater than or equal to the maximum number of iterations K, initialize the size of the experience replay pool to 20000, initialize K to 1000 times. If convergence still cannot be achieved after this training, the number of iterations can be increased. If the training result converges in advance, the maximum number of iterations can be decreased to obtain the training result faster; finally, synchronize the parameters of the target Q-network with the Q-network parameters;

[0110] S2: Initialize the time step t in the k-th iteration to 1;

[0111] S3: Input the current state s at the input end of the neural network t , and the neural network will output the Q-value functions corresponding to all action units a t , select the action a with the maximum value in the Q-value function n , that is, the scheduling server for the current task. At the same time, select the action a according to the state s t and the reward and punishment value r t after that, and transition to the next state s t , update the task execution cost Cost t+1 , that is, Cost total = Cost total = Cost total+Cost n , where Cost n is the cost of executing the current task. Record the training data (s t , a t , r t , s t+1 );

[0112] S4: Store (s t , a t , r t , s t+1 ) into the experience replay pool;

[0113] S5: The neural network randomly samples a batch of training data from the experience replay pool, calculates the target Q-value for each sample, and updates the Q-value function using the formula Q(s t , a t ) ← Q(s t , a t ) + α(r t + γmaxQ(s t+1 , a n ) - Q(s t , a t ))), where α is set to 0.001 to ensure that the learning does not get stuck in local optima and can also reach convergence quickly. γ is set to 0.5 to ensure the balance between long-term and short-term expectations. And update the parameters of the Q-network through backpropagation to maximize the reward value r t .

[0114] S6: Continuously and randomly sample training data from the experience replay pool, optimize the Q-network, and gradually learn the optimal scheduling strategy to maximize the reward value r t as much as possible.

[0115] S7: Judge whether t < T is satisfied, where T is the total number of time steps in one iteration k. If so, t = t + 1 and enter S3. If not, enter S8;

[0116] S8: Judge whether k < K is satisfied, where K is the maximum number of iterations. If so, k = k + 1 and enter S2. If not, the iteration ends, the overall system cost Cost toal tends to be stable, the entire learning process ends, and the optimal scheduling actions a t of all tasks in the action unit at the last round of iteration are obtained. At the same time, the total cost Cost toal of the system is the final value obtained based on algorithm optimization.

[0117] In this embodiment, since the combined execution time and energy consumption are considered, the two variables are mainly considered when setting the reward and penalty values, and then substituted into the optimization algorithm for iterative optimization until the result converges to the optimal solution. For specific steps, refer to the detailed description above.

[0118] pass Figure 3 It can be concluded that in the iterative process of the present invention, the sum of delay and energy consumption becomes lower and lower, and the convergence effect can be achieved quickly.

Claims

1. The collaborative computing strategy for AI model reasoning tasks under the edge architecture is characterized by The following steps are involved: Step 1. Build an edge architecture system model The model is divided into an edge terminal layer and an edge server layer. The edge terminal runs the artificial intelligence application, and the edge server runs the application responsible for executing the remaining neural network execution; There are N devices in the edge terminal layer, marked as N = {1, 2, ..., N}, and each terminal device n has corresponding performance information {cal n ,com n ,ram n ,bat n }, cal n Represents the computing power available to the terminal device, measured in FLOPS, i.e. floating point operations per second; n Represents the allocatable communication bandwidth of the terminal device, in MB / s; RAM n Represents the available memory size of the device in MB; bat n Represents the available energy of the device in joules; Edge terminal device applications have several neural network reasoning tasks Indicates the i-th task generated by the n-th device; Represents the input data size of the inference task model, in MB; Represents the computing resources required to execute the remaining neural network part, in FLOPS; Represents the maximum tolerable delay of the reasoning task; Represents the task from the neural network Starting from the first layer, the inference task is sent to the edge server for calculation; There are a total of M edge servers in the edge server layer, marked as M = {1, 2, ..., M}, and each edge server m has its status information S m ={cal m ,load m ,com m ,ram m ,Model m }, cal m Represents the computing power of the edge server, load m Represents the load of the edge server, com m Represents the available communication bandwidth, ram m Represents available memory resources, Model m Indicates whether there are resources for the neural model. If an inference task is received from an edge terminal but there is no corresponding neural network model, the model resources need to be pulled from the central server. Step 2. Communication Model In the communication model, the total communication delay includes the propagation delay D prop , the transmission delay is D trans , queuing delay D queue , processing delay D proc ; When the AI ​​application on the terminal device starts to generate reasoning tasks, the reinforcement learning algorithm is used to determine how to divide the tasks. The reasoning tasks are divided into two parts. The first part is executed locally by the terminal device, and the second part is executed by the edge server. Let H n represents the channel gain between the terminal device and its adjacent edge server, let represents the transmission power of the terminal equipment, σ 2 represents the noise power consumption, B represents the wireless channel bandwidth allocated to the terminal device by the edge server, then the transmission rate of the terminal device is defined as The transmission delay is calculated as The edge terminal transmission power consumption is calculated as Edge server side channel queuing delay Channel processing delay Indicates the transmission power of the edge server and the total transmission rate of the edge server Edge server m transmission power consumption Finally, the total transmission delay of task i is calculated as The total transmission power consumption is calculated as Step 3. Calculate the model When the inference task on the edge device decides to perform local inference, the local execution time of each task is defined as It is obtained by dividing the floating-point computing power required by each task by the floating-point computing power of the local device; the local computing energy consumption is defined as in Represents the average computing power of the terminal device; when the inference task is sent to the low-load edge server for inference execution, the execution time on the edge server is defined as Server computing energy consumption in represents the average computing power of the edge server; the queuing waiting time delay of the edge server processing the request is The delay of external memory access and memory bus transmission is recorded as a constant T proc ; The total computational delay is calculated as The total computing power consumption is calculated as Step 4. Joint optimization model of latency and energy consumption According to the calculation model and communication model, the total delay is calculated as The total power consumption is calculated as The total cost consumed by task i to complete the calculation is: The weights λ1 and λ2 are weighted parameters for the computing time and energy consumption required for the edge terminal device decision, where 0≤λ1,λ2≤1,λ1+λ2=1; The joint optimization model is: Step 5. Establish a multi-agent reinforcement learning model with task division, as follows: (1) Status Unit The status unit is used to reflect the task size and device status of the system on the terminal device at each moment, that is, In each time slot, the state space of each agent should contain the current resource usage of the device and the characteristics of the reasoning task; resource usage includes the CPU load of each terminal n. n , memory usage ram n 、Remaining battery n , bandwidth utilization com n ; Reasoning task characteristics include the model calculation amount of each task i Data size Delay Requirements (2) Action unit The action space defines a discrete range of layers, representing the model layer number; assuming a deep learning model with N layers, the action space is represented by a t =[L0,L1,L2,…,L n ], where L j represents the agent’s choice of partitioning from layer j in each time slot, L0 represents all offloading to the edge server, L n Represents all calculations locally; a t The choice is to select the action a with the maximum value in the Q value function n ; a t Defined as: a t =argmax(Q(s t ,a n ))#(9) Where Q(s t ,a n ) is the selection action a n Q-value function of (3) Q-value function The Q value function is used to update the parameters of the neural network and select action a t The corresponding Q value function Q(s t ,a t )The formula is: Q(s t ,a t )=r t +γmaxQ(s t+1 ,a n )#(10) Where γ is the discount factor, ranging from 0 to 1; r t To select action a t The corresponding reward and punishment value; input the current state s at the input end of the neural network t+1 , the neural network will output all action units a t The corresponding Q value function Q(s t+1 ,a n ), thereby using the maximum Q value function to update the existing Q value function Q(s t ,a t ); (4) Reward and Penalty Value The reward forms are defined as: in It is the total time from the start to the completion of the task, including local computing time, data transmission time and execution time on the server; is the total energy consumed to execute the task; λ1 and λ2 are the weight parameters of time and energy consumption, 0≤λ1,λ2≤1,λ1+λ2=1; Step 6. Establish a reinforcement learning model for edge-side task scheduling, as follows: (1) Status Unit In each time slot, the state space includes the computing power, current load, memory usage, and network bandwidth of each server in the edge server cluster, which is used to uniformly schedule the inference task flow; that is, the state unit (2) Action unit The action space is a t =[A1,A2,…,A m ], where A i Indicates that the task is assigned to the i-th server; (3) Q-value function Through the DQN network learning and updating the Q value, the system gradually adjusts the scheduling strategy to minimize the execution time and energy consumption; the Q value function is approximated by the neural network, and the following update formula is used during the training process Q(s t ,a t )←Q(s t ,a t )+α(r t +γmaxQ(s t+1 ,a n )-Q(s t ,a t ))#(12) Where α is the learning rate, ranging from 10 -6 to 10 0 between; r t To select action a t The corresponding reward and punishment value; γ is the discount factor, ranging from 0 to 1; the meaning of this formula is that when selecting action a t The next state after t+1 Under this condition, the Q value function maxQ(s) with the maximum reward value is used. t+1 ,a n ) to partially update the existing Q-value function Q(s t ,a t ); (4) Reward and Penalty Value By scheduling task execution to reduce power consumption, the lower the energy consumption, the higher the reward; the reward function is designed as In the reward function, is the execution time of the task on the edge side, and λ5 is the weight value of time; is the energy consumption generated by server computing, λ6 is the weight of energy consumption; L edge is the load balancing degree of the server, and λ7 is the weight of load balancing; the weight relationship and range between these three parts are 0≤λ5,λ6,λ7≤1,λ5+λ6+λ7=1.

Images