Collaborative reasoning task scheduling method based on multi-agent reinforcement learning

Abstract

The present application belongs to the field of multi-agent reinforcement learning, and specifically relates to a collaborative reasoning task scheduling method based on multi-agent reinforcement learning, which initializes a multi-agent environment, sets up multiple user devices, each of which is an agent, deploys an independent Actor neural network on each user device, sets up multiple cloud servers and deploys a central Critic neural network on a central control server, and provides multiple communication channel options for user devices. The advantage is that this solution is used to solve the combinatorial optimization problem of task segmentation points, communication channels, and server selection.

Description

Technical Field

[0001] The present application belongs to the field of multi-agent reinforcement learning, and specifically relates to a collaborative reasoning task scheduling method based on multi-agent reinforcement learning. Background Art

[0002] Separate computing is gradually being accepted by people as a third computing mode besides cloud computing and client computing. However, the traditional separate optimization problem is a simple scenario of one client, one cloud server, one communication channel and one deep learning reasoning task. This problem scenario does not match the actual situation. The actual situation is a complex situation with multiple user clients, multiple cloud servers, multiple communication channels and multiple deep learning tasks.

[0003] In actual situations, the impacts of multiple choices are not completely independent, and the optimal choice of a single user may not be the global optimal solution. Reinforcement learning is a common method to solve the problem of optimal combination with a large number of choices. However, traditional reinforcement learning and multi-agent reinforcement learning are difficult to make the global optimal choice for each client and cannot solve such problems. Summary of the invention

[0004] In view of the shortcomings of existing technologies and the need for improvement, this paper invented a highly robust collaborative computing resource and network resource scheduling method based on multi-agent reinforcement learning. The difficulty of this problem lies in how to design a multi-agent algorithm and make it highly resistant to interference.

[0005] In order to solve the above technical problems, the present invention provides a collaborative reasoning task scheduling method based on multi-agent reinforcement learning to achieve the above purpose.

[0006] A collaborative reasoning task scheduling method based on multi-agent reinforcement learning includes the following steps:

[0007] S1. Initialize the multi-agent environment, set up multiple user devices, each of which is an agent, deploy an independent Actor neural network on each user device, set up multiple cloud servers and deploy a central Critic neural network on the central control server, and provide multiple communication channel options for user devices;

[0008] S2. Collect the current state of the multi-agent environment and pass it to each Actor network to obtain output;

[0009] S3. Map the output of each Actor network to the decision space to obtain the decision of this cycle;

[0010] S4. Deploy the decision to the multi-agent environment and obtain the execution time of the current decision;

[0011] S5. Pass the execution time of the decision into the Critic neural network to obtain the score of the current decision;

[0012] S6. Update the Actor network and Critic network with the decision score;

[0013] S7. Repeat steps S2 to S6 until the decision converges, that is, the execution time of the decision is stable.

[0014] Preferably, the multi-agent environment in step S1 is a complex environment including multiple deep learning tasks, task segmentation points, cloud devices, and communication channels; assuming that the user device set is , including n user devices in total; let the cloud server set be , which contains m cloud servers in total; let the communication channel set be , contains a total of x communication channels, and , each user device can only select one cloud server and one communication channel during the execution of a cycle; the set of deep learning task reasoning is , the inference task performed on the elements in each user device set U is the element in the set T, that is, the i-th user client The reasoning task is ; Each reasoning task has 6 task split points , respectively in the deep learning task FLOP; let the task splitting point of the inference task of the i-th client be , then the task split point The task Divided into two parts, namely It is the task split point The previous task means it is executed on the client. It is the task split point The subsequent tasks represent tasks executed on the cloud server; Indicates that all tasks are executed on the cloud server. , Indicates that all tasks are executed locally on the client. , Respectively represent the split points at of In this case, the task will be divided into two parts and deployed on the client and server respectively.

[0015] Preferably, the Actor network deployed on the user device in step S1 includes a two-part structure of a weight calculation head for calculating the weight of the environment state and three tail outputters for determining the decision on the user device where the Actor is located.

[0016] Preferably, the state of the i-th multi-agent at time t in step S2 is The time is calculated locally by the user , cloud computing time , the size to be transferred , transmission time , waiting time It is composed of five parts, namely , the state of the multi-agent environment at time t is .

[0017] Preferably, the network calculation process of each Actor in step S3 is as follows:

[0018] The state of the multi-agent environment at time t is First, input the weight calculation head composed of two layers of fully connected neural network to obtain the weight matrix ;

[0019] The weight matrix is ​​multiplied by the state to obtain the mixed state matrix:

[0020] ;

[0021] The tail output unit consists of two layers of fully connected neural networks. For the i-th output unit, the input at time t is .

[0022] Preferably, the output and decision mapping specifications of the Actor network in step S3 are as follows: The range of the number of elements in the last layer of the three tail outputs is fixed, which are The output will go through The final three integer numerical results are obtained after the function; the result of the first output determines which cloud server in the server set S the user device will choose, the result of the second output determines which communication channel in the communication channel set C the user device will choose, and the result of the third output determines which split point in the split point set P the user device will choose.

[0023] Preferably, the decision score of the multi-agent environment in step S4 is defined as follows:

[0024] The state of the i-th agent at time t The time is calculated locally by the user , cloud computing time , the size to be transferred , transmission time , waiting time The final time consumed by the decision of the i-th agent at time t is: .

[0025] Preferably, the decision score of the multi-agent environment in step S5 is .

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention provides a collaborative computing resource scheduling method based on multi-agent reinforcement learning, which can significantly reduce the number of learning rounds in a multi-agent collaborative scenario and converge to the optimal result faster than traditional schemes. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the flow chart of this application.

[0029] Figure 2 A comparison chart is made for this application of hyper-ppo, mahppo and traditional solution fc-ppo (plot for client5*8). DETAILED DESCRIPTION

[0030] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0031] Figure 1 As shown, a collaborative reasoning task scheduling method based on multi-agent reinforcement learning includes the following steps:

[0032] S1. Initialize the multi-agent environment, set up multiple user devices, each of which is an agent, deploy an independent Actor neural network on each user device, set up multiple cloud servers and deploy a central Critic neural network on the central control server, and provide multiple communication channel options for user devices;

[0033] The multi-agent environment in step S1 is a complex environment including multiple deep learning tasks, task segmentation points, cloud devices, and communication channels;

[0034] Assume that the user device set is , including n user devices in total, assuming that the cloud server set is , which contains m cloud servers in total. Let the communication channel set be , contains a total of x communication channels, and , each user device can only select one cloud server and one communication channel during the execution of a cycle; the set of deep learning task reasoning is , the inference task performed on the elements in each user device set U is the element in the set T, that is, the i-th user client The reasoning task is ,Each reasoning task has 6 task split points , respectively in the deep learning tasks of At; let the task splitting point of the inference task of the i-th client be Task split point The task Divided into two parts, namely ; It is the task split point The previous task means that it is executed on the client side. It is the task split point The subsequent tasks represent tasks executed on the cloud server; Indicates that all tasks are executed on the cloud server. , Indicates that all tasks are executed locally on the client. ; Respectively represent the split points at of In this case, the task will be divided into two parts and deployed on the client and server respectively.

[0035] The Actor network deployed on the user device in step S1 includes a weight calculation head for calculating the weight of the environment state and three tail outputs for determining the decision on the user device where the Actor is located.

[0036] S2. Collect the current state of the multi-agent environment and pass it to each Actor network to obtain output;

[0037] The state of the i-th multi-agent at time t in step S2 The time is calculated locally by the user , cloud computing time , the size to be transferred , transmission time , waiting time It is composed of five parts, namely , the state of the multi-agent environment at time t is .

[0038] S3. Map the output of each Actor network to the decision space to obtain the decision of this cycle;

[0039] The calculation process of each Actor network is as follows:

[0040] The state of the multi-agent environment at time t is First, input the weight calculation head composed of two layers of fully connected neural network to obtain the weight matrix ;

[0041] The weight matrix is ​​multiplied by the state to obtain the mixed state matrix: ;

[0042] The tail output unit consists of two layers of fully connected neural networks. For the i-th output unit, the input at time t is: .

[0043] The output and decision mapping specifications of the Actor network in step S3 are as follows: The range of the number of elements in the last layer of the three tail outputs is fixed, which are , the output will go through The final three integer numerical results are obtained after the function; the result of the first output determines which cloud server in the server set S the user device will choose, the result of the second output determines which communication channel in the communication channel set C the user device will choose, and the result of the third output determines which split point in the split point set P the user device will choose.

[0044] S4. Deploy the decision to the multi-agent environment and obtain the execution time of the current decision;

[0045] The decision score of the multi-agent environment in step S4 is defined as follows:

[0046] The state of the i-th agent at time t The time is calculated locally by the user , cloud computing time , the size to be transferred , transmission time , waiting time The final time consumed by the decision of the i-th agent at time t is: .

[0047] S5. Pass the execution time of the decision into the Critic neural network to obtain the score of the current decision;

[0048] The decision score of the multi-agent environment is .

[0049] S6. Update the Actor network and Critic network with the decision scores.

[0050] S7. Repeat steps S2 to S6 until the decision converges, that is, the execution time of the decision is stable.

[0051] Figure 2 It can be seen that the solution (hyper-ppo) provided by this application has faster convergence than other solutions, and has the characteristics of slow performance degradation when the number of users increases, and has stronger stability than other solutions. This trend can be seen more clearly by summarizing each algorithm together.

[0052] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A collaborative reasoning task scheduling method based on multi-agent reinforcement learning, characterized in that: The following steps are involved: S1. Initialize the multi-agent environment, set up multiple user devices, each of which is an agent, deploy an independent Actor neural network on each user device, set up multiple cloud servers and deploy a central Critic neural network on the central control server, and provide multiple communication channel options for user devices; S2. Collect the current state of the multi-agent environment and pass it to each Actor network to obtain output; S3. Map the output of each Actor network to the decision space to obtain the decision of this cycle; S4. Deploy the decision to the multi-agent environment and obtain the execution time of the current decision; The decision score for a multi-agent environment is defined as follows: The state of the i-th agent at time t The time is calculated locally by the user Cloud computing time The size to be transferred Transmission time Waiting time The final time consumed by the decision of the i-th agent at time t is: S5. Pass the execution time of the decision into the Critic neural network to obtain the score of the current decision; The decision score of the multi-agent environment is S6. Update the Actor network and Critic network with the decision score; S7. Repeat steps S2 to S6 until the decision converges, that is, the execution time of the decision is stable; The multi-agent environment in step S1 is a complex environment that includes multiple deep learning tasks, task segmentation points, cloud devices, and communication channels. Let the user device set be U = U1, U2, ..., U n , including n user devices in total; let the cloud server set be S = S1, S2, ..., S m , including m cloud servers in total; let the communication channel set be C = {C1, C2, ..., C x }, including x communication channels, and n>m, n>x, each user device can only select one cloud server and one communication channel during the execution of one cycle; the set of deep learning task reasoning is T={T1,T2,…,T q }, the inference task performed on the elements in each user device set U is the element in the set T, that is, the i-th user client U i The reasoning task is T j ; Each inference task has 6 task split points P = {P0, P1, P2, ..., P5}, which are at {0%, 20%, 40%, 60%, 80%, 100%} of the FLOP of the deep learning task respectively; let the task split point of the inference task of the i-th client be P i , then the task split point P i Task T j Divided into two parts, namely T j =Task local +Task server ;Task local is the task split point P i The previous task means that it is executed on the client. server is the task split point P i The tasks after that are executed on the cloud server; P0 means that all tasks are executed on the cloud server. P5 means that all tasks are executed locally on the client. P1, P2, P3, and P4 represent the cases where the split points are 20%, 40%, 60%, and 80% of FLOPS respectively. The task will be divided into two parts and deployed on the client and server for execution respectively.

2. The collaborative reasoning task scheduling method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The Actor network deployed on the user device in step S1 includes a weight calculation head for calculating the weight of the environment state and three tail outputs for determining the decision on the user device where the Actor is located.

3. The collaborative reasoning task scheduling method based on multi-agent reinforcement learning according to claim 2 is characterized in that: The state of the i-th multi-agent at time t in step S2 The time is calculated locally by the user Cloud computing time The size to be transferred Transmission time Waiting time It is composed of five parts, namely The state of the multi-agent environment at time t is 4. The collaborative reasoning task scheduling method based on multi-agent reinforcement learning according to claim 3 is characterized in that: The network calculation process for each Actor in step S3 is as follows: The state of the multi-agent environment at time t is State t First, input the weight calculation head composed of two layers of fully connected neural network to obtain the weight matrix W = [W1, W2, ...W n ]; The weight matrix is ​​multiplied by the state to obtain the mixed state matrix: M t =[W1·State t , W2·State t ..., W n State t ]; The tail output unit consists of two layers of fully connected neural networks. For the i-th output unit, the input at time t is 5. The collaborative reasoning task scheduling method based on multi-agent reinforcement learning according to claim 2 is characterized in that: The output and decision mapping specifications of the Actor network in step S3 are as follows: the range of the number of elements in the last layer of the three tail outputs is fixed, namely m, x, and 6, and the output will be passed through the Softmax function to obtain the final three integer numerical results; the result of the first output determines which cloud server the user device will select in the server set S, the result of the second output determines which communication channel the user device will select in the communication channel set C, and the result of the third output determines which segmentation point the user device will select in the segmentation point set P.

Images