Computation offloading method and device based on partially observable multi-agent reinforcement learning

Abstract

The present application proposes a method and device for computational offloading based on partially observable multi-agent reinforcement learning, which relates to the technical field of reinforcement learning, wherein the method includes: obtaining the computational tasks generated by the user agent, and inputting the computational tasks into the policy neural network to generate an offloading decision; offloading the computational tasks to the computing nodes according to the offloading decision, defining the transmission delay and computational delay by means of queuing theory and feeding back to the user agent; obtaining feedback information based on probability control; optimizing the evaluation neural network's evaluation result of the current state according to the feedback information. The user agent is enabled to make decisions in a distributed manner under the premise of partial observability, and the model-free optimization problem is converted into a model-based optimization problem in a probabilistic control manner to more effectively search for iterative approximation to the global optimal solution, thereby ensuring the efficiency of task offloading and the privacy of the user offloading strategy.

Description

Technical Field

[0001] The present application relates to the technical field of reinforcement learning, and in particular to a computation offloading method and device based on partially observable multi-agent reinforcement learning. Background Art

[0002] In recent years, the surge in mobile applications such as artificial intelligence (AI), autonomous driving, industrial Internet of Things (Iot), and blockchain is reshaping our lifestyle and has received widespread attention from industry and academia. However, these applications are computationally intensive and latency sensitive. This makes it unlikely that the computing tasks of these applications can be processed by handheld devices with limited resources. The Cloud-Edge-Device (CED) system is a new computing paradigm that combines the advantages of cloud computing and edge computing. Compared with the traditional computing paradigm, the cloud-edge-device system provides mobile devices with elastic resources, which can simultaneously overcome the problems of resource shortage, low economic benefits of edge computing, and latency. In addition, with the support of cloud and edge computing, the cloud-edge-device system can also provide stable computing resources to meet user service quality requirements. However, the heterogeneity of mobile devices in the cloud-edge-device system and the high randomness of the computing resources it provides make it difficult to offload users' computing tasks to appropriate computing nodes in the cloud-edge-device system with high performance. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first objective of this application is to propose a computational offloading method based on partially observable multi-agent reinforcement learning.

[0005] The second object of the present application is to provide a device.

[0006] The third objective of the present application is to provide an electronic device.

[0007] A fourth objective of the present application is to provide a computer-readable storage medium.

[0008] A fifth object of the present application is to provide a computer program product.

[0009] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a computation offloading method based on partially observable multi-agent reinforcement learning, comprising:

[0010] Obtaining a computing task generated by a user agent, and inputting the computing task into a policy neural network to generate an offloading decision;

[0011] offloading the computing task to the computing node according to the offloading decision, defining the transmission delay and the computing delay by means of queuing theory and feeding back the delay to the user agent;

[0012] The user agent is made to observe the computation delay of the task generated by the user agent and the transmission delay of the link directly connected to the user agent, and obtain feedback information based on probability control;

[0013] The evaluation result of the current state of the neural network is optimized and evaluated according to the feedback information.

[0014] Optionally, inputting the computing task into a policy neural network to generate an offloading decision includes:

[0015] Generate the strategy probability distribution based on the current observable environment information and determine the state space S of the cloud-edge system; S = {Q, L, F}, where Q is the value set of the backlog of tasks on each computing node, L is the length of the virtual dynamic queue on each link, and F is the set of computing resources. Each computing node u can use F u (t) represents its computing resources, s∈S, s t Represents the state of the cloud-edge system at time point t;

[0016] The offloading decision of each computing task is determined by sampling the strategy probability distribution, wherein the offloading decision includes: the target node w i (t), represents the computing node to which task i is unloaded at time point t; data transmission rate x i (t) represents the data rate transmitted by task i.

[0017] Optionally, obtaining a strategy probability distribution according to the current observable environment input includes:

[0018] Introduce a Boolean variable M(t), where M(t) = 1 represents a decision (w i (t),x i (t)) is the current optimal strategy;

[0019] Determine the strategic probability distribution of M(t) as Pr(M(t)=1|s t ,a t )=exp(-r(w i (t),x i (t))), where a t is the joint action of each user agent, r(w i (t),x i (t)) is the direct reward of the user agent.

[0020] Optionally, defining the transmission delay and the calculation delay by means of queuing theory and feeding back the delay to the user agent includes:

[0021] Determine the direct reward r(w) of the user agent according to the target node and the data transmission rate i (t),x i (t));

[0022] Convert the direct reward into a probability instantaneous reward exp(-r(w i (t),x i (t)));

[0023] The offloading decision of each computing task is searched and determined according to the probability reward, wherein β is defined t (s t ,a t )=Pr(M t:T |s t ,a t ), where β t (s t ,a t ) indicates that when (s t ,a t ) The probability that the sequence of actions given by the starting strategy from t to T is the optimal strategy; define β t (s t )=Pr(M t:T |s t ), β t (s t ) means that when the state s t At the beginning, the probability that a sequence of actions from t to T given by the strategy is optimal;

[0024] The distributed random optimization data is obtained, and the strategy is distributed random optimization is performed to obtain the offloading decision.

[0025] Optionally, the performing distributed random optimization on the strategy to obtain the unloading decision includes:

[0026]

[0027] g'(w i (t),x i (t))=VΣ(αE i,t (w i (t),x i (t))+βw i (t)R u s i (t))+ΣV l (t)x i (t)

[0028] Among them, the target node w i (t+1), represents the computing node to which task i is unloaded at time point t+1; V l (t) is the link communication delay of the network link at time point t.

[0029] Optionally, optimizing the evaluation result of the neural network on the current state according to the feedback information includes:

[0030] According to the feedback information and the actual value function β t (s t ) calculates the loss function, wherein the calculation formula of the loss function is:

[0031] ||B(Φ θ (s t ))-Pr Π (s t+1 |s t )Φ θ (s t )-r(w i (t),x i (t))||

[0032] Among them, Pr Π (s t+1 |s t ) is the state s under this strategy t Transfer to state s t+1 The probability of θ (s t ) is the learning goal;

[0033] The parameters in the evaluation neural network are optimized with the goal of minimizing the loss function.

[0034] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a computation offloading device based on partially observable multi-agent reinforcement learning, comprising:

[0035] A decision module, used for acquiring computing tasks generated by the user agent and inputting the computing tasks into the policy neural network to generate an offloading decision;

[0036] A task offloading module, used for offloading the computing task to the computing node according to the offloading decision, defining the transmission delay and the computing delay by means of queuing theory and feeding back the delay to the user agent;

[0037] A feedback module, used to enable the user agent to observe the computation delay of the task generated by it and the transmission delay of the link directly connected to it, and obtain feedback information based on probability control;

[0038] The optimization module is used to optimize the evaluation result of the neural network on the current state according to the feedback information.

[0039] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0040] The memory stores computer-executable instructions;

[0041] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0042] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0043] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, which implements any method in the first aspect when executed by a processor.

[0044] The present application provides a method, device, electronic device and storage medium for offloading computation based on partially observable multi-agent reinforcement learning. The present invention enables user agents to make decisions in a distributed manner under the premise of partial observability, ensuring the efficiency of task offloading and the privacy of user offloading strategies, while not relying on a third-party central control node. This is currently the first distributed offloading decision-making scheme for multi-agent offloading reinforcement learning based on probability control.

[0045] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0047] Figure 1 This is a diagram of the cloud-edge-end system architecture of the present invention;

[0048] Figure 2 A schematic diagram of computing node calculation and network link transmission of the present invention;

[0049] Figure 3 A flowchart of a computation offloading method based on partially observable multi-agent reinforcement learning provided in an embodiment of the present application;

[0050] Figure 4A schematic diagram of a multi-layer perceptron combined with an LSTM network structure provided in this embodiment;

[0051] Figure 5 A schematic diagram of the structure of a computation offloading device based on partially observable multi-agent reinforcement learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0053] In recent years, the surge in mobile applications such as artificial intelligence (AI), autonomous driving, industrial Internet of Things (Iot), and blockchain is reshaping our lifestyle and has received widespread attention from industry and academia. However, these applications are computationally intensive and latency sensitive. This makes it unlikely that the computing tasks of these applications can be processed by handheld devices with limited resources. The Cloud-Edge-Device (CED) system is a new computing paradigm that combines the advantages of cloud computing and edge computing. Compared with the traditional computing paradigm, the cloud-edge-device system provides mobile devices with elastic resources, which can simultaneously overcome the problems of resource shortage, low economic benefits of edge computing, and latency. In addition, with the support of cloud and edge computing, the cloud-edge-device system can also provide stable computing resources to meet user service quality requirements. However, the heterogeneity of mobile devices in the cloud-edge-device system and the high randomness of the computing resources it provides make it difficult to offload users' computing tasks to appropriate computing nodes in the cloud-edge-device system with high performance.

[0054] A high-performance offloading solution must take into account both individual benefits and overall social benefits. If each user's offloading solution only considers individual benefits, it will easily lead to the homogenization of offloading solutions, resulting in the loss of individual benefits and the imbalance of cloud-edge system load. The imbalance of computing resource load will lead to the waste of computing resources and thus damage the overall social benefits. Only considering the overall social benefits will lead to individual users becoming "starved" or even "starved to death".

[0055] To address the high-performance offloading problem in the cloud-edge system mentioned above, existing solutions mainly use optimization methods and centralized reinforcement learning methods. The optimization method abstracts the computation offloading problem into a non-convex optimization problem and proves its NP-completeness. A suboptimal strategy is obtained through a heuristic algorithm. However, the modeling of the cloud-edge system in this solution is static. When encountering time-varying user demands and unpredictable network conditions, the performance of the algorithm will be greatly reduced. Reinforcement learning (RL) methods are often used for optimal control in dynamic systems. Based on the RL method, the task offloading problem is modeled as a Markov decision process (MDP), whose goal is to maximize the long-term cumulative reward (Reward). The learning agent interacts with the environment by observing the changes in the environment after making a decision. During the interaction, the agent estimates the long-term reward from the decision and establishes the optimal offloading strategy based on the observation results. The generated strategy determines the optimal task offloading in each time period in turn to approximate the long-term optimum. At present, in the cloud-edge system, the RL-based task offloading scheme mainly uses centralized learning with a central control node, which can observe all environmental state changes and formulate offloading strategies at the same time. In fact, users in the cloud-edge can only observe the state changes of part of the environment and make decisions asynchronously. If a centralized control node is used, it will also bring problems such as privacy leakage and single point failure.

[0056] Under the existing research technology, few solutions use distributed means to study the problem of task offloading in cloud-edge systems, and centralized offloading solutions have problems such as single point failure and user agents cannot independently determine their own offloading strategies. The existing distributed solutions still have the following three problems: 1. The existing RL-based task offloading assumes that the user agent can fully view the dynamics of the environmental system, which will bring about problems such as privacy leakage. 2. The communication overhead between user agents is large, and it will increase with the increase of system scale. 3. RL-based solutions guide decisions by maximizing cumulative rewards from a long-term perspective. The reward function is usually given in advance for some specific purposes, which makes these solutions less generalizable.

[0057] Before introducing the technical solution of the present invention, it is necessary to briefly explain some terms involved in the present invention, as follows:

[0058] The cloud-edge-device (CED) scenario can be abstracted into a three-layer computing architecture: the cloud layer consists of large data centers with huge computing power but far away from end users; the edge layer consists of edge computing servers with limited computing power; the device layer consists of nearby mobile devices (MDs), which have a large amount of redundant idle resources developed but unstable computing power. Users offload the generated random tasks to one of the above layers according to their offloading strategies.

[0059] In the present invention, the network topology of the cloud-edge-end system is modeled as an undirected graph G(v,e), where v and e are the sets of nodes and links in the cloud-edge-end system, respectively. Vu, Vc, Ve, Vm, and Vr represent users, cloud servers, edge servers, mobile devices, and network routers, respectively. Cloud servers, edge servers, and mobile devices have computing resources available for task offloading.

[0060] The network link state is a virtual dynamic queue Vl(t). The capacity of the queue indicates the maximum capacity of the link. The length of the real-time queue is the load of the current link. At the same time, the length of the link queue is stable in every time period.

[0061] Node Vk,k=c,e,m has computing resources available for task offloading and can process random tasks generated by users. The computing resources of each node u can be represented by Fu(t). At the same time, Fc(t)>>Fe(t)>>Fm(t) represents the computing capacity of each computing node by the number of CPU clocks in each time interval. For cloud servers and edge servers, the computing capacity is constant, and they will provide all their computing resources for computing task offloading at any time. For mobile devices (MD), their computing capacity is time-varying, and mobile devices can only use their idle resources at the current moment for computing offloading. At the same time, each computing node has a task queue Qu(t), the capacity of the queue is the server computing capacity, and the length of the queue is the current load of the server. For task i unloaded to computing node u at time t, its computing delay is Pi,u=(Qu(t)+si(t)) / Fu(t).

[0062] Figure 1 This is a diagram of the cloud-edge system architecture of the present invention. Figure 1 As shown in Figure 1, the components and organizational structure of the cloud-edge system architecture mainly include: central cloud nodes, edge cloud nodes, and mobile device nodes.

[0063] Each cloud-edge system has three types of computing nodes, namely central cloud nodes, edge cloud nodes, and mobile device nodes. All three types of computing nodes can provide computing resources. The central cloud node has sufficient computing resources and fast computing speed, but the corresponding number of nodes is small. At the same time, there is a long physical distance between it and the mobile device, which will generate a large communication overhead. Edge cloud nodes have a faster computing speed and are generally deployed at the telecommunications infrastructure facilities in street communities. The physical distance from the user is relatively short, but its capacity is limited. Mobile device nodes are both user agents and computing resource provision nodes. They have fewer computing resources and poor computing power. At the same time, their resource provision is random and dynamic, but the number of mobile device nodes is large, which can provide flexible computing resources.

[0064] These nodes are hierarchically constructed into a cloud-edge-end system, and the nodes are hierarchically connected through network links. Figure 1 As shown in the figure, each edge cloud node can be connected to the central cloud node, and the mobile device node is directly connected to its local edge cloud node. The connection between the external edge cloud and the central cloud needs to be forwarded through the local edge cloud. Each edge cloud node can connect to many mobile device nodes, build a local network and provide forwarding services.

[0065] The nodes in the cloud-edge system have the following basic functions: 1. Provide computing resources with a price. The computing resources and computing time are determined by S11. 2. Cloud computing nodes and edge cloud nodes provide users with their own computing resource queue status. 3. User agents have the task production function and can obtain the computing delay fed back to themselves by the task execution node and the end-to-end delay fed back by the link through task offloading, but their own computing resource queue can only be observed by themselves.

[0066] Figure 2 The schematic diagram of the computing node calculation and network link transmission of the present invention. For any network link, its capacity is cl, that is, the sum of the transmission speeds of all tasks in the link in each time period shall not exceed cl. For computing node u, the execution of its computing tasks is an execution queue Qu(t). The node computing power is quantified by the CPU frequency as Pu(t). For computing nodes, especially mobile device nodes, the computing power Pu(t) available for task offloading is dynamic. The computational complexity of each task i is quantified by the number of CPU clocks it needs to consume as si. Suppose the set of tasks unloaded to node u in time period t is Au(t), then the state transition equation of the execution queue of the computing node is:

[0067]

[0068] In view of the problems in the prior art, the embodiments of the present application provide a computation offloading method based on partially observable multi-agent reinforcement learning. Figure 3 The following is a flow chart of a method for offloading computation based on partially observable multi-agent reinforcement learning provided by an embodiment of the present application. Figure 3 As shown, the method comprises the following steps:

[0069] Step 101, obtaining a computing task generated by a user agent, and inputting the computing task into a policy neural network to generate an offloading decision;

[0070] In this embodiment, the user agent in the user agent first obtains the current observable environment information, and generates the corresponding strategy probability distribution through its own strategy network according to the partial environment information O observed by itself. The architecture of the strategy network can be composed of multiple architectures. Figure 4 A schematic diagram of a multi-layer perceptron combined with an LSTM network structure provided in this embodiment. This embodiment adopts Figure 4 The multi-layer perceptron shown in the figure is combined with an LSTM network as the architecture of our policy network and evaluation network, which can effectively adapt to time series data.

[0071] Step 102, offloading the computing task to the computing node according to the offloading decision, defining the transmission delay and computing delay by means of queuing theory and feeding back to the user agent;

[0072] In this embodiment, after the user agent generates a task, the user agent will sample its own strategy probability distribution for each task to determine the location of the computing node that needs to be unloaded for the task, and make an unloading action. When the task arrives at the computing node, the computing node obtains the computing delay and computing cost (expense, such as money) based on its own computing queue length and the number of computing resources Si required for the task, and passes it to the task source node. At the same time, the source node cannot obtain the queue state of the entire link due to partial observability, and receives the end-to-end delay of the task feedback as the communication overhead. The source node (i.e., the user agent) uses the sum of the computing delay and the communication delay and the inverse of the weighted sum of the computing cost as the direct reward for this unloading action.

[0073] Step 103, allowing the user agent to observe the computational delay of the task generated by it and the transmission delay of the link directly connected to it, and obtain feedback information based on probability control;

[0074] In this embodiment, in each time period, the user agent will uninstall tasks multiple times, and will receive direct rewards for multiple uninstall actions. It will convert the direct rewards into probability rewards and sum them up to get R. At the same time, each time an uninstall action is made, the strategy loss ploss will be obtained according to the probability distribution of the selected action and strategy. The definition of ploss for each uninstall action is log(p(a|Π(si))), that is, the probability of making the current uninstall action under strategy Π(si) and taking the logarithm. Since the user agent will uninstall tasks multiple times in the same time period, it is necessary to sum the strategy losses to get the complete ploss for a time period.

[0075] Step 104: Optimize and evaluate the evaluation result of the neural network on the current state according to the feedback information.

[0076] In this embodiment, the R value and the policy loss value are accumulated, and after a certain period of time, the accumulated R and the policy loss are back-propagated to obtain the gradient value of the neural network, and then the Adam optimizer is used to apply the gradient on the network to update the evaluation network and the policy network.

[0077] This solution is an on-line solution. The output of the evaluation network will converge to the probability of achieving the optimal state for the current state during continuous iterations, and the output of the policy network will converge to the probability of achieving the optimal state for the current decision under the current state during continuous iterations.

[0078] First, define the optimization goal of the solution. The optimization goal of the entire strategy is to maximize social benefits while ensuring individual offloading efficiency. Social benefits consist of two parts: 1. The sum of the unit computing delay and unit communication delay of all users participating in the offloading of computing tasks; 2. The cost (such as money) paid by all users to the computing nodes. The sum of the average unit computing delay and unit communication delay is Ei,t, and the cost of offloading task i to node u is Rusi(t). The optimization goal of the problem is to minimize the above formula under the premise that Vl(t) is stable.

[0079] According to the optimization goal, a partially observable multi-agent collaborative reinforcement learning model is established with Actor-Critic as the baseline. Actor-Critic is a method evolved from Policy Gradient. On the basis of Policy Gradient based on the policy value function, the value-evaluated Critic is added, which enables the strategy that originally required a batch to be updated to be updated in a single step, thereby improving the convergence speed of the algorithm. At the same time, it retains the characteristic of Policy Gradient that it can select the corresponding strategy in the continuous action space and state space. Based on the optimization formula described above, the following POMDP problem is defined, which is equivalent to the above optimization formula. This solution uses reinforcement learning to solve this POMDP problem:

[0080] User agent: a group of user agents in the cloud-edge environment. In each time period, they will generate some tasks that can be offloaded. Let the user agent set be N.

[0081] State space: S is the state space in the cloud-edge scenario. This solution defines S = {Q, L, F}. Q is the value set of the backlog of tasks on each computing node. L is the length of the virtual dynamic queue on each link, and its length represents the link bandwidth that has been used. F is the set of computing resources. Each computing node u can use Fu(t) to represent its computing resources.

[0082] Action space: Each agent needs to decide which computing node to offload the task to and the data rate of the task. For ease of description, this scheme assumes that each user generates a task at each t. The action (also called decision, strategy) of the agent corresponding to task i at t can be represented by a 2-tuple (wi(t),xi(t)), where wi(t) is a vector representing the computing node to which task i is offloaded. xi(t) represents the data rate of task transmission.

[0083] Reward mechanism: r(wi(t),xi(t)) represents the direct reward (instantaneous reward) given by the environment to the user agent after making a decision.

[0084] Observation space: Each user agent only has partial information oi about the system dynamics, rather than a complete view of the system dynamics. For example, the network links are controlled by the operator, and the user cannot observe the specific information of each link but can only observe that the end-to-end delay Di(t) is approximately equal to ΣVl(t) / cl.

[0085] Optionally, inputting the computing task into a policy neural network to generate an offloading decision includes:

[0086] Generate the strategy probability distribution based on the current observable environment information and determine the state space S of the cloud-edge system; S = {Q, L, F}, where Q is the value set of the backlog of tasks on each computing node, L is the length of the virtual dynamic queue on each link, and F is the set of computing resources. Each computing node u can use F u (t) represents its computing resources, s∈S, s t Represents the state of the cloud-edge system at time point t;

[0087] The offloading decision of each computing task is determined by sampling the strategy probability distribution, wherein the offloading decision includes: the target node w i (t), represents the computing node to which task i is unloaded at time point t; data transmission rate x i (t) represents the data rate transmitted by task i.

[0088] In this embodiment, when a computing task is generated by a user agent, the policy network of the user agent itself obtains the policy probability distribution according to the current observable environment input, and then assigns a policy (w i (t),x i (t)). The computing task is offloaded according to the strategy provided by the strategy network.

[0089] Optionally, obtaining a strategy probability distribution according to the current observable environment input includes:

[0090] Introduce a Boolean variable M(t), where M(t) = 1 represents a decision (w i (t),x i (t)) is the current optimal strategy;

[0091] Determine the strategic probability distribution of M(t) as Pr(M(t)=1|s t ,a t )=exp(-r(w i (t),x i (t))), where a t is the joint action of each user agent, r(w i (t),x i (t)) is the direct reward of the user agent. This equation shows that the optimality of state-action selection is related to the state s at time t. t Next select a t The value of the immediate reward.

[0092] Optionally, defining the transmission delay and the calculation delay by means of queuing theory and feeding back the delay to the user agent includes:

[0093] Determine the direct reward r(w) of the user agent according to the target node and the data transmission rate i (t),x i (t));

[0094] Convert the direct reward into a probability instantaneous reward exp(-r(w i (t),x i (t)));

[0095] The offloading decision of each computing task is searched and determined according to the probability reward, wherein β is defined t (s t ,a t )=Pr(M t:T |s t ,a t ), where β t (s t ,a t ) indicates that when (s t ,a t ) The probability that the sequence of actions given by the starting strategy from t to T is the optimal strategy; define β t (s t )=Pr(M t:T |s t ), β t (s t ) means that when the state s t At the beginning, the probability that a sequence of actions from t to T given by the strategy is optimal;

[0096] The distributed random optimization data is obtained, and the strategy is distributed random optimization is performed to obtain the offloading decision.

[0097] Optionally, the performing distributed random optimization on the strategy to obtain the unloading decision includes:

[0098]

[0099] g'(w i (t),x i (t))=VΣ(αE i,t (w i (t),x i (t))+βw i (t)R u s i (t))+ΣV l (t)x i (t)

[0100] Among them, the target node wi(t+1) represents the computing node to which task i is unloaded at time point t+1; Vl(t) is the link communication delay of the network link at time point t.

[0101] In this embodiment, for each user agent, a distributed random optimization method is used to solve the optimization problem proposed in 1 to iteratively approach the optimal offloading. The original optimization problem can be equivalent to minΣ(αEi,t(wi(t),xi(t))+βwi(t)Rusi(t)). For the computational delay, it can be decoupled by independently solving the computational delay of each user agent, so as to obtain the total delay in a distributed manner. As for the link communication delay, the communication delay of the task offloading between each user agent will depend on other user agents, so it is necessary to jointly optimize them to keep Vl(t) stable. Under this condition, the link communication delay is decoupled. At this time, for each i, it can be obtained independently by i itself without relying on other tasks. As for the network link Vl(t), it cannot be obtained directly due to partial observability. This solution replaces ΣVl(t) / cl with the end-to-end delay Di(t) to obtain an approximate value of Vl(t). At this time, all the distributed random optimization data can be obtained by the user agent, and the user agent performs distributed random optimization on the strategy to solve the optimal offloading.

[0102] Optionally, optimizing the evaluation result of the neural network on the current state according to the feedback information includes:

[0103] According to the feedback information and the actual value function β t (s t) calculates the loss function, wherein the calculation formula of the loss function is:

[0104] ||B(Φ θ (s t ))-Pr Π (s t+1 |s t )Φ θ (s t )-r(w i (t),x i (t))||

[0105] Among them, Pr Π (s t+1 |s t ) is the state s under this strategy t Transfer to state s t+1 The probability of θ (s t ) is the learning goal;

[0106] The parameters in the evaluation neural network are optimized with the goal of minimizing the loss function.

[0107] In this embodiment, through probabilistic control and RL estimated random optimization actions, each agent can independently search for the optimal unloading at time t. For RL based on probabilistic control in Pro-CoRL, this scheme applies an Actor-Critic network, where the Critic network estimates the strategy based on historical information and feedback rewards at each time step. The Actor network generates actions based on the parameters optimized by the values ​​calculated by the Critic network. According to the above optimization iteration, the Actor-Critic network is trained:

[0108] Critic training in Pro-CoRL: The Critic network itself is an evaluation of the state, and it is required to approach the modeled value function βt(st). This scheme comprehensively considers the policy gradient and the value function, and trains the parameters in the Critic network by minimizing the Bellman error. Define Φθ(st) as the learning goal of the Critic network, which approaches the true βt(st). PrΠ(st+1|st) is the probability of transitioning from state st to state st+1 under this strategy. Therefore, the training objective, that is, the loss function, can be described as ||B(Φθ(st))-PrΠ(st+1|st)Φθ(st)-r(wi(t),xi(t))||

[0109] Actor training in Pro-CoRL: Each agent trains a Critic model. In Pro-CoRL, the policy at each task t is parameterized as Π(wi(t)), where wi(t) is the network parameter for each task i. To derive the optimal control, for each agent, the actor network parameters are updated by maximizing the conditional probability that the current policy is the optimal policy under the constraint of KL divergence.

[0110] In order to implement the above embodiments, the present application also proposes a computational offloading device based on partially observable multi-agent reinforcement learning. Figure 5 The schematic diagram of the structure of a computation offloading device based on partially observable multi-agent reinforcement learning provided in the embodiment of the present application is shown in FIG. Figure 5 As shown, the device comprises:

[0111] A decision module 510 is used to obtain a computing task generated by a user agent and input the computing task into a policy neural network to generate an offloading decision;

[0112] A task offloading module 520 is used to offload the computing task to the computing node according to the offloading decision, define the transmission delay and the computing delay by means of queuing theory and feed back to the user agent;

[0113] A feedback module 530, configured to enable the user agent to observe the computation delay of the task generated by the user agent and the transmission delay of the link directly connected to the user agent, and obtain feedback information based on probability control;

[0114] The optimization module 540 is used to optimize the evaluation result of the current state of the evaluation neural network according to the feedback information.

[0115] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0116] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0117] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0118] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0119] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0120] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0121] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0122] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0123] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0124] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0125] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0126] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0127] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0128] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A computation offloading method based on partially observable multi-agent reinforcement learning, characterized in that: The following steps are involved: Obtaining a computing task generated by a user agent, and inputting the computing task into a policy neural network to generate an offloading decision; offloading the computing task to the computing node according to the offloading decision, defining the transmission delay and the computing delay by means of queuing theory and feeding back the delay to the user agent; The user agent is made to observe the computation delay of the task generated by the user agent and the transmission delay of the link directly connected to the user agent, and obtain feedback information based on probability control; Optimizing and evaluating the neural network's evaluation result of the current state according to the feedback information; The step of inputting the computing task into a policy neural network to generate an offloading decision comprises: Generate the strategy probability distribution based on the current observable environment information and determine the state space S of the cloud-edge system, where S = {Q, L, F}, Q is the value set of the backlog of tasks on each computing node, L is the length of the virtual dynamic queue on each link, F is the set of computing resources, and each computing node u can use F u (t) represents its computing resources, s∈S, s t Represents the state of the cloud-edge system at time point t; The offloading decision of each computing task is determined by sampling the strategy probability distribution, wherein the offloading decision includes: the target node w i (t), represents the computing node to which task i is unloaded at time point t; data transmission rate x i (t),represents the data rate transmitted by task i; The strategy probability distribution is obtained according to the current observable environment input, including: Introduce a Boolean variable M(t), where M(t) = 1 represents a decision (w i (t),x i (t)) is the current optimal strategy; Determine the strategic probability distribution of M(t) as Pr(M(t)=1|s t ,a t )=exp(-r(w i (t),x i (t))), where a t is the joint action of each user agent, r(w i (t),x i (t)) is the direct reward of the user agent; The method of defining transmission delay and calculation delay by means of queuing theory and feeding back the delay to the user agent includes: Determine the direct reward r(w) of the user agent according to the target node and the data transmission rate i (t),x i (t)); Convert the direct reward into a probability instantaneous reward exp(-r(w i (t),x i (t))); The offloading decision of each computing task is searched and determined according to the probability reward, wherein β is defined t (s t ,a t )=Pr(M t:T |s t ,a t ), where β t (s t ,a t ) indicates that when (s t ,a t ) The probability that the sequence of actions given by the starting strategy from t to T is the optimal strategy; define β t (s t )=Pr(M t:T |s t ), β t (s t ) means that when the state s t At the beginning, the probability that a sequence of actions from t to T given by the strategy is optimal; Obtaining distributed random optimization data, and performing distributed random optimization on the strategy to obtain the unloading decision; The optimizing and evaluating the evaluation result of the neural network on the current state according to the feedback information includes: According to the feedback information and the actual value function β t (s t ) calculates the loss function, wherein the calculation formula of the loss function is: ||B(Φ θ (s t ))-Pr Π (s t+1 |s t )Φ θ (s t )-r(w i (t),x i (t))|| Among them, Pr Π (s t+1 |s t ) is in Π Under the strategy, from state s t Transfer to state s t+1 The probability of θ (s t ) as the learning goal; The parameters in the evaluation neural network are optimized with the goal of minimizing the loss function.

2. The method according to claim 1, characterized in that The step of performing distributed random optimization on the strategy to obtain the unloading decision includes: g'(w i (t),x i (t))=VΣ(αE i,t (w i (t),x i (t))+βw i (t)R u s i (t))+ΣV l (t)x i (t) Among them, the target node w i (t+1), represents the computing node to which task i is unloaded at time point t+1; V l (t) is the link communication delay of the network link at time point t.

3. A computation offloading device based on partially observable multi-agent reinforcement learning, characterized in that: include: A decision module, used for acquiring computing tasks generated by the user agent and inputting the computing tasks into the policy neural network to generate an offloading decision; A task offloading module, used for offloading the computing task to the computing node according to the offloading decision, defining the transmission delay and the computing delay by means of queuing theory and feeding back the delay to the user agent; A feedback module, used to enable the user agent to observe the computation delay of the task generated by it and the transmission delay of the link directly connected to it, and obtain feedback information based on probability control; An optimization module, used for optimizing the evaluation result of the neural network on the current state according to the feedback information; The decision module is also used to generate a strategy probability distribution based on the current observable environment information and determine the state space S of the cloud-edge system, where S = {Q, L, F}, Q is the value set of the backlog of tasks on each computing node, L is the length of the virtual dynamic queue on each link, F is the set of computing resources, and each computing node u can use F u (t) represents its computing resources, s∈S, s t Represents the state of the cloud-edge system at time point t; The offloading decision of each computing task is determined by sampling the strategy probability distribution, wherein the offloading decision includes: the target node w i (t), represents the computing node to which task i is unloaded at time point t; data transmission rate x i (t),represents the data rate transmitted by task i; The task offloading module is also used to introduce a Boolean variable M(t), where M(t)=1 represents a decision (w i (t),x i (t)) is the current optimal strategy; Determine the strategic probability distribution of M(t) as Pr(M(t)=1|s t ,a t )=exp(-r(w i (t),x i (t))), where a t is the joint action of each user agent, r(w i (t),x i (t)) is the direct reward of the user agent; The method of defining transmission delay and calculation delay by means of queuing theory and feeding back the delay to the user agent includes: Determine the direct reward r(w) of the user agent according to the target node and the data transmission rate i (t),x i (t)); Convert the direct reward into a probability instantaneous reward exp(-r(w i (t),x i (t))); The offloading decision of each computing task is searched and determined according to the probability reward, wherein β is defined t (s t ,a t )=Pr(M t:T |s t ,a t ), where β t (s t ,a t ) indicates that when (s t ,a t ) The probability that the sequence of actions given by the starting strategy from t to T is the optimal strategy; define β t (s t )=Pr(M t:T |s t ), β t (s t ) means that when the state s t At the beginning, the probability that a sequence of actions from t to T given by the strategy is optimal; Obtaining distributed random optimization data, and performing distributed random optimization on the strategy to obtain the unloading decision; The optimization module is also used to calculate the value of the feedback information and the actual value function β t (s t ) calculates the loss function, wherein the calculation formula of the loss function is: ||B(Φ θ (s t ))-Pr Π (s t+1 |s t )Φ θ (s t )-r(w i (t),x i (t))|| Among them, Pr Π (s t+1 |s t ) is in Π Under the strategy, from state s t Transfer to state s t+1 The probability of θ (s t ) as the learning goal; The parameters in the evaluation neural network are optimized with the goal of minimizing the loss function.

4. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 2 when executed by a processor.

6. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 2 when being executed by a processor.

Images