A multi-agent based resource allocation and task offloading optimization method

Abstract

The present invention belongs to the field of mobile communication technology, and relates to a multi-agent-based resource allocation and task offloading optimization method; the method includes treating edge cloud servers and energy collection-enabled mobile devices in heterogeneous network environments as independent decision-making intelligent units, combining actual computing offloading scenarios, constructing a Stackelberg game model for the benefits of edge cloud resource allocation and mobile device task offloading, considering random time-varying edge environments and incomplete state observations, remodeling the Stackelberg game model as a partially observable Markov decision process, and establishing a multi-agent deep reinforcement learning algorithm to solve the partially observable Markov decision process game model, so as to learn the optimal edge cloud resource collaborative allocation strategy and task offloading strategy. The present invention can effectively improve the task computing success rate of edge cloud servers and reduce the task abandonment rate of mobile devices.

Description

Technical Field

[0001] The invention belongs to the technical field of mobile communications and relates to a resource allocation and task offloading optimization method based on multi-agents. Background Art

[0002] At present, the fifth generation mobile communication system (5G) is being rapidly deployed and gradually put into large-scale commercial use. Although the 5G mobile communication system has greatly improved in terms of transmission rate, connection density, transmission delay, energy efficiency, etc., it still cannot effectively support the rapid development of emerging industries such as augmented reality, virtual reality, telemedicine, smart cities, unmanned driving, online games, and smart factories, which are characterized by big data and intelligence, and are characterized by intensive computing and delay sensitivity. At the same time, driven by the Internet of Things (IoT) and mobile Internet technologies, the number of network edge device connections and data traffic has increased dramatically. Massive IoT applications with cloud-based business computing, ubiquitous connections, and heterogeneous business types pose a series of challenges to efficient mobile communication networks and wireless edge network resource allocation. Mobile Edge Computing (MEC), as an emerging computing model, can significantly improve users' service experience by offloading all or part of local computing tasks to MEC servers.

[0003] With the convergence of heterogeneous networks and MEC, task offloading and network resource allocation in converged scenarios face new challenges. Some of the main achievements are: (1) A Novel Mobility-aware Offloading Management Scheme in Sustainable Multi-access Edge Computing (Reference: Guan S, Boukerche AA Novel Mobility-aware Offloading Management Scheme in Sustainable Multi-access Edge Computing [J]. IEEE Transactions on Sustainable Computing, 2022, 7(1): 1-13. DOI: 10.1109 / TSUSC.2021.3065310): The algorithm considers the intermittent connection and load imbalance of mobile devices between servers in the converged environment of heterogeneous networks and MEC, and proposes a priority-based mobility-aware computing offloading model to optimize the execution efficiency, computing latency and energy efficiency of tasks. (2) SDN-based Service Mobility Management in MEC-enabled 5G and Beyond Vehicular Networks (Reference: Shah SDA, Gregory MA, Li S, et al. SDN-based Service Mobility Management in MEC-enabled 5G and Beyond Vehicular Networks [J]. IEEE Internet of Things Journal (Early Access), 2022, DOI: 10.1109 / JIOT.2022.3142157): This algorithm takes into account the different computing delays and bandwidth requirements of different tasks in the fusion of MEC and heterogeneous networks, and proposes a centralized control optimization method based on software-defined network (SDN) to optimize task offloading and network resource allocation of mobile devices.

[0004] These works studied the collaborative computing and task migration problems between servers. The offloaded tasks can further improve the network computing performance with the assistance of adjacent servers. However, these works use traditional optimization methods (such as convex optimization theory and game theory) and assume that the deep uncertain network model is completely known. They only optimize the computing offloading performance in an overly ideal network environment, which is of poor practicality. A few industry researchers use artificial intelligence or machine learning methods to study dynamic computing offloading algorithms and optimize communication resource allocation and task offloading strategies. However, most of these works model and optimize the compromise between communication and task offloading, and do not consider the distribution characteristics of computing resources in multi-source and heterogeneous network scenarios and the impact of random time-varying network environments on task offloading and resource allocation. In addition, in actual edge cloud networks, users often use autonomous and random cloud access methods, which results in insufficient and unbalanced utilization of edge cloud resources, which in turn affects the utilization rate of edge cloud resources and user service experience. Therefore, the intelligent collaboration and task offloading optimization of edge cloud servers in heterogeneous edge networks have important research value. Summary of the invention

[0005] In view of this, it is necessary to design a multi-agent based resource allocation and task offloading optimization method. The edge cloud servers and energy harvesting enabled mobile devices in heterogeneous network environments are regarded as agents that can make independent decisions. Combined with the actual computing offloading scenario, the benefits of edge cloud resource allocation and mobile device task offloading are constructed into a Stackelberg game model. Considering the random time-varying edge environment and incomplete state observation, the Stackelberg game model is remodeled as a partially observable Markov decision process, and a deep reinforcement learning algorithm using multi-agents is established to solve the partially observable Markov decision process game model, so as to learn the optimal edge cloud resource collaborative allocation strategy and task offloading strategy.

[0006] The technical solutions adopted by the present invention to solve the above technical problems include:

[0007] A multi-agent-based resource allocation and task offloading optimization method, the method comprising:

[0008] Step 1) for the edge cloud server network enabled by energy harvesting in a heterogeneous network environment, respectively establish a computing task model, a communication model for task offloading to the heterogeneous network, a task execution model, and an energy consumption model, and obtain the environmental state according to the established model;

[0009] Step 2) The edge cloud server is regarded as the master, and the master's immediate reward function is constructed according to the obtained environmental state, and the benefit objective function of the long-term cumulative reward of the master's execution of collaborative resource allocation decisions is established according to the master's immediate reward function;

[0010] Step 3) The mobile device is regarded as a slave, and an immediate reward function of the slave is constructed according to the obtained environmental state, and a benefit objective function of the long-term cumulative reward of the slave for executing the task offloading decision is established according to the immediate reward function of the slave;

[0011] Step 4) Maximizing the profit objective function of the master and the profit objective function of the slave, combining the decision constraints of the master and the slave, and constructing the Stackelberg game model;

[0012] Step 5) Based on the random time-varying network environment and the incomplete observation states of the master and the slave, the Stackelberg game model is modeled as a partially observable Markov decision process game model;

[0013] Step 6) Use a multi-agent deep reinforcement learning algorithm to solve the partially observable Markov decision process game model, and learn and optimize the edge cloud resource collaborative allocation decision and mobile device task offloading decision through centralized training and distributed execution.

[0014] Beneficial effects of the present invention:

[0015] The present invention studies the optimization method of edge cloud intelligent collaboration and task offloading in heterogeneous network scenarios. First, heterogeneous edge cloud servers and energy-harvesting enabled mobile devices are regarded as intelligent units that can make independent decisions. Combined with the actual computing offloading scenario, the edge cloud resource allocation and mobile device task offloading problems are abstracted into a typical distributed Stackelberg game model. The edge cloud acts as the master to allocate computing and transmission resources to mobile devices, and the mobile devices act as slaves to perform task offloading and local energy management. Then, considering the random time-varying edge environment and incomplete state observations, the intelligent agent decision problem is modeled as a partially observable Markov decision process, and a multi-agent deep deterministic policy gradient model based on Stackelberg dynamic game is established to learn the optimal edge cloud resource collaborative allocation and task offloading strategy. The model proposed in the present invention can effectively improve the task calculation success rate of edge cloud servers and reduce the task abandonment rate of mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a multi-agent-based edge cloud server intelligent collaboration and task offloading optimization method in the present invention;

[0017] Figure 2 This is a model diagram of the MEC offloading system supporting energy collection in a heterogeneous edge network in the present invention;

[0018] Figure 3 A typical DDPG framework diagram used in the present invention;

[0019] Figure 4This is a framework diagram of SG-MADDPG based on MEC in the present invention;

[0020] Figure 5 This is a graph showing changes in the average revenue per round during the training phase of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Figure 1 FIG. 1 is a flow chart of a multi-agent-based edge cloud server intelligent collaboration and task offloading optimization method in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0023] Step 1) For the MEC network enabled by Energy Harvesting (EH) in a heterogeneous network environment, a computing task model is established, and a communication model, a task execution model, and an energy consumption model for task offloading to the heterogeneous network are established, and the environmental state is obtained according to the established models;

[0024] Among them, the models established above are all system models, which are established for the MEC network enabled by energy harvesting (EH) in a heterogeneous network environment. The system model can reflect the environmental status of the above MEC network during resource allocation and task offloading. The environmental status here includes but is not limited to the available computing resources and wireless channel gain of the edge cloud server; the location of the mobile device, task backlog, available energy and collected energy. Through these environmental status, the benefits that can be obtained when the edge cloud server and / or mobile device execute relevant decisions can be obtained.

[0025] Step 2) The edge cloud server is regarded as the master, and the master's immediate reward function is constructed according to the obtained environmental state, and the benefit objective function of the long-term cumulative reward of the master's execution of collaborative resource allocation decisions is established according to the master's immediate reward function;

[0026] In an embodiment of the present invention, the edge server is regarded as the master (represented as L), and the goal is to maximize the collective benefit of the edge cloud servers, and collaborative resource allocation is performed according to the network environment status. Since the collective benefit of the edge cloud servers is taken as the goal here, all edge cloud servers can be regarded as an edge cloud server cluster. This edge cloud server cluster is regarded as a whole, and together as the master, a better collaborative work effect can be achieved.

[0027] Step 3) The mobile device is regarded as a slave, and an immediate reward function of the slave is constructed according to the obtained environmental state, and a benefit objective function of the long-term cumulative reward of the slave for executing the task offloading decision is established according to the immediate reward function of the slave;

[0028] In the embodiment of the present invention, each mobile device is regarded as a slave (denoted as F), and with the goal of maximizing the benefit of each mobile device, task offloading is performed according to status information such as local task characteristics, thereby achieving task offloading optimization for each mobile device.

[0029] In some preferred embodiments of the present invention, each mobile device can also perform local energy management based on status information such as battery energy level, wherein any mobile device captures energy in the surrounding environment through energy collection technology to support data communication and business processing of the mobile device; through the management of energy collection and energy output of mobile devices, the mobile device can maintain long-term and stable computing performance.

[0030] Step 4) Maximizing the profit objective function of the master and the profit objective function of the slave, combining the decision constraints of the master and the slave, and constructing the Stackelberg game model;

[0031] In an embodiment of the present invention, since the maximization of the benefits of resource allocation performed by the edge cloud server and the maximization of the benefits of task offloading performed by the mobile device are a one-to-many game problem, the present invention constructs it as a Stackelberg game model, allowing the edge cloud to act as the master to allocate computing and transmission resources to the mobile device, and the mobile device to act as the slave to perform task offloading and local energy management.

[0032] Step 5) Based on the random time-varying network environment and the incomplete observation states of the master and the slave, the Stackelberg game model is modeled as a partially observable Markov decision process game model;

[0033] In an embodiment of the present invention, considering that both the master and the slave can only observe incomplete state information, and the network environment is also randomly time-varying, it will make the optimization problem of the above-mentioned Stackelberg game model difficult to solve quickly. Therefore, the present invention models the game optimization problem of the master and the slave as a POMDP game problem. The basic elements in the POMDP game include a participant set, an environmental state space, an observation space, an action space, and a reward function. These basic elements constitute a partially observable Markov decision process game model.

[0034] Step 6) Use a multi-agent deep reinforcement learning algorithm to solve the partially observable Markov decision process game model, and learn and optimize the edge cloud resource collaborative allocation decision and mobile device task offloading decision through centralized training and distributed execution.

[0035] In an embodiment of the present invention, the edge cloud server cluster and each mobile device are respectively used as an intelligent agent. The heterogeneous network environment first sends an environmental state to the intelligent agent, and then the intelligent agent takes action to respond to the state based on its knowledge. After that, the heterogeneous network environment sends the next state and returns the reward to the intelligent agent. The intelligent agent updates its knowledge with the reward returned by the heterogeneous network environment and evaluates the previous action. This cycle continues until the heterogeneous network environment sends a termination state to end the event; during the training process, the critic centrally shares global information, and during the execution process, the actor obtains local information, that is, the observation information of the current intelligent agent, in a distributed manner; through this training execution process, it is possible to learn and optimize the edge cloud resource collaborative allocation decision and the mobile device task offloading decision.

[0036] The following will provide a detailed description of each step from step 1) to step 6), please refer to the following:

[0037] 1. System Model

[0038] like Figure 2 As shown in the figure, the present invention considers a typical EH-enabled MEC network scenario in a heterogeneous network environment. The scenario includes m edge mobile devices (Mobile Device, MD), which are defined as a set One macrocell base station (MBS) and n femtocell base stations (FBS), define the FBS set Assume that any mobile device supports energy harvesting technology; and any base station is equipped with an MEC server. The MEC server can only provide computing and data storage services to mobile devices within its wireless coverage. In addition, in order to avoid confusion, the present invention uses MBS and FBS to represent the MEC server connected to the MBS and FBS respectively. The system runs in discrete time slots and defines time as a set Let τ represent the time length of each time slot.

[0039] definition Represents the resource set of MBS, where: Indicates the maximum computing capacity of MBS, represents the available bandwidth resources of MBS; accordingly, let represents the resource set of the jth FBS. Usually, the wireless coverage of MBS is much larger than that of FBS. Therefore, assuming that all mobile devices All are within the coverage of MBS, while FBS can only cover some devices. represents the set of mobile devices within the coverage of the jth FBS in time slot t. In addition, let and denote the maximum battery capacity of device i and the battery energy level at time slot t, respectively.

[0040] 1. Computational Task Model

[0041] Assume that the arrival of MD tasks follows an independent and identically distributed Bernoulli process with parameter λ∈[0,1]. Indicates the task arrival indicator variable of device i in time slot t, i.e. b i (t) = 1 means a new task unit is generated, otherwise, b i (t) = 0. To simplify the analysis complexity, for any mobile device, the arriving task is represented as a triple (μ, ν, δ), where μ, ν, δ represent the task size (in bits), computational density (the number of CPU cycles required to execute), and maximum computational latency constraint, respectively. In addition, the present invention assumes that any task unit is an indivisible minimum task unit, and the local task processing capability of the mobile device is disabled.

[0042] In any time slot, the MD can offload the computational tasks to the MBS / FBS or backlog the tasks to the local task queue. Without loss of generality, the backlogged tasks are modeled as the first-input first-out (FIFO) rule in the device task queue for task unloading. i (t)∈{-1,0,1} represents the task offloading strategy of device i in time slot t, where a iWhen (t) = -1, it means that the calculation offloading function is disabled. Let a i When (t) = 0, it means that device i offloads the task to MBS. The task is unloaded to FBS j. Based on the above assumptions, we can get the task queue Q of device i: i The evolution process of (t) is in, is the maximum task queue backlog of device i, 1{x} is an indicator function that is equal to 1 when condition x is true and 0 otherwise.

[0043] When the task queue backlog reaches the upper limit (such as ), then the newly arrived task will be discarded, which can be expressed as represents the task abandonment indicator factor of mobile device i in time slot t.

[0044] 2. Task offloading to the MBS model

[0045] Since all mobile devices are within the wireless coverage of MBS, any mobile device Both can choose to offload tasks to MBS.

[0046] Communication Model: Assuming Devices The wireless channel between the device i and the macro base station is a Gaussian white noise channel. Let g0(t) represent the channel gain of MBS in time slot t, then the uplink data transmission rate r from device i to MBS can be obtained. i0 (t) is Among them, B i0 (t) represents the bandwidth resource allocated to device i by MBS in time slot t, I0(t) represents the average noise power in the MBS environment, P i0 is the data transmission power of device i under MBS, and the transmission delay of device i to offload the task to MBS can be obtained. for

[0047] Task execution model: using f i0 (t) represents the computing resources allocated by MBS to device i, then the task computing latency of device i can be obtained for If device i offloads the task to MBS, and the computing resources allocated by MBS are insufficient to complete the task processing within the delay constraint (e.g. ), then the task calculation offloading is considered to have failed, which can be expressed as It represents the task calculation failure indicator factor of MBS in time slot t.

[0048] 3. Task offloading to the FBS model

[0049] Communication model: In any time slot t, let Denotes the channel gain of FBSj, and defines B ij (t) is the bandwidth resource allocated by FBSj to device i in time slot t, I j (t) represents the average noise power in the FBS environment, P ij is the data transmission power of device i under FBSj, then the uplink data transmission rate from device i to FBSj can be defined as The task transmission delay from device i to FBSj can be obtained for

[0050] Task execution model: Assume that in time slot t, the computing resource allocated to device i by FBSj is f ij (t), then the task processing delay can be obtained as Similarly, if the task offloading delay exceeds the maximum computational delay limit of device i (e.g. ), then FBSj will be penalized, expressed as represents the task calculation failure indicator factor of FBSj in time slot t.

[0051] The communication energy consumption of device i is When the energy consumption of mobile device i (mainly communication energy consumption e i (t)) is greater than the battery energy level of the device in the current time slot t (such as e i (t)>E i (t)), it is considered that the calculation offloading of mobile device i fails, which can be expressed as φ i (t) = 1{e i (t)>E i (t)}. In time slot t, the task offloading delay of device i can be expressed as:

[0052] definition is the energy that device i can collect in time slot t, and the battery energy E of device i i The evolution process of (t) is

[0053] 2. Problem Description

[0054] The problem of edge cloud resource allocation and mobile device task offloading is modeled as a typical distributed Stackelberg game model, in which the edge server is regarded as the master (denoted as L), and the goal is to maximize the collective benefits of the edge cloud server and perform collaborative resource allocation according to the network environment state; the mobile device is regarded as the slave (denoted as F), and performs task offloading and local energy management according to status information such as local task characteristics and battery energy level. In order to quantify the performance of edge cloud services in heterogeneous networks, and They are the computing resources and bandwidth resource allocation strategy sets of edge cloud servers (F * (t) and B * (t) represents its optimal strategy); a(t) is the set of mobile device task offloading strategies; and They are respectively the satisfaction functions for the corresponding indicators of the edge cloud server and the mobile device, which are generally defined as positive monotonically decreasing functions; ω1, ω2, ω3 are the relevant weight coefficients of the edge cloud server, and λ1 and λ2 are the relevant weight coefficients of the mobile device. The instant reward of the master is defined as:

[0055]

[0056] Similarly, in time slot t, the immediate reward r from party i is i F (t) can be expressed as:

[0057]

[0058] In order to maintain the long-term computing performance of the system in a random time-varying network environment, the long-term cumulative rewards R L and The revenue maximization problem is:

[0059]

[0060] sta i (t)∈{-1,0,1}

[0061]

[0062]

[0063]

[0064]

[0065] Among them, F(t) is the computing resource allocation strategy set of the edge cloud server, B(t) is the bandwidth resource allocation strategy set of the edge cloud server; a(t) is the task offloading strategy set of the mobile device; R L represents the long-term cumulative reward of the edge cloud server, r L (t) represents the instantaneous reward of the edge cloud server in time slot t; represents the long-term cumulative reward of mobile device i, r i F (t) represents the instantaneous reward of mobile device i in time slot t, Represents the collection of mobile devices in the network; a i (t) represents the task offloading strategy of mobile device i in time slot t; 1{x} is an indicator function, which is equal to 1 when condition x is true and 0 otherwise; f i0 (t) represents the computing resources allocated by MBS to mobile device i in time slot t, B i0 (t) represents the bandwidth resources allocated by MBS to mobile device i in time slot t; Indicates the maximum computing capacity of MBS, Indicates the available bandwidth resources of MBS; f ij (t) represents the computing resources allocated by FBSj to mobile device i in time slot t; represents the maximum computing capacity of FBSj; B ij (t) represents the bandwidth resources allocated by FBSj to mobile device i in time slot t; represents the available bandwidth resources of FBSj; represents the set of mobile devices within the coverage of FBSj; m represents the total number of mobile devices, MBS represents the macro base station, and FBSj represents the jth femto base station.

[0066] It can be understood that the above constraints represent, in turn: the task offloading method that the mobile device can choose; the computing resources allocated by the edge cloud cannot exceed the maximum computing resource constraints of MBS and FBS; the bandwidth resources allocated by the edge cloud cannot exceed the maximum bandwidth constraints of MBS and FBS.

[0067] 3. Edge cloud intelligent collaboration and distributed task offloading optimization

[0068] The problem is a typical mixed integer nonlinear programming (MINLP) problem, which is difficult to solve quickly using the traditional Stackelberg game model. The allocation of edge cloud resources and the task offloading strategy of mobile devices are only related to their current state, but not to their past state. It is a typical Markov decision process (MDP) problem. Therefore, the optimization problem is transformed into an MDP game problem and solved using the deep reinforcement learning (DRL) method.

[0069] 1. Analysis of resource allocation and task offloading issues

[0070] In order to achieve efficient task offloading and edge resource allocation, protect user privacy, and reduce frequent information interaction between edge cloud and mobile devices. In the present invention, the edge cloud server / master is considered to be able to observe only incomplete channel information and partial information of mobile devices (such as the location information of mobile devices); at the same time, any mobile device / slave can only observe partial information of the edge cloud server (such as adjacent FBS). Based on the above assumptions, the game optimization problem of the master and the slave is modeled as a POMDP game problem.

[0071] use and They represent the set of participants, the environment state space, the observation space, and the action space in the game, respectively. and are the observation spaces of the edge cloud server and device i, respectively; and are the action spaces of the edge cloud server and device i respectively. In any time slot, the basic elements of the POMDP game are summarized as follows:

[0072] Participant Collection In the above game model, there are 1 edge cloud agent L and m mobile device agents F1,…,F i ,…,F m Among them, 1 edge cloud agent refers to the edge cloud server cluster, all edge cloud servers work together as a cluster to execute allocation decisions, and m+1 participants can be represented as

[0073] Environmental state space At time t, the network environment status It mainly includes three parts:

[0074] Edge cloud server / host state: In any time slot, the state information of the edge cloud server mainly consists of the available computing resources. The available computing resources of MBS in time slot t are expressed as The available computing resources of FBSj in time slot t are expressed as

[0075] Mobile device / slave status: For any mobile device, its status mainly includes location, task backlog Q i (t), available energy E i (t) and the energy collected in the current time slot set up Represents the location information of device i, where l i (t) = 0 means that mobile device i is not within the coverage of any FBS, l i(t)=j indicates that device i is within the wireless coverage of FBSj.

[0076] Channel state: The channel state mainly includes the wireless channel gain g0(t) and

[0077] Therefore, the environmental state space s(t) at time t can be defined as:

[0078]

[0079] Q1(t),…,Q i (t),…,Q m (t),

[0080] E1(t),…,E i (t),…,E m (t),

[0081] l1(t),…,l i (t),…,l m (t),

[0082]

[0083] g0(t),g1(t),…,g j (t),…,g m (t)}

[0084] Observation state space O L and At time t, the observable space of the edge cloud agent It mainly includes the remaining local computing resources and the locations of all mobile devices, denoted as For any mobile device agent, the observable space It can be modeled as

[0085] Action Space and set up represents the action space of the edge cloud server at time t, then the definition of Z(t) can be expressed as:

[0086]

[0087] in, represents the number of mobile devices covered by FBSj, i (j) represents the i-th mobile device under FBSj. is the action space of device i at time t. According to the above analysis, it can be defined as

[0088] Reward function: The instant reward functions of the edge cloud and mobile device can be defined as r L (t) and Each agent performs an action in the current time slot and receives a reward (such as r L (t) or r i F (t)). The agents will then update their respective strategies based on the rewards they receive.

[0089] 2. Edge cloud resource allocation and task offloading based on multi-intelligence

[0090] In the above optimization problem, there are multiple agents (a master agent and m slave agents), and the action strategy of each agent will affect the strategies of other agents. For example, when most mobile devices offload tasks to the MBS, the available computing resources and bandwidth resources of the MBS will be reduced, while other mobile devices will choose to offload tasks to neighboring FBSs with more abundant resources to maximize their returns. For traditional single-agent deep reinforcement learning models (such as Q-learning and Deep Q Network (DQN)), each agent learns and improves its strategy through interaction with the environment. Independent learning cannot effectively solve the policy optimization problem in multi-agent scenarios. To this end, in this section, a multi-agent reinforcement learning framework is used to solve the above optimization problem.

[0091] As an extension of Deep Deterministic Policy Gradient (DDPG), Multi Agent Deep Deterministic Policy Gradient (MADDPG) provides a collaborative decision-making framework for multi-agent environments. Below, we first give a brief overview of DDPG to clearly show the training and execution process of MADDPG.

[0092] (1) DDPG model

[0093] Generally speaking, reinforcement learning (RL) algorithms can be divided into two categories: action value methods (such as the DQN series) and policy gradient methods (such as policy gradient (PG), deterministic policy gradient (DPG), deep deterministic policy gradient (DDPG), etc.). The former is often used to solve discrete action decision-making problems in reinforcement learning, but it still has many limitations in dealing with continuous action problems and random policy problems.

[0094] Taking mobile device i as an example, we explain how to use PG, DPG and DDPG algorithms to solve the corresponding MDP problem. First, define the state-action value function Q of device i i (s(t), a(t)), that is, the reward obtained by the device taking action a(t) in state s(t). Given the current state s of device i i (t), action a i (t), Q i (s(t),a(t)) is expressed as Among them, π i is the policy of device i, and γ is the discount factor.

[0095] In addition, according to the Bellman optimality equation, the state-action value function can be recursively expressed as

[0096] Policy Gradient (PG): The policy gradient algorithm aims to maximize the benefit and learns the parameterized policy (such as ). For example, using the PG algorithm, the optimization action of mobile device i can be expressed as in, is the parameterized strategy of mobile device i, θ i is the policy parameter. Accordingly, the objective function of mobile device i can be expressed as in, is the state distribution. According to the policy gradient theory, α is defined as a non-negative step size, and the policy gradient is related to the parameter θ i The updating process can be expressed as

[0097] Deterministic Policy Gradient (DPG): The policy learned by the PG algorithm is a random policy. Therefore, to select the optimal action for device i, it is necessary to calculate the value of each possible action and select the optimal policy. Obviously, the computational complexity of the PG algorithm increases as the action space increases.

[0098] As an extension of policy gradient, DPG can be achieved through a deterministic strategy (such as ) to obtain the optimal action value. Mobile device i is in s i The optimal action selected in state (t) can be expressed as In DPG, the objective function of mobile device i is expressed as

[0099] Furthermore, the gradient of the policy can be expressed as

[0100] Deep Deterministic Policy Gradient (DDPG): As a combination of deep learning and DPG, DDPG is an Actor-Critic framework, mainly composed of Actor and Critic. The Actor part is based on the input state s i (t) Output a deterministic strategy (such as );The Critic part is based on the input state s i (t) and the resulting deterministic strategy a i (t) Output an estimated state-action Q value.

[0101] Similar to the DQN algorithm, DDPG also uses experience replay cache and target network in, Used to store experience from the training phase, the target network is used to reduce the relevance of training data. Figure 3 As shown, both the Actor and the Critic have an online network and a target network. and They represent the online network parameters of the Actor and Critic of mobile device i respectively. We randomly sample the empirical data in , set N as the size of the mini-batch, and use Ornstein-Uhlenbeck (OU) noise for exploration. Then, we update the parameters of the Actor and Critic online networks based on the sampled empirical data. Among them, in order to maximize the cumulative return, the Actor online network parameters The update of can be obtained by maximizing the policy objective function, which is

[0102] In addition, the parameters of the Critic online network can be updated by minimizing the loss function It is expressed as:

[0103]

[0104] Among them, y i (t) can be expressed as

[0105] In addition, in order to improve the stability of the algorithm, DDPG uses soft updates for the target network, which can be expressed as:

[0106]

[0107]

[0108] in, and

[0109] (2) Multi-Agent Deep Deterministic Policy Gradient (MADDPG)

[0110] Multi-agent Deep Deterministic Policy Gradient (MADDPG) extends DDPG to multi-agent environments through centralized training and distributed execution. Unlike single-agent reinforcement learning algorithms, during the training phase of MADDPG, each agent uses the information of other agents (such as observations and actions of other agents) to improve its local strategy. For example, during the training phase, the observed state of mobile device i is represented as:

[0111]

[0112] in, And k≠i.

[0113] In order to adapt to the dynamic environment of multiple agents, MADDPG further improves the experience replay buffer based on DDPG. Any experience data for any agent includes not only the local observation of the current agent, but also the observations and actions of other agents. For example, the experience playback of mobile device i Any experience in can be expressed as Similarly, the observed states and experience replay values ​​of other agents during the training phase can be inferred.

[0114] In the execution phase of MADDPG, for each agent, the Actor only needs to observe the local state to determine the strategy to be executed. For example, the Actor of mobile device i determines the strategy to be executed according to Select a strategy. That is, during the MADDPG execution phase, each agent is able to select a local execution strategy without knowing the information of other agents.

[0115] In the POMDP problem, the SG-MADDPG model based on multi-agent Stackelberg game is used to compare the edge cloud agent (L) and the mobile device agent. Conduct centralized training, such as Figure 4As shown in the figure, each agent can update its strategy based on its independent reward function, Actor network and Critic network. The trained model is executed in a distributed manner to obtain the resource allocation strategy of the edge cloud agent and the task offloading strategy of the mobile device.

[0116] It can be understood that in the embodiments of the present invention, the multi-agent deep reinforcement learning algorithm used to solve the partially observable Markov decision process game model can be any existing deep reinforcement learning algorithm, such as the DQN algorithm, DDPG algorithm and MADDPG algorithm in the above embodiments, and the present invention does not make specific limitations on this.

[0117] The task arrival probability of each device is set to 0.5 and the average energy harvesting power is set to 100 mW. Figure 5 The average rewards obtained by the master and slave in each round during the training phase are shown. Figure 5 It can be seen that at the beginning of training, the fluctuations are large, and the overall trend of the rewards obtained by the master and the slave is gradually increasing. This is because the initial parameters of the Actor network and the Critic network for each agent are random. As the training continues, the parameters are continuously optimized and gradually approach the optimal strategy and the state-action value function corresponding to the optimal strategy, so that the rewards obtained by the agent continue to increase. In the later stage of network training (after about 2000 rounds), the rewards of each round gradually stabilize, the parameters of the Actor network and the Critic network are optimized, and the state-action value function values ​​of each agent gradually stabilize.

[0118] In the description of the present invention, it is necessary to understand that the terms "coaxial", "bottom", "one end", "top", "middle", "the other end", "upper", "one side", "top", "inside", "outside", "front", "center", "both ends" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0119] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0120] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-agent based resource allocation and task offloading optimization method, characterized in that: The method comprises: Step 1) For the MEC network enabled by energy harvesting in a heterogeneous network environment, a computing task model is established, and a communication model, a task execution model, and an energy consumption model for task offloading to the heterogeneous network are established, and the environmental state is obtained according to the established models; Step 2) The edge cloud server is regarded as the master, and the master's immediate reward function is constructed according to the obtained environmental state, and the benefit objective function of the long-term cumulative reward of the master's execution of collaborative resource allocation decisions is established according to the master's immediate reward function; Step 3) The mobile device is regarded as a slave, and an immediate reward function of the slave is constructed according to the obtained environmental state, and a benefit objective function of the long-term cumulative reward of the slave for executing the task offloading decision is established according to the immediate reward function of the slave; Step 4) Maximizing the profit objective function of the master and the profit objective function of the slave, combining the decision constraints of the master and the slave, and constructing the Stackelberg game model; The constructed Stackelberg game model is expressed as: Among them, F(t) is the computing resource allocation strategy set of the edge cloud server, B(t) is the bandwidth resource allocation strategy set of the edge cloud server; a(t) is the task offloading strategy set of the mobile device; R L represents the long-term cumulative reward of the edge cloud server, r L (t) represents the instantaneous reward of the edge cloud server in time slot t; represents the long-term cumulative reward of mobile device i, r i F (t) represents the instantaneous reward of mobile device i in time slot t, Represents the collection of mobile devices in the network; a i (t) represents the task offloading strategy of mobile device i in time slot t; 1{x} is an indicator function, which is equal to 1 when condition x is true and 0 otherwise; f i0 (t) represents the computing resources allocated by MBS to mobile device i in time slot t, B i0 (t) represents the bandwidth resources allocated by MBS to mobile device i in time slot t; Indicates the maximum computing capacity of MBS, Indicates the available bandwidth resources of MBS; f ij (t) represents the computing resources allocated by FBSj to mobile device i in time slot t; represents the maximum computing capacity of FBSj; B ij (t) represents the bandwidth resources allocated by FBSj to mobile device i in time slot t; represents the available bandwidth resources of FBSj; represents the set of mobile devices within the coverage of FBSj; m represents the total number of mobile devices, MBS represents the macro base station, and FBSj represents the jth femto base station; The instant reward of the edge cloud server is expressed as: Among them, r L [F(t), B(t), a(t)] represents the instant reward of the edge cloud server under F(t), B(t) and a(t); is the indicator satisfaction function to measure the edge cloud server, is a function to measure the satisfaction of the corresponding indicators of mobile devices; ω1, ω2, ω3 are the relevant weight coefficients of edge cloud servers, λ1 and λ2 are the relevant weight coefficients of mobile devices; represents the task discard indicator factor of mobile device i in time slot t; represents the task offloading delay of the task of mobile device i in the MBS in time slot t; represents the task calculation failure indicator factor of FBSj in time slot t; represents the task calculation failure indicator factor of MBS in time slot t; The instant reward for mobile devices is expressed as: Among them, r i F [F * (t),B * (t),a(t)] represents F * (t), B * (t) and a(t), the instant reward of mobile device i, F * (t) is the optimal computing resource allocation strategy set of edge cloud servers, B * (t) is the optimal bandwidth resource allocation strategy set of the edge cloud server; Q i (t) represents the local task queue of mobile device i at time slot t, φ i (t) represents the task computation probability of mobile device i in time slot t; Step 5) Based on the random time-varying network environment and the incomplete observation states of the master and the slave, the Stackelberg game model is modeled as a partially observable Markov decision process game model; Step 6) Use a multi-agent deep reinforcement learning algorithm to solve the partially observable Markov decision process game model, and learn and optimize the edge cloud resource collaborative allocation decision and mobile device task offloading decision through centralized training and distributed execution.

2. A multi-agent based resource allocation and task offloading optimization method according to claim 1, characterized in that: The environmental status includes the available computing resources and wireless channel gain of the edge cloud server; the location, task backlog, available energy, and collected energy of the mobile device.

3. The multi-agent-based resource allocation and task offloading optimization method according to claim 1, characterized in that: The Stackelberg game model is modeled as a partially observable Markov decision process game model, including that the edge cloud server can only observe incomplete channel information and partial information of the mobile device; any mobile device can only observe partial information of the edge cloud server; thereby constructing an observation space; and combining the participant set, environmental state space, observation space and action space to establish a partially observable Markov decision process game model; wherein, the observable space for the edge cloud server agent includes the local remaining computing resources and the locations of all mobile devices; the observable space for any mobile device agent includes the location of the mobile device, task backlog and available energy.

4. The multi-agent-based resource allocation and task offloading optimization method according to claim 1, characterized in that: Through centralized training and distributed execution, learning and optimizing edge cloud resource collaborative allocation decisions and mobile device task offloading decisions include treating the master and each slave as intelligent agents. Each intelligent agent performs a decision action in the current time slot and will receive a corresponding instant reward in the next time slot based on its independent instant reward function. The intelligent agents use the Actor network and the Critic network to update their respective decisions based on the received instant rewards.

5. The method for optimizing resource allocation and task offloading based on multi-agent according to claim 1, characterized in that: The method also includes managing local energy of mobile devices, wherein any mobile device captures energy in the surrounding environment through energy harvesting technology to support data communication and business processing of the mobile device.

Images