Edge-cloud task unloading method based on multi-agent deep reinforcement learning

Abstract

The invention belongs to the technical field of mobile communication, and particularly relates to an edge-cloud task unloading method based on multi-agent deep reinforcement learning, which is mainly embodied in the following aspects: conversion of an optimization problem, energy consumption and time delay optimization problems under an edge-cloud task unloading scene are NP-Hard and are difficult to solve by using a traditional optimization method, and the problem is solved by using a traditional optimization method. The optimization problem is converted into a partially observable Markov decision process, and an unloading decision is optimized by using an MADRL method; a reward function is designed, the reward function of the MADRL comprises energy consumption and time delay, and a weighting factor is used for balancing the proportion of the energy consumption and the time delay in the optimization process; according to the introduction of the attention mechanism, the attention mechanism is added into a loss function of MADRL, so that the model can better distinguish the importance degrees of different features of the intelligent agent. According to the method, the energy consumption and the time delay of task unloading can be effectively balanced, and meanwhile, a more accurate unloading decision is made based on an attention mechanism.

Description

Technical Field

[0001] The present invention belongs to the field of mobile communication technology, and specifically relates to an edge-cloud task offloading method based on multi-agent deep reinforcement learning. Background Art

[0002] With the continuous development of mobile communication and network technologies, the application demands of mobile users are increasing, leading to the emergence of many computationally intensive and latency-sensitive applications, such as augmented reality, autonomous driving, and smart homes. These applications place a significant computational and communication burden on the network. In the Internet of Things (IoT) and Intelligent Traffic Systems (ITS) fields, massive amounts of sensor data and computational tasks need to be analyzed and processed. Mobile user devices, with their limited computing power, struggle to handle such a massive workload in a short period of time. Mobile Edge Computing (MEC) processes the computational tasks generated by mobile users by deploying computing resources at the edge of the network close to the mobile users, offering low latency, high reliability, and strong security. However, compared to central servers, edge servers typically have very limited computing resources. Properly allocating these resources is crucial to effectively handle the large number of computational tasks generated by mobile users. Therefore, computation offloading has become a research area of ​​great interest.

[0003] Generally speaking, in MEC, edge servers are able to process computing tasks generated by mobile users relatively quickly. However, with the increasing amount of data in the network, the computing power required to handle certain complex tasks may far exceed the computing power of edge servers. In this case, without the help of cloud servers, the quality of service (QoS) of mobile users will inevitably decline significantly, seriously affecting the mobile user experience. Therefore, to solve this problem, researchers have proposed an edge-cloud computing architecture consisting of edge servers and cloud servers. Under this architecture, computing offloading and resource allocation problems will become more complex, and how to quickly and effectively determine offloading targets becomes a new challenge.

[0004] When performing computational offloading, the latency of data transmission and processing is an important indicator for measuring the pros and cons of the offloading method. In addition, whether it is today's 5G network or the future 6G network, how to reduce the energy consumption of the system is also a key issue that researchers focus on. If these two factors are considered at the same time, the optimization problem is often expressed as a mixed integer nonlinear programming (MINLP) problem. This problem is NP-hard, and it is usually very difficult to obtain its optimal solution. Therefore, the present invention proposes a computational offloading method based on multi-agent deep reinforcement learning MADRL to optimize the task offloading problem under the edge-cloud computing architecture. The rapid development of artificial intelligence (AI) technology has provided new ideas and methods for computational offloading, and reinforcement learning (RL) and deep reinforcement learning (DRL) technologies have been applied in this field. It can flexibly and effectively handle various nonlinear relationships and autonomously select the optimal strategy according to different environmental conditions. Summary of the Invention

[0005] The purpose of this invention is to provide an edge-cloud task offloading method based on multi-agent deep reinforcement learning. By combining MADDPG and the attention mechanism, it effectively balances the energy consumption and latency of task offloading under the edge-cloud computing architecture, and can improve the energy efficiency of the system while meeting the latency requirements of data transmission and processing.

[0006] The technical solutions adopted by the present invention are as follows:

[0007] A method for edge-to-cloud task offloading based on multi-agent deep reinforcement learning, comprising the following steps:

[0008] S1: Establish a communication system; including an edge-cloud computing scenario consisting of mobile users, roadside units (RSUs), and base stations (BSs);

[0009] S2: Optimization problem transformation; the goal of the optimization problem transformation is to minimize the long-term energy consumption and delay of the communication system in step 1; the "process of minimizing the long-term energy consumption and delay of the communication system" is transformed into a partially observable Markov decision process POMDP, and then the mixed integer nonlinear programming MADRL method is used to minimize the long-term energy consumption and delay of the communication system in step 1;

[0010] S3: Design based on MADDPG-AT method and construction of MADDPG-AT model;

[0011] We use the attention mechanism AT to distinguish the importance of different features and design a computation offloading method based on the attention mechanism AT and MADDPG (MADDPG-AT method). The MADDPG-AT method is used to minimize the long-term energy consumption and latency of the communication system. We also build a MADDPG-AT model based on the MADDPG-AT method.

[0012] The MADDPG-AT method uses an actor-critic architecture to learn and optimize strategies. At the same time, multiple agents cooperate by sharing a pool of experience, which has good convergence and scalability. For the above optimization problem, the actor network can generate a specific action strategy based on the state information of the environment, that is, to allocate power; the critic network will evaluate the expected long-term return of the action and feed it back to the actor network; the state information that the agent can obtain may be time-varying, and the importance of these features may be different for different time slots. By introducing the attention mechanism, the MADDPG-AT model is assisted in distinguishing the importance of different features, thereby making more accurate unloading decisions; the present invention combines the MADDPG method with the attention mechanism and proposes a deep reinforcement learning (DRL) method called MADDPG-AT.

[0013] S4: MADDPG-AT model training and performance evaluation; in the edge-cloud computing scenario in S1, the MADDPG-AT model based on the MADDPG-AT method in step S3 is used to perform task offloading, analyze the convergence of the MADDPG-AT model and the energy consumption and latency of the communication system; and output the MADDPG-AT model after training and evaluation.

[0014] Preferably, in S1, the mobile user generates a computing task, and the roadside unit and the base station are equipped with an edge server and a cloud server, respectively; the computing task generated by the mobile user is processed through three methods: local computing, edge computing, and cloud computing; in S1, if the mobile user cannot complete the computing task locally, it will be offloaded to a nearby edge server, and the edge server will select and process the corresponding task based on the maximum tolerable delay of the task. For tasks with high delay tolerance or excessive computational complexity, the edge server will transmit them to the cloud server.

[0015] Preferably, in S2, the partially observable Markov decision process POMDP represents the "process of minimizing the long-term energy consumption and delay of the system model" as a six-tuple (M, S, O, A, P, R), where the six parameters are agent, state space, observation space, action space, joint transition and reward respectively;

[0016] Preferably, in S2:

[0017] The intelligent agent refers to all mobile users in the system model. Mobile can observe the current state of the environment and choose a reasonable action to maximize the reward;

[0018] The state space contains the characteristics of all agents in the system model in their current state, and the agents need to make reasonable decisions based on these characteristics;

[0019] The observation space refers to the information each agent obtains by observing the current environment;

[0020] The action space refers to the decisions that the agent makes based on the current state information;

[0021] Joint transition means that multiple agents cooperate with each other to jointly change the state of the environment;

[0022] Rewards are used to clarify the learning goals of the agent, guide the agent's behavior, and evaluate the performance of the strategy.

[0023] Preferably, in S4, after outputting the MADDPG-AT model after training and evaluation, the MADDPG-AT model based on the MADDPG-AT method in step S3 is used to perform task offloading in the edge-cloud computing scenario in S1, and iterative training and iterative performance evaluation are performed.

[0024] The technical effects achieved by the present invention are:

[0025] The present invention discloses a task offloading method based on multi-agent deep reinforcement learning (MADRL) and attention mechanism under the edge-cloud computing architecture, which effectively improves the energy efficiency of the system while meeting the delay requirements of data transmission and processing. The innovation of this technical solution is mainly reflected in the following aspects: the transformation of the optimization problem. The energy consumption and delay optimization problem in the edge-cloud task offloading scenario is NP-Hard and difficult to solve using traditional optimization methods. The optimization problem is transformed into a partially observable Markov decision process (POMDP), and the MADRL method is used to optimize the offloading decision; the design of the reward function. The reward function of MADRL contains two parts: energy consumption and delay. A weighting factor is used to balance the proportion of the two in the optimization process; the introduction of the attention mechanism. The attention mechanism is added to the loss function of MADRL, so that the model can better distinguish the importance of different features of the agent. Through the above method, this solution can effectively balance the energy consumption and delay of task offloading, and make more accurate offloading decisions based on the attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic diagram of a task offloading scenario of the present invention;

[0027] Figure 2 is a schematic diagram of the Markov decision process of the present invention;

[0028] Figure 3 is a schematic diagram of the MADDPG-AT method of the present invention;

[0029] Figure 4 It is a curve diagram of the changes in the reward function value, power and delay of each user during the training process in the MADDPG-AT method of the present invention;

[0030] Figure 5 is a comparison chart of energy consumption performance between the MADDPG-AT method of the present invention and the baseline method;

[0031] Figure 6 2 is a comparison chart of the delay performance of the MADDPG-AT method of the present invention and the baseline method;

[0032] Figure 7 This is a comparison chart of the total training cost of the MADDPG-AT method of the present invention and the other two DRL methods. DETAILED DESCRIPTION

[0033] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is described in detail below with reference to the following examples. It should be understood that the following text is only used to describe one or more specific embodiments of the present invention and does not strictly limit the scope of protection of the present invention.

[0034] like Figure 1-Figure 7 As shown, an edge-cloud task offloading method based on multi-agent deep reinforcement learning includes the following steps:

[0035] S1: Establish a communication system; including an edge-cloud computing scenario consisting of mobile users, roadside units (RSUs), and base stations (BSs);

[0036] In S1, mobile users generate computing tasks, and roadside units and base stations are equipped with edge servers and cloud servers, respectively. The computing tasks generated by mobile users are processed through three methods: local computing, edge computing, and cloud computing. In S1, if the mobile user cannot complete the computing task locally, it will be offloaded to a nearby edge server. The edge server will select and process the corresponding task based on the maximum tolerable delay of the task. For tasks with high delay tolerance or excessive computational complexity, the edge server will transmit it to the cloud server. In this process, the communication between the mobile user and the edge server is achieved through wireless transmission, while the edge server and the cloud server are connected through optical fiber. In addition, it should be noted that the present invention believes that the computing tasks generated by mobile users are fine-grained, that is, divisible. Moreover, this scenario does not include multi-hop communication between mobile users, so there is no network jitter or node failure.

[0037] MADRL-based edge-cloud task offloading process:

[0038] exist Figure 1 In the scenario shown, consider an edge-cloud computing system consisting of m mobile users, n RSUs and a base station. The sets of mobile users and RSUs are denoted by Ψ M = {1, 2, ... M} and Ψ N ={1,2,…N}. Time is divided into discrete time slots, and the index of the time slot is represented by the set Ψ T ={1,2,…T} means that the length of each time slot is τ0. For mobile user m∈Ψ M For example, in time slot t∈Ψ T The computational tasks generated within can be represented by a two-tuple

[0039]

[0040] Where D m(t) is the size of the task (in bits), subject to the parameter λ m Poisson distribution, called λ m is the task arrival rate, T m (t) is the maximum tolerable delay of the task (in seconds).

[0041] For time slot t, assume that the number of bits that mobile user m needs to process locally is d m,l (t), then the local processing power of mobile user m is

[0042]

[0043] Where, ρ m The number of CPU cycles required to process one bit of data for mobile user m, κ is the effective switching capacitance, which depends on the chip structure. Therefore, the local computing energy consumption of mobile user m in time slot t is

[0044] E m,l (t) = p m,l (t)τ0 (3)

[0045] The number of bits d that mobile user m offloads to the edge server in time slot t m,e (t) and its own transmission power p m,e (t) satisfies the following relationship:

[0046]

[0047] Where W is the transmission bandwidth and N0 is the noise power spectrum density. Modifying equation (4) can yield the transmission power p m,e The expression of (t) is

[0048]

[0049] According to formula (5), the edge computing energy consumption of mobile user m in time slot t can be calculated as

[0050] E m,e (t) = p m,e (t)τ0 (6)

[0051] When the computing task is offloaded to the cloud server, the offloading process includes two stages: wireless transmission and wired transmission. Therefore, in time slot t, the number of bits d that mobile user m offloads to the cloud server is m,c (t) and its own transmission power p m,c (t) The following relationship should be satisfied:

[0052]

[0053] Where R0 is the data transmission rate of the optical fiber. Modifying formula (7) can obtain the transmission power p m,c The expression of (t) is

[0054]

[0055] According to formula (8), the cloud computing energy consumption of mobile user m in time slot t can be calculated as

[0056] E m,c (t) = p m,c (t)τ0 (9)

[0057] Assume that the computational tasks generated by mobile user m in time slot t are first stored in a buffer Z m In (t), in each time slot, the amount of tasks processed locally by mobile user m is d m,l (t), and at the same time, the amount of tasks offloaded to the edge server and cloud server are d m,e (t) and d m,c (t). Therefore, the evolution rule of the queue length in the buffer is

[0058]

[0059] Where {x} + =max(x,0). According to Little’s theorem in queuing theory, the average queuing delay in the buffer is Average queue length in the buffer Its size is equal to the average queue length in the buffer divided by the task arrival rate, that is,

[0060]

[0061] Since the length of each time slot is fixed, the task processing delay mainly depends on its queuing delay in the buffer.

[0062] When the task needs to be offloaded to an edge server or cloud server for computation, mobile user m establishes a connection with RSUn via wireless communication. The path loss (in dB) of this process can be calculated as follows:

[0063]

[0064] Where c is the speed of light, f c is the carrier frequency, r mn (t) is the distance between mobile user m and RSU n in time slot t. According to this formula, the receiving end SINR when mobile user m communicates with RSU n can be calculated as

[0065]

[0066] Where h mn (t) is the channel gain between mobile user m and RSUn in time slot t, which satisfies the Gauss-Markov block fading autoregressive model. The numerator represents the useful signal power received by RSU n from mobile user m, and the denominator represents the sum of the interference power and noise power received by RSUn. It should be noted that p in Equation (13) m,e (t) is the transmission power when mobile user m adopts edge computing. When mobile user m adopts cloud computing, this variable needs to be replaced by p m,c (t).

[0067] S2: Optimization problem transformation; the goal of the optimization problem transformation is to minimize the long-term energy consumption and delay of the communication system in step 1; the "process of minimizing the long-term energy consumption and delay of the communication system" is transformed into a partially observable Markov decision process POMDP, and then the mixed integer nonlinear programming MADRL method is used to minimize the long-term energy consumption and delay of the communication system in step 1;

[0068] In S2, the partially observable Markov decision process POMDP represents the process of minimizing the long-term energy consumption and delay of the system model as a six-tuple (M, S, O, A, P, R). The six parameters are the agent, state space, observation space, action space, joint transition, and reward.

[0069] The intelligent agent refers to all mobile users in the system model. Mobile can observe the current state of the environment and choose a reasonable action to maximize the reward;

[0070] The state space contains the characteristics of all agents in the system model in their current state, and the agents need to make reasonable decisions based on these characteristics;

[0071] The observation space refers to the information each agent obtains by observing the current environment;

[0072] The action space refers to the decisions that the agent makes based on the current state information;

[0073] Joint transition means that multiple agents cooperate with each other to jointly change the state of the environment;

[0074] Rewards are used to clarify the learning goals of the agent, guide the agent's behavior, and evaluate the performance of the strategy;

[0075] In the actual operation of the present invention, in order to consider both energy cost and time cost, a weighting factor is used to express the total cost of mobile user m as the weighted sum of its own energy consumption and the average waiting delay in the buffer:

[0076]

[0077] Where ω∈[0,1] is a weighting factor. By varying the value of ω, the weight of energy consumption and latency in the cost function can be adjusted, thereby achieving a trade-off between the two. A value of ω > 0.5 indicates that energy consumption has a greater impact on the total cost. Conversely, a value of ω < 0.5 indicates a greater focus on latency costs.

[0078] The optimization goal is to minimize the long-term energy consumption and delay of the system. Therefore, the optimization problem can be expressed as

[0079]

[0080] In equation (15), the power allocation p of the mobile user is m,l (t), p m,e (t) and p m,c (t) is a continuous variable, and the task offloading decision can be expressed as a binary variable. From Equations (5) and (8), it can be seen that the relationship between the mobile user's transmission power and the amount of data offloaded is nonlinear, so γ in Equation (13) mn The calculation of (t) is also nonlinear. Furthermore, the objective function is composed of the weighted sum of the energy consumption and latency of each mobile user, which also involves nonlinear terms. Therefore, the optimization problem described by Equation (15) can be reduced to a MINLP problem. This problem is NP-hard, and obtaining its optimal solution is usually very difficult. Using traditional methods to solve it will result in extremely high computational complexity. In this invention, the problem is first converted into a POMDP, and then the MADRL method is used to solve it.

[0081] Figure 1 The unloading scenario shown in Figure 1 contains a series of time-varying parameters and continuous actions. Therefore, the optimization problem described in Equation (15) can be reformulated as Figure 2 The POMDP shown in the figure represents the process as a sextuple (M, S, O, A, P, R). The six parameters represent, in order, agent, state space, observation space, action space, joint transition, and reward. The agent refers to all mobile users in the system, which can observe the current state of the environment and choose a reasonable action to maximize the reward. The observation space refers to the information each agent obtains from observing the current environment. The joint transition refers to the collaborative effort of multiple agents to change the environment. The state space, action space, and reward are the three most important variables in DRL. The following will explain their specific meanings.

[0082] State space: The state space contains the characteristics of all agents in the system in their current state. Agents need to make reasonable decisions based on these characteristics. For mobile user m, in each time slot t, the queue length Z in its buffer is m (t) will be updated according to formula (10). In addition, the channel gain h mn (t) is a time-varying variable that results in the receiver SINRγ at the RSU mn (t) changes with time. Based on these three variables, the agent can judge the delay in the buffer in the current state, as well as the channel quality and signal quality during the communication process. Therefore, the state space of mobile user m in the tth time slot is defined as

[0083] S m (t) = {Z m (t),h mn (t),γ mn (t-1)} (16)

[0084] Action space: The agent makes a judgment based on the current state space and then selects a suitable action in the action space to perform computation offloading. Figure 1 In the edge-cloud computing model shown, mobile user m needs to determine the power p used for local computing in each time slot. m,l (t) and the transmission power p for edge computing and cloud computing m,e (t) and p m,c (t). Therefore, the action space of mobile user m in the t-th time slot is defined as

[0085] A m (t) = {p m,l (t),p m,e (t),p m,c (t)} (17)

[0086] Reward: For the optimization problem described by Equation (15), the optimization goal is to minimize the long-term total cost of the system, which includes the energy and time costs of all mobile users in the system. Since the energy consumption of mobile user m in time slot t is proportional to its own power, and the average queuing delay in its buffer is also proportional to the average queue length in the buffer, the power and buffer length of mobile user m can be directly used to define its reward function in time slot t:

[0087] R m (t)=-ω[p m,l (t)+p m,e (t)+p m,c (t)]-(1-ω)Z m (t) (18)

[0088] S3: Design based on MADDPG-AT method and construction of MADDPG-AT model;

[0089] The attention mechanism AT is used to distinguish the importance of different features. A computation offloading method based on the attention mechanism AT and MADDPG (MADDPG-AT method) is designed. The MADDPG-AT method is used to minimize the long-term energy consumption and latency of the communication system. A MADDPG-AT model is constructed based on the MADDPG-AT method.

[0090] The MADDPG-AT method uses an actor-critic architecture to learn and optimize strategies. At the same time, multiple agents collaborate by sharing a pool of experience, which has good convergence and scalability. For the above optimization problem, the actor network can generate specific action strategies based on the state information of the environment, that is, to allocate power; the critic network will evaluate the expected long-term return of the action and feed it back to the actor network; the state information that the agent can obtain may be time-varying, and the importance of these features may be different for different time slots. By introducing the attention mechanism, the MADDPG-AT model is assisted in distinguishing the importance of different features, thereby making more accurate unloading decisions; the present invention combines the MADDPG method with the attention mechanism and proposes a deep reinforcement learning (DRL) method called MADDPG-AT.

[0091] In this paper, the attention mechanism is used to distinguish the importance of different features, and a computation offloading algorithm based on the attention mechanism and MADDPG (MADDPG-AT method) is proposed to minimize the long-term energy consumption and latency of the system. Figure 3 This is a schematic diagram of the MADDPG-AT method. The algorithm uses an actor-critic architecture to learn and optimize strategies. At the same time, multiple agents cooperate by sharing experience pools, which has good convergence and scalability.

[0092] In the MADDPG-AT method, the actor network can generate a specific action strategy based on the state information of the environment, that is, to perform power allocation. Its policy gradient can be calculated by the following formula:

[0093]

[0094] Where θ μ and θ Q are the parameters of the actor network and the critic network, respectively, which can be updated by soft updating; μ(s|θ μ ) and Q(s,a|θ Q) are the actor function and the critic function (Q function). The action policy generated by the actor network is stored in an experience buffer, and the critic network evaluates the expected long-term reward of the action based on the samples in the buffer and feeds it back to the actor network.

[0095] The role of the critic network is to estimate the value of the Q function, so its loss function is generally written as the mean square error between the actual Q value and the target Q value. In the POMDP of step S2, the state space contains three time-varying features of the agent, namely the buffer queue length, channel gain, and receiver SINR. However, the importance of these features may be different for different time slots. Therefore, when designing the loss function of the critic network, an attention mechanism is introduced to distinguish the importance of these three features. The loss function of the critic network can be expressed as

[0096]

[0097] Where y m is the target value, β m is the attention weight, which is calculated by the softmax function:

[0098]

[0099] Where, is the feature matrix, which describes the three time-varying features of agent m; is the weight matrix, which is a learnable parameter and its size can be continuously adjusted during the model training process. m The direction and magnitude of the update are determined by L c To X m The gradient determines .

[0100] S4: MADDPG-AT model training and performance evaluation; In the edge-cloud computing scenario in S1, the MADDPG-AT model based on the MADDPG-AT method in step S3 is used to perform task offloading, analyze the convergence of the MADDPG-AT model and the energy consumption and latency of the communication system; and output the MADDPG-AT model after training and evaluation.

[0101] In actual use, S4 of the present invention is as follows: in the edge-cloud computing scenario of step S1, the MADDPG-AT method of step S3 and five other baseline methods are used to perform task offloading, and the convergence of the algorithm, training cost, and energy consumption and latency of the system are analyzed.

[0102] The baseline methods include two other DRL methods and three greedy algorithms:

[0103] Multi-Agent Deep Deterministic Policy Gradient-Based Computation Offloading (MADDPG): This algorithm does not use the attention mechanism to distinguish the importance of the three features of the agent. The loss function of the critic network is directly written as the mean square error between the actual Q value and the target Q value, without the attention weight β m The rest of the algorithm is the same as the MADDPG-AT method.

[0104] Computational offloading based on deep Q network (DQN): The action space of this algorithm is discretized, that is, p m,l (t), p m,e (t) and p m,c (t) needs to be selected from several discrete power values ​​that have been quantized in advance, instead of being able to select power values ​​in the continuous domain like the MADDPG-AT method.

[0105] Greedy Local Computing (LGD): In each time slot, mobile users prioritize processing computing tasks locally. If the task cannot be processed locally, it is offloaded to edge servers or cloud servers for processing.

[0106] Greedy Edge Computing (EGD): In each time slot, mobile users prioritize offloading computing tasks to edge servers for processing, and then consider local computing and cloud computing.

[0107] Greedy Cloud Computing (CGD): In each time slot, mobile users prioritize offloading computing tasks to cloud servers for processing, followed by local computing and edge computing.

[0108] In S4, after outputting the trained and evaluated MADDPG-AT model, the MADDPG-AT model based on the MADDPG-AT method in step S3 is used to perform task offloading in the edge-cloud computing scenario in S1, and iterative training and iterative performance evaluation are performed.

[0109] A specific embodiment of the present invention: The edge-cloud computing scenario consists of one cloud server, four edge servers, and six mobile users. The task arrival rates of the six mobile users are 0.5Mbps, 1Mbps, 1.5Mbps, 2Mbps, 2.5Mbps, and 3Mbps, respectively. The time slot length τ0 is set to 1ms, and the mobile users are stationary in each time slot. The system bandwidth W is set to 10MHz, and the carrier frequency f when the mobile users communicate wirelessly with the edge servers is set to 0. c The fiber transmission rate R0 between the edge server and the cloud server is set to 2.4GHz and 1Gbps. The channel gain h between the mobile user and the edge server is mn (t) varies in the range of [-20, -5]dB, and the maximum tolerable delay of the task is T m(t) is randomly generated in the range of [0.5, 10] s. The maximum power of mobile users for local processing and task offloading and Both are set to 2W, and the number of CPU cycles required to locally process 1 bit of data is ρ m Set to 500, the effective switching capacitance κ is 10 -27 .

[0110] Both the actor network and the critic network contain two hidden layers, with 400 and 300 neurons, respectively. The activation function for all hidden layers is Reinforced Luminance (ReLU). The output layer of the actor network uses a sigmoid function to bind actions. The Adam optimizer is used to update the neural network parameters during training. The number of training epochs, K, is 1500, with a maximum step size, T, of 200 per epoch and a batch size of 16.

[0111] Figure 4 The MADDPG-AT method is used to show how the reward function values, power, and latency of six mobile users change during training. It can be seen that through the continuous interaction between the agents and the environment, as well as the mutual cooperation between the agents, the reward function values ​​of each agent converge to a large and stable value. Simultaneously, the power and latency of the six mobile users also converge to a stable value. This demonstrates the good convergence and stability of the MADDPG-AT method.

[0112] Figure 5 The average energy consumption of the MADDPG-AT method and five other baseline methods at different task arrival rates is shown. When the task arrival rate of mobile users is low, the average power of the three DRL methods is similar. However, when the task arrival rate of mobile users increases to 2Mbps or above, the average power of the MADDPG-AT method is significantly lower than that of the other two algorithms. This is because the attention mechanism can accurately capture changes in environmental conditions and promptly adjust the power of mobile users based on these changes, resulting in better energy consumption performance than the MADDPG method. The DQN algorithm requires discretization of the power values ​​in the agent's action space, which results in a certain degree of precision loss, resulting in worse energy consumption performance than the MADDPG method.

[0113] Figure 6The average latency of the MADDPG-AT method and five other baseline methods at different task arrival rates is shown. The MADDPG method has better latency performance because it makes decisions based on a multi-agent framework, allowing agents to cooperate and optimize together. Although the MADDPG-AT method is also based on a multi-agent framework, the introduction of the attention mechanism increases the complexity of the algorithm, slowing down its execution to a certain extent and sacrificing some of the algorithm's latency performance in exchange for better energy efficiency. The DQN algorithm is forced to discretize the continuous action space, so its offloading decisions are often suboptimal, resulting in poor latency and energy efficiency performance.

[0114] Figure 7 The total training costs of the three DRL methods were compared for different numbers of mobile users. When the number of mobile users was 3, 6, or 9, the MADDPG-AT method had the highest total training cost due to the increased complexity of the algorithm caused by the attention mechanism. However, when the number of mobile users increased to 12, the cost of the MADDPG method increased dramatically, surpassing that of the MADDPG-AT method. This is because the attention mechanism in the MADDPG-AT method plays a significant role when the number of agents is large. By accurately capturing the important characteristics of the agents, the attention mechanism enables the algorithm to select more appropriate actions within each time slot, significantly reducing the total training cost of the algorithm.

[0115] The above simulation results demonstrate the effectiveness and feasibility of the edge-cloud task offloading method based on multi-agent deep reinforcement learning proposed in this invention, indicating that the proposed scheme has broad application prospects in the field of task offloading.

[0116] The present invention discloses a task offloading method based on multi-agent deep reinforcement learning (MADRL) and attention mechanism under the edge-cloud computing architecture, which effectively improves the energy efficiency of the system while meeting the delay requirements of data transmission and processing. The innovation of this technical solution is mainly reflected in the following aspects: the transformation of the optimization problem. The energy consumption and delay optimization problem in the edge-cloud task offloading scenario is NP-Hard and difficult to solve using traditional optimization methods. The optimization problem is transformed into a partially observable Markov decision process (POMDP), and the MADRL method is used to optimize the offloading decision; the design of the reward function. The reward function of MADRL contains two parts: energy consumption and delay. A weighting factor is used to balance the proportion of the two in the optimization process; the introduction of the attention mechanism. The attention mechanism is added to the loss function of MADRL, so that the model can better distinguish the importance of different features of the agent. Through the above method, this solution can effectively balance the energy consumption and delay of task offloading, and make more accurate offloading decisions based on the attention mechanism.

[0117] The foregoing is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained herein shall, unless otherwise specified or limited, be implemented in accordance with conventional means in the art.

Claims

1. An edge-to-cloud task offloading method based on multi-agent deep reinforcement learning, characterized by: The following steps are involved: S1: Establish a communication system; including an edge-cloud computing scenario consisting of mobile users, roadside units (RSUs), and base stations (BSs); S2: Optimization problem transformation; the goal of the optimization problem transformation is to minimize the long-term energy consumption and delay of the communication system in step 1; the "process of minimizing the long-term energy consumption and delay of the communication system" is transformed into a partially observable Markov decision process POMDP, and then the mixed integer nonlinear programming MADRL method is used to minimize the long-term energy consumption and delay of the communication system in step 1; S3: Design based on MADDPG-AT method and construction of MADDPG-AT model; We use the attention mechanism AT to distinguish the importance of different features and design a computation offloading method based on the attention mechanism AT and MADDPG (MADDPG-AT method). The MADDPG-AT method is used to minimize the long-term energy consumption and latency of the communication system. We also build a MADDPG-AT model based on the MADDPG-AT method. The MADDPG-AT method uses an actor-critic architecture to learn and optimize strategies. At the same time, multiple agents cooperate by sharing a pool of experience, which has good convergence and scalability. For the above optimization problem, the actor network can generate a specific action strategy based on the state information of the environment, that is, to allocate power; the critic network will evaluate the expected long-term return of the action and feed it back to the actor network; the state information that the agent can obtain may be time-varying, and the importance of these features may be different for different time slots. By introducing the attention mechanism, the MADDPG-AT model is assisted in distinguishing the importance of different features, thereby making more accurate unloading decisions; the present invention combines the MADDPG method with the attention mechanism and proposes a deep reinforcement learning (DRL) method called MADDPG-AT. S4: MADDPG-AT model training and performance evaluation; In the edge-cloud computing scenario in S1, the MADDPG-AT model based on the MADDPG-AT method in step S3 is used to perform task offloading, and the convergence of the MADDPG-AT model and the energy consumption and latency of the communication system are analyzed; And output the MADDPG-AT model after training and evaluation.

2. The edge-to-cloud task offloading method based on multi-agent deep reinforcement learning according to claim 1, characterized in that: In S1, the mobile user generates a computing task, and the roadside unit and the base station are equipped with an edge server and a cloud server respectively; the computing task generated by the mobile user is processed through three methods: local computing, edge computing and cloud computing.

3. The edge-to-cloud task offloading method based on multi-agent deep reinforcement learning according to claim 2, characterized in that: In S1, if the mobile user cannot complete the computing task locally, it will be offloaded to a nearby edge server. The edge server will select and process the corresponding task based on the maximum tolerable delay of the task. For tasks with high delay tolerance or excessive computational complexity, the edge server will transmit them to the cloud server.

4. The edge-to-cloud task offloading method based on multi-agent deep reinforcement learning according to claim 3 is characterized by: In S2, the partially observable Markov decision process POMDP represents the "process of minimizing the long-term energy consumption and delay of the system model" as a sextuple (M, S, O, A, P, R), where the six parameters are agent, state space, observation space, action space, joint transition, and reward.

5. The edge-to-cloud task offloading method based on multi-agent deep reinforcement learning according to claim 4 is characterized in that: In S2: The intelligent agent refers to all mobile users in the system model. Mobile can observe the current state of the environment and choose a reasonable action to maximize the reward; The state space contains the characteristics of all agents in the system model in their current state, and the agents need to make reasonable decisions based on these characteristics; The observation space refers to the information each agent obtains by observing the current environment; The action space refers to the decisions that the agent makes based on the current state information; Joint transition means that multiple agents cooperate with each other to jointly change the state of the environment; Rewards are used to clarify the learning goals of the agent, guide the agent's behavior, and evaluate the performance of the strategy.

6. The edge-to-cloud task offloading method based on multi-agent deep reinforcement learning according to claim 5, characterized in that: In S4, after outputting the trained and evaluated MADDPG-AT model, the MADDPG-AT model based on the MADDPG-AT method in step S3 is used to perform task offloading in the edge-cloud computing scenario in S1, and iterative training and iterative performance evaluation are performed.

Images