Collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning

Abstract

A collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning, which method belongs to the technical field of mobile communications. The method comprises: constructing an MEC system in an MEC network scenario where there are multiple base stations and diversified service requests; modeling a collaborative task offloading and service caching problem by taking as an objective the maximization of a system utility based on QoE; converting the collaborative task offloading and service caching problem into a decentralized partially observable Markov decision process; and using an algorithm based on graph attention multi-agent reinforcement learning to solve the decentralized partially observable Markov decision process obtained by means of conversion. By means of a simulation experiment, compared with a benchmark algorithm, significant improvements are made in terms of a cache hit rate, system utility and computation success rate to the algorithm proposed herein.

Description

Collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 8, 2023, with application number 202311160072.8 and invention name “Collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention belongs to the field of mobile communication technology, and in particular relates to a collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning. Background Art

[0003] Mobile Edge Computing (MEC) is considered an effective solution for supporting a large number of ultra-low-latency computing services. It effectively reduces task processing latency by pushing cloud functions to the edge network closer to user devices. MEC servers equipped with storage and computing resources can cache various services (including program code and databases) and enable computing tasks requested by user equipment (UE) to be processed on the edge network, thereby reducing service latency for a wide range of emerging applications such as autonomous driving, augmented reality, and interactive online gaming.

[0004] However, the limited storage resources of the MEC server mean that it can only cache a small portion of popular service content at the same time. At the same time, in densely deployed small cell networks, the offloading requests of a large number of heterogeneous services will also cause serious interval interference, resulting in poor quality of experience (QoE). Obviously, the computing performance of the MEC network is closely related to the computing offloading and service caching strategies. Therefore, efficient task offloading and service caching strategies are crucial to improving the QoE of UEs. Domestic and foreign researchers have conducted a lot of in-depth research on this issue, and some of the main results are:

[0005] (1) Service Caching, Computation Offloading and Resource Allocation Algorithm in Mobile Edge Computing (Reference: Zhang G, Zhang S, Zhang W, et al. Joint Service Caching, Computation Offloading and Resource Allocation in Mobile Edge Computing Systems[J]. IEEE Transactions on Wireless Communications, 2021, 20(8): 5288-5300.): This algorithm considers the scenario of multiple user devices and jointly optimizes transmission, computing resource allocation, service caching and task offloading, with the aim of minimizing the energy consumption of all user devices.

[0006] (2) Service caching and task offloading algorithm in high-altitude platform assisted intelligent transportation system (Reference: Ren Q, Abbasi O, Kurt GK, et al. Caching and Computation Offloading in High Altitude Platform Station (HAPS) Assisted Intelligent Transportation Systems [J]. IEEE Transactions on Wireless Communications, 2022, 21(11): 9010-9024.): This algorithm considers service caching and task offloading in intelligent transportation scenarios, and jointly optimizes task offloading, service caching, bandwidth and computing resource allocation decisions to minimize system latency.

[0007] However, with the diversification and regionalization of service requests and the continuous expansion of MEC scale, the complex spatial correlation between edge nodes makes it difficult for existing optimization algorithms to obtain efficient task offloading and resource allocation strategies in such MEC environments. Specifically, the following two aspects of spatial correlation need to be considered:

[0008] (1) The service request types between different edge nodes are different, and adjacent nodes usually have the same popular service request types due to the similarity of UEs;

[0009] (2) The uplink transmission rate that a UE can achieve mainly depends on the radio resource allocation strategy and the interference between other cells. The closer the cells are, the greater the interference they cause to each other.

[0010] Therefore, edge nodes should pay more attention to the service requests and wireless network status information of their neighboring edge nodes before formulating task offloading, resource allocation, and service caching strategies.

[0011] Summary of the Invention

[0012] To solve the above problems, the present invention considers the collaborative task offloading and service caching scenarios of MEC networks with multiple base stations and diversified service requests, and proposes a collaborative task offloading and service caching algorithm based on graph attention multi-agent reinforcement learning.

[0013] The technical solution adopted by the present invention comprises the following steps:

[0014] S1. Build an MEC system in a MEC network scenario with multiple base stations and diverse service requests, where:

[0015] Each base station in the MEC system is configured with a cache and computing resources of a MEC server to provide computing services to its associated user equipment; the base station set is defined as is the number of base stations; definition For base station The associated user device collection, Indicates the number of user equipment associated with base station n; the MEC system is time slot dynamic and uses Represents a set of time slots, each of which has the same length, denoted as The MEC system also includes a task model, a service cache model, a communication model, a task offloading model and a utility model;

[0016] S2. Based on the MEC system, we model the collaborative task offloading and service caching problem with the goal of maximizing the system utility based on QoE.

[0017] S3. Transform the collaborative task offloading and service caching problem into a distributed partially observable Markov decision process, including:

[0018] S31. Convert the collaborative task offloading and service caching problem into a distributed partially observable Markov decision process with N base stations, and treat each base station as an intelligent agent.

[0019] S32. Set the environment state space, local observation space, action space and reward function;

[0020] S4. Solving the transformed distributed partially observable Markov decision process using a graph attention-based multi-agent reinforcement learning algorithm, including:

[0021] S41. Construct a graph-attention-based multi-agent reinforcement learning network, which includes a graph attention module, a joint action-value hybrid module, and a local action-value network corresponding to each agent;

[0022] S42. Each agent inputs its received local observation value and previous action into its corresponding local action value network and outputs a local action value function;

[0023] S43. The graph attention module takes the environmental state of the MEC system as input and outputs the weight of each local action value function;

[0024] S44. The joint action value mixing module adds all local action value functions according to the weights to obtain a joint action value function, and trains it by minimizing the loss function;

[0025] S45. After the training is completed, the global optimal task offloading and service caching strategy is obtained based on the local observation values ​​of the base station.

[0026] Furthermore, the task model establishment method includes: each user device regularly generates computing tasks with different QoE and service requirements, and defines is the set of service types in the MEC system. The number of service types; define user equipment The computation task generated at time t consists of 6 parameters, expressed as in, For user equipment m n The task data size of the computing task generated at time t, For user equipment m n The service type of the computing task generated at time t, For user equipment m n The number of CPU cycles required for each bit of input data of the computation task generated at time t; It is based on the user device m n The service type of the computing task generated at time t predefines three service delay thresholds, is the minimum service delay threshold, is the maximum service delay threshold, is the user device m n You begin to feel the QoE drop to a point where the experience is poor.

[0027] Furthermore, the method for establishing the service cache model includes: the service cached by the base station in time slot t is used to perform the corresponding task in time slot t+1; defining a binary variable The service caching decision for service k in base station n for time slot t, where if service k is cached in base station n, then otherwise

[0028] Furthermore, the communication model establishment method includes: the MEC system bandwidth is divided into C orthogonal channels, allocated in each time slot, and each channel fully reuses the spectrum resources of the MEC system; defining Represents the set of available channels for each base station, and represents the channel allocation variable of base station n in time slot t. If base station n allocates channel Assigned to user equipment m n ,but otherwise

[0029] set up is the user equipment m in time slot t n The channel gain of user equipment m on channel c and its local base station n is n Uplink transmission rate on channel c at time slot t Expressed as:

[0030] Therefore, user equipment m n The uplink transmission rate at the local base station n in time slot t Expressed as:

[0031] Where w is the channel bandwidth, For user equipment m n The transmission power, σ 2 is the background noise power, is the interference received by base station n from other user equipment on channel c in time slot t.

[0032] Furthermore, in the collaborative task offloading and service caching method of graph attention multi-agent reinforcement learning, the task offloading model establishment method includes: the computing task of each user device has two offloading strategies: local base station offloading and collaborative offloading; defining Indicates user equipment m n Task offloading decision in time slot t, if user device m n The computational task generated at time t is offloaded to its local base station n for execution, then If the user device m n The computational tasks generated at time t are offloaded to the base station Execute, then If the user device m n The computational task generated at time t is discarded, then Wherein, t time slot is a time period, and time t represents the start time of t time slot;

[0033] If the local base station offloading strategy is selected, the user equipment m n The uplink delay of the computing task generated at time t to be sent to its local base station n The calculation formula is:

[0034] in, For user equipment m n The task data size of the computing task generated at time t, Indicates user equipment m n The uplink transmission rate at local base station n in time slot t;

[0035] User equipment m n The computation delay of the computation task at its local base station n is The calculation formula is:

[0036] Among them, f n is the CPU frequency of base station n, Indicates user equipment m n The number of CPU cycles required for each bit of input data of the computation task generated at time t;

[0037] Therefore, user equipment m n The processing delay of the computation task generated at time t to be offloaded to its local base station n for:

[0038] If the collaborative offloading strategy is selected, the user device m n Migrating the computing task generated at time t to base station n′ will cause additional transmission delay. n The uplink delay of the computation task sent to base station n' at time t The calculation formula is:

[0039] in, represents the average transmission rate between base station n and base station n′;

[0040] User equipment m n The computation delay of the computation task at base station n′ The calculation formula is:

[0041] Among them, f n′ represents the CPU frequency of base station n′;

[0042] Therefore, user equipment m n The processing delay of the computation task generated at time t being offloaded to base station n′ for:

[0043] Furthermore, the method for establishing the utility model includes: defining a utility function to evaluate the service quality of the user equipment, wherein the utility function is expressed as:

[0044] in, Indicates user equipment m n The utility obtained in time slot t, Indicates processing of user equipment m n The service delay of the computing task generated at time t, Indicates user equipment m n The minimum service delay requirement of the computing task generated at time t, Indicates user equipment m n The maximum service delay requirement of the computing task generated at time t.

[0045] Furthermore, step S32 sets the environment state space, local observation space, action space and reward function, including:

[0046] In the environment state space:

[0047] Among them, s t represents the environmental state of the MEC system in time slot t, It is the set of task computing requirement information requested by all user devices at time t. For user equipment m n Computational tasks generated at time t; is the set of channel gains of all user equipment associated with base station n on each channel, is the user equipment m in time slot t n The channel gain on channel c with its local base station n; is the set of interference channel gains between other user equipment and base station n on each channel, is the user equipment m in time slot t n′ Interference channel gain with base station n on channel c; The service cache decision set at time t-1;

[0048] In the local observation space:

[0049] in, represents the local observation value of agent n in time slot t;

[0050] In action space:

[0051] in, represents the action space of agent n in time slot t, is the agent n service cache decision, represents the service caching decision of service k in time slot t in base station n; is the channel allocation decision for all user devices under the coverage of agent n, Indicates user equipment m n Channel allocation variables at time slot t; is the task offloading decision of all user devices under the coverage of agent n, Indicates user equipment m n Task offloading decision at time slot t;

[0052] The reward function is defined as:

[0053] Among them, r t represents the rewards obtained by all agents in time slot t, a t represents the joint action of all agents in time slot t, Indicates user equipment m n The utility obtained in time slot t, R() represents the difference between s t and a t Related reward functions.

[0054] Furthermore, in step S43, the attention module takes the environmental state of the MEC system as input and outputs the weight of each local action value function, including:

[0055] S431. Construct the multi-agent environment of the MEC system as an undirected graph in is a collection of nodes, each node represents an agent, It is a set of edges that represent the connectivity between nodes. Each node has a set of edges. Determined set of neighbor nodes

[0056] S432. In time slot t, the environment state s t Input into the MLP encoder and output the local latent representation vector in Represents the local potential representation vector of node n, and then the local potential vector is input into the GAT network;

[0057] S433. Use the self-attention mechanism in the GAT network to calculate the attention coefficient between each node and its neighboring nodes, and use the softmax function to normalize the attention coefficient to obtain the normalized attention coefficient;

[0058] S434. Based on the normalized attention coefficient, the multi-head attention mechanism is used to calculate the output feature representation vector of each node, and input into the MLP network to obtain the weight of the local action value function of each intelligent agent.

[0059] Furthermore, the loss function used in step S44 is expressed as: y tot =r+γmax a′ Q tot (τ′,a′;θ - );

[0060] Among them, θ is the parameter of the prediction network, X represents the number of samples randomly sampled from the experience replay pool, x represents the sample number, θ - is the target network parameter, r represents the reward of all agents in the current time slot, γ represents the discount factor, τ′ represents the joint action observation history of all agents in the next time slot, a′ represents the joint action of all agents in the next time slot, Q tot () represents the joint action value function of all agents in the current time slot.

[0061] Beneficial effects of the present invention:

[0062] The present invention aims at the collaborative task offloading and service caching problems in MEC networks with multiple base stations and diversified service requests, and proposes a multi-agent reinforcement learning algorithm based on graph attention. First, the MEC network scenario with multiple base stations and diversified service requests is modeled, and a utility function based on QoE is introduced to represent the UE's satisfaction with service latency. Under the constraints of storage resources and wireless resources, the optimization problem is modeled as a collaborative task offloading and service caching problem with the goal of maximizing the QoE-based system utility. Then, a multi-agent reinforcement learning algorithm based on graph attention is proposed to learn the optimal task offloading and service caching strategy. Through simulation experiments, compared with the baseline algorithm, the proposed algorithm has obvious improvements in cache hit rate, system utility and calculation success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] FIG1 is a flow chart of a collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning according to the present invention;

[0064] FIG2 is a diagram of a system model of the present invention;

[0065] FIG3 is a schematic diagram of the utility function line of the present invention;

[0066] FIG4 is the architecture of the collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning of the present invention;

[0067] FIG5 is a simulation scene diagram of an embodiment of the present invention;

[0068] FIG6 is a diagram showing how cache hit rate, system utility, and calculation success rate vary with base station storage capacity under different algorithms of the present invention;

[0069] FIG7 is a diagram showing how the cache hit rate, average utility per user, and calculation success rate change with the number of users under different algorithms of the present invention. DETAILED DESCRIPTION

[0070] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0071] The present invention provides a collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning, as shown in Figure 1, including the following steps:

[0072] S1. Build an MEC system in a multi-base station MEC network scenario;

[0073] S2. Based on the MEC system, we model the collaborative task offloading and service caching problem with the goal of maximizing the system utility based on QoE.

[0074] S3. Convert the collaborative task offloading and service caching problem into a distributed partially observable Markov decision process;

[0075] S4. Use a graph attention-based multi-agent reinforcement learning algorithm to solve the distributed partially observable Markov decision process of the transformation.

[0076] Example 1, as shown in Figure 2, consider a MEC system consisting of N base stations (BSs). Each base station in the MEC system is configured with a cache and computing resources of a MEC server to provide computing services to its associated user equipment; the base station set is defined as is the number of base stations; definition To connect to BSn (base station ) associated user device set, Indicates the number of user equipment associated with BSn; the MEC system is time slot dynamic and uses Represents a set of time slots, each of which has the same length, denoted as The MEC system also includes a task model, a service cache model, a communication model, a task offloading model and a utility model.

[0077] Specifically, the method for establishing the task model includes:

[0078] Each user device regularly generates computing tasks with different QoE and service requirements. Once a computing task with a specific service requirement is generated, the relevant data of the service (such as program code and database) needs to be configured to process the computing task. is the set of service types in the MEC system. is the number of service types; define UEm n (i.e. user equipment ) The computation task generated at time t consists of 6 parameters, expressed as in, UEM n The task data size of the computing task generated at time t, UEM n The service type of the computing task generated at time t, UEM n The number of CPU cycles required for each bit of input data of the computation task generated at time t; According to UEm n The service type of the computing task generated at time t predefines three service delay thresholds, It is the minimum service delay threshold, which means that even if the service delay is lower than this value, UEm n No further improvement in QoE can be perceived; Is the maximum service delay threshold. If the service delay is greater than This indicates that the QoE exceeds the acceptable range, that is, the task execution fails; when the service delay is and Between n The QoE is within an acceptable range, but will decrease as the service delay increases. It is UEM n You begin to feel the QoE drop to a point where the experience is poor.

[0079] Specifically, the method for establishing the service cache model includes:

[0080] BSs with caching and computing capabilities can cache services required by computationally intensive and latency-sensitive applications, allowing the corresponding tasks to be processed on the edge network instead of being handed over to the remote cloud center. However, the limited storage capacity of the BS cannot cache all services. Therefore, the BS needs to decide the type of service to cache. Define the binary variable The service caching decision for service k in BSn for time slot t, where if service k is cached in BSn, then otherwise In addition, the present invention considers that the service cached by the base station in time slot t can be used to perform the corresponding task in time slot t+1.

[0081] Specifically, the method for establishing the communication model includes:

[0082] The present invention considers a MEC system based on orthogonal frequency division multiple access (OFDMA). The system bandwidth is divided into C orthogonal channels, which can be allocated in each time slot. Each channel fully reuses the system's spectrum resources. Therefore, during the uplink transmission from the UE to its local base station, it is subject to interval interference caused by the uplink transmission of UEs in other base stations. Definition Represents the set of available channels for each BS, and represents the channel allocation variable for time slot t. If BSn allocates the channel Assigned to UEm n ,but otherwise

[0083] set up is time slot t UEM n The channel gain of UEm on channel c and its local BSn is n Uplink transmission rate on channel c at time slot t Expressed as:

[0084] Where w is the channel bandwidth, UEM n The transmission power, σ 2 is the background noise power, is the interference received by BSn from other user equipment on channel c in time slot t, specifically expressed as:

[0085] in, It is UEM n Interference channel gain with BSn on channel c;

[0086] Based on the above assumptions, UEm n Uplink transmission rate at local BSn in time slot t Expressed as:

[0087] Specifically, the method for establishing the task offloading model includes:

[0088] The present invention considers the collaboration between BSs to provide computing services for UEs. Specifically, each UE has two offloading strategies: 1) the task can be offloaded to the MEC server of the local BS for execution; 2) if the local BS does not cache the service type requested by the UE, the local BS can migrate the task to the MEC server of another BS that has cached the service for execution. Represents UEm n Task offloading decision in time slot t, if UEm n The computational task generated at time t is offloaded to its local BSn for execution, then If UEM n The computational tasks generated at time t are offloaded to BSn′, n′∈N\{n} for execution. If UEM n The computational task generated at time t is discarded, then Wherein, t time slot is a time period, and t time indicates the start time of t time slot.

[0089] The task offloading model is divided into the local base station offloading model and the collaborative offloading model, where:

[0090] 1) Local base station offloading model: Once the UE decides to offload its task to the local BS, the task will first be uploaded to the BS and then executed on the edge server configured by the BS. After the local BS completes the task, it returns the calculation result to the corresponding UE. This paper mainly studies the uplink transmission delay and task calculation delay. Since the amount of data of the calculation result is small, the transmission delay of the result is ignored. According to the above communication model, UEm n The uplink delay of the computing task generated at time t to be sent to its local BSn The calculation formula is:

[0091] in, UEM n The task data size of the computing task generated at time t, Represents UEm n The uplink transmission rate at local base station n in time slot t;

[0092] The MEC server configured in the local BS is responsible for processing computing tasks. nThe computing delay of the computing task in its local BSn The calculation formula is:

[0093] Among them, f n is the CPU frequency of BSn, Represents UEm n The number of CPU cycles required for each bit of input data of the computation task generated at time t;

[0094] Combining task communication and computation delay, we can get UEm n The processing delay of the computation task generated at time t to be offloaded to its local BSn for:

[0095] 2) Collaborative offloading model: Migrating tasks to another BS will generate additional transmission delay. To simplify the system model, is the average transmission rate between BSn and BSn′. n The uplink delay of the computing task generated at time t to be sent to BSn′ The calculation formula is:

[0096] UEm n The computing delay of the computing task in BSn′ The calculation formula is:

[0097] Among them, f n′ Indicates the CPU frequency of BSn′;

[0098] Therefore, UEM n The processing delay of the computation task generated at time t being offloaded to BSn′ for:

[0099] Knowing the service cache status of time slot t (i.e., the service cache decision of time slot t-1), as well as the task offloading and channel allocation decision of time slot t, the UE m that requests the kth service n The service delay can be expressed as:

[0100] Among them, 1 {·} is an indicator function that is 1 if the event {·} is true. {·} =1, otherwise 1 {·} =0.

[0101] Specifically, the method for establishing the utility model includes:

[0102] In order to evaluate the QoE of UE, the present invention adopts a utility function based on service delay. The utility function is defined as follows:

[0103] in, Indicates user equipment m n The utility obtained in time slot t, Indicates processing of user equipment m n The service delay of the computing task generated at time t, Indicates user equipment m n The minimum service delay requirement of the computing task generated at time t, Indicates user equipment m n The maximum service delay requirement of the computing task generated at time t.

[0104] As shown in Figure 3, the utility function is a non-increasing piecewise linear utility function that maps the service delay of each UE to a mean opinion rating scale, i.e., excellent, good, fair, poor, and unsatisfactory. Specifically, when When UEM n With excellent QoE, even if the service delay is lower than this value, UEm n Nor can further improvement of QoE be perceived. In this case, the utility remains unchanged, i.e. When service delays meet When UEM n The QoE of the service is within the acceptable range, but decreases as the service delay increases. At this time, the utility is between 0 and 1. In this case, the present invention uses the point Represents UEm n As shown in Figure 3, when the QoE drops to a point where the user experiences a poor quality of life, When UEM n The QoE is good; when When UEM n The QoE is poor. Note that It does not change the slope of the utility function; the utility function is only affected by and If If the service delay exceeds the acceptable range, the QoE is unsatisfactory. This means that the corresponding task offloading fails, and the utility is negative, that is,

[0105] In Example 2, based on the relevant model constructed in Example 1, the present invention formulates task offloading, channel allocation, and service caching strategies to maximize the total utility of all UEs in the network as the optimization goal, and models the collaborative task offloading and service caching problem as shown below:

[0106] The constraints included are:

[0107] C1:

[0108] C2:

[0109] C3:

[0110] C4:

[0111] C5:

[0112] C6:

[0113] in, is a set of service cache decisions, is the channel allocation decision set, is the set of offloading decisions. Constraint C4 indicates that the service cached by each BS cannot exceed the storage capacity; where Z n is the storage capacity of BSn, z k Serving the cache The present invention assumes that all BSs in the network can exchange service cache information at the beginning of a time slot. Because each BS uses OFDMA for communication, constraint C5 indicates that each UE can be assigned at most one channel; at the same time, constraint C6 indicates that each channel in a BS can be assigned to at most one UE.

[0114] Specifically, the problem in Equation (12) can be solved by finding the optimal decision variables β, α, and b in all time slots, but the huge number of states makes the problem particularly complex. In addition, the present invention takes into account the distributed dynamic MEC environment, and the BSs in the environment cannot obtain complete system information. Therefore, it is difficult to solve this problem using traditional optimization methods. In this case, the problem in Equation (12) can be transformed into a distributed partially observable Markov decision process (Decentralized Partial Observable Markov Decision Process, Dec-POMDP), and the problem is solved by utilizing an algorithm based on multi-agent reinforcement learning (MARL). In addition, the present invention also combines the Graph Attention Network (GAT) with MARL to capture the potential spatial correlation between BSs, thereby better performing collaborative learning.

[0115] Example 3, the problem raised in Example 2 is transformed into a distributed partially observable Markov decision process with N base stations, and each base station is regarded as an intelligent agent; specifically, a tuple can be used To describe Dec-POMDP, where represents the environmental state of all agents, is the set of local observation spaces of agent n, is the action space set of agent n, R is the reward function, and γ∈[0,1) represents the discount factor. At time t, each agent n receives a local observation and select an action Afterwards, the joint actions of all agents can be obtained using After the agent performs the joint action, the environment returns a global reward r t =R(s t ,a t ), and transfer the state to the next state s t+1 In the following, the present invention defines the environment state space, local observation space, action space and reward function of BSs at time t.

[0116] Specifically, at time t, the environment state includes the task computing requirements of all UEs, the wireless channel status in the network, and the current service cache status of all base stations. Therefore, the environment state space Defined as:

[0117] Among them, s t represents the environmental state of the MEC system in time slot t, It is the set of task computing requirement information requested by all UEs at time t. UEM n The computing task generated at time t; for UEm n , whose task computing requirements include the task data size of the computing task generated at time t Service type of computing tasks Required computing resources (unit is CPU cycles / bit) and three predefined service delay thresholds, namely is the set of channel gains of all UEs associated with BSn on each channel, is the user equipment m in time slot t n The channel gain on channel c with its local base station n; is the set of interference channel gains between other user equipment and base station n on each channel, is the user equipment m in time slot t n′ Interference channel gain with base station n on channel c; is the service cache decision set for time slot t-1.

[0118] Specifically, in a partially observable MEC environment, the local observation value of agent n at time slot t can be expressed as:

[0119] in, represents the local observation value of agent n in time slot t; the agent can exchange service cache information with all BSs at the beginning of time slot t, so the agent can observe the service cache status of all BSs, that is, β t-1 .

[0120] Specifically, according to the problem in Equation (12), the action space of agent n in time slot t can be expressed as:

[0121] in, represents the action space of agent n in time slot t, is the agent n service cache decision, represents the service caching decision of service k in time slot t in base station n; is the channel allocation decision for all user devices under the coverage of agent n, Indicates user equipment m n Channel allocation variables at time slot t; is the task offloading decision for all user devices under the coverage of agent n, Indicates user equipment m n Task offloading decision at time slot t.

[0122] Specifically, when performing the joint action a t After that, the environment will return a reward r t To evaluate the joint action. According to formula (12), the goal of the agent is to maximize the sum of the QoE-based utilities of all associated UEs. Therefore, the reward function of the MEC system can be defined as:

[0123] The reward function is defined as:

[0124] Among them, r t represents the rewards obtained by all agents in time slot t, a t represents the joint action of all agents in time slot t, Indicates user equipment m nThe utility obtained in time slot t, R() represents the difference between s t and a t Related reward functions.

[0125] In a partially observable environment, each agent n receives local observations And according to its local strategy π n (a n |τ n )Select an action where τ n Observe the history of actions. represents the joint strategy of all agents. The ultimate goal of all agents is to learn a strategy that jointly optimizes task offloading, channel allocation, and service caching to maximize the decaying cumulative global reward: Therefore, the joint action-value function can be defined as:

[0126] in, is the expected operation, the action value function Q π (s t ,a t ) indicates that the t is the initial state, π is the initial joint strategy and a t is the expectation of the decayed cumulative global reward under the initial action, the optimal joint strategy π * Is to make Q π (s t ,a t )The largest joint strategy.

[0127] Example 4: Traditional MARL methods, such as independent Q-learning, are feasible for solving the Dec-POMDP of N BSs. However, this method has the following challenges: 1) Since the agents will independently update the task offloading and resource allocation strategies during the learning process, the training environment of each agent is unstable, resulting in low learning efficiency; 2) It cannot effectively utilize the potential spatial correlation between agents. For example, different BSs have different popular service requests, and due to the similarity of UEs, adjacent BSs usually have similar popular service requests; at the same time, the uplink transmission rate that the UE can achieve is closely related to the channel allocation strategy and interval interference, and the closer the BSs are, the greater the interference they cause to each other. This suggests that before making an offloading decision, BSs should pay more attention to the status information of their neighboring BSs.

[0128] To address these challenges, the present invention proposes a GatMARL algorithm to solve the task offloading and service caching problems of multi-cell MEC networks. Specifically, the present invention constructs the multi-agent environment as an undirected graph. in is a collection of nodes, each node represents an agent, It is a set of edges that represent the connectivity between nodes. Due to this one-to-one correspondence, BS, agents and nodes all represent the same object. Then, the present invention adopts a value decomposition network based on graph attention to decompose the joint action value function into a combination of local action value functions based on attention relationships. The graph attention mechanism can mine the spatial correlation between BSs and calculate the weight of the local action value function of each agent. The present invention adopts the architecture of Centralized Training with Decentralized Execution (CTDE) for learning. Centralized training can access the state information of all agents in the environment and use this information to update the network parameters of the agents. During the execution phase, each BS only needs to select its task offloading, channel allocation and service caching actions based on its local observation information, without obtaining global state information.

[0129] Specifically, the GatMARL algorithm framework proposed in this invention is shown in FIG4 , which consists of three independent modules: 1) a local action-value network for each agent, 2) a graph attention module, and 3) a joint action-value hybrid module.

[0130] Specifically, each agent is equipped with a Deep Recurrent Q-network (DRQN) consisting of a Gate Recurrent Unit (GRU) and a Multilayer Perceptron (MLP) as its corresponding local action value network. As shown in Figure 4, at time t, the DRQN ​​of agent n receives the local observation value and the previous action As input, and output a local action value function In partially observable MEC networks, the GRU-based Q-network can help the agent memorize its historical service cache and channel allocation actions, as well as historical observations, which is conducive to long-term learning.

[0131] Specifically, the MARL algorithm based on value decomposition aims to transform the joint action-value function Q tot (τ,a) is decomposed into a local action-value function to guide the strategy of a single agent, where τ is the joint action observation history and a is the joint action. For example, the Value Decomposition Network (VDN) decomposes Q tot It is expressed as the sum of local action value functions, and QMIX introduces a hypernetwork to convert Qtot Decomposed into a weighted sum of local action value functions. In a dense MEC system based on OFDMA, due to the spatial correlation of service type requests and wireless network status, the weight of the local action value function of an agent is affected by all its neighboring agents. Therefore, this paper designs a value decomposition network based on graph attention to learn the appropriate Q tot .

[0132] As shown in Figure 4, the environment state s is transformed into t Encoded as a local latent representation vector Then, the local latent representation vector is input into the GAT network to adaptively capture the correlation between agents. Since the present invention constructs the multi-agent environment as an undirected graph G. At time t, the feature of each node n is the local latent representation vector Each node n has a Determine the set of neighbor nodes In the present invention, if there is an edge between nodes, they are neighbors.

[0133] In GAT, node n and its adjacent node i The attention coefficient e n,i can be Calculated, where att(·) is the self-attention mechanism and W is the corresponding learnable weight matrix. Attention coefficient e n,i Indicates the importance of the feature of node n to its neighbor node i. This paper only considers the first-level neighbors of node n (including node n). Then, the softmax function is used to normalize the attention coefficient to facilitate comparison between different nodes, as shown below:

[0134] α n,i Represents the attention coefficient e n,i The normalized attention coefficient after normalization. In order to stabilize the learning process, the present invention adopts a multi-head attention mechanism. After obtaining the normalized attention coefficient, the output feature representation vector of node n with L independent attention mechanisms is It can be given by the following formula:

[0135] Among them, σ is a nonlinear function, || represents the concatenation operation, and l represents the sequence number of the attention mechanism. By applying the graph attention mechanism, represents the attention-based feature representation between nodes. Then, the MLP network is based on the agent n as input and generate appropriate weights for the local action-value function of agent n

[0136] According to the above analysis, the present invention can obtain the graph attention weight of the hybrid module Then, the joint action value function Q tot Can be broken down into:

[0137] Finally, GatMARL is trained by minimizing the loss function, which is expressed as: y tot =r+γmax a′ Q tot (τ′,a′;θ - ) (twenty two);

[0138] Among them, θ is the parameter of the prediction network (i.e., the multi-agent reinforcement learning network based on graph attention), X represents the number of samples randomly sampled from the experience replay pool, x represents the sample sequence number, and θ - is the target network parameter, r represents the reward of all agents in the current time slot, γ represents the discount factor, τ′ represents the joint action observation history of all agents in the next time slot, a′ represents the joint action of all agents in the next time slot, Q tot () represents the joint action value function of all agents in the current time slot.

[0139] In Example 5, in order to verify the effectiveness of the method provided in this embodiment, relevant simulation experiments were carried out. The present invention uses the Pytorch platform and the Adam optimizer to simulate and verify a collaborative task offloading and service caching algorithm based on graph attention multi-agent reinforcement learning (GatMARL). In order to verify the performance of the proposed algorithm, the comparison algorithms include:

[0140] Independent Q-Learning (IQL): IQL allows BSs to independently explore and learn based on their local observations. In IQL-based task offloading and service caching algorithms, if the local BS does not cache the requested service, it migrates the task to other BSs that have cached the service based on the service cache information shared between BSs at the beginning of each time step.

[0141] VDN: The VDN algorithm does not consider the spatial correlation of service request types and wireless network states between BSs, and directly decomposes the joint action value function into the sum of the local action value functions of all agents.

[0142] QMIX: The QMIX algorithm utilizes a hypernetwork to decompose the joint action-value function into a weighted sum of local action-value functions. Similarly, the QMIX-based task offloading and service caching algorithms do not consider the spatial correlation between BSs agents.

[0143] GatMARL-OL: Different from the GatMARL algorithm, the GatMARL-OL algorithm only processes tasks on local BSs.

[0144] Figure 5 shows the simulation scenario of the present invention. The present invention considers 9 BSs with known location coordinates in a 1600m×1600m area. The coverage radius of each BS is set to 200m, and each BS provides services to 5 randomly moving UEs evenly distributed within its coverage area. The present invention divides all BSs into 4 clusters based on the distance between them, namely {BS1, BS2, BS3}, {BS4}, {BS5, BS6} and {BS7, BS8, BS9}. The BSs in the cluster are neighbors. The channel gain is set to d -2 , where d is the distance between the transmitter and the receiver. The transmission power of the UE Evenly distributed in the range of [5,33]dBm. Average transmission rate between BSs Set to 10Mbps. The amount of data generated by the task Evenly distributed between [10,30]Mbit, the number of CPU cycles required for each bit of data Evenly distributed between [500,1000] cycles / bit. The present invention sets 20 service types with storage requirements randomly determined between [10,20] GB. The storage capacity of each base station is Z n The number of orthogonal channels is set to 16, and the bandwidth w of each channel is set to 10 MHz. 2 Set to -50dBm. In the experiment, different services have different delay thresholds. The present invention sets three predefined latency threshold categories for UE service requirements: for latency-sensitive services such as virtual reality, the latency thresholds are set to (0.5s, 1.25s, 2.5s); for some cloud-based gaming, the latency thresholds are set to (1.25s, 2.5s, 3.75s); and for latency-tolerant services such as web services, the latency thresholds are set to (2.5s, 12.5s, 20s). The latency threshold for each service is randomly selected from these three categories.

[0145] Assume that UE generates one task per time slot and service preference follows Zipf distribution. The ranking of service k’s popularity in BS is recorded as Then the popularity of service k can be defined as:

[0146] Among them, δ nrepresents the skewness coefficient of popularity in BSn. In the experiment, unless otherwise specified, the skewness coefficient of all BSs is 0.8. Then, the present invention defines a popularity vector φ for each BSn n ={φ n,1 ,φ n,2 ,…,φ n,K}. UEs are assigned a value based on the φ of their local BS. n Generate tasks with specific service requirements. To simulate the regional popularity of service requirements, the present invention sets different popularity vectors for different clusters. Specifically, the service popularity ranking of each cluster is randomly sorted according to the service directory in the network.

[0147] For GatMARL and GatMAR-OL algorithms, the present invention uses a GAT layer with 4 attention heads. The attention mechanism att(·) is a single-layer feedforward neural network activated by the LeakyReLU activation function. The hidden layer dimension of the GRU layer is set to 64, and the hidden layer dimension of the hybrid network of QMIX is set to 64. The present invention reduces the exploration rate from 1 to 0.05 in 1,000,000 time slots. The learning rate is 0.0005. In addition, in order to improve the learning efficiency of the decision model, the present invention deletes the local BS offloading action (i.e. ), and use the local BS service cache status to represent That is, if the local BS has cached the corresponding service type, the agent directly performs the local BS offloading action without further exploration.

[0148] The proposed GatMARL algorithm is verified based on three performance indicators: utility, computation success rate, and cache hit rate. The computation success rate is the number of tasks successfully processed by the entire MEC system within the maximum time threshold divided by the total number of tasks; the cache hit rate is the number of tasks processed by the local base station divided by the total number of tasks in the network.

[0149] Figure 5 shows the impact of BS storage capacity on system utility, computation success rate, and cache hit rate. In this experiment, we set the storage capacity of each BS between 60 and 180, with a step size of 20, and kept the default values ​​mentioned above for other parameters. As shown in Figure 6(a), the curves show an upward trend as the storage capacity of each BS increases. This is because a larger storage capacity allows each BS to cache more services, thereby satisfying more UE requests from the local BS or other BSs in the network, thereby improving the computation success rate. Similarly, as shown in Figure 6(b), increasing storage capacity can improve the cache hit rate because more popular services can be cached on the local BS. Increased computation success rate and cache hit rate also mean improved system utility, as shown in Figure 6(c). Compared to the other four benchmarks, the proposed GatMARL-based task offloading and service caching algorithm consistently performs best.

[0150] Figure 7 shows the performance of different algorithms under different numbers of UEs. In this experiment, the present invention sets the number of UEs within each BS range to 2 to 8, and all other parameters are default values. As shown in Figure 7(a), the offloading success rate of the IQL, VND, QMIX, and GatMARL algorithms is inversely proportional to the number of UEs. The reason is that as the number of UEs increases, the interference between cells will also increase, resulting in an increase in the uplink transmission delay from the UE to the local base station, which will lead to the failure of task offloading, especially in the cooperative offloading mode with additional inter-base station transmission delay. As can be seen from Figure 7(b), the local hit rate of all algorithms is almost constant under different numbers of UEs. From the trend of the curve, it can be seen that the agent can always learn an effective service caching strategy based on the historical request service type status of the UE. Figure 7(c) shows the average utility of each UE achieved by different algorithms under different numbers of UEs. It can be observed that the utility gradually decreases with the increase of the number of UEs. This is because the decrease in the calculation success rate reduces the utility. At the same time, according to formula (11), the increase in the transmission delay from the UE to the local BS may also lead to a decrease in utility. In addition, Figure 7 also shows that under different numbers of UEs, the performance of GatMARL is significantly better than other algorithms.

[0151] In the present invention, unless otherwise clearly stipulated and limited, the terms "installation", "setting", "connection", "fixation", "rotation" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0152] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning, characterized in that: The following steps are involved: S1. Build a MEC system in a MEC network scenario with multiple base stations and diversified service requests, where: In the MEC system, each base station is configured with a cache and computing resources for the MEC server to provide computing services to its associated user equipment; the base station set is defined as is the number of base stations; definition For base station The associated user device collection, Represents the number of user equipment associated with base station n; the MEC system is time slot dynamic and adopts Represents a set of time slots, each of which has the same length, denoted by The MEC system also includes a task model, a service cache model, a communication model, a task offloading model and a utility model; S2. Based on the MEC system, the collaborative task offloading and service caching problems are modeled with the goal of maximizing the system utility based on QoE; S3. Transform the collaborative task offloading and service caching problem into a distributed partially observable Markov decision process, including: S31. Convert the collaborative task offloading and service caching problem into a distributed partially observable Markov decision process with N base stations, and regard each base station as an intelligent agent; S32. Setting the environment state space, local observation space, action space and reward function; S4. Using a graph attention-based multi-agent reinforcement learning algorithm to solve the transformed distributed partially observable Markov decision process, including: S41. Construct a multi-agent reinforcement learning network based on graph attention, which includes a graph attention module, a joint action value hybrid module, and a local action value network corresponding to each agent; S42. Each agent inputs the local observation value and the previous action it receives into its corresponding local action value network, and outputs the local action value function; S43. The graph attention module takes the environmental state of the MEC system as input and outputs the weight of each local action value function; S44. The joint action value mixing module adds all local action value functions according to weights to obtain a joint action value function, and trains by minimizing the loss function; S45. After the training is completed, the global optimal task offloading and service caching strategy is obtained according to the local observation values ​​of the base station.

2. The collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning according to claim 1 is characterized in that: The method for establishing the task model includes: each user device periodically generates computing tasks with different QoE and service requirements, and defines is a set of service types in the MEC system. is the number of service types; defines user equipment The computing task generated at time t consists of 6 parameters, expressed as in, For user equipment m n The task data size of the computing task generated at time t, For user equipment m n The service type of the computing task generated at time t, For user equipment m n The number of CPU cycles required for each bit of input data of the computation task generated at time t; According to the user device m n The service type of the computing task generated at time t predefines three service delay thresholds: is the minimum service delay threshold, is the maximum service delay threshold, is the user device m n You start to feel the QoE drop to a point where the experience is poor.

3. The collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning according to claim 1 is characterized in that: The method for establishing a service cache model includes: the service cached by the base station in the t time slot is used to perform the corresponding task in the t+1 time slot; defining a binary variable The service cache decision for service k in base station n in time slot t, where if service k is cached in base station n, then otherwise 4. The collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning according to claim 1 is characterized in that: The method for establishing the communication model includes: the bandwidth of the MEC system is divided into C orthogonal channels, which are allocated in each time slot, and each channel fully reuses the spectrum resources of the MEC system; define Represents the set of available channels for each base station, and represents the channel allocation variable of base station n in time slot t. If base station n allocates channel Assigned to user equipment m n ,but otherwise set up is the user equipment m in time slot t n The channel gain of user equipment m on channel c and its local base station n is n The uplink transmission rate on channel c at time slot t It is expressed as: Therefore, user device m n The uplink transmission rate at local base station n in time slot t is It is expressed as: Where w is the channel bandwidth, For user equipment m n The transmission power, σ 2 is the background noise power, is the interference from other user equipment received by base station n on channel c in time slot t.

5. The collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning according to claim 1 is characterized in that: The method for establishing the task offloading model includes: each user equipment's computing task has two offloading strategies: local base station offloading and collaborative offloading; defining Indicates user equipment m n Task offloading decision in time slot t, if user device m n The computing task generated at time t is offloaded to its local base station n for execution, then If the user device m n The computational tasks generated at time t are offloaded to the base station Execute, then If the user device m n The computational task generated at time t is discarded, then Wherein, t time slot is a time period, and time t represents the start time of t time slot; If the local base station offloading strategy is selected, the user equipment m n The uplink delay of the computing task generated at time t to be sent to its local base station n The calculation formula is: in, For user equipment m n The task data size of the computing task generated at time t, Indicates user equipment m n The uplink transmission rate at local base station n in time slot t; User equipment m n The computation delay of the computation task at its local base station n is The calculation formula is: Among them, f n is the CPU frequency of base station n, Indicates user equipment m n The number of CPU cycles required for each bit of input data of the computation task generated at time t; Therefore, user device m n The processing delay of the computation task generated at time t to be offloaded to its local base station n for: If the collaborative offloading strategy is selected, the user device m n The migration of the computing task generated at time t to base station n′ will cause additional transmission delay. n The uplink delay of the computing task generated at time t to be sent to base station n′ The calculation formula is: in, represents the average transmission rate between base station n and base station n′; User equipment m n The computing delay of the computing task at base station n′ The calculation formula is: Among them, f n′ represents the CPU frequency of base station n′; Therefore, user device m n The processing delay of the computational task generated at time t being offloaded to base station n′ for:

6. The collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning according to claim 1 is characterized in that: The method for establishing the utility model includes: defining a utility function to evaluate the service quality of the user equipment, wherein the utility function is expressed as: in, Indicates user equipment m n The utility obtained in time slot t, Indicates processing user equipment m n The service latency of the computing task generated at time t, Indicates user equipment m n The minimum service delay requirement of the computing task generated at time t, Indicates user equipment m n The maximum service delay requirement of the computing task generated at time t.

7. The collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning according to claim 1 is characterized in that: Step S32 sets the environment state space, local observation space, action space and reward function, including: In the environment state space: Among them, s t represents the environmental state of the MEC system in time slot t, is the set of task computing requirement information requested by all user devices at time t, For user equipment m n At t The computational tasks generated at that moment; is the set of channel gains of all user equipments associated with base station n on each channel, is the user equipment m in time slot t n The channel gain on channel c with its local base station n; is the set of interference channel gains between other user equipment and base station n on each channel, is the user equipment m in time slot t n′ Interference channel gain with base station n on channel c; The service cache decision set at time t-1; In the local observation space: in, represents the local observation value of agent n in time slot t; In action space: in, represents the action space of agent n in time slot t, is the agent n service cache decision, represents the service cache decision of service k in time slot t in base station n; is the channel allocation decision for all user devices under the coverage of agent n, Indicates user equipment m n Channel allocation variables at time slot t; is the task offloading decision of all user devices under the coverage of agent n, Indicates user equipment m n Task offloading decision at time slot t; The reward function is defined as: Among them, r t represents the rewards obtained by all agents in time slot t, a t represents the joint action of all agents in time slot t, Indicates user equipment m n The utility obtained in time slot t, R() represents the difference with s t and a t The associated reward function.

8. The collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning according to claim 1 is characterized in that: Step S43: The attention module takes the environmental state of the MEC system as input and outputs the weight of each local action value function, including: S431. Construct the multi-agent environment of the MEC system as an undirected graph in is a set of nodes, each node represents an agent, ε is a set of edges representing the connectivity between nodes, and each node has a set of neighbor nodes determined by the edge set ε S432. In time slot t, the environment state s t Input into the MLP encoder and output the local latent representation vector in Represents the local potential representation vector of node n, and then the local potential representation vector is input into the GAT network; S433. The self-attention mechanism in the GAT network is used to calculate the attention coefficient between each node and its neighboring nodes, and the softmax function is used to normalize the attention coefficient to obtain a normalized attention coefficient; S434. According to the normalized attention coefficient, a multi-head attention mechanism is used to calculate the output feature representation vector of each node, and the vector is input into the MLP network to obtain the weight of the local action value function of each agent.

9. The collaborative task offloading and service caching method based on graph attention multi-agent reinforcement learning according to claim 1 is characterized in that: The loss function used in step S44 is expressed as: y tot = r + γmax a′ Q tot (τ′,a′;θ - ); Among them, θ is the parameter of the prediction network, X represents the number of samples randomly sampled from the experience replay pool, x represents the sample number, and θ - is the target network parameter, r represents the reward of all agents in the current time slot, γ represents the discount factor, τ′ represents the joint action observation history of all agents in the next time slot, a′ represents the joint action of all agents in the next time slot, Q tot () represents the joint action value function of all agents in the current time slot.

Images