A Multi-Agent Cooperative Computing Offloading Method in a 5G Mobile Edge Computing Environment

Abstract

The present invention discloses a multi-agent collaborative computing offloading method in a 5G mobile edge computing environment. By initializing the parameters of the equipment, 5G network and optical fiber network in the MEC environment, the task queue information of the mobile device, the task information that needs to be calculated, and the network information of the device and the base station are collected based on the initialization parameters; according to the collected information, if the task information that needs to be calculated is local calculation, the task is added to the local task queue of the mobile device; if it is calculation offloading, the task is transmitted to the MEC server of the connected base station; if it is transmitted to the MEC server for calculation, the calculation task is transmitted to the MEC server of the nearby base station, and the task is added to the task queue of the server; if it is cloud computing, the calculation task is transmitted to the cloud server for calculation, thereby realizing multi-agent collaborative computing offloading. The powerful computing power of cloud computing and the close distance advantage of MEC of the present invention provide users with a high-quality experience and minimize the application execution delay.

Description

Technical Field

[0001] The present invention belongs to the field of computing offloading strategies, and specifically relates to a multi-agent collaborative computing offloading method in a 5G mobile edge computing environment. Background Art

[0002] With the proliferation of mobile smart devices and the development of wireless communication technologies like 5G, numerous computationally intensive applications with low latency requirements have emerged, such as autonomous driving, augmented reality, virtual reality, and video streaming analysis. Traditional cloud computing cannot meet the low-latency demands of these applications. Mobile Edge Computing (MEC) is a promising computing paradigm within the 5G architecture. It can work in tandem with cloud computing to migrate workloads from mobile terminals to closely located MEC servers or central clouds, providing computing power to users' mobile devices and improving their Quality of Experience (QoE). However, combining the powerful computing power of cloud computing with the proximity advantages of MEC to provide users with a high-quality experience and minimize application execution latency has become an urgent challenge. Summary of the Invention

[0003] The purpose of the present invention is to provide a multi-agent collaborative computing offloading method in a 5G mobile edge computing environment to overcome the shortcomings of the existing technology. The present invention effectively combines the powerful computing power of cloud computing and the close distance advantage of MEC to provide users with a high-quality experience and minimize application execution delay.

[0004] A multi-agent collaborative computing offloading method in a 5G mobile edge computing environment comprises the following steps:

[0005] S1: Initialize the parameters of devices, 5G networks, and optical networks in the MEC environment. Based on the initialization parameters, collect information about the task queues of mobile devices, information about tasks to be calculated, and network information between devices and base stations.

[0006] S2, calculates based on the collected information. If the task information to be calculated is local calculation, the task is added to the local task queue of the mobile device; if it is calculation offloading, the task is transmitted to the MEC server of the connected base station; if it is transmitted to the MEC server for calculation, the calculation task is transmitted to the MEC server of the nearby base station, and the task is added to the task queue of the server; if it is cloud computing, the calculation task is transmitted to the cloud server for calculation, realizing multi-agent collaborative calculation offloading.

[0007] Preferably, a time slot model is established according to the initialization parameters, and communication and calculation offloading is performed based on the time slot model.

[0008] Preferably, an orthogonal frequency division multiplexing method is used to allocate different sub-channels to different devices.

[0009] Preferably, at the tth time slot, the mobile device u n With MEC server m The channel gain h of the wireless channel between n,m (t) is calculated by formula (3):

[0010]

[0011] Where: β n，m (t) represents the t-th time slot u n With e m The channel fading factor between them obeys the Rayleigh distribution, and its probability distribution function is f B (β), A d Indicates e m The radar gain of the connected base station, c0 represents the speed of light in vacuum, f c Indicates e m The carrier frequency of the connected base station, d n，m (t) represents the t-th time slot u n With e m The distance between e Represents the path loss exponent.

[0012] Preferably, according to Shannon's formula and formula (3), at the tth time slot, the mobile device u n With MEC server m The transmission rate between them is calculated by formula (4):

[0013]

[0014] Where: B represents u n With e m The channel bandwidth between them, p0 represents the mobile device u n The transmission power of N0 is the Gaussian white noise power.

[0015] Preferably, the tasks are allocated based on the collected information and calculated with minimizing the computational delay as the optimization goal.

[0016] Preferably, at the tth time slot, the local task queue of the mobile device un is set to As shown in formula (6):

[0017]

[0018] Where: Indicates the number of tasks in the local queue at the beginning of the current time slot, Indicates the total number of CPU cycles that the local queue needs to calculate at the beginning of the current time slot. At t = 0, That is, there are no tasks in the queue;

[0019] At the t+1th time slot, if the computation task d n (t) Total CPU cycles in the local queue during local execution Calculated by formula (7):

[0020]

[0021] If d n (t) Total CPU cycles in the local queue that are not executed locally Calculated by formula (8):

[0022]

[0023] The number of tasks in the local queue It is related to the number of tasks completed within time T, but the number of cycles for each task is different;

[0024] Set mobile device u n Completed in time slot t tasks, if the task is executed locally, then the number of tasks in the local queue at the beginning of the next time slot is Calculated by formula (9):

[0025]

[0026] If the task is not executed locally, then Calculated by formula (10):

[0027]

[0028] If the mobile device u n The computation task d at the tth time slot n (t) If the task is executed locally on the device, it will be added to the local queue and its execution delay will be Calculated by formula (11):

[0029]

[0030] Preferably, at the tth time slot, the MEC server e m Task Queue As shown in formula (12):

[0031]

[0032] Where: Indicates the number of tasks in the MEC server queue at the beginning of the current time slot, Indicates the total number of CPU cycles that the MEC server queue needs to calculate at the beginning of the current time slot; at t = 0,

[0033] Preferably, if the mobile device u n The computation task d at the tth time slot n (t) The MEC server e of a base station near the base station it is connected to m′ Execute, the task will be added to the MEC server e m′ In the task queue, the total number of CPU cycles and the number of tasks in the next time slot of the queue are calculated by formulas (13), (14), (15) and (16), and the task execution delay is Calculated by formula (18):

[0034]

[0035] Preferably, the problem of minimizing the long-term computational delay of all tasks in the entire MEC system under limited resource constraints is expressed as formula (21):

[0036]

[0037]

[0038] Where A(t) represents the set of locations where all mobile devices execute computational tasks at the tth time slot, D represents the coverage radius of the base station, constraint C1 ensures that the computational task can only be performed once, and constraint C2 ensures that all user mobile devices are active within the coverage area of the base station. After a mobile device exceeds the maximum service distance of a base station, another base station will continue to provide service.

[0039] Compared with the prior art, the present invention has the following beneficial technical effects:

[0040] The present invention provides a multi-agent collaborative computing offloading method in a 5G mobile edge computing environment. By initializing the parameters of the equipment, 5G network and optical fiber network in the MEC environment, the task queue information of the mobile device, the task information that needs to be calculated, and the network information of the device and the base station are collected based on the initialization parameters; calculation is performed according to the collected information. If the task information that needs to be calculated is local calculation, the task is added to the local task queue of the mobile device; if it is calculation offloading, the task is transmitted to the MEC server of the connected base station; if it is transmitted to the MEC server for calculation, the calculation task is transmitted to the MEC server of the nearby base station, and the task is added to the task queue of the server; if it is cloud computing, the calculation task is transmitted to the cloud server for calculation, thereby realizing multi-agent collaborative computing offloading. The powerful computing power of cloud computing and the close distance advantage of MEC of the present invention provide users with a high-quality experience and minimize the application execution delay.

[0041] Furthermore, an orthogonal frequency division multiplexing method is used to allocate different sub-channels to different devices to reduce mutual interference between sub-channels and ensure the transmission needs of the devices.

[0042] Furthermore, the present invention uses multiple intelligent agents at the end layer, edge layer and cloud layer to offload task calculations based on the time slot model on mobile device information, effectively reducing the average task calculation time and improving the efficiency of edge computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is the MEC system architecture diagram for cloud-edge-end collaboration in an embodiment of the present invention.

[0044] Figure 2 2 is a diagram of the training process of a mobile device agent in an embodiment of the present invention.

[0045] Figure 3 This is a training diagram of the MEC server agent in an embodiment of the present invention.

[0046] Figure 4 This is a comparative experimental diagram of the baseline method in an embodiment of the present invention.

[0047] Figure 5 This is a comparison chart of the embodiments of the present invention and other algorithm experiments. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0049] like Figure 1 As shown in the figure, the multi-agent collaborative computing offloading system in the 5G mobile edge computing environment includes the cloud layer, edge layer, and terminal layer, forming a collaborative MEC system architecture, including the following:

[0050] The cloud layer includes a remote central cloud server cluster, which has much greater computing and storage capabilities than MEC servers and mobile devices. However, it is far away from the mobile devices. Offloading tasks to the central cloud requires more core network resources, resulting in longer transmission delays. The cloud computing time of the task can be relatively ignored.

[0051] The edge layer includes base stations and MEC servers connected to the base stations. The computing power of MEC servers is much greater than that of mobile devices. Offloading tasks to MEC servers requires a certain amount of computing time. However, the distance between the MEC servers and mobile devices is very close, and the transmission delay is large. In the case of poor channel conditions, there will be a relatively high transmission delay.

[0052] The end layer includes mobile devices, which have the lowest computing power. However, when the network is very poor, local computing will provide the shortest result.

[0053] Users move within the coverage area of base station signals, and their mobile devices run computationally intensive and time-sensitive applications. Users' mobile devices move within the coverage area of all base stations, which makes the 5G wireless network between mobile devices and base stations more variable and the computational offloading problem of device tasks more complicated.

[0054] In the architecture, the scenario is seamlessly covered by many base stations (BSs). These base stations provide computing offload services to a large number of mobile devices (MDs) of users distributed within the base station range through the 5G wireless communication network, and the base stations are connected by optical fibers.

[0055] Each base station deploys a MEC server to increase computing power, enabling the base station to compute a variety of tasks to meet user needs. A user's mobile device generates computing tasks, which it can handle using its own processor or a connected MEC server. Furthermore, the base station is connected to the core network via high-speed fiber optic cables, exchanging data with the central cloud. This allows mobile devices to offload tasks to the central cloud through the base station.

[0056] A mobile device agent is designed using a deep reinforcement learning approach based on Double DQN (DDQN). Its state space is designed to be the acquired information, its action space is designed to be composed of two actions: local computation and computation offloading. The reward function is designed to be the negative of the task computation time. If the acquired task information is for local computation, the task is added to the local task queue; if it is for computation offloading, the computation task is transferred to the MEC server of the connected base station.

[0057] A MEC server agent is designed using a deep reinforcement learning method based on Dueling DDQN (D3QN). Its state space is designed to be the acquired information, and its action space is designed to be three actions: local computation, computation on a nearby base station MEC server, and cloud computing. The reward function is designed to be the negative of the task computation time. If the computation is done locally, the task is added to the local task queue. If the computation is done on a nearby base station MEC server, the computation task is transferred to the nearby base station MEC server and added to the server's task queue. If the computation is done on cloud computing, the task is transferred to the cloud server for computation.

[0058] The set U of user mobile devices is shown in formula (1):

[0059] U={u1,u2,…,u n ,...,u N} (1)

[0060] Where: u n Represents the nth mobile device, and N represents the number of mobile devices.

[0061] The set E of MEC servers is shown in formula (2):

[0062] E={e1,e2 ， ..., e m ,...,e M}#(2)

[0063] Where: e m Indicates the mth MEC server, and M indicates the number of servers.

[0064] This application establishes a time slot model, which discretizes time and divides it into time intervals of equal length. These intervals are called time slots, whose length is recorded as T and the index is represented by t = 0, 1, 2, .... The size of T is determined by the coherence time of the channel, that is, the maximum time difference range within which the channel remains constant. In a communication system, the communication channel may change over time. Due to the Doppler effect, this channel change is more significant in a wireless communication system. In the system model, the mobile device and the base station communicate through a 5G wireless network, so this application establishes a time slot model based on the coherence time of the channel. Based on the time slot model, the communication model and the calculation model are introduced in detail below.

[0065] In the MEC system network that collaborates with the cloud, edge, and end, there are mainly two communication modes: wireless communication and wired communication; wireless communication between the edge layer and the end layer, that is, wireless communication between mobile devices and base stations; wired communication between edge layers and between edge layers and cloud layers, that is, wired communication between base stations and between base stations and central clouds.

[0066] Among them, any base station e m and e m′ Between and base stations m The transmission rates with the central cloud are expressed as and Both obey stable and independent random processes, and their probability distribution functions are f E (v E ), f E，C (v E，C ). Next, the wireless communication method between mobile devices and base stations is introduced in detail and modeled.

[0067] To minimize the execution latency of computing tasks on mobile devices, this application builds a wireless communication model based on 5G technology. Mobile devices may need to offload computations and transmit data to the base station every T time interval. Therefore, Orthogonal Frequency Division Multiplexing (OFDM) technology is used to allocate different sub-channels to different devices, reducing mutual interference between sub-channels and ensuring device transmission needs.

[0068] Based on the free space path loss model, this application constructs a Rayleigh fading model to simulate the channel in a densely populated urban environment. In this scenario, there is no direct path between the transmitter and the receiver, and dense buildings and other objects attenuate, reflect, refract, and diffract the wireless signal. The channel remains stable within a time slot, but changes between different time slots. At the tth time slot, the mobile device un and the MEC server e m The channel gain h of the wireless channel between n，m (t) is calculated by formula (3):

[0069]

[0070] Where: β n，m (t) represents the t-th time slot u n With e m The channel fading factor between them obeys the Rayleigh distribution, and its probability distribution function is f B (β), A d Indicates e m The radar gain of the connected base station, c0 represents the speed of light in vacuum, f c Indicates e m The carrier frequency of the connected base station, d n，m (t) represents the t-th time slot u n With e m The distance between e Represents the path loss exponent.

[0071] Since the base station and its deployed MEC server are very close, there is almost no transmission delay, which is ignored in this application. According to Shannon formula and formula (3), at the tth time slot, the mobile device u n With MEC server m The transmission rate between them is calculated by formula (4):

[0072]

[0073] Where: B represents u n With e m The channel bandwidth between them, p0 represents the mobile device u n The transmission power of OFDM is 0, and N0 represents the Gaussian white noise power. Since the OFDM sub-channels do not interfere with each other, the noise in the channel is only Gaussian white noise.

[0074] At the beginning of the tth time slot, N mobile devices generate N tasks simultaneously. The tasks are independent of each other and each task is inseparable. n The size of the generated calculation data is represented as s n (t), use c n (t) represents the number of CPU cycles required to process 1 bit of data. If no data is generated at the beginning of the tth time slot, then s n (t) = 0; s n (t) is u n The data generated in the t-1th time slot can be processed starting from the tth time slot. Therefore, in order to facilitate the representation of the model, it is considered that s n (t) is generated at the beginning of the t-th time slot.

[0075] To simplify the notation, at the beginning of the tth time slot, the mobile device u n The generated computing task d n (t) is shown in formula (5):

[0076] d n (t): = {s n (t), c n (t)}#(5)

[0077] Where: s n (t) and c n (t) is modeled to obey different stable and independent random processes, whose probability distribution functions are f S (s), f C (c).

[0078] The CPU frequency (i.e., computing power) of the central cloud can be considered infinite compared to that of the mobile device and MEC server. Therefore, this application sets the CPU frequency of the central cloud to infinity and the CPU frequency of the mobile device and MEC server to fixed values, which are respectively denoted as f and f. U and f E Indicates that f E >>f U The computing power of mobile devices and MEC servers is relatively limited. If a task is calculated on a mobile device or MEC server, it may not be executed immediately. Therefore, this application sets up task queues in the mobile device and MEC server respectively to store tasks to be executed, where the queues adopt the "first in, first out" principle. At the tth time slot, the mobile device u n The task queue is represented as MEC Server m The task queue is represented as

[0079] Computational tasks on mobile devices can be performed locally, offloaded to MEC servers at connected base stations, offloaded to MEC servers at nearby base stations, or offloaded to a central cloud. Different computing modes produce different latency, and these four computing modes are analyzed in detail below.

[0080] Local computing:

[0081] At time slot t, let the local task queue of mobile device un be As shown in formula (6):

[0082]

[0083] Where: Indicates the number of tasks in the local queue at the beginning of the current time slot, Indicates the total number of CPU cycles that the local queue needs to calculate at the beginning of the current time slot. At t = 0, That is, there are no tasks in the queue;

[0084] At the t+1th time slot, if the computation task d n (t) Total CPU cycles in the local queue during local execution Calculated by formula (7):

[0085]

[0086] If d n (t) Total CPU cycles in the local queue that are not executed locally Calculated by formula (8):

[0087]

[0088] The number of tasks in the local queue It is related to the number of tasks completed within time T, but the number of cycles for each task is different;

[0089] Set mobile device u n Completed in time slot t tasks, if the task is executed locally, then the number of tasks in the local queue at the beginning of the next time slot is Calculated by formula (9):

[0090]

[0091] If the task is not executed locally, then Calculated by formula (10):

[0092]

[0093] If the mobile device u n The computation task d at the tth time slot n (t) If the task is executed locally on the device, it will be added to the local queue and its execution delay will be Calculated by formula (11):

[0094]

[0095] Mobile Edge Computing (MEC):

[0096] At the tth time slot, similar to formula (6), let the MEC server e m Task Queue As shown in formula (12):

[0097]

[0098] Where: Indicates the number of tasks in the MEC server queue at the beginning of the current time slot, Indicates the total number of CPU cycles that the MEC server queue needs to calculate at the beginning of the current time slot; at t = 0,

[0099] By analogy with formulas (7), (8), (9) and (10), we can calculate the MEC server e at the beginning of the next time slot: m The total number of CPU cycles and number of tasks that the task queue needs to calculate, where Indicates e m The number of tasks completed in the tth time slot.

[0100] If task d n(t) is executed on the MEC server. The total number of CPU cycles and the number of tasks in the next time slot of the queue can be calculated by formulas (13) and (14) respectively:

[0101]

[0102]

[0103] If task d n (t) is not executed on the MEC server. The total number of CPU cycles and the number of tasks in the next time slot of the queue can be calculated by formulas (15) and (16) respectively:

[0104]

[0105]

[0106] If the mobile device u n The computation task d at the tth time slot n (t) MEC server e at its connected base station m If the task is executed, it will be added to the task queue of the MEC server and its execution delay Calculated by formula (17). Since the result data after task calculation is very small compared to the task data, the time it takes to transmit the result data back to the device is ignored.

[0107]

[0108] Neighboring MEC calculation:

[0109] If the mobile device u n The computation task d at the tth time slot n (t) The MEC server e of a base station near the base station it is connected to m′ Execute, the task will be added to the MEC server e m′ In the task queue, the total number of CPU cycles and the number of tasks in the next time slot of the queue are calculated by formulas (13), (14), (15) and (16), and the task execution delay is Calculated by formula (18).

[0110]

[0111] cloud computing:

[0112] If the mobile device u n The computation task d at the tth time slot n (t) is executed in the central cloud, and its execution delay Calculated by formula (19). Since the central cloud computing capacity is infinite, the computing time of the task is very small compared to the transmission time, so it is ignored.

[0113]

[0114] Move your mobile device to n The computation task d at the tth time slot n The execution location of (t) is represented by a n (t), when a n When (t) = 0, it means d n (t) is calculated locally; when a n When (t) = 1, it means d n (t) is calculated in the MEC server of the base station connected to it; when a n When (t) = 2, it means d n (t) is calculated by the MEC server of a base station near the base station connected to it; when a n When (t)=3, it means d n (t) is calculated in the central cloud server. Combine formulas (11), (17), (18) and (19) to calculate task d n The execution delay of (t) is calculated using formula (20).

[0115]

[0116] Where: represents the indicator function, when If the condition in the brackets is true, the function value is 1, otherwise the function value is 0.

[0117] Then, the problem of minimizing the long-term computing delay of all tasks in the entire MEC system under the constraint of limited resources is expressed as formula (21):

[0118]

[0119]

[0120] Where A(t) represents the set of locations where all mobile devices execute computational tasks at the tth time slot, D represents the coverage radius of the base station, constraint C1 ensures that the computational task can only be performed once, and constraint C2 ensures that all user mobile devices are active within the coverage area of the base station. After a mobile device exceeds the maximum service distance of a base station, another base station will continue to provide service.

[0121] like Figure 2 As shown in Figure 2, the training process of the mobile device agent includes the following:

[0122] The DDQN-based mobile device agent is designed to address the issue of whether a mobile device can perform computational tasks locally. Its state space, action space, and reward function are defined as follows.

[0123] State Space:

[0124] Considering the limited computing power of mobile devices, it is not suitable to place complex agents. This application minimizes the state space and defines the state of the mobile device agent in the tth time slot. As shown in formula (22), where S U Represents the state space of the mobile device agent. The agent is based on the current mobile device task, queue and the connection with the base station e m The status of the wireless network can make the decision of computing offloading; since the mobile device agent is deployed on the mobile device, the agent can easily obtain the queue, tasks and communication with the base station. m Wireless network information;

[0125]

[0126] Where: d n (t) represents the computational task generated by the mobile device at the beginning of the t-th time slot, Represents the local task queue of the mobile device, v n，m (t) represents the transmission rate between the mobile device and the MEC server, s n (t), c n (t) represents the data size of the computing task and the number of CPU cycles required to process 1 bit of the data, They represent the number of tasks in the local queue and the total number of CPU cycles required to calculate the queue.

[0127] Action Space:

[0128] The goal of the mobile device agent is to map the state space to the action space, that is, to select the optimal action based on the current state to minimize the delay in executing the computing task. In the MEC system of this application, the mobile device agent is responsible for deciding whether to execute the computing task locally in each time slot. The action of the agent in the tth time slot is used It is defined as shown in formula (23):

[0129]

[0130] Where: A U Represents the action space of the mobile agent. Indicates task d n (t) executed locally on the mobile device, Indicates that task d n (t) Uninstall, by em The agent makes the next decision.

[0131] Reward function:

[0132] Generally speaking, the reward function is related to the target problem. In the optimization problem considered in this application, the goal is to minimize the long-term computational delay of all tasks in the entire MEC system, while reinforcement learning is to learn "what to do (i.e., how to map the current situation into actions) to maximize the numerical benefit signal." Therefore, the reward function value needs to be negatively correlated with the target problem value. Combined with formula (11), at the tth time slot, the agent is in state Next, take action The reward function value of Calculated by formula (24), where R U Represent the reward function space of the mobile agent:

[0133]

[0134] Where: time n，m (t) represents task d n (t) Transmitted to the MEC server e of the base station connected to the mobile device m The specific calculation formula for the subsequent execution time can be found in formula (28) in the MEC server agent.

[0135] The training process of a single episode of a mobile agent based on DDQN consists of the following steps, where a single episode refers to the sequence of states, actions, and rewards that a mobile agent uses to interact with the system model for computational offloading. First, the system model is initialized, the current network Q parameters θ and the target network Q parameters θ are initialized, and the training process of a single episode of a mobile agent is carried out. parameter and experience replay pool Next, iterative training begins by obtaining the initial state s0. In each time slot, an ε-greedy strategy is used to select actions. This strategy behaves greedily most of the time, occasionally selecting actions randomly from all actions with equal probability, independent of the action value. This ensures that after the neural network has roughly converged, the agent can explore other actions and prevent itself from falling into a local optimum. The action is then executed in the system model, resulting in the corresponding reward and next state. This transition information is then stored in the experience replay pool. The agent then selects batches of samples from the experience replay pool to calculate the gradient of the loss function, which is then minimized by backpropagating the gradient using gradient descent. Finally, through continuous iterative training, the current network Q converges to the optimal action-value function.

[0136] like Figure 3 As shown in Figure 1, the training process of the MEC server agent includes the following:

[0137] The D3QN-based MEC server agent is designed to solve the problem of mobile device tasks executing in one of three computing modes: MEC computing, adjacent MEC computing, or cloud computing. Its state space, action space, and reward function are defined as follows:

[0138] State Space:

[0139] At time slot t, move device u n When a computing task arrives, the MEC server agent e m Status The definition is as shown in formula (25), where S E Represents the state space of the MEC server agent. The agent makes computation offloading decisions based on the tasks offloaded by the mobile device and the queue status of itself and nearby MEC servers. Because the MEC server agent is deployed on the base station's MEC server, and the operator's base stations trust each other, the agent can easily obtain the queue status of itself and nearby MEC servers.

[0140]

[0141] Where: d n，m (t) represents the mobile device u at the beginning of the tth time slot n Offload to MEC server m Computational tasks; Indicates that the MEC server e m The task queue; Indicates that the task is coming m Task queues of nearby MEC servers, where Indicates e m A collection of nearby MEC servers, Indicates e m The number of nearby MEC servers; S n，m (t), c n，m (t) represents the computing task d n，m (t) The data size and the number of CPU cycles required to process 1 bit of the data; Respectively represent e m The number of tasks in the task queue and the total number of CPU cycles the queue needs to calculate; and Respectively represent e m The set of task numbers in the task queues of adjacent MEC servers and the total number of CPU cycles that the queues need to calculate.

[0142] Action Space:

[0143] The goal of the MEC server agent is to further decide the execution location of the computing tasks offloaded from the mobile device, that is, to select the optimal action based on the current state to minimize the task execution delay. In the MEC system of this application, the MEC server agent is responsible for deciding the computing mode of the computing task in each time slot. The tth time slot agent is responsible for the task d n，m (t) action It is represented by , which is defined as shown in formula (26).

[0144]

[0145] Where: A E Represents the action space of the MEC server agent. Indicates task d n，m (t) executed locally on the MEC server, Indicates that task d n，m (t) Offload to the central cloud for execution, Indicates that task d n，m (t) Offload to a nearby MEC server for execution.

[0146] Reward function:

[0147] Similar to the mobile device agent, combined with formulas (17), (18) and (19), at the tth time slot, the agent is in state Next, take action The reward function value of Calculated by formula (27), where R E Represents the reward function space of the MEC server agent:

[0148]

[0149] Where: time n，m (t) represents task d n (t)Transmit to e m The subsequent execution time is calculated using formula (28):

[0150]

[0151] The multi-scene training process of the multi-agent collaborative computation offloading strategy includes the following steps. First, the system model and the parameters of each network are initialized. Second, multi-scene iterative training begins. Each mobile device agent in a single scene obtains the current initial state and starts single-scene training from the initial state. Among them, if a mobile device agent offloads the computation task to the MEC server to which it is connected, the agent of the MEC server begins single-scene training. Finally, through continuous iterative training and learning, the current network of each mobile device agent and the MEC server agent converges to the optimal action-value function, that is, the optimal offloading strategy is learned, which minimizes the computational delay of all tasks in the entire MEC system.

[0152] Example:

[0153] like Figure 4 The following is a baseline comparison experiment chart, which vividly illustrates the comparison between our method and the baseline methods. The baseline methods include local computation only on mobile devices, computation offloading only, and random offloading. Compared with the baseline methods, our method reduces the average task computation time by 46.6%, 2.7%, and 21.4%, respectively. In a single scene, our method reduces the average task computation time by 49.1%, 9.8%, and 23.6%, respectively, compared with the baseline methods.

[0154] like Figure 5 The figure below is a comparative experimental diagram with other algorithms, which vividly demonstrates the comparison between this method and other algorithms. The end-layer multi-agent approach uses DDQN to design an agent on a mobile device, whose decision-making task is to calculate the positions of the cloud, edge, and end layers. The edge-layer single-agent approach uses D3QN to design an agent at the edge layer, whose decision-making task is to calculate the positions of the cloud, edge, and end layers. Compared with the end-layer multi-agent approach and the edge-layer single-agent approach, this method reduces the average task computation time by 6.6% and 40.8%, respectively. In a single scene, this method reduces the average task computation time by 12.4% and 48.5%, respectively, compared with the end-layer multi-agent approach and the edge-layer single-agent approach.

[0155] This paper discloses a multi-agent collaborative computation offloading method in a 5G mobile edge computing environment. This method effectively overcomes the shortcomings of existing technologies and combines the powerful computing power of cloud computing with the proximity advantage of MEC to provide users with a high-quality experience and minimize application execution latency. Starting from the actual environment of 5G mobile edge computing, this paper formulates user needs and uses two types of deep reinforcement learning agents: a mobile device agent based on DDQN and an MEC server agent based on D3QN. This method constructs a multi-agent collaborative method to improve the quality of experience (QoE) for users executing computationally intensive and time-sensitive applications.

Claims

1. A multi-agent collaborative computing offloading method in a 5G mobile edge computing environment, characterized in that: The following steps are involved: S1: Initialize the parameters of devices, 5G networks, and optical networks in the MEC environment. Based on the initialization parameters, collect information about the task queues of mobile devices, information about tasks to be calculated, and network information between devices and base stations. S2, calculate based on the collected information. If the task information to be calculated is local calculation, add the task to the local task queue of the mobile device; If it is computation offloading, the task is transferred to the MEC server of the connected base station; If the task is transferred to the MEC server for calculation, the calculation task will be transferred to the MEC server of the nearby base station and the task will be added to the task queue of the server; If it is cloud computing, the computing task is transferred to the cloud server for computing, and the multi-agent collaborative computing offloading is realized. The time slot model is established according to the initialization parameters, and communication and computing offloading are carried out based on the time slot model. Different sub-channels are allocated to different devices using the orthogonal frequency division multiplexing method. At the tth time slot, the mobile device u n With MEC server m The channel gain h of the wireless channel between n,m (t) is calculated by formula (3): Where: β n,m (t) represents the t-th time slot u n With e m The channel fading factor between them obeys the Rayleigh distribution, and its probability distribution function is f B (β), A d Indicates e m The radar gain of the connected base station, c0 represents the speed of light in vacuum, f c Indicates e m The carrier frequency of the connected base station, d n,m (t) represents the t-th time slot u n With e m The distance between e Represents the path loss index, which is calculated based on the collected information and allocates tasks with the minimum calculation delay as the optimization goal; At the tth time slot, let the mobile device u n Local task queue As shown in formula (6): Where: Indicates the number of tasks in the local queue at the beginning of the current time slot, Indicates the total number of CPU cycles that the local queue needs to calculate at the beginning of the current time slot. At t = 0, That is, there are no tasks in the queue; At the t+1th time slot, if the computation task d n (t) Total CPU cycles in the local queue during local execution Calculated by formula (7): If d n (t) Total CPU cycles in the local queue that are not executed locally Calculated by formula (8): The number of tasks in the local queue It is related to the number of tasks completed within time T, but the number of cycles for each task is different; Set mobile device u n Completed in time slot t tasks, if the task is executed locally, then the number of tasks in the local queue at the beginning of the next time slot is Calculated by formula (9): If the task is not executed locally, then Calculated by formula (10): If the mobile device u n The computation task d at the tth time slot n (t) If the task is executed locally on the device, it will be added to the local queue and its execution delay will be Calculated by formula (11): If the mobile device u n The computation task d at the tth time slot n (t) The MEC server e of a base station near the base station it is connected to m' Execute, the task will be added to the MEC server e m' In the task queue, the total number of CPU cycles and tasks in the next time slot of the queue, and the task execution delay Calculated by formula (18):

2. The multi-agent collaborative computing offloading method in a 5G mobile edge computing environment according to claim 1 is characterized in that: According to Shannon's formula and formula (3), at the tth time slot, the mobile device u n With MEC server m The transmission rate between them is calculated by formula (4): Where: B represents u n With e m The channel bandwidth between them, p0 represents the mobile device u n The transmission power of N0 is the Gaussian white noise power.

3. The multi-agent collaborative computing offloading method in a 5G mobile edge computing environment according to claim 1 is characterized in that: At the tth time slot, set the MEC server e m Task Queue As shown in formula (12): Where: Indicates the number of tasks in the MEC server queue at the beginning of the current time slot, Indicates the total number of CPU cycles that the MEC server queue needs to calculate at the beginning of the current time slot; At t = 0, 4. The multi-agent collaborative computing offloading method in a 5G mobile edge computing environment according to claim 1, characterized in that: The problem of minimizing the long-term computational delay of all tasks in the entire MEC system under limited resource constraints is expressed as formula (21): Where A(t) represents the set of locations where all mobile devices execute computational tasks at the tth time slot, D represents the coverage radius of the base station, constraint C1 ensures that the computational task can only be performed once, and constraint C2 ensures that all user mobile devices are active within the coverage area of the base station. After a mobile device exceeds the maximum service distance of a base station, another base station will continue to provide service.

Images