Task copying and unloading method based on game theory in mobile edge computing

Abstract

The invention provides a task copying and unloading method based on a game theory in mobile edge computing. The task copying and unloading method comprises the following steps: constructing a mobile edge system architecture; a task copying and unloading strategy of each IoT device is used as a to-be-solved strategy, overall service performance maximization of the mobile edge system in a period T is used as a target function, constraint conditions are combined, and an optimization model of task copying and unloading is constructed and converted into a game theory model; and solving the game theory model by adopting a Nash equilibrium solving method based on multi-agent reinforcement learning to obtain an optimal task copying and unloading strategy of each IoT device. According to the task copying and unloading method based on the game theory in the mobile edge computing, a steady game theory model and a multi-agent reinforcement learning algorithm are effectively combined, and the system service performance and the system reliability can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile edge computing, and specifically relates to a task duplication and offloading method based on game theory in mobile edge computing. Background Art

[0002] The popularity of the fifth generation of mobile communication technology (5G) has ushered in a rapid development era for the Internet of Things (IoT), in which a large number of computationally intensive and latency-sensitive applications have become ubiquitous. Applications such as real-time video analysis, augmented reality (AR), and autonomous driving require a large amount of computing resources and ultra-low latency to operate effectively. This surge in demand has prompted the emergence of a new paradigm, Mobile Edge Computing (MEC), which brings computing power to the edge of the network to bring computing resources closer to the data source. MEC provides many advantages, including reduced latency, improved bandwidth utilization, and enhanced user experience by offloading tasks from centralized cloud computing centers to edge servers (MEC servers) closer to end devices.

[0003] In the context of MEC, task offloading strategies have attracted much attention because they are used to optimize the allocation of computing tasks between terminal devices and MEC servers. These strategies are crucial to maximizing the efficiency and performance of edge computing environments. By intelligently deciding which tasks need to be offloaded and where to offload them, the global task execution time can be significantly reduced, the energy consumption of terminal devices can be reduced, and the overall system throughput can be improved.

[0004] However, the successful implementation of task offloading technology requires not only efficient decision-making algorithms and communication protocols, but also fault tolerance to ensure the reliability of task execution in the event of network failures, communication delays, or computing node failures. In the mobile edge computing environment, the fault tolerance of task offloading is particularly important because terminal devices and edge nodes often encounter unstable network connections, device failures, limited computing resources, and unpredictable working environments. In addition, many application scenarios, such as autonomous driving, telemedicine, and smart manufacturing, require high availability and continuity of the system. Therefore, designing task offloading technology with fault tolerance is crucial to ensure the high availability of edge computing systems.

[0005] Task offloading technology in MEC systems has attracted great attention from academia and industry in the past few years. From the perspective of task fault tolerance, task offloading strategies can be divided into two categories: those with and without fault tolerance.

[0006] 1. Task offloading strategy without fault tolerance

[0007] As the name implies, the task offloading strategy without fault tolerance does not take the fault tolerance of the system as an indicator to evaluate the performance of the offloading strategy when considering the task offloading strategy in the MEC environment. Instead, the overall execution delay or average execution delay of all tasks is used as the main optimization target. This type of method can be divided into the following two categories:

[0008] (1.1) Decentralized offloading strategy

[0009] This category is also called ubiquitous edge computing, which refers to edge computing that relies only on edge devices with perception, storage, and communication capabilities to achieve task offloading without centralized management. The typical technology is to convert the problem into a stochastic game theory model based on the complete observation of the system state and derive the Nash equilibrium between edge devices. On this basis, the Nash equilibrium point is solved through general adversarial imitation learning or deep reinforcement learning, thereby deriving the optimal task offloading strategy.

[0010] (1.2) Centralized offloading strategy

[0011] This type of edge computing is also called edge computing based on a centralized controller. Its basic idea is to rely on a central controller that has global network information, combined with network status information, edge device computing resources and other information to decide how to offload tasks. Typical technologies include the use of Deep Deterministic Policy Gradient (DDPG) learning algorithms to implement multi-priority task scheduling strategies for the interdependence characteristics between multiple tasks; another technology is mainly aimed at computing offloading and service caching in the three-layer mobile cloud edge computing architecture (edge ​​devices, MEC servers and cloud computing centers). In this structure, mobile users subscribe to the cloud service center to obtain computing offloading services and pay related fees on a monthly or annual basis. In addition, the cloud service center can purchase some computing and communication resources from MEC servers with limited cache capacity and computing resources to help mobile users offload computing.

[0012] However, these studies are usually limited to the assumption of a reliable edge network environment and ignore potential failure scenarios when implementing task offloading decisions, such as sudden failures of the equipment performing the task, power outages, etc.

[0013] 2. Fault-tolerant task offloading strategy

[0014] At present, a number of task offloading strategies that take fault tolerance into consideration have emerged in academia and industry, which can be summarized into two categories: system-level task fault tolerance solutions and task-level task fault tolerance solutions.

[0015] (2.1) System-level task fault-tolerant offloading solution

[0016] The characteristic of the system-level task fault-tolerant offloading scheme is that it needs to support the offloading decision based on global information. Typical technologies are as follows: First, based on the auction mechanism, the interaction between MEC participants and the probability of successful user offloading are modeled to ensure the provision of fault-tolerant edge services under unreliable conditions; second, based on the modeling of the dynamic mobile edge environment, user costs, fault penalties and diverse quality of service (QoS) requirements, the task offloading problem is transformed into an online decision-making problem in a random process, and a fault recovery strategy is implemented in the decision-making process to deal with different types of faults; third, the Dueling Deep Q network algorithm is used to determine the user offloading behavior, and the adaptive checkpoint mechanism is used to improve the task reliability, thereby realizing a semi-online fault-tolerant offloading method to optimize the service offloading efficiency and system reliability.

[0017] However, the above technologies are all fault-tolerant task offloading solutions implemented at the global level of the system, and all require the acquisition of corresponding global information, which will incur additional costs because it is difficult to easily obtain global network, device or task information when actually used.

[0018] (2.2) Task-level fault-tolerant offloading scheme

[0019] The second type of offloading scheme with fault tolerance is the task-level offloading scheme, which is to ensure the fault tolerance of the MEC system from the perspective of the edge device computing task itself without the global information perspective. Typical technologies are: First, the fault tolerance method based on checkpoints and replication. The basic idea is to use intelligent checkpoints for IoT application tasks executed in the edge network and to improve the reliability of the system by replicating the checkpoint files on nearby alternative MEC servers, thereby improving the overall availability of the system. Second, it is based on task offloading and service replication technology. This type of technology usually models the offloading strategy problem as an integer linear programming problem, with the goal of minimizing the response time of all users and modeling it with the time and time difference of users being satisfied at the same time as the constraints. On this basis, linear relaxation programming is used to solve the model based on Lagrangian analysis, so as to obtain the optimal offloading strategy to improve the availability of the system; third, the problem is modeled as a nonlinear integer programming problem based on the interruption probability of a single replication task. Under the constraints of the hardware resources of the user terminal and the MEC server, the replication decision of the task is jointly optimized to minimize the probability of system interruption. However, these technologies have the following two main shortcomings: first, they are implemented through heuristic solutions; second, they lack comprehensive theoretical analysis support. These shortcomings make it difficult for these solutions to adapt to the dynamically changing characteristics of edge computing scenarios.

[0020] From the above analysis, it can be seen that the existing edge computing task replication and offloading strategies have the following shortcomings:

[0021] 1) Task offloading strategies without fault tolerance do not consider unexpected situations such as sudden failure of the device executing the task or task transmission delay due to network reasons when making task offloading decisions, resulting in task offloading or execution failure. Such task offloading strategies are not suitable for business scenarios that need to provide high availability to users;

[0022] 2) The existing system-level fault-tolerant task offloading scheme assumes the existence of a centralized controller that can obtain global information, such as network status information, device status information, and task status information. However, this assumption is too ideal. It is difficult to obtain global information in actual application. From the perspective of technical implementation, obtaining global information will introduce additional costs, such as communication delays.

[0023] 3) The commonality of the technical principles of the existing task-level fault-tolerant offloading solutions is achieved through heuristic solutions. At the same time, these solutions lack rigorous theoretical analysis support. When the edge network environment changes, it is difficult to adapt to the changes, so it is difficult to achieve good results. In the MEC environment, changes in the edge network status are a common phenomenon. Summary of the invention

[0024] In view of the defects of the prior art, the present invention provides a task replication and offloading method based on game theory in mobile edge computing, which can effectively solve the above problems.

[0025] The technical solution adopted by the present invention is as follows:

[0026] The present invention provides a task replication and offloading method based on game theory in mobile edge computing, comprising the following steps:

[0027] Step S1, constructing a mobile edge system architecture; the mobile edge system architecture includes N IoT devices D1, D2, ..., D N The formed set D={D1,D2,…,D N}, M MEC servers E1, E2, …, E M The formed set ε={E1,E2,…,E M}, and a cloud computing center r; wherein, and Respectively represent the set {1,2,…,N} and the set {1,2,…,M};

[0028] Step S2, within the period T, the task replication and offloading strategy of each IoT device is to be solved, wherein the task replication and offloading strategy of each IoT device includes the original task replication strategy x, the original task offloading strategy y and the replication task offloading strategy z; taking the maximization of the overall service performance of the mobile edge system in the period T as the objective function, combined with the constraint conditions, an optimization model of task replication and offloading is constructed;

[0029] Step S3, converting the optimization model of task replication and offloading into a game theory model;

[0030] Step S4, using a Nash equilibrium solution method based on multi-agent reinforcement learning to solve the game theory model and obtain the optimal task replication and offloading strategy for each IoT device.

[0031] Preferably, step S2 specifically comprises:

[0032] Step S2.1, in the mobile edge environment, define the parameters and meanings related to task replication and offloading:

[0033] Step S2.1.1, assuming IoT device D n Generate the original task at time t The collection formed Where Q(n,t) represents the IoT device D n The number of all original tasks generated at time t;

[0034] Step S2.1.2, for IoT device D n The original task generated at time t q∈{1,2,…,Q(n,t)}, its input data volume and computational complexity are and The number of copies at time t is expressed as Therefore, the replication tasks; G represents the maximum number of replications, and the number of replications for a task Therefore, if the original task If no replication is performed,

[0035] Step S2.1.3, in the mobile edge environment, IoT device D n Each original task or each replicated task generated at time t can only be executed on one node, that is, either on the local IoT device D n Execute or be offloaded to a MEC server E in the MEC server set ε m Execute, or be offloaded to a cloud computing center for execution;

[0036] Therefore, define the original task The decision outcome variable And the decision result variable of copy task x i∈{r∪n∪ε}, i represents the node that executes the task, which means:

[0037] If the original task Local IoT device D n Execute, then i = n, so,

[0038] If the replication task x is on the local IoT device D n Execute, then i = n, so,

[0039] If the original task Executed in cloud computing center r, then i = r, so,

[0040] If the replication task x is executed in the cloud computing center r, then i = r, so,

[0041] If the original task Offloaded to MEC server E m Execute, then i = m, so,

[0042]

[0043] If replication task x is offloaded to MEC server E m Execute, then i = m, so,

[0044]

[0045] Since each original task Or each replication task x can only be executed on one node i, so, and

[0046] Step S2.2, evaluate and obtain IoT device D n Transmission delay at time t

[0047] Step S2.3, evaluate and obtain IoT device D n Execution delay at time t

[0048] Step S2.4, evaluate and obtain IoT device D n Reliability Υ at time t n (t);

[0049] Step S2.5, using formula (1), for IoT device D n Transmission delay at time t Execution delay and reliability n (t) Perform weighted summation to obtain IoT device D n The service cost u at time t n (t):

[0050]

[0051] Among them: α1 and α2 are weight parameters of weighted summation;

[0052] Step S2.6, using formula (2), obtain the overall service cost U of the mobile edge system at time t g (t):

[0053]

[0054] Step S2.7, with the goal of minimizing the overall service cost of the mobile edge system in period T, that is, maximizing the overall service performance, and combining constraints, an optimization model for task replication and offloading is constructed.

[0055] Preferably, step S2.2 is specifically:

[0056] Step S2.2.1, IoT device D n The original task generated at time t It is from IoT device D n To MEC server E m The transmission delay between Use formula (3) to determine:

[0057]

[0058] Where: r n,m (t) represents IoT device D n To MEC server E m The wireless link transmission rate between is determined by formula (4):

[0059]

[0060] Where: W represents the link bandwidth; N0 is the noise power spectral density; h n,m (t) and p n,m (t) represent IoT devices D n To MEC server E m The wireless channel gain and transmission power between them;

[0061] Step S2.2.2, IoT device D n The original task generated at time t It is from IoT device D n Transmission delay to cloud computing center r Use formula (5) to determine:

[0062]

[0063] in: Indicates IoT device D n The transmission rate between the cloud computing center r;

[0064] Step S2.2.3, using formula (6), obtain IoT device D n The total transmission delay of all original tasks at time t

[0065]

[0066] IoT Device D n The total transmission delay of all original tasks at time t IoT device D n Transmission delay at time t

[0067] Preferably, step S2.3 is specifically:

[0068] Step S2.3.1, if IoT device D n The original task generated at time t Local IoT device D n Execution, then the execution delay of its local execution Using formula (7), we get:

[0069]

[0070] Where: f n (t) represents IoT device D n The computing power at time t;

[0071] Step S2.3.2, if IoT device D n At time t, the original task Offload to MEC server E m Execute, then it is on the MEC server E m Execution delay of execution Using formula (8), we get:

[0072]

[0073] Where: f n,q,m (t) represents the MEC server E m Assigned to original task computing power;

[0074] Step S2.3.3, if IoT device D n At time t, the original task Offload to cloud computing center r for execution, and its execution delay is calculated as 0;

[0075] Step S2.3.4, using formula (9), obtain IoT device D n The total execution delay of all original tasks at time t

[0076]

[0077] IoT Device D n The total execution delay of all original tasks at time t IoT device D n Execution delay at time t

[0078] Preferably, step S2.4 is specifically as follows:

[0079] Step S2.4.1, MEC server E m The failure rate is expressed as δ m (t) indicates that the failure rate follows a Poisson distribution, so at time t, the MEC server E m Occurrence m The probability of failure MEC Server E m No failure occurred, that is, r m =0 is

[0080] Therefore, the MEC server E m The probability of failure at time t

[0081] Therefore, for IoT device D n The original task at time t The cumulative failure probability at time t is expressed as

[0082] Step S2.4.2, IoT device D n The original task at time t Reliability Determine by formula (10):

[0083]

[0084] Step S2.4.3, IoT device D n The reliability of all original tasks at time t is γ n (t), is determined by formula (11):

[0085]

[0086] IoT Device D n The reliability of all original tasks at time t is γ n (t), which is the IoT device D n Reliability γ at time t n (t).

[0087] Preferably, in step S2.7, the optimal model for task replication and offloading is:

[0088] Objective function P1:

[0089] The constraints C1-C7 are:

[0090] C1:

[0091] C2:

[0092] C3:

[0093] C4:

[0094] C5: 0 ≤ p n,m (t)≤P n (t)

[0095] C6: q∈{1,2,…,Q(n,t)}

[0096] C7: i∈{r∪n∪ε}

[0097] in:

[0098] C1 means The value range of

[0099] C2 represents and All are binary variables;

[0100] C3 means for any original task It can only be executed on one node, namely: the local IoT device D n Execution, cloud computing center r execution, or MEC server Em Execution, i.e.: original task Cannot be split to multiple nodes for execution;

[0101] C4, F m (t) represents the MEC server E m The overall computing power of n,q,m (t) represents the MEC server E m Assigned to the original task Therefore, C4 means: on MEC server E m In the example, the allocated computing resources cannot exceed its total computing resources;

[0102] In C5, p n,m (t) represents IoT device D n To MEC server E m The transmission power; P n (t) represents IoT device D n Maximum power of IoT device D n To MEC server E m The transmission power of IoT device D n Maximum power;

[0103] C6 represents the number of IoT devices, the number of MEC servers, and the value range of the number of original tasks;

[0104] C7 indicates the value range of the number of task replications and the value range of the node type on which the task is executed.

[0105] Preferably, step S3 specifically comprises:

[0106] N IoT devices D1, D2, ..., D N As a player in game theory; for player D n , that is, IoT device D n , whose task replication and offloading strategy is represented by π n ={x n ,y n ,z n}; where x n Indicates IoT device D n The original task replication strategy, y n represents the offloading strategy of the original task, z n Indicates the offloading strategy of the replication task;

[0107] When IoT devices n Adopting task replication and offloading strategyπ n When the service cost u n (πn ) as the revenue in game theory, thus establishing the game theory model

[0108] Preferably, step S4 is specifically:

[0109] Step S4.1, transform the game theory model into a multi-agent partially observable Markov decision process POMDP;

[0110] Specifically, each player D in the game theory model n Considered as agent D n ;POMDP consists of five parts, denoted as

[0111] Represents the state space, representing the global state of the entire system;

[0112] Represents the observation space, represents the system state observable by the agent, and is part of the global state; at time t, the observation space of the agent Among them, the intelligent agent D n The observed state L n (t) represents the agent D n The location information, f n (t) represents the agent D n The computing power of Represents the agent D n The number of original tasks generated and the size of the original task input data, Σ ε Represents the global information of the MEC server, Σ r Represents the global information of the cloud computing center;

[0113] Represents the action space, which contains all possible actions of all agents, that is, a n (t) = {x n (t),y n (t),z n (t)}; where a n (t) represents agent D n The action taken at time t; x n (t),y n (t),z n (t), respectively represent the agent D n The actions taken at time t include the original task replication strategy, the original task offloading strategy, and the replication task offloading strategy;

[0114] p represents the observed state transition probability function, which represents the probability of receiving a specific observation given the current state and the action taken;

[0115] Represents the reward space, expressed as R n (t) = u n (t); where R n (t) represents agent D n The reward obtained at time t; u n (t) represents agent D n The cost of service at time t;

[0116] Based on the five parts of POMDP The problem of multi-task copying and offloading is expressed as:

[0117] Objective:

[0118]

[0119] Where: t represents time t, T represents the time interval; γ(t) represents the attenuation coefficient, 0≤γ(t)≤1, which represents the impact of the reward value at a certain moment in the future on the current reward value; R n (t) represents the agent D n The reward received at time t; In the objective function Objective, the goal of the task duplication and offloading strategy is to find appropriate x, y, and z to maximize the cumulative decay reward value of all agents in the time interval T; Represents the optimal task replication and offloading strategy found. They are IoT devices D n The optimal original task replication strategy, the original task offloading strategy and the replication task offloading strategy;

[0120] Step S4.2, using the task replication and offloading Nash equilibrium solution method based on MATD3, solve the objective function Objective:

[0121] Specifically, each agent D n With a MATD3 controller, each agent D n By performing its own action a n (t) Interact with the environment and obtain the corresponding reward R n (t) and observe the next state; the algorithm includes the experience replay pool D, the critic network and the actor network;

[0122] Experience Replay Pool Used to store existing experience samples, expressed as in, Represents the agent D n The current observation state, a n (t) indicates the current action taken, Indicates the current reward. Indicates the next observed state; the experience samples stored in the experience replay pool D are used to train the critic network and the actor network;

[0123] The actor network actor network includes the evaluation network Evaluation Network and the target network TargetNetwork; the evaluation network Evaluation Network is used to receive the agent D n Current observation status Through the strategic network Generate action a n (t); This action is continuously improved by the gradient update optimizer; the target network Target Network adopts the strategy Used for the next observation status Add noise to generate target action And synchronize parameters from the evaluation network through soft update to ensure that the policy network Stability and consistency; through the experience replay pool Based on the experience samples, the actor network continuously learns and optimizes strategies to achieve efficient decision-making for edge computing tasks.

[0124] The critic network includes two independent evaluation networks and two independent target networks. The parameters of the two evaluation networks are and Optimization is performed using the gradient descent algorithm, with the goal of minimizing the loss function This loss function measures the difference between the Q value output by each evaluation network and the target Q value; each evaluation network receives the agent D n Current observation status and the action taken n (t), calculate the target Q value respectively, that is: and Update its parameters via a gradient descent optimizer and The two target networks generate target Q values ​​respectively: and in, and They represent the parameters of the two target networks respectively, and the parameters of the network are evaluated synchronously through soft updating.

[0125] The game theory-based task replication and offloading method in mobile edge computing provided by the present invention has the following advantages:

[0126] The present invention proposes a task replication and offloading method based on game theory in mobile edge computing, which effectively combines a robust game theory model with a multi-agent reinforcement learning algorithm, and can effectively improve system service performance and system reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0127] Figure 1 A flowchart of a task replication and offloading method based on game theory in mobile edge computing provided by the present invention;

[0128] Figure 2 The mobile edge system architecture diagram provided by the present invention;

[0129] Figure 3 A schematic diagram of the Nash equilibrium solution method for task replication and offloading based on the MATD3 algorithm provided by the present invention. DETAILED DESCRIPTION

[0130] In order to make the technical problems, technical solutions and beneficial effects solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0131] The present invention proposes a task replication and offloading method based on game theory in mobile edge computing, which effectively combines a robust game theory model with a multi-agent reinforcement learning (MARL) algorithm, and can effectively improve system service performance and system reliability.

[0132] The present invention proposes a game-theoretical task replication and offloading method (GTRO) in MEC. Specifically, GTRO first uses a multi-party game theory model to model the proposed task replication and offloading problem, analyzes the Nash equilibrium (NE) point in the game model, and studies the existence and uniqueness conditions of NE to ensure that the game model has a unique solution. Secondly, GTRO uses a multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm to overcome the problem of dynamic changes in various resources in the edge network, and uses multiple agents to learn and adjust strategies through interaction in a simulated environment.

[0133] The method proposed in the present invention improves the robustness of the MEC system through a flexible, efficient and optimized task replication and offloading strategy. It can ensure the fault tolerance of computing tasks in the face of intermittent network failures, hardware failures and lack of computing resources, thereby improving the overall availability of the system.

[0134] See also Figure 1 The present invention provides a task replication and offloading method based on game theory in mobile edge computing, comprising the following steps:

[0135] Step S1, constructing a mobile edge system architecture; the mobile edge system architecture includes N IoT devices D1, D2, ..., D N The collection formed M MEC servers E1, E2, …, E M The formed set ε={E1,E2,…,E M}, and a cloud computing center r; wherein, and Respectively represent the set {1,2,…,N} and the set {1,2,…,M};

[0136] See also Figure 2 , is a specific example diagram of a mobile edge system architecture. Figure 2 In the figure, IoT devices are represented by ED1, ED2, ED3, etc. MEC servers are represented by "MEC" and cloud computing centers are represented by "cloud". The communication between IoT devices and MEC servers is based on the local area network (LAN), and the communication between IoT devices / MEC servers and cloud computing centers is based on the wide area network.

[0137] Step S2, within the period T, the task replication and offloading strategy of each IoT device is to be solved, wherein the task replication and offloading strategy of each IoT device includes the original task replication strategy x, the original task offloading strategy y and the replication task offloading strategy z; taking the maximization of the overall service performance of the mobile edge system in the period T as the objective function, combined with the constraint conditions, an optimization model of task replication and offloading is constructed;

[0138] Step S3, converting the optimization model of task replication and offloading into a game theory model;

[0139] Step S4, using a Nash equilibrium solution method based on multi-agent reinforcement learning to solve the game theory model and obtain the optimal task replication and offloading strategy for each IoT device.

[0140] The following is a detailed introduction to steps S2 to S4:

[0141] Step S2, within the period T, the task replication and offloading strategy of each IoT device is to be solved, wherein the task replication and offloading strategy of each IoT device includes the original task replication strategy x, the original task offloading strategy y and the replication task offloading strategy z; taking the maximization of the overall service performance of the mobile edge system in the period T as the objective function, combined with the constraints, an optimization model of task replication and offloading is constructed.

[0142] Step S2 is specifically implemented through steps S2.1 to S2.7:

[0143] Step S2.1, in the mobile edge environment, define the parameters and meanings related to task replication and offloading:

[0144] Step S2.1.1, assuming IoT device D n Generate the original task at time t The collection formed Where Q(n,t) represents the IoT device D n The number of all original tasks generated at time t;

[0145] Step S2.1.2, for IoT device D n The original task generated at time t q∈{1,2,…,Q(n,t)}, the input data volume and computational complexity are and The number of copies at time t is expressed as Therefore, the replication tasks; G represents the maximum number of replications, and the number of replications for a task Therefore, if the original task If no replication is performed,

[0146] Step S2.1.3, in the mobile edge environment, IoT device D n Each original task or each replicated task generated at time t can only be executed on one node, that is, either on the local IoT device D n Execute or be offloaded to a MEC server E in the MEC server set ε m Execute, or be offloaded to a cloud computing center for execution;

[0147] Therefore, define the original task The decision outcome variable And the decision result variable of copy task x i∈{r∪n∪ε}, i represents the node that executes the task, which means:

[0148] If the original task Local IoT device D n Execute, then i = n, so,

[0149] If the replication task x is on the local IoT device D n Execute, then i = n, so,

[0150] If the original task Executed in cloud computing center r, then i = r, so,

[0151] If the replication task x is executed in the cloud computing center r, then i = r, so,

[0152] If the original task Offloaded to MEC server E m Execute, then i = m, so,

[0153] If replication task x is offloaded to MEC server E m Execute, then i = m, so,

[0154] Since each original task Or each replication task x can only be executed on one node i, so, and

[0155] Step S2.2, evaluate and obtain IoT device D n Transmission delay at time t

[0156] exist Figure 2 In the system architecture, for IoT devices D n , there are two communication links, namely IoT device D n and MEC server E m The communication link between IoT devices D n The communication link between IoT device D and cloud computing center r. Therefore, the transmission delay of these two communication links is calculated separately, and then summed up to get n Transmission delay at time t

[0157] The specific steps are as follows:

[0158] Step S2.2.1, IoT device D n and MEC server E m The transmission delay caused by the communication link between them:

[0159] IoT Device D n The original task generated at time t It is from IoT device D n To MEC server E m The transmission delay between Use formula (3) to determine:

[0160]

[0161] Where: r n,m (t) represents IoT device D n To MEC server E m The wireless link transmission rate between is determined by formula (4):

[0162]

[0163] Where: W represents the link bandwidth; N0 is the noise power spectral density; h n,m (t) and p n,m (t) represent IoT devices D n To MEC server E m The wireless channel gain and transmission power between them;

[0164] Step S2.2.2, IoT device D n The transmission delay caused by the communication link between the cloud computing center r is:

[0165] Specifically, when MEC server E m Unable to execute IoT device D nWhen the task is generated, IoT device D n The task needs to be offloaded to the cloud computing center r. Since the cloud computing center r has sufficient computing power to ensure the successful execution of the task, the replication of the task is not considered, so there is no need to consider the transmission delay of the replicated task.

[0166] Therefore, IoT device D n The original task generated at time t It is from IoT device D n Transmission delay to cloud computing center r Use formula (5) to determine:

[0167]

[0168] in: Indicates IoT device D n The transmission rate between the cloud computing center r;

[0169] Step S2.2.3, using formula (6), obtain IoT device D n The total transmission delay of all original tasks at time t

[0170]

[0171] IoT Device D n The total transmission delay of all original tasks at time t IoT device D n Transmission delay at time t

[0172] Step S2.3, evaluate and obtain IoT device D n Execution delay at time t

[0173] exist Figure 2 In the system architecture, IoT device D n There are three task execution modes for tasks generated at time t: local IoT device D n Execute and offload to MEC server E m Execute, or offload to the cloud computing center r for execution, thus generating the following three task execution delays; then sum up the three task execution delays to get the IoT device D n Execution delay at time t

[0174] The specific steps are as follows:

[0175] Step S2.3.1, local IoT device D n Execution delay:

[0176] If IoT device D n The original task generated at time t Local IoT device D n Execution, then the execution delay of its local execution Using formula (7), we get:

[0177]

[0178] Where: f n (t) represents IoT device D n The computing power at time t;

[0179] Step S2.3.2, MEC server E m Execution delay:

[0180] If IoT device D n At time t, the original task Offload to MEC server E m Execute, then it is on the MEC server E m Execution delay of execution Using formula (8), we get:

[0181]

[0182] Where: f n,q,m (t) represents the MEC server E m Assigned to original task computing power;

[0183] Step S2.3.3, cloud computing center r execution delay:

[0184] If IoT device D n At time t, the original task Offload to cloud computing center r for execution. Since cloud computing center r has sufficiently powerful and abundant computing resources, the delay caused by task execution in cloud computing center r can be ignored. Therefore, its execution delay is calculated as 0.

[0185] Step S2.3.4, using formula (9), obtain IoT device D n The total execution delay of all original tasks at time t

[0186]

[0187] IoT Device D n The total execution delay of all original tasks at time t IoT device D nExecution delay at time t

[0188] Step S2.4, evaluate and obtain IoT device D n Reliability γ at time t n (t);

[0189] Step S2.4 is specifically as follows:

[0190] Step S2.4.1, MEC server E m The failure rate is expressed as δ m (t) indicates that the failure rate follows a Poisson distribution, so at time t, the MEC server E m Occurrence m The probability of failure is: MEC Server E m No failure occurred, that is, r m =0 is

[0192] Therefore, the MEC server E m The probability of failure at time t

[0193] Therefore, for IoT device D n The original task at time t The cumulative failure probability at time t is expressed as

[0194] Step S2.4.2, IoT device D n The original task at time t Reliability Determine by formula (10):

[0195]

[0196] Step S2.4.3, IoT device D n The reliability of all original tasks at time t is Υ n (t), is determined by formula (11):

[0197]

[0198] IoT Device D n The reliability of all original tasks at time t is Υ n (t), which is the IoT device D n Reliability γ at time t n (t).

[0199] Step S2.5, using formula (1), for IoT device Dn Transmission delay at time t Execution delay and reliability n (t) Perform weighted summation to obtain IoT device D n The service cost u at time t n (t):

[0200]

[0201] Among them: α1 and α2 are weight parameters of weighted summation;

[0202] Step S2.6, using formula (2), obtain the overall service cost U of the mobile edge system at time t g (t):

[0203]

[0204] Step S2.7, with the goal of minimizing the overall service cost of the mobile edge system in period T, that is, maximizing the overall service performance, and combining constraints, an optimization model for task replication and offloading is constructed.

[0205] Specifically, the present invention proposes a task replication and offloading method based on game theory to minimize the service cost of each IoT device, that is, to maximize the service performance, thereby maximizing the service performance of the overall system. n Service cost u n (t) is defined as the weighted sum of reliability, execution delay and transmission delay, and the service cost U of the overall system is g (t) is defined as the sum of the service costs of all IoT devices in the edge computing system, that is: Among them: α1 and α2 are weight parameters of weighted summation.

[0206] The present invention models the task replication and offloading problem as an optimization problem of finding the maximum service performance within time T for the original task replication strategy x, the original task offloading strategy y, and the replication task offloading strategy z (i.e., the service cost minimum problem of the following P1 model); the specific objective function and constraints are as follows:

[0207] Objective function P1:

[0208] The constraints C1-C7 are:

[0209] C1:

[0210] C2:

[0211] C3:

[0212] C4:

[0213] C5: 0 ≤ p n,m (t)≤P n (t)

[0214] C6: q∈{1,2,…,Q(n,t)}

[0215] C7: i∈{r∪n∪ε}

[0216] in:

[0217] C1 means The value range of

[0218] C2 represents and All are binary variables;

[0219] C3 means for any original task It can only be executed on one node, namely: the local IoT device D n Execution, cloud computing center r execution, or MEC server E m Execution, i.e.: original task Cannot be split to multiple nodes for execution;

[0220] C4, F m (t) represents the MEC server E m The overall computing power of n,q,m (t) represents the MEC server E m Assigned to the original task Therefore, C4 means: on MEC server E m In the example, the allocated computing resources cannot exceed its total computing resources;

[0221] In C5, p n,m (t) represents IoT device D n To MEC server E m The transmission power; P n (t) represents IoT device D n Maximum power of IoT device D n To MEC server E m The transmission power of IoT device D n Maximum power;

[0222] C6 represents the number of IoT devices, the number of MEC servers, and the value range of the number of original tasks;

[0223] C7 represents the value range of the number of task copies and the value range of the node type for task execution, namely: execution by local IoT device, execution by MEC server, or execution by cloud computing center.

[0224] Step S3, converting the optimization model of task replication and offloading into a game theory model;

[0225] The essence of solving the P1 model problem is to find the appropriate x, y and z so that P1 can achieve the minimum value. The present invention proposes a solution based on multi-party game theory. In this solution, N IoT devices D1, D2, ..., D N As players in game theory; for player D n , that is, IoT device D n , its task duplication and offloading strategy is the strategy in game theory, expressed as π n ={x n ,y n ,z n}; where x n Indicates IoT device D n The original task replication strategy, y n represents the offloading strategy of the original task, z n Represents the offloading strategy of the replication task; the player (i.e., D n ) can choose their own strategies according to their respective goals; the present invention uses the maximization of the service performance of IoT devices as an indicator to represent the payoffs in game theory.

[0226] Therefore, when IoT device D n Adopting task replication and offloading strategyπ n When the service cost u n (π n ) as the revenue in game theory, thus establishing the game theory model For this game theory model, finding the Nash equilibrium of the model means finding π n ={x n ,y n ,z n}.

[0227] Step S4, using a Nash equilibrium solution method based on multi-agent reinforcement learning to solve the game theory model and obtain the optimal task replication and offloading strategy for each IoT device.

[0228] In the mobile edge computing environment, factors such as the dynamic variability of IoT device computing resources and the dynamic variability of the edge network environment lead to obvious deficiencies in the solution methods of traditional game theory models in capturing these dynamics. This paper proposes a method based on Multi-Agent Reinforcement Learning (MARL) to solve the Nash equilibrium of the game theory model.

[0229] Specifically, MARL first transforms the game theory model into a multi-agent partially observable Markov decision process (POMDP). In POMDP, the game theory players (i.e., the aforementioned IoT devices) are regarded as agents. The agents make decisions on task duplication and offloading based on their own observations of the environment. The environment here refers to the network delay state, task execution state, and device computing resources. The state is determined by the actions of the agent and other agents, because the actions of the agents will change the current state of the system and make the system enter the next state. Secondly, the present invention proposes to interact with the environment based on the Multi-Agent Twin Delayed Deep Deterministic policy gradient algorithm (MATD3) to obtain the optimal offloading and duplication strategy. The specific method is described as follows:

[0230] Step S4 is specifically implemented through steps S4.1 to S4.2:

[0231] Step S4.1, transform the game theory model into a multi-agent partially observable Markov decision process POMDP;

[0232] Specifically, each player D in the game theory model n Considered as agent D n ;POMDP consists of five parts, denoted as

[0233] It represents the state space, representing the global state of the entire system; it should be noted that in the scenario targeted by the present invention, the agent cannot obtain the global state of the system.

[0234] Represents the observation space, represents the system state observable by the agent, and is part of the global state; at time t, the observation space of the agent Among them, the intelligent agent D n The observed state L n (t) represents the agent D r The location information, f n (t) represents the agent D n The computing power of Represents the agent D n The number of original tasks generated and the size of the original task input data, Σ ε Represents the global information of the MEC server, Σ r Represents the global information of the cloud computing center;

[0235] Represents the action space, which contains all possible actions of all agents, that is, Among them, a n (t) represents agent D n The action taken at time t; x n (t),y n (t),z n (t), respectively represent the agent D n The actions taken at time t include the original task replication strategy, the original task offloading strategy, and the replication task offloading strategy;

[0236] represents the observation state transition probability function, which represents the probability of receiving a specific observation given the current state and the action taken;

[0237] Represents the reward space, expressed as R n (t) = u n (t); where R n (t) represents agent D n The reward obtained at time t; u n (t) represents agent D n The cost of service at time t;

[0238] Based on the five parts of POMDP The problem of multi-task copying and offloading is expressed as:

[0239] Objective:

[0240]

[0241] Where: t represents time t, T represents the time interval; γ(t) represents the attenuation coefficient, 0≤γ(t)≤1, which represents the impact of the reward value at a certain moment in the future on the current reward value; R n(t) represents the agent D n The reward received at time t; In the objective function Objective, the goal of the task duplication and offloading strategy is to find appropriate x, y, and z to maximize the cumulative decay reward value of all agents in the time interval T; Represents the optimal task replication and offloading strategy found. They are IoT devices D n The optimal original task replication strategy, the original task offloading strategy and the replication task offloading strategy;

[0242] Step S4.2, using the task replication and offloading Nash equilibrium solution method based on MATD3, solve the objective function Objective:

[0243] The present invention proposes a task replication and offloading Nash equilibrium solution method based on MATD3. The overall architecture of the method is shown in FIG. Figure 3 shown.

[0244] Specifically, each IoT device D n That is, an intelligent agent D n , Agent D n With a MATD3 controller, each agent D n By performing its own action a n (t) Interact with the environment and obtain the corresponding reward R n (t) and observe the next state. Figure 3 Shows the algorithm design details of one of the IoT devices, which includes the experience replay pool critic network and actor network;

[0245] Experience Replay Pool Used to store existing experience samples, expressed as in, Represents the agent D n The current observation state, a n (t) indicates the current action taken, Indicates the current reward. Indicates the next observed state; experience replay pool The stored experience samples are used to train the critic network and the actor network to improve the learning efficiency and performance of the algorithm.

[0246] The actor network actor network includes the evaluation network Evaluation Network and the target network TargetNetwork; the evaluation network Evaluation Network is used to receive the agent D n Current observation status Through the strategic network Generate action a n (t); This action is continuously improved through the gradient update optimizer to improve the effect of the strategy; the target network Target Network adopts the strategy Used for the next observation status Add noise to generate target action And synchronize parameters from the evaluation network through soft update to ensure that the policy network stability and consistency; through the experience samples in the experience replay pool D, the actor network actornetwork continuously learns and optimizes strategies to achieve efficient decision-making for edge computing tasks;

[0247] The critic network includes two independent evaluation networks and two independent target networks for more stable and efficient strategy learning. The parameters of the two evaluation networks are and Optimization is performed using the gradient descent algorithm, with the goal of minimizing the loss function This loss function measures the difference between the Q value output by each evaluation network and the target Q value; each evaluation network receives the agent D n Current observation status and the action taken n (t), calculate the target Q value respectively, that is: and Update its parameters via a gradient descent optimizer and The two target networks generate target Q values ​​respectively: and in, and They represent the parameters of the two target networks respectively, and the parameters of the networks are evaluated synchronously through soft updates to ensure stability.

[0248] The task replication and offloading algorithm based on MATD3 uses MARL to optimize the task allocation between multiple devices and obtain the Nash equilibrium in game theory. The algorithm first initializes the actor network and the critic network and uses the experience replay pool Stores experience tuples. During training, each player chooses actions that increase exploration noise to encourage diverse experiences. The environment executes these actions and returns the next observation, reward, and completion signal. Players update their critic networks by minimizing the mean squared error between predicted Q-values ​​and target Q-values, using the minimum of the target critic network in the calculation to reduce overestimation bias. Using deterministic policy gradients, the actor network is updated less frequently to ensure stable learning. The target network is updated less frequently to slowly track the learned network, thereby improving the stability of training. This approach allows for efficient task replication and offloading decisions in dynamic edge computing environments, thereby improving the overall performance of the system.

[0249] The present invention proposes a task replication and offloading method in a mobile edge computing environment, the key points of which are as follows:

[0250] 1) Aiming at the problem of task duplication and offloading in mobile edge computing scenarios, the present invention aims to improve the overall service performance of the system, comprehensively considers factors such as transmission delay, computing delay and system reliability, and proposes a new problem modeling method, which transforms the problem into a model for optimal solution under certain constraints (i.e., the above-mentioned P1 model).

[0251] 2) The present invention proposes a new model solving method based on game theory, which converts the above P1 model into a game theory model. The players in the game theory are IoT devices, the strategies are task replication and offloading strategies, and the benefits are the overall service costs of the edge computing system. Therefore, obtaining the Nash equilibrium of the game theory model means obtaining the optimal task replication and offloading strategy.

[0252] 3) This paper proposes a new method for solving Nash equilibrium in game theory. This method solves Nash equilibrium based on multi-agent reinforcement learning technology to solve the dynamic uncertainty problem in game theory. Specifically, this method transforms the game theory model into a partially observable Markov decision process, and designs a method based on a multi-agent double-delay deep deterministic policy gradient algorithm to obtain Nash equilibrium.

[0253] The present invention proposes a task replication and offloading method in a mobile edge computing environment, which has the following effects:

[0254] 1) This method can support the edge computing system to provide strong fault tolerance when facing network failure, equipment failure, insufficient computing resources or other unpredictable failures, so that the edge computing tasks can be successfully executed.

[0255] 2) The task replication and offloading method proposed in the present invention can comprehensively consider the three factors of task computing delay, transmission delay and system fault tolerance to form the overall service performance of the system, and based on the proposed modeling method and model solving method, the overall service performance of the edge computing system can be optimized.

[0256] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A task replication and offloading method based on game theory in mobile edge computing, characterized in that: The following steps are involved: Step S1, constructing a mobile edge system architecture; the mobile edge system architecture includes N IoT devices D1, D2, ..., D N The collection formed M MEC servers E1, E2, …, E M The formed set ε={E1,E2,…,E M }, and a cloud computing center r; wherein, and Respectively represent the set {1,2,…,N} and the set {1,2,…,M}; Step S2, within the period T, the task replication and offloading strategy of each IoT device is to be solved, wherein the task replication and offloading strategy of each IoT device includes the original task replication strategy x, the original task offloading strategy y and the replication task offloading strategy z; taking the maximization of the overall service performance of the mobile edge system in the period T as the objective function, combined with the constraint conditions, an optimization model of task replication and offloading is constructed; Step S3, converting the optimization model of task replication and offloading into a game theory model; Step S4, using a Nash equilibrium solution method based on multi-agent reinforcement learning to solve the game theory model and obtain the optimal task replication and offloading strategy for each IoT device.

2. According to a game theory-based task replication and offloading method in mobile edge computing according to claim 1, it is characterized in that: Step S2 is specifically as follows: Step S2.1, in the mobile edge environment, define the parameters and meanings related to task replication and offloading: Step S2.1.1, assuming IoT device D n Generate the original task at time t The collection formed Where Q(n,t) represents the IoT device D n The number of all original tasks generated at time t; Step S2.1.2, for IoT device D n The original task generated at time t The input data volume and computational complexity are and The number of copies at time t is expressed as Therefore, the replication tasks; G represents the maximum number of replications, and the number of replications for a task Therefore, if the original task If no replication is performed, Step S2.1.3, in the mobile edge environment, IoT device D n Each original task or each replicated task generated at time t can only be executed on one node, that is, either on the local IoT device D n Execute or be offloaded to a MEC server E in the MEC server set ε m Execute, or be offloaded to a cloud computing center for execution; Therefore, define the original task The decision outcome variable And the decision result variable of copy task x i represents the node that executes the task, which means: If the original task Local IoT device D n Execute, then i = n, so, If the replication task x is on the local IoT device D n Execute, then i = n, so, If the original task Executed in cloud computing center r, then i = r, so, If the replication task x is executed in the cloud computing center r, then i = r, so, If the original task Offloaded to MEC server E m Execute, then i = m, so, If replication task x is offloaded to MEC server E m Execute, then i = m, so, Since each original task Or each replication task x can only be executed on one node i, so, and Step S2.2, evaluate and obtain IoT device D n Transmission delay at time t Step S2.3, evaluate and obtain IoT device D n Execution delay at time t Step S2.4, evaluate and obtain IoT device D n Reliability γ at time t n (t); Step S2.5, using formula (1), for IoT device D n Transmission delay at time t Execution delay and reliability n (t) Perform weighted summation to obtain IoT device D n The service cost u at time t n (t): Among them: α1 and α2 are weight parameters of weighted summation; Step S2.6, using formula (2), obtain the overall service cost U of the mobile edge system at time t g (t): Step S2.7, with the goal of minimizing the overall service cost of the mobile edge system in period T, that is, maximizing the overall service performance, and combining constraints, an optimization model for task replication and offloading is constructed.

3. According to a game theory-based task replication and offloading method in mobile edge computing according to claim 2, it is characterized in that: Step S2.2 is specifically as follows: Step S2.2.1, IoT device D n The original task generated at time t It is from IoT device D n To MEC server E m The transmission delay between Use formula (3) to determine: Where: r n,m (t) represents IoT device D n To MEC server E m The wireless link transmission rate between is determined by formula (4): Where: W represents the link bandwidth; N0 is the noise power spectral density; h n,m (t) and p n,m (t) represent IoT devices D n To MEC server E m The wireless channel gain and transmission power between them; Step S2.2.2, IoT device D n The original task generated at time t It is from IoT device D n Transmission delay to cloud computing center r Use formula (5) to determine: in: Indicates IoT device D n The transmission rate between the cloud computing center r; Step S2.2.3, using formula (6), obtain IoT device D n The total transmission delay of all original tasks at time t IoT Device D n The total transmission delay of all original tasks at time t IoT device D n Transmission delay at time t 4. According to a game theory-based task replication and offloading method in mobile edge computing according to claim 2, it is characterized in that: Step S2.3 is specifically as follows: Step S2.3.1, if IoT device D n The original task generated at time t Local IoT device D n Execution, then the execution delay of its local execution Using formula (7), we get: Where: f n (t) represents IoT device D n The computing power at time t; Step S2.3.2, if IoT device D n At time t, the original task Offload to MEC server E m Execute, then it is on the MEC server E m Execution latency Using formula (8), we get: Where: f n,q,m (t) represents the MEC server E m Assigned to original task computing power; Step S2.3.3, if IoT device D n At time t, the original task Offload to cloud computing center r for execution, and its execution delay is calculated as 0; Step S2.3.4, using formula (9), obtain IoT device D n The total execution delay of all original tasks at time t IoT Device D n The total execution delay of all original tasks at time t IoT device D n Execution delay at time t 5. The method for task replication and offloading based on game theory in mobile edge computing according to claim 2 is characterized in that: Step S2.4 is specifically as follows: Step S2.4.1, MEC server E m The failure rate is expressed as δ m (t) indicates that the failure rate follows a Poisson distribution, so at time t, the MEC server E m Occurrence m The probability of failure MEC Server E m No failure occurred, that is, r m =0 is Therefore, the MEC server E m The probability of failure at time t Therefore, for IoT device D n The original task at time t The cumulative failure probability at time t is expressed as Step S2.4.2, IoT device D n The original task at time t Reliability Determine by formula (10): Step S2.4.3, IoT device D n The reliability of all original tasks at time t is Υ n (t), is determined by formula (11): IoT Device D n The reliability of all original tasks at time t is γ n (t), which is the IoT device D n Reliability Υ at time t n (t).

6. The method for task replication and offloading based on game theory in mobile edge computing according to claim 2 is characterized in that: In step S2.7, the optimal model for task replication and offloading is: Objective function P1: The constraints C1-C7 are: C1： C2： C3： C4： C5:0≤p n,m (t)≤P n (t) C6: q∈{1,2,…,Q(n,t)} C7: i∈{r∪n∪ε} in: C1 means The value range of C2 represents and All are binary variables; C3 means for any original task It can only be executed on one node, namely: the local IoT device D n Execution, cloud computing center r execution, or MEC server E m Execution, i.e.: original task Cannot be split to multiple nodes for execution; C4, F m (t) represents the MEC server e m The overall computing power of n,q,m (t) represents the MEC server E m Assigned to the original task Therefore, C4 means: on MEC server E m In the example, the allocated computing resources cannot exceed its total computing resources; In C5, p n,m (t) represents IoT device D n To MEC server E m The transmission power; P n (t) represents IoT device D n Maximum power of IoT device D n To MEC server E m The transmission power of IoT device D n Maximum power; C6 represents the number of IoT devices, the number of MEC servers, and the value range of the number of original tasks; C7 indicates the value range of the number of task replications and the value range of the node type on which the task is executed.

7. The method for task replication and offloading based on game theory in mobile edge computing according to claim 2 is characterized in that: Step S3 is specifically as follows: N IoT devices D1, D2, ..., D N As a player in game theory; for player D n , that is, IoT device D n , whose task replication and offloading strategy is represented by π n ={x n ,y n ,z n }; where x n Indicates IoT device D n The original task replication strategy, y n represents the offloading strategy of the original task, z n Indicates the offloading strategy of the replication task; When IoT devices n Adopting task replication and offloading strategyπ n When the service cost u n (π n ) as the revenue in game theory, thus establishing the game theory model 8. The method for task replication and offloading based on game theory in mobile edge computing according to claim 7 is characterized in that: Step S4 is specifically as follows: Step S4.1, transform the game theory model into a multi-agent partially observable Markov decision process POMDP; Specifically, each player D in the game theory model n Considered as agent D n ;POMDP consists of five parts, denoted as Represents the state space, representing the global state of the entire system; Represents the observation space, represents the system state observable by the agent, and is part of the global state; at time t, the observation space of the agent Among them, the intelligent agent D n The observed state L n (t) represents the agent D n The location information, f n (t) represents the agent D n The computing power of Represents the agent D n The number of original tasks generated and the size of the original task input data, Σ ε Represents the global information of the MEC server, Σ r Represents the global information of the cloud computing center; Represents the action space, which contains all possible actions of all agents, that is, a n (t) = {x n (t),y n (t),z n (t)}; where a n (t) represents agent D n The action taken at time t; x n (t),y n (t),z n (t), respectively represent the agent D n The actions taken at time t include the original task replication strategy, the original task offloading strategy, and the replication task offloading strategy; represents the observation state transition probability function, which represents the probability of receiving a specific observation given the current state and the action taken; Represents the reward space, expressed as R n (t) = u n (t); where R n (t) represents agent D n The reward obtained at time t; u n (t) represents agent D n The service cost at time t; Based on the five parts of POMDP The problem of multi-task copying and offloading is expressed as: Objective: Where: t represents time t, T represents the time interval; γ(t) represents the attenuation coefficient, 0≤γ(t)≤1, which represents the impact of the reward value at a certain moment in the future on the current reward value; R n (t) represents the agent D n The reward received at time t; In the objective function Objective, the goal of the task duplication and offloading strategy is to find appropriate x, y, and z to maximize the cumulative decay reward value of all agents in the time interval T; Represents the optimal task replication and offloading strategy found. They are IoT devices D n The optimal original task replication strategy, the original task offloading strategy and the replication task offloading strategy; Step S4.2, using the task replication and offloading Nash equilibrium solution method based on MATD3, solve the objective function Objective: Specifically, each agent D n With a MATD3 controller, each agent D n By performing its own action a n (t) Interact with the environment and obtain the corresponding reward R n (t) and observe the next state; the algorithm includes an experience replay pool critic network and actor network; Experience Replay Pool Used to store existing experience samples, expressed as in, Represents the agent D n The current observation state, a n (t) indicates the current action taken, Indicates the current reward. Indicates the next observed state; experience replay pool The stored experience samples are used to train the critic network and the actor network; The actor network actor network includes the evaluation network Evaluation Network and the target network TargetNetwork; the evaluation network Evaluation Network is used to receive the agent D n Current observation status Through the strategic network Generate action a n (t); This action is continuously improved by the gradient update optimizer; the target network Target Network adopts the strategy Used for the next observation status Add noise to generate target action And synchronize parameters from the evaluation network through soft update to ensure that the policy network Stability and consistency; through the experience replay pool Based on the experience samples, the actor network continuously learns and optimizes strategies to achieve efficient decision-making for edge computing tasks. The critic network includes two independent evaluation networks and two independent target networks. The parameters of the two evaluation networks are and Optimization is performed using the gradient descent algorithm, with the goal of minimizing the loss function This loss function measures the difference between the Q value output by each evaluation network and the target Q value; each evaluation network receives the agent D n Current observation status and the action taken n (t), calculate the target Q value respectively, that is: and Update its parameters via a gradient descent optimizer and The two target networks generate target Q values ​​respectively: and in, and They represent the parameters of the two target networks respectively, and the parameters of the network are evaluated synchronously through soft updating.

Images