Agent policy learning method with privacy protection in mobile edge computing

Abstract

An agent policy learning method with privacy protection in mobile edge computing, which belongs to the technical field of mobile communications. The method comprises: establishing an edge collaborative computing unloading model which comprises a service caching model, a task unloading model and a system cost model; on the basis of the edge collaborative computing unloading model, using the minimization of the processing cost of a task as a target to establish an optimization problem of joint task unloading, service caching, computing power resource allocation and transmission power control; abstracting an optimal problem into a partially observable Markov decision process; and on the basis of the Markov decision process, using a multi-agent deep reinforcement learning algorithm based on a federated learning method to autonomously learn a task unloading, service caching, computing power resource allocation and transmission power control policy. In the present invention, an edge collaborative computing unloading model is established, such that an agent network protects privacy security of data and sensitive information of users by means of training based on federated learning.

Description

A privacy-preserving agent policy learning method in mobile edge computing

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 12, 2023, with application number 202310686533.9 and invention name “A method for learning intelligent agent strategies with privacy protection in mobile edge computing”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention belongs to the field of mobile communication technology, and in particular relates to a privacy-preserving intelligent agent strategy learning method in mobile edge computing. Background Art

[0003] Mobile Edge Computing (MEC) pushes the storage and processing of user tasks to the edge of the mobile communication network, allowing users to enjoy highly reliable and low-latency services at the edge of the network. It provides strong technical support for the efficient processing of user services, and can well meet users' requirements for efficient and fast service quality. However, with the mutual integration and vigorous development of communication technology and Internet of Things technology, the edge network structure is becoming increasingly dense and heterogeneous. At the same time, in the edge network environment, the wide-area differentiation of services, the highly dynamic network environment, and the decentralized deployment of computing network resources restrict the efficiency of network service caching and computing network resource allocation. A key issue in MEC is to design and implement efficient task offloading, service caching, and resource allocation solutions for the decentralized edge network structure and the diverse business needs of users.

[0004] Deep reinforcement learning (DRL) has the advantages of both deep learning and reinforcement learning. It can both perceive and make decisions. The relevant theoretical technologies have also been applied by researchers to the field of wireless communications. Some of the main achievements include: (1) Task scheduling in vehicular edge computing based on deep reinforcement learning (Zhan W, Luo C, Wang J, et al. Deep-reinforcement-learning-based offloading scheduling for vehicular edge computing. IEEE Internet of Things Journal, 2020, 7(6): 5449-5465.): The computational offloading scheduling problem in the vehicular edge computing scenario is studied. With the goal of minimizing the long-term processing cost of the task, a stochastic optimization problem of task offloading and scheduling is established. A deep reinforcement learning algorithm based on a progressive optimization strategy is proposed. The method of combining parameter sharing network and convolutional neural network is used to approximate the policy function and value function. (2) Dynamic offloading for multiuser muti-CAP MEC networks: a deep reinforcement learning approach (Li C, Xia J, Liu F, et al. Dynamic offloading for multiuser muti-CAP MEC networks: a deep reinforcement learning approach [J]. IEEE Transactions on Vehicular Technology, 2021, 70 (3): 2922-2927.): Aiming at the dynamic offloading problem in multiuser MEC networks, this paper first abstracts it into a Markov decision process, and then designs a DQN-based offloading strategy, which allows users to dynamically adjust the proportion of task offloading to ensure system performance. However, traditional DRL algorithms require terminal devices to transmit their private data to edge servers or remote cloud centers for processing or training. However, the data may be stolen or tampered with by third parties during transmission and processing, which puts user data and sensitive information at risk of leakage.

[0005] Therefore, as people pay more and more attention to privacy and security issues, protecting users' privacy and security while designing more flexible and efficient distributed task offloading, resource allocation and service caching strategies has become an urgent problem to be solved in current research.

[0006] To summarize, the existing technical problem is that traditional DRL algorithms require terminal devices to transmit their private data to edge servers or remote cloud centers for processing or training. However, the data may be stolen or tampered with by third parties during transmission and processing, which puts users' data and sensitive information at risk of leakage.

[0007] Summary of the Invention

[0008] To solve the above technical problems, the present invention proposes a privacy-preserving agent strategy learning method in mobile edge computing, comprising:

[0009] S1: For decentralized MEC scenarios, establish an edge collaborative computing offloading model, including a service cache model, a task offloading model, and a system cost model;

[0010] S2: Based on the edge collaboration computing offloading model, with the goal of minimizing the processing cost of tasks, under the constraints of multi-dimensional resources, an optimization problem of joint task offloading, service caching, computing resource allocation and transmission power control is established;

[0011] The multi-dimensional resources include: computing resources and storage resources;

[0012] S3: Abstract the optimization problem of joint task offloading, service caching, computing resource allocation, and transmission power control into a partially observable Markov decision process;

[0013] S4: Based on the Markov decision process, a multi-agent deep reinforcement learning algorithm based on the federated learning method is used to autonomously learn task offloading, service caching, computing resource allocation and transmission power control strategies.

[0014] Beneficial effects of the present invention:

[0015] The present invention studies the service caching and resource allocation in the decentralized MEC scenario while taking into account the user privacy protection issue. First, an edge-cloud collaborative computing offloading model is established. Then, with the goal of minimizing the processing cost of the task, task offloading, service caching, computing resource allocation and transmission power control are jointly optimized, and abstracted into a partially observable Markov decision process. Then, a multi-agent deep reinforcement learning algorithm based on the federated learning method is proposed to autonomously learn task offloading, service caching, computing resource allocation and transmission power control decisions. There are data security and privacy leakage issues in the centralized training stage of the multi-agent model, and a distributed model training method based on federated learning is adopted. During the training process, the current actor network and the critic network update the network parameters by maximizing the policy gradient and the loss function respectively, and the target actor network and the critic network are updated by soft update. The strategy learning is performed by the multi-agent model after training, which fully protects the privacy of user data and sensitive information. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG1 is a schematic diagram of the MEC system model of the present invention;

[0017] FIG2 is a block diagram of a MADDPG-based service caching and resource allocation algorithm of the present invention;

[0018] FIG3 is a diagram of the model training based on federated learning of the present invention;

[0019] FIG4 is a diagram showing how the average cost of the present invention changes with the number of training iterations;

[0020] FIG5 is a diagram showing how the average cache hit rate of the present invention changes with the number of training iterations. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] A privacy-preserving agent policy learning method in mobile edge computing, comprising:

[0023] S1: For decentralized MEC scenarios, establish an edge collaborative computing offloading model, including a service cache model, a task offloading model, and a system cost model;

[0024] S2: Based on the edge collaboration computing offloading model, with the goal of minimizing the processing cost of tasks, under the constraints of multi-dimensional resources, an optimization problem of joint task offloading, service caching, computing resource allocation and transmission power control is established;

[0025] The multi-dimensional resources include: computing resources and storage resources;

[0026] S3: Abstract the optimization problem of joint task offloading, service caching, computing resource allocation, and transmission power control into a partially observable Markov decision process;

[0027] S4: Based on the Markov decision process, a multi-agent deep reinforcement learning algorithm based on the federated learning method is used to autonomously learn task offloading, service caching, computing resource allocation and transmission power control strategies.

[0028] 1. System Model

[0029] As shown in Figure 1, the present invention considers a typical MEC system, which contains M base stations (BSs). The base station set is defined as Each base station is equipped with a MEC server with certain computing and storage capabilities. There are N m End User (EU), define the user set as The system operates in discrete time slots, defined as At time slot t, user EUi m The resulting task is defined as in, Indicates the data size of the task, in bits; Represents user i m The maximum tolerable delay of task processing; Indicates the number of CPU cycles required to process a unit bit task; represents the service type required to process the task, then the task set of all users under base station BS m can be defined as

[0030] 1. Service caching model

[0031] In this invention, it is assumed that there are K types of services in the network, and the service type set is defined as Let a k,m (t)∈{0,1} means that at time slot t, BS m The cache indicator function of service k in a k,m (t) = 1 means that BSm caches service k, otherwise BSm will not cache service k. Further, the service caching decision of BSm in time slot t can be expressed as the service caching strategy set am (t) = {a 1,m (t),…,a k,m (t),…,a K,m (t)}. Limited by the storage space of the MEC server, the storage space occupied by the cached service cannot exceed the storage capacity of the MEC server. The size of the storage space of MECm is defined as R m , then: Among them, l k Indicates the size of the storage space occupied by service k.

[0032] 2. Task Offloading Model

[0033] User EUi m The generated tasks can be processed locally or offloaded to the base station or cloud for processing. m The generated tasks can be processed in the following ways: 1) local processing; 2) offloading to the associated base station BSm for processing; 3) forwarding to the nearby base station BSn through the associated base station BSm ( and n≠m) processing; 4) offloading to the cloud for processing. Definition For EUi m The uninstall decision variables. User EUi's tasks are processed locally, otherwise Likewise, User EUi m The task is offloaded to the associated base station BSm for processing, otherwise User EUi m The task is forwarded to the base station BSm through the base station BSn for processing, otherwise User EUi m The task is offloaded to the cloud for processing, otherwise and satisfy Therefore, at time slot t, EUi m The task offloading strategy can be expressed as Then the task offloading decision of all users under BSm can be expressed as

[0034] 1) Local processing

[0035] When tasks are processed locally, Let f im,m Indicates user EUi m The local CPU frequency of the task can be expressed as: Correspondingly, the task processing energy consumption is: where k is the effective capacitance coefficient that depends on the chip architecture.

[0036] 2) Offload to the associated base station for processing

[0037] When the base station BSm caches the service k required to process the user task, if the user EUi m The task can be directly offloaded to the base station BSm for processing, that is, The bandwidth of base station BSm is B m , the total number of uplink channels is H m , then the subchannel bandwidth is According to Shannon's formula, the task upload rate is: in, Indicates user EUi m Transmit power in time slot t; Indicates user EUi m Channel gain with BSm; σ 2 (t) represents the additive white Gaussian noise power at time slot t.

[0038] User EUi m When the task is offloaded to the associated base station BSm for processing, the task processing delay includes transmission delay and execution delay, that is: in, is the total computing resource of base station BSm; is allocated to user EUi by BSm in time slot t m The CPU frequency allocation coefficient satisfies Indicates that BSm is allocated to user EUi m The CPU frequency of , the computing resource allocation strategy of BS m can be expressed as

[0039] Correspondingly, the task processing energy consumption is: Among them, e bs Indicates the energy consumption of the base station in processing a unit bit task.

[0040] 3) Offload to nearby base stations for processing

[0041] If the associated base station BSm does not cache the service k required to process the user task, and the nearby base station BSn caches the service k, then the user EUi m The task can be forwarded by base station BSm to the nearby base station BSn for processing, that is, Then the forwarding rate of BS m is: Among them, P m (t) represents the transmission power of BSm in time slot t; G m,n represents the channel gain between BSm and BSn. Correspondingly, the task processing delay includes transmission delay, forwarding delay and execution delay, that is:

[0042] Correspondingly, the task processing energy consumption is:

[0043] 4) Offload to the cloud for processing

[0044] If the associated base station BSm does not have the service k required to process the user task in the cache, the user EUi m You can also choose to offload tasks to the cloud for processing, i.e. Ignoring the task execution delay and energy consumption, the task processing delay is: Among them, r m,c (t) represents the transmission rate from base station BS m to the cloud. Correspondingly, the energy consumption of task processing is: Among them, P m,c (t) represents the transmission power from base station BSm to the cloud.

[0045] 3. System cost model

[0046] Given task offloading, computing resource allocation and service caching decisions, correspondingly, user EUi m Mission The processing delay is:

[0047] Correspondingly, the task processing energy consumption is:

[0048] Then, process user EU i m Mission The cost is: Among them, α im,m and ε im,m are the weight coefficients of delay and energy consumption, respectively, satisfying: They represent the task processing delay in the local area, the associated base station processing delay, the nearby base station processing delay, and the cloud processing delay. They represent user i’s task being processed locally, offloaded to the associated base station m, forwarded to base station n through base station m, and offloaded to the cloud. They represent the energy consumption corresponding to task processing locally, processing at the associated base station, processing at a nearby base station, and processing in the cloud.

[0049] 2. Problem Description

[0050] Due to the limitations of server resources (such as computing and storage space), task offloading and resource allocation are coupled. Therefore, this paper establishes a joint optimization problem for service caching, computing resource allocation, and transmission power control, with the goal of minimizing the long-term average processing cost of tasks. The joint optimization problem is modeled as follows:

[0051] Where a(t)={a1(t),…,a M (t)} represents the base station service cache strategy, b(t)={b1(t),…,b M (t)} represents the task offloading strategy, β(t)={β1(t),…,β M (t)} represents the base station computing resource allocation strategy, P(t)={P1(t),P2(t),…,P M (t)}、 represents the transmission power control decision, M represents the number of base stations, T represents the time slot, Nm represents the number of end users, Indicates processing user i m Mission The cost, Represents user i m Mission Processing delay, a k,m (t) represents the cache decision service k of base station m in time slot t, l k Indicates the size of the storage space occupied by service k, R m Indicates the MEC scenario m The size of the storage space of each base station server, Indicates that base station m allocates to user i in time slot t m The CPU frequency allocation coefficient, They represent the task of user i being processed locally, offloaded to the associated base station m, forwarded to base station m through base station n, and offloaded to the cloud. K represents the service type and N represents the number of users. Indicates that the task processing delay cannot exceed the maximum tolerance delay, constraint Indicates that the cached service cannot exceed the storage capacity of the BS, constraining Indicates that the total amount of allocated computing resources does not exceed the total computing capacity of the server. and constraints Indicates that the user selects only one way to handle their task.

[0052] 3. Problem Solving Based on Federated Multi-Agent Deep Reinforcement Learning

[0053] This paper designs a distributed service caching and resource allocation algorithm (DSCRA) based on federated multi-agent deep reinforcement learning. The base station acts as an agent and learns task offloading strategies, service caching strategies, computing resource allocation strategies, and transmission power control strategies, while also protecting user privacy. Taking into account the diversity of local models, an attention mechanism is used during parameter aggregation to assign different parameter weights to different local models.

[0054] 1. Problem transformation

[0055] The above cost minimization problem is abstracted into a partially observable Markov decision process, with the base station acting as the intelligent agent, and the tuple is defined Describe the above Markov game process, where Represents the global state space, and the environment of time slot t is the global state is the observation space set of the agent, is the global action space set, is the reward set. At time slot t, the agent m Based on local observations Adopt strategy π m : Select the corresponding action Get corresponding rewards

[0056] 1) State Space

[0057] At time slot t, the environmental state can be defined as s(t) = {d1(t), d2(t), ..., d M (t),P1(t), in, is the local CPU frequency set of all users under base station BSm; is the set of channel gains between all users and BSm under the base station BSm. At time slot t, the environment state observed by agent m is The definition is as follows:

[0058] 2) Action Space

[0059] Agent m observes the environment state o m (t) and the current policy π m , select the corresponding action from the action space, time slot t, the action of agent m The definition is as follows: m (t) = {bm (t),β m (t),a m (t),P m (t)}, where b m (t) represents the task offloading actions of all users under BS m; β m (t) represents the computing resource allocation action of BS m; a m (t) represents the service cache action of BS m; P m (t) represents the transmission power control actions of all users under BS m.

[0060] 3) Reward Function

[0061] The reward function measures the effect of an agent taking an action in a given state. During the training process, the agent takes an action in the t-1 time slot, and the corresponding reward will be returned to the agent in the t time slot. Based on the reward obtained, the agent will update its strategy to obtain the optimal result. Since the reward causes each agent to reach its optimal strategy and directly determines the corresponding task offloading strategy, the base station's computing resource allocation strategy, the service cache strategy, and the transmission power control decision, the reward function should be designed according to the original optimization problem. The reward in this article consists of three parts: the first part is the reward for the task processing cost, and the second part is the reward for the task processing delay meeting the delay constraint, that is: The third part is the reward for caching within the storage capacity limit of the edge server, namely: The optimization goal is to minimize the long-term average processing cost of the task and maximize the long-term reward, so the cumulative reward of agent m should be: Where H(□) is the Heaviside step function, and λ1 and λ2 are weight coefficients.

[0062] 2DSCRA algorithm

[0063] As shown in Figure 2, the MADDPG model is an actor-critic based algorithm. The base station is used as an intelligent agent. The actor network and critic network of each agent have two deep neural networks, including the current network and the target network. During the training phase, the actor network and the critic network update the network parameters through federated learning: the critic current network updates the network parameters by minimizing the loss function, and the actor current network updates the network parameters θ by maximizing the policy gradient based on the centralized Q function calculated by the critic current network and its own observation information. The parameters of the actor target network and the critic target network are updated through soft updates, and the attention mechanism is used for parameter aggregation. Experience Replay Memory Used to save the tuples related to observations and actions in the training phase, D = {o m (t),a m (t),r m (t),o′ m (t+1)}, where o m (t) represents the observed state of agent i in time slot t; a m (t) represents the time slot t where agent m is based on the current observation o m (t) the action taken; r m (t) indicates that agent m takes action a in time slot t m (t) The reward obtained after; o′ m (t+1) represents the state of agent m at time slot t+1.

[0064] In the decentralized execution phase, at time slot t, each agent’s actor network observes the state o according to its local state. m (t) and its own strategy Select an action: in, represents the set of observed states of agent m, represents the action decision set of agent m, θ m Represents the current network parameters of the actor of agent m.

[0065] During the centralized training phase, each critic network can obtain observations o of other agents. m (t) and action a m (t), then the Q function of agent m can be expressed as: Q m (o1(t),o2(t),…,o M (t),a1(t),a2(t),…,a M (t);ω m )

[0066] Among them, Q m () represents the centralized Q function, o1(t), o2(t),…, o M (t) represents the observed state of each agent, a1(t), a2(t),…, a M (t) represents the action taken by each agent, ω m Indicates the current network parameters of the critic.

[0067] The Q function evaluates the actions of the actor network from a global perspective and guides the actor network to choose a better action. During training, the critic network updates the network parameters by minimizing the loss function, which is defined as follows:

[0068] where y m =r m +γQ′ m (o′1(t+1),o′2(t+1),…,o′ M (t+1),a′1(t+1),a′2(t+1),…,a′ M (t+1);ω′ m ), γ is the discount factor.

[0069] At the same time, the actor network updates the network parameters θ based on the centralized Q function calculated by the critic network and its own observation information, and outputs an action a. The actor network parameters θ are updated by maximizing the policy gradient, that is:

[0070] The parameters of the target network are updated by soft updating, namely:

[0071] Among them, represents the gradient, J() represents the policy objective function to be optimized, represents the expected cumulative reward, θ m Represents the current network parameters of the actor m, o m (t) represents the observed state of agent m, a m (t) represents the action decision of agent m, Q m () represents the centralized Q function, o1(t), o2(t),…, o M (t) represents the observed state of each agent, a1(t), a2(t),…, a M (t) represents the action taken by each agent, ω m Indicates the current network parameters of critic, represents the strategy of agent m itself, θ′ m represents the updated target actor network parameters of agent m, ω′ m represents the updated target critic network parameters of agent m, and Represent the update coefficients of the actor network and critic network respectively.

[0072] 3. Model training based on federated learning

[0073] During the centralized training phase of the MADDPG model, there are data security and privacy leakage issues. In order to solve the problem of sensitive information leakage and reduce the computing pressure at the edge while improving network performance, federated learning is used for training. The training model is shown in Figure 3: In the initial stage, the base station obtains the global MADDPG model from the cloud center. The base station then uses local data and the global model to train the local model The trained local model is then uploaded to the cloud center for parameter aggregation. Taking into account the differences in local models across base stations, an attention mechanism is used during parameter aggregation to assign different parameters to different local models. Rewards and other device-specific metrics are used to represent the contribution of the local model to the global model.

[0074] The weighted federated aggregation problem can be formulated as: Among them, ξ m is a weight factor used to measure the contribution of the local to the global model. For agent m, the weight ξ is calculated using the average reward, average loss, and cache hit rate. m .

[0075] Average reward: the average reward of agent m For all local rewards m (t) average.

[0076] Average loss: the average loss of agent m is the average of the loss function output during the training process.

[0077] Cache hit ratio: average cache hit ratio is the cache hit rate h in T time slots m average.

[0078] The above evaluation indicators can be described as The evaluation index vector K m and the local model parameters of agent m are modeled as the key and value of the attention mechanism respectively. The goal of the model is to obtain a more powerful agent to obtain greater rewards, less losses and higher cache hit rates, then:

[0079] The input of the base station includes Q, dimension d k Key K m Sum Calculate the dot product of Q and all keys and divide by Use the softmax function to obtain the weight of the value, that is, the weight factor ξ m for:

[0080] As shown in Figure 4, with the increase in the number of training times, the average task processing cost continues to decrease, gradually stabilizing, and eventually reaching convergence. The cost of the DSCRA algorithm is minimized, indicating that the DSCRA algorithm can make better offloading and resource allocation strategies, thereby achieving a lower task processing cost and realizing on-demand resource allocation, proving the effectiveness of the algorithm. As shown in Figure 5, with the increase in the number of training times, the cache hit rate curve shows an upward trend and eventually reaches convergence. The cache hit rate of DSCRA is the highest, proving the effectiveness of the algorithm.

[0081] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A privacy-preserving agent strategy learning method in mobile edge computing, characterized in that: include: S1: For decentralized MEC scenarios, an edge collaborative computing offloading model is established, including a service cache model, a task offloading model, and a system cost model; S2: Based on the edge collaboration computing offloading model, with the goal of minimizing the processing cost of tasks, under the constraints of multi-dimensional resources, an optimization problem of joint task offloading, service caching, computing resource allocation and transmission power control is established; The multi-dimensional resources include: computing resources and storage resources; S3: The optimization problem of joint task offloading, service caching, computing resource allocation and transmission power control is abstracted into a partially observable Markov decision process; S4: Based on the Markov decision process, a multi-agent deep reinforcement learning algorithm based on the federated learning method is used to autonomously learn task offloading, service caching, computing resource allocation and transmission power control strategies.

2. According to claim 1, a privacy-preserving agent strategy learning method in mobile edge computing is characterized in that: The decentralized MEC scenario includes: In the MEC system scenario, there are M base stations (BS), and the base station set is defined as Each base station is equipped with a MEC server with computing and storage capabilities. There are N m End User (EU), define the user set as The system operates in discrete time slots, defined as At time slot t, user i m The resulting task is defined as in, Indicates the data size of the task, in bits. Indicates user i m Task Processing The maximum tolerable delay, Indicates the number of CPU cycles required to process a unit bit task. represents the service type required to process the task, then the task set of all users under base station m can be defined as 3. According to the method for learning agent strategy with privacy protection in mobile edge computing in claim 1, it is characterized in that: The service cache model includes: Assume that there are K types of services in the network, and define the service type set as Let a k,m (t)∈{0,1} represents the cache indicator function of service k in base station m at time slot t, a k,m (t) = 1 means that base station m caches service k, otherwise base station m will not cache service k; the service caching decision of base station m in time slot t can be expressed as the service caching strategy set a m (t) = {a 1,m (t),…,a k,m (t),…,a K,m (t)}, limited by the storage space of the MEC server, the storage space occupied by the cached service cannot exceed the storage capacity of the MEC server. The size of the storage space of the mth base station server in the MEC scenario is defined as R m , then: Among them, l k Indicates the size of the storage space occupied by service k.

4. The method for learning agent strategies with privacy protection in mobile edge computing according to claim 1, characterized in that: The task offloading model includes: User m The generated tasks can be processed locally or offloaded to the base station or cloud for processing. m The task offloading decision variable is in, Indicates that the task of user i is processed locally, otherwise Indicates user i m The task is offloaded to the associated base station m for processing, otherwise Indicates user i m The task is forwarded to base station m through base station n for processing, otherwise Indicates user i m Offloading tasks to the cloud Otherwise And meet Therefore, at time slot t, EUi m The task offloading strategy can be expressed as Then the task offloading decision of all users under base station m can be expressed as 5. The method for learning agent strategies with privacy protection in mobile edge computing according to claim 1, characterized in that: The system cost model includes: Given task offloading and service caching decisions, user i m Mission The processing delay is: Correspondingly, the task processing energy consumption is: Then process user i m Mission The cost is: in, and are the weight coefficients of delay and energy consumption, respectively, satisfying: They represent the task processing delay in the local area, the associated base station processing delay, the nearby base station processing delay, and the cloud processing delay. They represent that the task of user i is processed locally, offloaded to the associated base station m, forwarded to base station m through base station n, and offloaded to the cloud. Respectively indicate that the tasks are The energy consumption corresponding to ground processing, associated base station processing, nearby base station processing and cloud processing.

6. The method for learning agent strategies with privacy protection in mobile edge computing according to claim 1, characterized in that: The optimization problems of joint task offloading, service caching, computing resource allocation and transmission power control include: Where a(t)={a1(t),…,a M (t)} represents the base station service cache strategy, b(t) = {b1(t),…,b M (t)} represents the task offloading strategy, β(t)={β1(t),…,β M (t)} represents the base station computing resource allocation strategy, P(t) = {P1(t), P2(t),…, P M (t)} represents the transmission power control decision, M represents the number of base stations, T represents the time slot, Nm represents the number of end users, Indicates processing user i m Mission The cost, Indicates user i m Mission Processing delay, a k,m (t) represents the cache decision service k of base station m in time slot t, l k represents the size of the storage space occupied by service k, R m Indicates the size of the storage space of the mth base station server in the MEC scenario. Indicates that base station m allocates to user i in time slot t m The CPU frequency allocation coefficient, They respectively represent that the task of user i is processed locally, offloaded to the associated base station m for processing, forwarded to base station n through base station m for processing, and offloaded to the cloud for processing. K represents the service type, and N represents the number of users.

7. The method for learning agent strategies with privacy protection in mobile edge computing according to claim 1, characterized in that: The above task processing cost minimization problem is abstracted into a partially observable Markov decision process, including: The base station acts as an intelligent agent and defines the tuple Describe the above Markov game process, where Represents the global state space, and the environment of time slot t is the global state is the observation space set of the agent, is the global action space set, is the reward set; at time slot t, agent m observes Adopt strategy π m : Select the corresponding action Get corresponding rewards 8. The method for learning agent strategies with privacy protection in mobile edge computing according to claim 1, characterized in that: A multi-agent deep reinforcement learning algorithm based on a federated learning method is used to autonomously learn task offloading, service caching, computing resource allocation, and transmission power control strategies, including: The base station is used as an intelligent agent. The actor network and critic network of each intelligent agent have two deep neural networks, including the current network and the target network. The intelligent agent also includes an experience replay memory. In the training phase, the actor network and the critic network update network parameters through federated learning: the critic current network updates the network parameters by minimizing the loss function, the actor current network updates the network parameters θ by maximizing the policy gradient based on the centralized Q function calculated by the critic current network and its own observation information, the parameters of the actor target network and the critic target network are updated by soft update, and the attention mechanism is used for parameter aggregation; In the decentralized execution phase, the actor network with updated parameters takes actions based on the state of the agent. After making decisions, the critic network with updated parameters evaluates the actions taken by the actor network and guides the actor network to choose a better action. Experience replay memory Used to store tuples related to observations and actions in the training phase, D = {o m (t),a m (t),r m (t),o′ m (t+1)}, where o m (t) represents the observed state of agent m at time slot t, a m (t) represents the time slot agent m based on the current observation o m (t) the action taken, r m (t) indicates that agent m takes action a in time slot t m (t) The reward obtained after that, o′ m (t+1) represents the state of agent m at time slot t+1; The actor network makes action decisions based on the state of the agent, including: in the decentralized execution phase, the actor network of each agent in time slot t makes action decisions based on the local observation state o m (t) and its own strategy Select an action: in, represents the set of observed states of agent m, represents the action decision set of agent m, θ m Represents the current network parameters of the actor of agent m; The action decisions include: task offloading, service caching, computing resource allocation and transmission power control strategy.

9. The method for learning agent strategies with privacy protection in mobile edge computing according to claim 8, characterized in that: The centralized Q function includes: Q m (o1(t),o2(t),…,o M (t),a1(t),a2(t),…,a M (t);ω m ) Among them, Q m () represents the concentrated Q function, o1(t), o2(t),…,o M (t) represents the observed state of each agent, a1(t), a2(t),…, a M (t) represents the action taken by each agent, ω m Indicates the current network parameters of critic.

10. The method for learning agent strategy with privacy protection in mobile edge computing according to claim 8, characterized in that: The parameter updates of the actor and critic current networks and the actor and critic target networks include: The critic current network updates the network parameters by minimizing the loss function, which includes: The actor's current network parameters θ are updated by maximizing the policy gradient: The parameters of the actor and critic target networks are updated through soft updates, including: Among them, L m (ω m ) represents the loss function, represents the gradient, J() represents the strategy objective function that needs to be optimized, represents the expected cumulative reward, θ m Represents the current network parameters of the actor of agent m, o m (t) represents the observed state of agent m, a m (t) represents the action decision of agent m, Q m () represents the concentrated Q function, o1(t), o2(t),…,o M (t) represents the observed state of each agent, a1(t), a2(t),…, a M (t) represents the action taken by each agent, y m represents the target Q-value function, ω m Indicates the current network parameters of critic, represents the strategy of agent m itself, θ′ m represents the updated actor target network parameters of agent m, ω′ m represents the updated critic target network parameters of agent m, and Represent the update coefficients of the actor network and the critic network respectively.

Images