Unmanned aerial vehicle assisted mobile edge computing task unloading privacy protection method and device

Abstract

The invention discloses an unmanned aerial vehicle assisted mobile edge computing task unloading privacy protection method and device, and the method comprises the steps: obtaining the state information of a multi-user IoT device and a multi-UAV edge server, and building a system model; according to the system model, establishing an optimization control problem by taking the minimization of the total cost as a target; establishing a Markov decision process MDP model of the privacy protection unloading system according to the system model and the optimization problem; and according to the MDP model, adopting a deep reinforcement learning algorithm based on multi-agent near-end strategy optimization to obtain an intelligent optimization unloading strategy. According to the method, the samples are generated through continuous interaction with the environment, the neural network is trained, and experience is continuously accumulated, so that an intelligent optimized unloading strategy of the IoT equipment is obtained, delay and energy consumption of the IoT equipment can be minimized, and meanwhile it is guaranteed that privacy protection and efficient unloading can be achieved in an actual complex multi-UAV dynamic scene.

Description

Technical Field

[0001] This invention relates to the field of drone-assisted mobile edge computing technology, and in particular to a task offloading privacy protection method and apparatus based on deep reinforcement learning. Background Technology

[0002] Mobile Edge Computing (MEC) is an emerging computing paradigm that deploys computing resources at the network edge to support the low-latency, high-bandwidth application requirements. With the rapid development of Internet of Things (IoT) and Unmanned Aerial Vehicle (UAV) technologies, user devices (such as IoT devices) can offload computationally intensive tasks to edge servers (such as UAVs), thereby reducing latency and energy consumption in local computing. However, traditional MEC systems often face privacy risks in multi-access-point environments, such as the inference of user location information and the theft or tampering of task data. These problems stem from data transmission during the offloading process and the potential insecurity of edge servers.

[0003] In existing technologies, single-access-point networks struggle to meet the demands of a large number of users and neglect privacy leaks during the offloading process, such as channel attacks or information injection. Existing offloading strategies largely rely on static optimization, failing to adapt to privacy risks, multi-user contention, and changes in UAV computing power in dynamic environments. Furthermore, privacy protection mechanisms (such as differential privacy) are computationally complex in practical applications, making them difficult to integrate with multi-agent optimization, leading to a decline in overall system performance.

[0004] To address these issues, an intelligent optimization offloading method is needed that can balance latency, energy consumption, and privacy protection in multi-UAV environments, enabling dynamic decision-making. Currently, there is no privacy-preserving offloading method based on deep reinforcement learning that can effectively adapt to dynamically changing network conditions, privacy leakage risks, and resource constraints, allowing IoT devices to safely and efficiently offload tasks, thereby improving the overall service quality of the system. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a privacy-preserving mobile edge computing offloading method and apparatus based on deep reinforcement learning.

[0006] This invention provides a privacy-preserving mobile edge computing offloading method based on deep reinforcement learning, comprising:

[0007] Acquire status information from multiple user IoT devices and multiple UAV edge servers to establish a system model;

[0008] Based on the system model, an optimization control problem is constructed with the goal of minimizing total cost (including latency, energy consumption, and privacy leakage).

[0009] A Markov Decision Process (MDP) model of the privacy-preserving offloading system is established based on the system model and optimization problem;

[0010] Based on the MDP model, an intelligent optimization offloading strategy is obtained by using a deep reinforcement learning algorithm based on Multi-Agent Proximal Policy Optimization (MAPPO).

[0011] This invention provides a privacy-preserving mobile edge computing offloading device based on deep reinforcement learning, comprising:

[0012] The system modeling module is used to acquire status information of multiple user IoT devices and multiple UAV edge servers to build a system model.

[0013] The problem construction module is used to construct optimization control problems based on the system model;

[0014] The MDP building module is used to build an MDP model of the privacy-preserving offloading system based on the system model and optimization problem.

[0015] The computational processing module is used to continuously interact with the environment using a MAPPO-based deep reinforcement learning algorithm to generate samples and train, thereby obtaining an intelligent optimization offloading strategy.

[0016] The present invention provides a privacy-preserving mobile edge computing offloading method and apparatus based on deep reinforcement learning. By comprehensively analyzing the performance of IoT devices and the computing characteristics of UAVs, it constructs an overall system dynamic equation, considers the impact of dynamic time-varying network latency and privacy leakage (including differential privacy and traffic privacy), establishes an optimization control problem, and thus constructs an MDP model. It adopts a MAPPO-based intelligent algorithm, generates samples through continuous interaction with the environment and trains the neural network, and continuously accumulates experience to obtain an intelligent optimization offloading strategy for IoT devices. This not only enables IoT devices to minimize latency and energy consumption, but also ensures that they can achieve privacy protection and efficient offloading in complex multi-UAV dynamic scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a privacy-preserving mobile edge computing offloading method based on deep reinforcement learning, provided according to an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of a scenario for a privacy-preserving mobile edge computing offloading method based on deep reinforcement learning, provided in an embodiment of the present invention.

[0020] Figure 3 This is an architecture diagram of a privacy-preserving mobile edge computing offloading method based on deep reinforcement learning, provided according to an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Figure 1 The flowchart of the privacy-preserving mobile edge computing offloading method based on deep reinforcement learning provided in the embodiments of the present invention is as follows: Figure 1 As shown, it includes:

[0023] S1. Obtain the status information of multiple user IoT devices and multiple UAV edge servers, and establish a system model.

[0024] It should be noted that the system in this embodiment of the invention includes ground-level IoT devices and air-level UAV servers. IoT devices perform computationally intensive tasks (such as earthquake monitoring and fire monitoring) in remote areas. Due to limited local computing power and energy constraints, these tasks can be offloaded to UAVs. Through wireless communication, the IoT devices acquire status information such as task size, location coordinates, computational complexity, and UAV location, thereby modeling the entire system. Furthermore, this embodiment considers multi-user, multi-UAV environments, dynamically adjusting the offloading ratio to adapt to privacy risks and resource competition. To clearly describe the technical solution, this embodiment uses multiple UAVs and multiple IoT users as an example. Moreover, the intelligent optimization offloading method provided in this embodiment is also applicable to more complex models. When the system model changes, the modeling method proposed in this embodiment can be used to construct corresponding optimization equations according to the specific circumstances of the system.

[0025] S2. Based on the system model, construct an optimization control problem with the goal of minimizing total cost (including latency, energy consumption, and privacy leakage).

[0026] It should be noted that the goal of privacy-preserving offloading is to minimize latency and energy consumption when IoT devices offload tasks, while quantifying and reducing the risk of privacy leaks (such as location inference). Therefore, an optimization control problem can be constructed with the objective of minimizing the weighted sum of latency, energy consumption, differential privacy leaks, and traffic privacy leaks. However, on the one hand, due to the high-dimensional state space and complex privacy mechanisms, this optimization control problem is difficult to solve directly analytically. On the other hand, due to the influence of actual network latency, location dynamics, and time-varying characteristics of privacy parameters, traditional optimization decision-making methods relying on fixed-parameter models and static policies often suffer from high robustness and stability risks. Therefore, this invention proposes an intelligent optimization offloading method based on deep reinforcement learning to improve the adaptability and privacy protection capabilities of IoT devices under complex dynamic conditions.

[0027] S3. Establish an MDP model for the privacy-protected uninstallation system based on the system model and optimization problem.

[0028] It should be noted that reinforcement learning (RL) problems are usually described using MDPs. An MDP generally includes a state, an action, a state transition function, and a reward function. The MDP model of the system is established based on the system model and the optimization problem.

[0029] S4. Based on the MDP model, a deep reinforcement learning algorithm based on MAPPO is used to obtain an intelligent optimization unloading strategy.

[0030] It should be noted that traditional single-agent-based reinforcement learning algorithms, such as PPO, often suffer from performance degradation due to poor convergence and stability when dealing with continuous action spaces involving multiple users and multiple UAVs. This invention, based on the MAPPO algorithm in deep reinforcement learning, uses a predefined MDP model to continuously interact with the environment for sample collection and training. It continuously optimizes the neural network parameters with the goal of maximizing the reward function, ultimately generating a real-time intelligent optimization offloading strategy output signal based on the current state input of the IoT device. This achieves both privacy protection and efficient offloading of the IoT device.

[0031] The privacy-preserving mobile edge computing offloading method and apparatus based on deep reinforcement learning provided in this invention analyzes the task characteristics of IoT devices and wireless network characteristics, constructs system dynamic equations, considers the impact of dynamically time-varying network latency, differential privacy, and traffic privacy, establishes an optimization control problem, and thus constructs an MDP model. Using the MAPPO algorithm, samples are generated through continuous interaction with the environment and the neural network is trained to finally obtain an intelligent optimization offloading strategy for IoT devices. This enables IoT devices to minimize latency and energy consumption and reduce the risk of privacy leakage, while ensuring the stable operation of the system under dynamic network conditions.

[0032] Based on the above embodiments, as an optional embodiment, embodiment S1 of the present invention establishes a system model according to the system, including:

[0033] The task size, location coordinates, computational complexity, and UAV location information of IoT devices are collected based on wireless communication.

[0034] Based on task size, location, and computational complexity, dynamic equations for IoT devices and UAVs are established, including local processing and edge offloading models.

[0035] Based on location information, a differential privacy model is established, and the risk of location leakage is quantified using a truncated Laplace distribution;

[0036] By solving the dynamic equations simultaneously, we can obtain the system state equations based on continuous time. After discretization, we can obtain the system model based on discrete time.

[0037] It should be noted that dynamic analysis is performed separately for local processing and edge offloading. System state information, such as task size, location, and computational complexity, is obtained through wireless communication. Dynamic equations can then be established based on the relationships between these information. Using the UAV as an edge server, channel gain, transmission rate, and computational resource allocation are calculated. After obtaining the state information, delay and energy consumption equations can be derived. Privacy protection follows a differential privacy strategy:

[0038]

[0039] in, Indicates the position after the disturbance. Indicates the actual location. and Indicates the cutoff range. This indicates a privacy budget. Furthermore, , Let be the normalization constant, defined as follows:

[0040] By analyzing the impact of privacy metrics on offloading, dynamic equations for latency and energy consumption are obtained. Then, the state equations of all IoT devices and UAVs are combined to obtain the continuous-time system state equation. Finally, this embodiment of the invention discretizes the continuous-time system state equation through sampling to obtain a discrete-time system model.

[0041] This invention, through overall modeling of a multi-user, multi-UAV system, and considering scenarios with dynamically changing network latency and privacy risks, combines optimization control theory and artificial intelligence methods to derive a privacy-preserving offloading strategy based on deep reinforcement learning, thereby achieving stable control of IoT devices. The advantage of this invention lies in applying deep reinforcement learning and privacy-preserving technologies to a mobile edge computing offloading system, considering the impact of complex dynamic environments on the system, and designing a MAPPO-based method to obtain intelligent optimization strategies, thus promoting the environmental adaptability and self-learning capability of the offloading system.

[0042] Figure 2 This is a schematic diagram illustrating a privacy-preserving mobile edge computing offloading method based on deep reinforcement learning according to an embodiment of the present invention. For ease of understanding, the system of this embodiment consists of multiple IoT users and multiple UAVs, where the UAVs act as edge servers, providing computing power. Each device in the system is equipped with a communication module, allowing IoT devices to offload tasks to the UAVs via wireless communication. To clearly illustrate the technical solution of this embodiment, the UAV positions are fixed but support dynamic adjustment.

[0043] according to Figure 2 The local processing performance equation for IoT devices can be defined as follows:

[0044]

[0045]

[0046] in, Indicates local latency. Indicates local energy consumption. This indicates the total uninstallation percentage. Indicates the task size. This represents the number of CPU cycles required to compute 1 bit of data. Indicates the local CPU frequency. This represents the capacitance coefficient of the chip.

[0047] The performance equation for UAV edge unloading can be defined as follows:

[0048]

[0049]

[0050] in, Indicates device and drones The data transmission rate between channels is calculated based on the Loss channel model. Indicates the association indicator, To the first The percentage of UAV tasks unloaded. For transmission power, It is assigned from IoT devices CPU cycle frequency of the unloading task.

[0051] Privacy leaks include differential privacy (location perturbation) and traffic privacy (offload preference entropy). Differential privacy leaks are calculated as follows:

[0052]

[0053] in, Right now Used to measure a given disturbance range Generate a false distance. The aim is to ensure that the generated spurious distances are within a reasonable range while satisfying differential privacy.

[0054] It is the attacker's prior knowledge, that is, without obtaining any information about the user's task uninstallation, the attacker has prior knowledge of the user's location (distance). The guessed distribution of ).

[0055] Traffic privacy calculation is as follows:

[0056]

[0057] in This is indicated as an uninstallation preference. For local privacy entropy, This represents a collection of drones.

[0058] The system objective is to minimize the total cost of unloading, thereby maintaining the overall system in equilibrium. To achieve optimal control, the cost function is defined as follows:

[0059]

[0060] in, It is a non-weighting factor. It's privacy entropy. It's a differential privacy leak. It is the total delay. It is the total energy consumption.

[0061] In summary, the optimization problem for privacy-preserving uninstallation can be constructed as follows:

[0062]

[0063] In view of the impact of dynamic privacy risks, in order to improve the environmental adaptability and self-learning ability of the privacy protection uninstallation system, this invention proposes an intelligent optimization uninstallation method based on MAPPO to solve the above-mentioned optimization problems.

[0064] MDP is typically used to formally describe RL problems. In each time slot, the agent observes the current state from the environment and makes a decision, executes an action to obtain the next state, and adjusts the policy based on the feedback reward value. This invention defines the state, action, state transition function, and reward function in MDP based on a constructed privacy-preserving offloading system model in a dynamic scenario and an optimization problem.

[0065] 1) Status

[0066] Considering that the optimization of the unloading strategy is jointly affected by the current task size, location, computational complexity, and privacy parameters, the state vector is defined as:

[0067]

[0068] 2) Actions

[0069] For privacy-protected uninstallation systems, actions can be defined as uninstallation percentage and privacy budget:

[0070]

[0071] 3) Reward function

[0072] Unlike minimizing the cost function in optimization theory, the goal of intelligent algorithms is to maximize the long-term cumulative reward value. Therefore, the reward function can be defined as:

[0073]

[0074] The long-term accumulated reward value is called the return, and is represented as follows:

[0075]

[0076] in This is used to discount rewards from future time slots to the current time slot, allowing for a balance between immediate rewards and long-term benefits when optimizing task unloading strategies.

[0077] Since the actions of the unloading system are continuous, and the MAPPO method in deep reinforcement learning can effectively solve the problem of continuous action space in multi-agent systems, this invention proposes an intelligent optimization unloading method based on MAPPO to obtain an intelligent control strategy, thereby improving the system's convergence and stability.

[0078] The architecture of the privacy-preserving mobile edge computing offloading method based on deep reinforcement learning proposed in this invention is as follows: Figure 3As shown, MAPPO mainly consists of an actor network and a critic network. The actor network is used for policy generation, and the critic network is used for value evaluation, representing the corresponding neural network parameters. The agent learns the policy by training the actor network and evaluates the policy by training the critic network to obtain corresponding values.

[0079] The privacy-preserving mobile edge computing offloading method based on deep reinforcement learning proposed in this invention can be divided into two steps: sampling and training.

[0080] 1) Sampling

[0081] First, sufficient samples need to be collected for training. In each time slot, based on the input state, the actor network outputs a corresponding action policy. To ensure effective exploration in the continuous action space, random noise is added to obtain the exploration policy. The policy is executed, and the state transition function yields the next time step state, while the reward function calculates the corresponding reward. The samples are then stored in an experience replay buffer. This process is repeated continuously to generate sufficient samples.

[0082] 2) Training

[0083] In this embodiment of the invention, the training process consists of 300 time slots per episode. In each episode, a small batch of samples is randomly selected for training to reduce the correlation of sample data and improve training efficiency.

[0084] The critic network updates its parameters by minimizing the loss function.

[0085] The actor network updates its parameters using the policy gradient function.

[0086] Finally, after training with a sufficient number of scenarios, optimized actor network parameters can be obtained. Thus, based on each acquired input state, the actor network can generate an optimized strategy for the privacy-preserving uninstallation system in real time.

[0087] By continuously interacting with the environment, the intelligent optimization offloading method proposed in this embodiment of the invention can continuously and intelligently learn and adjust the privacy-preserving mobile edge computing offloading strategy, thereby adapting to the complex and ever-changing dynamic scenarios and realizing privacy protection and efficient offloading of IoT devices.

Claims

1. A method for privacy protection in drone-assisted mobile edge computing task offloading, characterized in that, include: Acquire status information from multiple user IoT devices and multiple UAV edge servers to establish a system model; Based on the system model, an optimization control problem is constructed with the goal of minimizing the total cost; the total cost includes latency, energy consumption, and privacy leakage. A Markov Decision Process (MDP) model for a privacy-preserving offloading system is established based on the system model and optimization problem. Based on the MDP model, a deep reinforcement learning algorithm based on multi-agent proximal policy optimization of MAPPO is used to obtain an intelligent optimization unloading strategy.

2. The method for protecting privacy by offloading unmanned aerial vehicle (UAV)-assisted mobile edge computing tasks according to claim 1, characterized in that, The establishment of the system model includes: The task size, location coordinates, computational complexity, and UAV location information of IoT devices are collected based on wireless communication. Based on task size, location, and computational complexity, dynamic equations for IoT devices and UAVs are established, including local processing and edge offloading models. Based on location information, a differential privacy model is established, and the risk of location leakage is quantified using a truncated Laplace distribution. By solving the dynamic equations simultaneously, we obtain the system state equations based on continuous time. After discretization, we obtain the system model based on discrete time.

3. The method for protecting privacy by offloading unmanned aerial vehicle (UAV)-assisted mobile edge computing tasks according to claim 2, characterized in that, Dynamic analysis is performed on local processing and edge offloading separately. System state information such as task size, location, and computational complexity is obtained through wireless communication, and dynamic equations are established based on the relationships between them. Using the UAV as an edge server, channel gain, transmission rate, and computational resource allocation are calculated. After obtaining the state information, delay and energy consumption equations are derived. Privacy protection follows a differential privacy strategy: ； in, Indicates the position after the disturbance. Indicates the actual location. and Indicates the cutoff range. Indicates privacy budget; , Let be the normalization constant, which is defined as: ; By analyzing the impact of privacy metrics on offloading, dynamic equations with time delay and energy consumption are obtained; then, the state equations of all IoT devices and UAVs are combined to obtain the continuous-time system state equation; the continuous-time system state equation is discretized by sampling to obtain a discrete-time system model.

4. The method for protecting privacy by offloading unmanned aerial vehicle (UAV)-assisted mobile edge computing tasks according to claim 3, characterized in that, The local processing performance equation for IoT devices is defined as follows: ； ； in, Indicates local latency. Indicates local energy consumption. This indicates the total uninstallation percentage. Indicates the task size. This represents the number of CPU cycles required to compute 1 bit of data. Indicates the local CPU frequency. This represents the capacitance coefficient of the chip.

5. The method for protecting privacy by offloading unmanned aerial vehicle (UAV)-assisted mobile edge computing tasks according to claim 4, characterized in that, The performance equation for UAV edge unloading is defined as follows: ； ； in, Indicates device and drones The data transmission rate between channels is calculated based on the Loss channel model. Indicates the association indicator, To the first The percentage of UAV tasks unloaded. For transmission power, It is assigned from IoT devices CPU cycle frequency of the unloading task; Privacy leaks include differential privacy and traffic privacy; differential privacy leaks are calculated as follows: ； in, Right now Used to measure a given disturbance range Generate a false distance. The aim is to ensure that the generated spurious distances are within a reasonable range while satisfying differential privacy. It is the attacker's prior knowledge, that is, the attacker's guesses about the user's location without obtaining any information about the user's task uninstallation. Traffic privacy calculation is as follows: ； in This is indicated as an uninstallation preference. For local privacy entropy, This represents a collection of drones.

6. The method for protecting privacy by offloading unmanned aerial vehicle (UAV)-assisted mobile edge computing tasks according to claim 5, characterized in that, The system objective is to minimize the total cost of unloading, thereby maintaining the entire system in equilibrium. To achieve optimal control, the cost function is defined as follows: ； in, It is a non-weighting factor. It's privacy entropy. It's a differential privacy leak. It is the total delay. It is the total energy consumption.

7. The method for protecting privacy by offloading unmanned aerial vehicle (UAV)-assisted mobile edge computing tasks according to claim 5, characterized in that, The optimization problem for building a privacy-protected uninstallation system is as follows: ； Based on the constructed privacy-preserving offloading system model in a dynamic scenario and the optimization problem, the states, actions, state transition functions, and reward functions in the MDP are defined. 1) Status; Considering that the optimization of the unloading strategy is jointly affected by the current task size, location, computational complexity, and privacy parameters, the state vector is defined as: ； 2) Actions; For the privacy-protected uninstallation system, the actions are defined as uninstallation ratio and privacy budget: ； 3) Reward function; Unlike minimizing the cost function in optimization theory, the goal of intelligent algorithms is to maximize the long-term cumulative reward value, and the reward function is defined as: ； The long-term accumulated reward value is called the return, and is represented as follows: ； in This is used to discount rewards from future time slots to the current time slot, allowing for a balance between immediate rewards and long-term benefits when optimizing task unloading strategies.

8. The method for protecting privacy by offloading unmanned aerial vehicle (UAV)-assisted mobile edge computing tasks according to claim 7, characterized in that, MAPPO consists of an actor network and a critic network. The actor network is used for policy generation, and the critic network is used for value evaluation, representing the corresponding neural network parameters. The agent learns the policy by training the actor network and evaluates the policy by training the critic network to obtain the corresponding values.

9. The method for protecting privacy by offloading unmanned aerial vehicle (UAV)-assisted mobile edge computing tasks according to claim 8, characterized in that, The privacy-preserving mobile edge computing offloading method based on deep reinforcement learning consists of two steps: sampling and training; 1) Sampling; First, sufficient samples need to be collected for training. In each time slot, the actor network will output the corresponding action policy according to the input state. To ensure effective exploration in the continuous action space, random noise is added to obtain the exploration policy. The policy is executed, the state at the next time step is obtained according to the state transition function, and the corresponding reward is obtained according to the reward function. Then the samples are stored in the experience replay buffer. 2) Training; The training process consists of 300 time slots per episode. In each episode, a small batch of samples is randomly selected for training to reduce the correlation of sample data and improve training efficiency. The critic network updates its parameters by minimizing the loss function; The actor network updates its parameters using the policy gradient function; Finally, the optimized actor network parameters are obtained through multi-scenario training; based on the input state acquired each time, the actor network can generate an optimized strategy for the privacy-protected uninstallation system in real time.

10. An apparatus for implementing the privacy protection method for unmanned aerial vehicle-assisted mobile edge computing task offloading as described in any one of claims 1-9, characterized in that, include: The system modeling module is used to acquire status information of multiple user IoT devices and multiple UAV edge servers to build a system model. The problem construction module is used to construct optimization control problems based on the system model; The MDP building module is used to build an MDP model of the privacy-preserving offloading system based on the system model and optimization problem. The computational processing module is used to continuously interact with the environment using a MAPPO-based deep reinforcement learning algorithm to generate samples and train, thereby obtaining an intelligent optimization offloading strategy.

Images