Reinforcement Learning-Based Edge Computing Privacy Task Offloading Method, System, and Device

Abstract

This invention relates to the field of Internet of Things (IoT) technology, specifically to a method, system, and device for offloading privacy-preserving tasks in edge computing based on reinforcement learning. The method includes: acquiring computing task information and task offloading selections generated by each terminal device, whereby task offloading selections include either local execution or offloading to an edge server; calculating the local execution cost or offloading cost of each task based on this information; calculating the total cost of all tasks and establishing an initial global optimization objective; calculating the privacy entropy of each task and embedding this entropy as an additional cost of offloading into the initial global optimization objective to update the global optimization objective; and employing a single-agent reinforcement learning algorithm to find the optimal strategy for the updated global optimization objective. This method significantly reduces algorithm complexity and improves algorithm efficiency while enhancing privacy protection and adaptability to dynamic scenarios, making it suitable for edge computing applications with large-scale nodes, high dynamism, and strong privacy constraints.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and more specifically to edge computing privacy task offloading technology. Background Technology

[0002] With the development of mobile communication technology, 5G bearer networks have introduced new architectures such as resource pool cloudification and control plane / user plane separation to solve the problem of supporting different 5G application scenarios on the transmission side. Among them, mobile edge computing (MEC) technology is one of the core technologies to solve the diverse network requirements brought about by different applications. Edge computing technology increases computing power in the equipment room close to the access network, which can significantly reduce service latency, reduce bandwidth pressure on the transmission network, reduce transmission costs, and further improve content distribution efficiency and enhance user experience. It has been applied in many fields such as smart cities, smart homes, vehicle networking, industrial automation, and the Internet of Things.

[0003] In the practical deployment of edge computing technology, the importance of compute offloading methods is becoming increasingly prominent. As a key technology for achieving efficient resource scheduling in MEC (Multi-access Edge Computing), compute offloading dynamically distributes complex computing tasks from terminals to edge servers or the cloud for processing. This fully utilizes the computing power of edge nodes while avoiding performance bottlenecks caused by insufficient computing power in terminal devices. With the increasing demands for real-time performance and reliability in scenarios such as smart cities and connected vehicles, the optimization of compute offloading decisions has become a core element in ensuring service quality.

[0004] However, the task offloading process involves sensitive information such as the location information of terminal devices and user behavior data. Without effective privacy protection mechanisms, the efficiency gains brought by edge computing may actually lead to new security risks. Therefore, research on privacy protection during the task offloading process in edge computing has gradually become a research hotspot. For example, Chinese patent CN116405975A – a privacy-aware computational offloading method based on privacy entropy for IoT application environments – optimizes task offloading by using an improved Harris Hawk (OHHO) algorithm combined with Gaussian-Cauchy operators while protecting privacy. However, this patent still has the following technical problems:

[0005] 1. Although the improved Harris Hawk (OHHO) algorithm enhances global search capabilities through Gaussian-Cauchy mutation, the swarm intelligence algorithm inherently relies on the initial population distribution and parameter sensitivity, and may still get trapped in local optima in high-dimensional decision spaces (such as multi-user, multi-server scenarios). For example, the algorithm in this patent needs to handle a solution space of dimension N*K*M, and the mutation strategy is only triggered by threshold judgment, which will lead to unstable convergence speed;

[0006] 2. This patent relies on the energy state control algorithm of the prey to switch stages, but the energy model is static decay and cannot respond to network changes (such as channel fluctuations) in real time. Moreover, the weight g (Formula 12) of privacy entropy and resource consumption needs to be preset manually and cannot be adaptively adjusted according to dynamic task requirements (such as sudden high privacy tasks), which may sacrifice privacy protection in time-latency sensitive scenarios.

[0007] 3. Privacy protection requires complex multi-objective optimization algorithms, resulting in a large solution set that requires post-processing. At the same time, the OHHO algorithm needs to maintain the population and calculate Gaussian-Cauchy mutations. In large-scale IoT node scenarios (such as N>100), the algorithm complexity is O(N×K×M×T), which is too computationally expensive and difficult to apply to large-scale node scenarios. Summary of the Invention

[0008] To address the aforementioned technical issues, this invention proposes an edge computing privacy task offloading method, system, and device based on reinforcement learning. The aim is to significantly reduce algorithm complexity and improve algorithm efficiency while enhancing privacy protection and adaptability to dynamic scenarios. This approach is applicable to edge computing applications characterized by large-scale nodes, high dynamism, and strong privacy constraints.

[0009] In the first aspect, this application provides a method for offloading privacy tasks in edge computing based on reinforcement learning, including the following steps:

[0010] Obtain computing task information and task unloading options generated by each terminal device, wherein the task unloading options include selecting to execute the task locally or selecting to unload the task to the edge server;

[0011] Based on the computing task information and task unloading selections generated by each terminal device, the local execution cost or unloading cost of each task is calculated, and the total cost of all tasks is calculated.

[0012] Based on the total cost of all tasks, establish an initial global optimization objective;

[0013] Calculate the privacy entropy of the task and embed the privacy entropy of the task as an additional cost of unloading into the initial global optimization objective to obtain the updated global optimization objective;

[0014] A single-agent reinforcement learning algorithm is used to find the optimal strategy for the updated global optimization objective.

[0015] In some embodiments, the local execution cost or unloading cost of each task is calculated based on computation task information and task unloading selection, including:

[0016] When the task unloading option is to select local execution for the task, the local execution latency and local execution energy consumption are calculated based on the information of each computing task.

[0017] The local execution latency and local execution energy consumption are weighted and normalized to obtain the local execution cost.

[0018] When the task unloading option is to unload the task to the edge server, the transmission latency, server processing latency, transmission energy consumption, and idle energy consumption are calculated based on the information of each computing task.

[0019] The total offloading delay is obtained by summing the transmission delay and the server processing delay.

[0020] The total offloading energy consumption is obtained by summing the transmission energy consumption and the idle energy consumption.

[0021] The total unloading delay and total unloading energy consumption are weighted and normalized to obtain the unloading cost.

[0022] In some embodiments, an initial global optimization objective is established based on the total cost of all tasks, expressed as:

[0023]

[0024] Where n represents the total number of terminal devices. This represents the local execution cost of terminal device i. a represents the offloading cost of terminal device i. i ∈{0,1}, where 0 represents local execution and 1 represents offloading to an edge server for processing.

[0025] In some embodiments, the privacy entropy of the task is embedded as an additional cost of unloading into the initial global optimization objective, resulting in an updated global optimization objective, expressed as:

[0026]

[0027] Where n represents the total number of terminal devices. This represents the local execution cost of terminal device i. a represents the offloading cost of terminal device i. i ={0,1}, where 0 represents local execution and 1 represents offloading to an edge server for processing, H i This represents the privacy entropy of the task.

[0028] In some embodiments, a single-agent reinforcement learning algorithm is used to find the optimal policy for the updated global optimization objective, including:

[0029] Define the state space as s = C all ;

[0030] The reward function is designed such that the reward value is proportional to the cost reduction rate.

[0031] The Q-learning algorithm is used to iteratively update the Q-value until the preset convergence condition is met, thus obtaining the optimal strategy for the updated global optimization objective.

[0032] In some embodiments, the strategy for finding the optimal global optimization objective using a single-agent reinforcement learning algorithm further includes:

[0033] Define the action space as a set of binary actions, where each action corresponds to the decision of local execution or unloading of a task.

[0034] In some embodiments, the Q-learning algorithm iteratively updates the Q-value until a preset convergence condition is met, obtaining the optimal strategy for the updated global optimization objective, including:

[0035] Step 201, define a linear decay function for the exploration probability ∈(t);

[0036] Step 202: Generate a random number θ. If θ < ∈ (t), then randomly select action a. rand If θ>∈(t), then select the action with the maximum current Q value;

[0037] Step 203: Execute the selected action a, calculate the reward value R(s,a), and update the Q value of (s,a) using the update mechanism of the Q-learning algorithm;

[0038] Step 204: Update the new exploration probability ∈(t) according to the linear decay function of the exploration probability;

[0039] Step 205: Repeat steps 202-204 until the preset convergence condition is met, and obtain the optimal strategy for the updated global optimization objective.

[0040] Secondly, this application provides an edge computing privacy task offloading system based on reinforcement learning, including:

[0041] The data acquisition module is used to acquire computing task information and task unloading selections generated by each terminal device. The task unloading selections include selecting to execute the task locally or selecting to unload the task to the edge server.

[0042] The cost calculation module is used to calculate the local execution cost or unloading cost of each task based on the computing task information generated by each terminal device and the task unloading selection, and to calculate the total cost of all tasks.

[0043] The optimization goal establishment module is used to establish an initial global optimization goal based on the total cost of all tasks, calculate the privacy entropy of the tasks, and embed the privacy entropy of the tasks as an additional cost of unloading into the initial global optimization goal to obtain the updated global optimization goal.

[0044] The optimal policy output module is used to find the optimal policy for the updated global optimization objective using a single-agent reinforcement learning algorithm.

[0045] Thirdly, an electronic device including a processor and a memory;

[0046] The processor is connected to the memory;

[0047] The memory is used to store executable program code;

[0048] The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, in order to execute the aforementioned reinforcement learning-based edge computing privacy task offloading method.

[0049] Fourthly, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned reinforcement learning-based edge computing privacy task offloading method.

[0050] The beneficial technical effects of the present invention include at least the following:

[0051] 1. An edge computing privacy task offloading method, system, and device based on reinforcement learning are adopted. By designing a cost function embedded in privacy entropy, the local and offloading cost calculations can quantify latency, energy consumption, and privacy risks, providing a unified optimizable objective for reinforcement learning algorithms. Its deep collaboration with the privacy entropy model, Q-learning single-agent decision-making framework, and dynamic reward feedback mechanism creatively solves the technical problems of privacy protection-resource optimization trade-off, high algorithm complexity and low efficiency, and insufficient dynamic adaptability in existing technologies. The technical unit of this embodiment takes "extremely simplified modeling-dynamic feedback-real-time optimization" as the core link, transforming the trade-off between privacy protection and resource consumption from static multi-objective optimization to dynamic single-objective reinforcement learning tasks. Through the real-time feedback mechanism, it adapts to environmental changes and achieves efficient global convergence. By compressing the state and binary action space, it ensures the feasibility of the algorithm in large-scale scenarios, achieving a breakthrough of "1+1>2" in algorithm efficiency, edge computing scenario adaptability, and privacy protection strength.

[0052] 2. Traditional methods typically require independent optimization of privacy and resource metrics through complex multi-objective optimization algorithms, resulting in a large solution set and complex multi-objective post-processing. Furthermore, existing technologies rely on prey energy state control algorithms for stage switching, but the energy model is statically decaying and cannot respond in real-time to network changes (such as channel fluctuations). Additionally, the weights g of privacy entropy and resource consumption need to be manually preset and cannot be adaptively adjusted according to dynamic task requirements (such as sudden high-privacy tasks), potentially sacrificing privacy protection in latency-sensitive scenarios. Therefore, this application dynamically balances latency and energy consumption by normalizing the local execution and offloading costs to adapt to different scenario requirements. Privacy entropy is directly embedded as the optimization objective of the offloading decision into the cost function, transforming the multi-objective problem into a single-objective reinforcement learning task, significantly simplifying the decision-making logic while ensuring that privacy protection becomes the core constraint of the offloading decision. This achieves adaptive joint optimization of privacy protection and resource efficiency without manual intervention in weight allocation.

[0053] 3. Existing swarm intelligence algorithms (such as the improved OHHO algorithm) rely on the initial population distribution and parameter sensitivity. They explore the solution space through multi-agent collaboration, making them prone to getting trapped in local optima. Furthermore, existing technologies rely on the energy state of the prey to control algorithm phase switching, but the energy model is statically decaying and cannot respond to real-time network changes (such as channel fluctuations). Moreover, in high-dimensional decision spaces, maintaining the population and complex mutation operations increases computational overhead with the number of nodes, making them unsuitable for large-scale node scenarios. Therefore, this application models the optimization objective as a Markov Decision Process (MDP). Through a real-time feedback mechanism of the reward function, the strategy is dynamically adjusted, significantly improving adaptability to dynamic environments. A Q-learning single-agent decision mechanism is adopted, requiring only the maintenance of a Q-table (state-action pairs). The optimal action strategy can be directly learned through dynamic Q-value iteration, achieving model-free optimization and avoiding the limitations of traditional heuristic algorithms that rely on precise mathematical modeling. It eliminates the need to maintain the population and rely on initial parameters, avoiding the local optima problem caused by the initial population distribution in swarm intelligence algorithms. Finally, it utilizes a binary action space and a state s=C all Compressing high-dimensional network parameters into a combination of single metrics significantly reduces computational complexity and accelerates convergence. This design is particularly suitable for edge computing applications with large-scale nodes, high dynamism, and strong privacy constraints.

[0054] 4. By using a random exploration strategy to escape the local high point of the current Q value, a high exploration rate is used in the early stage to cover the global action space, and a low exploration rate is used in the later stage to focus on the optimal strategy. This balances convergence speed and stability, achieves efficient global exploration, avoids local traps, and has a computational complexity that is significantly lower than that of high-dimensional continuous optimization. This further overcomes the technical problems of local optima and low algorithm convergence efficiency in existing technologies.

[0055] Other features and advantages of the present invention will be disclosed in detail in the following detailed description and accompanying drawings. Attached Figure Description

[0056] The invention will be further described below with reference to the accompanying drawings:

[0057] Figure 1 This is a schematic diagram of the execution environment for the uninstallation task in an embodiment of the present invention.

[0058] Figure 2 This is a flowchart of an edge computing privacy task offloading method based on reinforcement learning, according to an embodiment of the present invention.

[0059] Figure 3 This is a schematic diagram of the edge computing privacy task offloading system based on reinforcement learning, according to an embodiment of the present invention.

[0060] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of the present invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of the present invention.

[0062] In the following description, terms such as “inner,” “outer,” “upper,” “lower,” “left,” and “right” are used only to indicate orientation or positional relationship for the convenience of describing the embodiments and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0063] Before describing several embodiments of this specification, the execution environment for the uninstallation task of these embodiments will be introduced. (See attached...) Figure 1 As shown in the embodiment of this specification, the execution environment for the offloading task is as follows: In an edge computing network, an edge server (MEC) and n terminal devices (such as smart terminals and IoT devices) are deployed. Each terminal device has data processing capabilities, and each terminal device generates an independent computing task. When the task is offloaded to the MEC server for processing, the terminal will transmit the task payload to the MEC server wirelessly through the nearest roadside unit (RSU). The server then processes the task and sends the processing result back to the terminal device through the roadside unit.

[0064] Please see the appendix Figure 2 , Figure 2A flowchart illustrating an embodiment of the reinforcement learning-based edge computing privacy task offloading method provided in this specification is shown.

[0065] like Figure 2 As shown, this reinforcement learning-based edge computing privacy task offloading method may include at least the following steps:

[0066] Step 101: Obtain computing task information and task unloading options generated by each terminal device. Task unloading options include selecting to execute the task locally or selecting to unload the task to the edge server.

[0067] The computing task information generated by each terminal device includes the workload (data size, unit: bits), the number of CPU cycles required for the task (unit: cycles), and the X types of data contained in each task. Each terminal device can choose to execute the task locally or offload the task workload to the MEC server for processing.

[0068] Step 102: Based on the computing task information generated by each terminal device and the task unloading selection, calculate the local execution cost or unloading cost of each task, and calculate the total cost of all tasks.

[0069] Furthermore, in this embodiment, based on the computation task information and task unloading selection, the local execution cost or unloading cost of each task is calculated, including:

[0070] When the task unloading option is to select local execution for the task, the local execution latency and local execution energy consumption are calculated based on the information of each computing task.

[0071] Specifically, local execution latency can be expressed as Among them, D i f represents the number of CPU cycles required for the task (unit: cycles). i l This represents the local CPU frequency of terminal device i, i.e., CPU cycles per second (unit: Hz, i.e., cycles / s), and the local execution latency. This represents the time required for the task to complete its computation on terminal device i, determined by the CPU's processing power (frequency). Local execution energy consumption can be expressed as... Among them, z i This represents the CPU power consumption of terminal device i in a single cycle, i.e., the power consumption per CPU cycle (unit: J / cycle), which is set to z in this embodiment. i =10 -27 (f i l ) 2(Based on actual experimental fitting), local execution energy consumption represents the energy consumed by the task in local computation, and is set to be proportional to the square of the CPU frequency, reflecting the energy cost brought by high computing power.

[0072] The local execution latency and local execution energy consumption are weighted and normalized to obtain the local execution cost.

[0073] Specifically, the local execution cost obtained after weight normalization can be expressed as: in, The normalized weights representing local execution latency This represents the normalized weight of local execution energy consumption. and

[0074] When the task unloading option is to unload the task to the edge server, the transmission latency, server processing latency, transmission energy consumption, and idle energy consumption are calculated based on the information of each computing task.

[0075] The total offloading delay is obtained by summing the transmission delay and the server processing delay.

[0076] The total offloading energy consumption is obtained by summing the transmission energy consumption and the idle energy consumption.

[0077] Understandably, when a task is offloaded to the MEC server for processing, the terminal transmits the task payload wirelessly to the MEC server via the nearest Roadside Unit (RSU). The server then processes the task and sends the processing result back to the terminal device via the RSU. Therefore, the task offloading to the MEC server for processing consists of three parts: data payload uploading, server processing, and result feedback. Because the server has sufficient computing power and the amount of processed result data is small enough, the latency and energy consumption of result feedback are negligible in this embodiment.

[0078] Specifically, the data transmission latency from the terminal device to the server can be expressed as: Among them, B i This indicates the amount of data (in bits) that the task needs to upload. i This represents the wireless transmission rate (in bits per second) from the terminal device i to the MEC server. Transmission energy consumption can be expressed as... in, This represents the transmission power (in W) of terminal device i. Server processing latency can be expressed as... Among them, f s This represents the server's CPU frequency (unit: Hz). The energy consumption of terminal devices while waiting for server processing to complete is called idle energy consumption, which can be expressed as... Among them, Pi,I This represents the power (in W) of terminal device i when it is idle (waiting for server processing).

[0079] Summing the transmission delay and the server processing delay, we can obtain the total offloading delay as follows: Summing the transmission energy consumption and idle energy consumption, we can obtain the total offloading energy consumption as follows:

[0080] The total unloading delay and total unloading energy consumption are weighted and normalized to obtain the unloading cost.

[0081] Specifically, the unloading cost obtained after weight normalization can be expressed as: in, The normalized weights represent the total unloading delay. This represents the normalized weight of the total energy consumption for unloading. and

[0082] It is understandable that this embodiment can achieve this by normalizing the weights. and Latency (seconds) and energy consumption (joules) are unified into dimensionless costs, allowing users to adjust priorities as needed by adjusting normalized weights. and It can flexibly adapt to the latency and energy consumption requirements of different scenarios. For example, it can be set for latency-sensitive tasks (such as autonomous driving). Prioritize reducing latency; settings can be configured for energy-sensitive tasks (such as IoT devices). Extend battery life.

[0083] Step 103: Based on the total cost of all tasks, establish an initial global optimization objective.

[0084] Specifically, in this embodiment, an initial global optimization objective is established based on the total cost of all tasks, expressed as:

[0085]

[0086] Where n represents the total number of terminal devices. This represents the local execution cost of terminal device i. a represents the offloading cost of terminal device i. i ∈{0,1}, where 0 represents local execution and 1 represents offloading to an edge server for processing.

[0087] Step 104: Calculate the privacy entropy of the task and embed the privacy entropy of the task as an additional cost of unloading into the initial global optimization objective to obtain the updated global optimization objective.

[0088] The implementation method of privacy entropy of computing task in this embodiment is similar to that of privacy entropy of computing task in the prior art, and will not be described again in this embodiment.

[0089] Specifically, in this embodiment, the privacy entropy of the task is embedded as an additional cost of unloading into the initial global optimization objective, resulting in an updated global optimization objective, expressed as:

[0090]

[0091] Where n represents the total number of terminal devices. This represents the local execution cost of terminal device i. a represents the offloading cost of terminal device i. i ∈{0,1}, where 0 represents local execution and 1 represents offloading to an edge server for processing, H i This represents the privacy entropy of the task.

[0092] Understandably, privacy entropy can quantify the privacy risk of a task (the more uniform the data type distribution, the higher the privacy entropy). Local task execution does not involve data transfer, so the privacy entropy is 0. Unloading a task incurs additional privacy leakage costs. Therefore, this embodiment embeds privacy entropy directly as an additional cost of unloading, which to some extent encourages agents to prioritize local execution in privacy-sensitive tasks. For example, in high-privacy tasks (such as medical data), the privacy entropy is high; even if the resource cost of unloading is low, the total cost may be higher than local execution, triggering a... i =0; In low-privacy tasks (such as environmental sensor data), privacy entropy is low, resource efficiency dominates the decision, and option a may be chosen. i =1.

[0093] Existing privacy protection technologies (such as independent optimization in Chinese patent CN116405975A) typically require complex multi-objective processing. However, this embodiment creatively embeds privacy entropy as a direct cost item in the offloading decision directly into the optimization objective, rather than as an independent objective. That is, privacy protection does not need to be modeled independently, but is endogenous in the optimization objective. This transforms the complex multi-objective problem into a single-objective reinforcement learning task, ensuring that privacy protection becomes the core constraint of the offloading decision. This achieves adaptive joint optimization of privacy protection and resource optimization without the need for manual intervention in weight allocation, while significantly reducing algorithm complexity.

[0094] Step 105: Use a single-agent reinforcement learning algorithm to find the optimal strategy for the updated global optimization objective.

[0095] Specifically, in this embodiment, a single-agent reinforcement learning algorithm is used to find the optimal strategy for the updated global optimization objective, including:

[0096] Define the state space as s = C all ;

[0097] The reward function is designed such that the reward value is proportional to the cost reduction rate.

[0098] The Q-learning algorithm is used to iteratively update the Q-value until the preset convergence condition is met, thus obtaining the optimal strategy for the updated global optimization objective.

[0099] It is understandable that in this embodiment, the state is defined as the current total system cost (s = C). all The state input is used as the Q-learning state to reflect the combined cost of all current tasks (i.e., global resource consumption and privacy risks), rather than the multi-dimensional position vector (e.g., X(t)∈R) used in existing technologies. N×K×M ).

[0100] Specifically, the dynamic reward function designed in this embodiment can be expressed as:

[0101]

[0102] Among them, tc l Let tc(s,a) represent the total cost of performing all tasks locally, and let tc(s,a) represent the actual total cost of the current state, driving the agent to choose C. all The action that minimizes resource consumption (i.e., the optimal privacy-resource trade-off strategy). A higher reward value indicates a better action.

[0103] Specifically, the Q-value update mechanism of the Q-learning algorithm can be expressed as:

[0104]

[0105] Where α∈{0,1} represents the learning rate, which is used to control the rate at which new information overwrites old knowledge, and γ∈{0,1} represents the discount factor, which is used to balance immediate rewards with long-term benefits.

[0106] For example, the preset convergence condition in this embodiment can be: when C in K consecutive iterations all If the fluctuation amplitude is less than the threshold δ (e.g., δ = 1%), then convergence is determined. This embodiment does not impose any limitations on this.

[0107] Furthermore, in this embodiment, the strategy for finding the optimal global optimization objective using a single-agent reinforcement learning algorithm further includes:

[0108] Define the action space as a set of binary actions, where each action corresponds to the decision of local execution or unloading of a task.

[0109] Specifically, the action space in this embodiment can be represented as a = [a1, a2, ..., a...].n ], where a i ∈{0,1} indicates whether the task is executed locally (a i =0) or uninstall (a i =1), the action space size is 2 n (e.g., the operating space for 10 terminal devices is only 2) 10 =1024 types).

[0110] This embodiment compresses the action space by defining an action set a∈{0,1} (binary unloading decision), simplifying the unloading decision to a 0 / 1 choice, avoiding the high computational overhead of continuous action spaces. The combination of binary action space and the model-independent nature of Q-learning significantly reduces computational complexity, making it suitable for large-scale terminal scenarios.

[0111] In summary, this embodiment, through the design of the cost function embedded in privacy entropy, enables the quantification of latency, energy consumption, and privacy risks in local and offloading cost calculations, providing a unified optimizable objective for reinforcement learning algorithms. Its deep synergy with the privacy entropy model, the single-agent decision-making framework of Q-learning, and the dynamic reward feedback mechanism creatively solves the technical problems of difficulty in balancing privacy protection and resource optimization, high algorithm complexity and low efficiency, and insufficient dynamic adaptability in existing technologies. The technical unit of this embodiment takes "extremely simplified modeling - dynamic feedback - real-time optimization" as the core link, transforming the trade-off between privacy protection and resource consumption from static multi-objective optimization to a dynamic single-objective reinforcement learning task. Through the real-time feedback mechanism, it adapts to environmental changes and achieves efficient global convergence. By compressing the state and binary action space, it ensures the feasibility of the algorithm in large-scale scenarios, achieving a breakthrough of "1+1>2" in algorithm efficiency, edge computing scenario adaptability, and privacy protection strength.

[0112] Specifically, on the one hand, traditional methods typically require independent optimization of privacy and resource metrics through complex multi-objective optimization algorithms, resulting in a large solution set and complex multi-objective post-processing. Furthermore, existing technologies rely on prey energy state control algorithms for stage switching, but the energy model is statically decaying and cannot respond in real-time to network changes (such as channel fluctuations). Moreover, the weights g of privacy entropy and resource consumption need to be manually preset and cannot be adaptively adjusted according to dynamic task requirements (such as sudden high-privacy tasks), potentially sacrificing privacy protection in latency-sensitive scenarios. Therefore, this embodiment dynamically balances latency and energy consumption by normalizing the local execution and offloading costs to adapt to different scenario requirements. It also directly embeds privacy entropy as the optimization objective of the offloading decision into the cost function, transforming the multi-objective problem into a single-objective reinforcement learning task, significantly simplifying the decision-making logic while ensuring that privacy protection becomes the core constraint of the offloading decision. This achieves adaptive joint optimization of privacy protection and resource efficiency without manual intervention in weight allocation. On the other hand, existing technologies typically employ swarm intelligence algorithms (such as the improved OH algorithm). The HO algorithm relies on the initial population distribution and parameter sensitivity, exploring the solution space through multi-agent collaboration, which is prone to getting trapped in local optima. Furthermore, existing technologies rely on the energy state of the prey to control algorithm phase switching, but the energy model is statically decaying and cannot respond to real-time network changes (such as channel fluctuations). Moreover, in high-dimensional decision spaces, maintaining the population and complex mutation operations requires computational overhead that increases dramatically with the number of nodes, making it unsuitable for large-scale node scenarios. Therefore, this embodiment models the optimization objective as a Markov Decision Process (MDP), dynamically adjusting the strategy through a real-time feedback mechanism of the reward function, significantly improving adaptability to dynamic environments. It adopts a Q-learning single-agent decision mechanism, requiring only the maintenance of a Q-table (state-action pairs), and directly learning the optimal action strategy through dynamic Q-value iteration, achieving model-free optimization. This avoids the limitations of traditional heuristic algorithms that rely on precise mathematical modeling, eliminating the need to maintain the population or rely on initial parameters, thus avoiding the local optima problem caused by the initial population distribution in swarm intelligence algorithms. Finally, it utilizes a binary action space and state s=C... all Compressing high-dimensional network parameters into a combination of single metrics significantly reduces computational complexity and accelerates convergence. This design is particularly suitable for edge computing applications with large-scale nodes, high dynamism, and strong privacy constraints.

[0113] In another embodiment of this description, to further avoid local optima, the following design is added. Specifically, the Q-learning algorithm's update mechanism iteratively updates the Q-value until a preset convergence condition is reached, obtaining the optimal strategy for the updated global optimization objective, including:

[0114] Step 201, define the linear decay function of the exploration probability ∈(t), which can be expressed as:

[0115] ∈(t)=max(∈min ,∈0-ηt)

[0116] Where η represents the decay rate (e.g., decreasing by 0.01 every 100 iterations), used to gradually reduce the exploration ratio, ∈ min This indicates a preset minimum exploration probability, which is used to respond to dynamic changes in the environment by maintaining a minimum exploration rate.

[0117] Step 202: Generate a random number θ. If θ < ∈ (t), then randomly select action a. rand (Uniform sampling), if θ>∈(t), then select the action a with the maximum current Q value. opt , can be represented as:

[0118] a opt =arg max a Q(s,a)

[0119] Understandably, this embodiment determines whether to randomly select an action (exploring an unknown strategy) based on the exploration probability ∈ , requiring only O(1) complexity, significantly lower than the probability distribution sampling of the policy gradient method (such as the O(n) complexity of Softmax), or to select the action with the maximum current Q value (using a known optimal strategy), requiring only O(n) comparison operations, significantly lower than the traditional continuous action space (such as DDPG) which requires calculating the policy gradient (O(n)). 2 (complexity).

[0120] Step 203: Execute the selected action a, calculate the reward value R(s,a), and update the Q value of (s,a) using the update mechanism of the Q-learning algorithm;

[0121] Step 204: Update the new exploration probability ∈(t) according to the linear decay function of the exploration probability.

[0122] Specifically, after each iteration, ∈ is dynamically adjusted according to ∈(t) to gradually reduce the exploration ratio in order to converge to a stable strategy.

[0123] Step 205: Repeat steps 202-204 until the preset convergence condition is met, and obtain the optimal strategy for the updated global optimization objective.

[0124] This embodiment uses a random exploration strategy to escape local highs in the current Q value. The high exploration rate in the early stage covers the global action space, while the low exploration rate in the later stage focuses on the optimal strategy. It balances convergence speed and stability, achieves efficient global exploration, avoids local traps, and has a computational complexity that is significantly lower than that of high-dimensional continuous optimization. This further overcomes the technical problems of local optima and low algorithm convergence efficiency in the prior art.

[0125] In summary, this embodiment combines a dynamic reward feedback mechanism with a stochastic exploration strategy. Q-learning achieves an efficient balance between global exploration and local utilization in a discrete action space. Specifically, dynamic rewards reflect the impact of actions on the multi-objective system in real time, avoiding the bias of manually designed reward functions. The stochastic exploration strategy uses controllable randomness to escape local optima while maintaining low computational complexity. The binary action space reduces the decision dimensionality, avoiding gradient calculations and the curse of dimensionality in high-dimensional continuous optimization. Thus, while ensuring the privacy-resource joint optimization objective, it significantly improves the algorithm's convergence speed and the real-time response capability of the edge computing system.

[0126] Please see the appendix Figure 3 , Figure 3 This is a schematic diagram of the structure of an edge computing privacy task offloading system based on reinforcement learning, provided as another embodiment of this specification.

[0127] like Figure 3 As shown, the reinforcement learning-based edge computing privacy task offloading system may include at least a data acquisition module 1, a cost calculation module 2, an optimization target establishment module 3, and an optimal policy output module 4, wherein:

[0128] Data acquisition module 1 is used to acquire computing task information and task unloading selections generated by each terminal device. The task unloading selections include selecting to execute the task locally or selecting to unload the task to the edge server.

[0129] The cost calculation module 2 is used to calculate the local execution cost or unloading cost of each task based on the computing task information generated by each terminal device and the task unloading selection, and to calculate the total cost of all tasks.

[0130] The optimization goal establishment module 3 is used to establish an initial global optimization goal based on the total cost of all tasks, calculate the privacy entropy of the tasks, and embed the privacy entropy of the tasks as an additional cost of unloading into the initial global optimization goal to obtain the updated global optimization goal.

[0131] The optimal policy output module 4 is used to find the optimal policy for the updated global optimization objective using a single-agent reinforcement learning algorithm.

[0132] It is understood that the technical concept of the edge computing privacy task offloading system based on reinforcement learning provided in this embodiment is similar to the technical concept of the aforementioned edge computing privacy task offloading method based on reinforcement learning, and will not be repeated here.

[0133] Please see the appendix Figure 4 , Figure 4 This is a schematic diagram of an electronic device structure provided as another embodiment of this specification. For example... Figure 4As shown, the electronic device 400 may include at least one processor 401, at least one network interface 404, a user interface 403, a memory 405, and at least one communication bus 402.

[0134] The communication bus 402 can be used to realize the connection and communication of the above components.

[0135] The user interface 403 may include buttons, and the optional user interface may also include a standard wired interface or a wireless interface.

[0136] Among them, network interface 404 may include, but is not limited to, Bluetooth module, NFC module, Wi-Fi module, etc.

[0137] The processor 401 may include one or more processing cores. The processor 401 connects to various parts within the electronic device 400 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 405, and by calling data stored in the memory 405. Optionally, the processor 401 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor 401 may integrate one or more of the following: CPU, GPU, and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 401 and may be implemented as a separate chip.

[0138] The memory 405 may include RAM or ROM. Optionally, the memory 405 may include a non-transitory computer-readable medium. The memory 405 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 405 may also be at least one storage device located remotely from the aforementioned processor 401. As a computer storage medium, the memory 405 may include an operating system, a network communication module, a user interface module, and application programs. The processor 401 may be used to call the application programs stored in the memory 405 and execute the methods in one or more of the above embodiments.

[0139] Another embodiment of this specification provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. The constituent modules of the above-described electronic device, if implemented as software functional units and used as independent downstream task predictions or applications, can be stored in a computer-readable storage medium.

[0140] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this specification is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0141] The above description is merely a preferred embodiment disclosed in this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of protection involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0142] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

Claims

1. A method for offloading privacy tasks in edge computing based on reinforcement learning, characterized in that, Includes the following steps: Obtain computing task information and task unloading options generated by each terminal device, wherein the task unloading options include selecting to execute the task locally or selecting to unload the task to the edge server; Based on the computing task information and task unloading selections generated by each terminal device, the local execution cost or unloading cost of each task is calculated, and the total cost of all tasks is calculated. Based on the total cost of all tasks, an initial global optimization objective is established, expressed as: ， wherein n represents the total number of terminal devices, represents the local execution cost of the terminal device i, represents the offloading cost of the terminal device i, wherein 0 represents local execution and 1 represents offloading to an edge server for processing; Calculate the privacy entropy of the task, and embed this entropy as an additional cost of unloading into the initial global optimization objective to obtain the updated global optimization objective, expressed as: ， wherein, represents the privacy entropy of the task; A single-agent reinforcement learning algorithm is used to find the optimal strategy for the updated global optimization objective.

2. The edge computing privacy task offloading method based on reinforcement learning as described in claim 1, characterized in that, Based on the computation task information and task unloading selection, the local execution cost or unloading cost of each task is calculated, including: When the task unloading option is to select local execution for the task, the local execution latency and local execution energy consumption are calculated based on the information of each computing task. The local execution latency and local execution energy consumption are weighted and normalized to obtain the local execution cost. When the task unloading option is to unload the task to the edge server, the transmission latency, server processing latency, transmission energy consumption, and idle energy consumption are calculated based on the information of each computing task. The total offloading delay is obtained by summing the transmission delay and the server processing delay. The total offloading energy consumption is obtained by summing the transmission energy consumption and the idle energy consumption. The total unloading delay and total unloading energy consumption are weighted and normalized to obtain the unloading cost.

3. The edge computing privacy task offloading method based on reinforcement learning as described in claim 1, characterized in that, A single-agent reinforcement learning algorithm is used to find the optimal policy for the updated global optimization objective, including: The state space is defined as ; The reward function is designed such that the reward value is proportional to the cost reduction rate. The Q-learning algorithm is used to iteratively update the Q-value until the preset convergence condition is met, thus obtaining the optimal strategy for the updated global optimization objective.

4. The edge computing privacy task offloading method based on reinforcement learning as described in claim 3, characterized in that, The method of using a single-agent reinforcement learning algorithm to find the optimal policy for the updated global optimization objective also includes: Define the action space as a set of binary actions, where each action corresponds to the decision of local execution or unloading of a task.

5. The edge computing privacy task offloading method based on reinforcement learning as described in claim 3, characterized in that, The Q-learning algorithm iteratively updates the Q-value until a preset convergence condition is met, yielding the optimal strategy for the updated global optimization objective, including: Step 201, define the linear decay function of the exploration probability ϵ(t); Step 202, generate a random number θ, if θ < ε(t), then randomly select an action if θ > ε(t), then select the action with the current maximum Q value; Step 203: Execute the selected action a, calculate the reward value R(s,a), and update the Q value of (s,a) using the update mechanism of the Q-learning algorithm; Step 204: Update the new exploration probability ϵ(t) according to the linear decay function of the exploration probability; Step 205: Repeat steps 202-204 until the preset convergence condition is met, and obtain the optimal strategy for the updated global optimization objective.

6. An edge computing privacy task offloading system based on reinforcement learning, characterized in that, include: The data acquisition module is used to acquire computing task information and task unloading selections generated by each terminal device. The task unloading selections include selecting to execute the task locally or selecting to unload the task to the edge server. The cost calculation module is used to calculate the local execution cost or unloading cost of each task based on the computing task information generated by each terminal device and the task unloading selection, and to calculate the total cost of all tasks. The optimization goal establishment module is used to establish an initial global optimization goal based on the total cost of all tasks, calculate the privacy entropy of the tasks, and embed the privacy entropy of the tasks as an additional cost of unloading into the initial global optimization goal to obtain the updated global optimization goal. The optimal policy output module is used to find the optimal policy for the updated global optimization objective using a single-agent reinforcement learning algorithm. When the optimization objective establishment module establishes an initial global optimization objective based on the total cost of all tasks, the expression is: ， Where n represents the total number of terminal devices. This represents the local execution cost of terminal device i. This represents the offloading cost of terminal device i. , where 0 represents local execution and 1 represents offloading to an edge server for processing; When the optimization goal establishment module embeds the task's privacy entropy as an additional cost of unloading into the initial global optimization goal to obtain the updated global optimization goal, the expression is: ， wherein, represents the privacy entropy of the task.

7. An electronic device, comprising: Including the processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to execute the reinforcement learning-based edge computing privacy task offloading method according to any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the reinforcement learning-based edge computing privacy task offloading method according to any one of claims 1 to 5.

Images