Reinforcement learning-based edge computing privacy task unloading method, system and equipment

Abstract

The invention relates to the technical field of the Internet of Things, in particular to an edge computing privacy task unloading method, system and device based on reinforcement learning, and the method comprises the steps: obtaining computing task information and task unloading selection generated by each terminal device, and the task unloading selection comprises task selection local execution or task selection unloading to an edge server, local execution cost or unloading cost of each task is obtained based on the calculation; performing statistics to obtain the total cost of all tasks, and establishing an initial global optimization target; the privacy entropy of the task is calculated, and the privacy entropy of the task is used as unloading additional cost to be embedded into the initial global optimization target to update the global optimization target; and searching an optimal strategy of the updated global optimization target by adopting a single-agent reinforcement learning algorithm. The privacy guarantee strength and the dynamic scene adaptability are improved while the algorithm complexity is greatly reduced and the algorithm efficiency is improved, and the method can be suitable for large-scale node, high-dynamic and strong-privacy-constraint application scenes in edge calculation.

Description

Technical Field

[0001] The present invention relates to the field of Internet of Things technology, and in particular to edge computing privacy task offloading technology. Background Art

[0002] With the development of mobile communication technology, 5G bearer networks have introduced new architectures such as cloud-based resource pools and control plane / user plane separation to solve the problem of supporting different 5G application scenarios on the transmission side. Among them, mobile edge computing (MEC) technology is one of the core technologies to solve the diversified network requirements brought by different applications. Edge computing technology increases computing power in computer rooms close to the access network, which can significantly reduce service latency, reduce bandwidth pressure on the transmission network, reduce transmission costs, further improve content distribution efficiency, and enhance user experience. It has been applied to smart cities, smart homes, Internet of Vehicles, industrial automation, Internet of Things and other fields.

[0003] Compute offloading methods are becoming increasingly important in the practical deployment of edge computing technologies. As a key technology for efficient resource scheduling in MEC (Meanwhile Integrated Circuits), compute offloading dynamically allocates complex terminal computing tasks to edge servers or cloud processing. This fully utilizes the computing power of edge nodes while avoiding performance bottlenecks caused by insufficient computing power in terminal devices. As scenarios like smart cities and the Internet of Vehicles (IoV) continue to demand greater real-time performance and reliability, optimizing compute offloading decisions has become a key component in ensuring service quality.

[0004] However, the task offloading process involves sensitive information such as the location information of terminal devices and user behavior data. Without an effective privacy protection mechanism, the efficiency improvement brought by edge computing may lead to new security risks. Therefore, the research on the protection of privacy information during the task offloading process in edge computing has gradually become a research hotspot. For example, Chinese patent CN116405975A - A privacy-aware computation offloading method based on privacy entropy for IoT application environment, optimizes task offloading while protecting privacy by using an improved Harris Hawk (OHHO) algorithm combined with the Gauss-Cauchy operator. However, this patent still has the following technical problems:

[0005] 1. While the Improved Harris Hawk (OHHO) algorithm enhances global search capabilities through Gaussian-Cauchy mutation, swarm intelligence algorithms inherently rely on the initial population distribution and parameter sensitivity, and can still fall into local optima in high-dimensional decision spaces (such as multi-user, multi-server scenarios). For example, the algorithm in this patent needs to process a solution space of dimension N*K*M, and the mutation strategy is triggered only by threshold judgment, which will lead to unstable convergence speed.

[0006] 2. This patent relies on the energy state of the prey to control the algorithm phase switching, but the energy model is static attenuation and cannot respond to network changes (such as channel fluctuations) in real time. In addition, the weight g (Formula 12) between privacy entropy and resource consumption must be manually preset and cannot be adaptively adjusted according to dynamic task requirements (such as sudden high-privacy tasks). As a result, privacy protection may be sacrificed in delay-sensitive scenarios.

[0007] 3. Privacy protection requires processing through complex multi-objective optimization algorithms, resulting in a large solution set and requiring post-processing. At the same time, the OHHO algorithm needs to maintain population individuals and calculate Gaussian-Cauchy mutations. In the scenario of large-scale IoT nodes (such as N>100), the algorithm complexity is O(N×K×M×T), and the computational overhead is too high, making it difficult to apply to large-scale node scenarios. Summary of the Invention

[0008] In response to the above technical problems, the present invention proposes an edge computing privacy task offloading method, system and device based on reinforcement learning, aiming to significantly reduce the algorithm complexity, improve the algorithm efficiency, while improving the privacy protection strength and adaptability to dynamic scenarios. It can be applied to application scenarios with large-scale nodes, high dynamics and strong privacy constraints in edge computing.

[0009] In the first aspect, this application provides an edge computing privacy task offloading method based on reinforcement learning, comprising the following steps:

[0010] Obtaining computing task information generated by each terminal device and task offloading options, wherein the task offloading options include selecting to execute the task locally or selecting to offload the task to an edge server;

[0011] Based on the computing task information generated by each terminal device and the task offloading selection, the local execution cost or offloading cost of each task is calculated, and the total cost of all tasks is calculated;

[0012] Establish an initial global optimization goal based on the total cost of all tasks;

[0013] Calculate the privacy entropy of the task and embed the privacy entropy of the task as the additional cost of offloading into the initial global optimization objective to obtain the updated global optimization objective;

[0014] A single-agent reinforcement learning algorithm is used to find the optimal strategy for the updated global optimization objective.

[0015] In some embodiments, based on the computing task information and the task offloading selection, the local execution cost or offloading cost of each task is calculated, including:

[0016] When the task offloading option is local execution, the local execution delay and local execution energy consumption are calculated based on the information of each computing task;

[0017] Perform weight normalization on the local execution latency and local execution energy consumption to obtain the local execution cost;

[0018] When the task offloading selection is to offload the task to the edge server, based on the information of each computing task, the transmission delay, server processing delay, transmission energy consumption and idle energy consumption are calculated;

[0019] The total offloading delay is obtained by summing the transmission delay and the server processing delay.

[0020] The total offloading energy consumption is obtained by summing the transmission energy consumption and the idle energy consumption;

[0021] The total unloading delay and total unloading energy consumption are weighted and normalized to obtain the unloading cost.

[0022] In some embodiments, an initial global optimization objective is established based on the total cost of all tasks, expressed as:

[0023]

[0024] Where n represents the total number of terminal devices, represents the local execution cost of terminal device i, represents the offloading cost of terminal device i, a i ∈{0,1}, where 0 represents local execution and 1 represents offloading to the edge server for processing.

[0025] In some embodiments, the privacy entropy of the task is embedded into the initial global optimization objective as an additional cost of offloading to obtain an updated global optimization objective, which is expressed as:

[0026]

[0027] Where n represents the total number of terminal devices, represents the local execution cost of terminal device i, represents the offloading cost of terminal device i, a i ={0,1}, where 0 represents local execution and 1 represents offloading to the edge server for processing. i Represents the privacy entropy of the task.

[0028] In some embodiments, a single-agent reinforcement learning algorithm is used to find the optimal strategy for the updated global optimization objective, including:

[0029] Define the state space as s = C all ;

[0030] Design the reward function so that the reward value is proportional to the cost reduction;

[0031] The update mechanism of the Q-learning algorithm is used to iteratively update the Q value until the preset convergence condition is reached, and the optimal strategy for the updated global optimization objective is obtained.

[0032] In some embodiments, using a single-agent reinforcement learning algorithm to find the optimal strategy for the updated global optimization objective further includes:

[0033] The action space is defined as a set of binary actions, each of which corresponds to a decision on local execution or offloading of a task.

[0034] In some embodiments, the Q-value is iteratively updated using an update mechanism of a Q-learning algorithm until a preset convergence condition is reached, thereby obtaining an updated optimal strategy for the global optimization objective, including:

[0035] Step 201, define a linear decay function of exploration probability ∈(t);

[0036] Step 202: Generate a random number θ. If θ < ∈(t), randomly select action a. rand , if θ>∈(t), then select the action with the maximum current Q value;

[0037] Step 203: Execute the selected action a, calculate the reward value R(s,a), and use the Q-learning algorithm to update the Q value corresponding to (s,a);

[0038] Step 204: Update the new exploration probability ∈(t) according to the linear decay function of the exploration probability;

[0039] Step 205, repeating steps 202 to 204 until a preset convergence condition is reached, and obtaining an updated optimal strategy for the global optimization objective.

[0040] Secondly, this application provides an edge computing privacy task offloading system based on reinforcement learning, including:

[0041] A data acquisition module is used to obtain computing task information generated by each terminal device and task offloading options, wherein the task offloading options include selecting to execute the task locally or selecting to offload the task to an edge server;

[0042] The cost calculation module is used to calculate the local execution cost or offload cost of each task based on the computing task information generated by each terminal device and the task offload selection, and to obtain the total cost of all tasks;

[0043] The optimization target establishment module is used to establish the initial global optimization target based on the total cost of all tasks, calculate the privacy entropy of the task, and embed the privacy entropy of the task as the additional cost of offloading into the initial global optimization target to obtain the updated global optimization target;

[0044] The optimal strategy output module is used to find the optimal strategy for the updated global optimization objective using a single-agent reinforcement learning algorithm.

[0045] In a third aspect, an electronic device includes a processor and a memory;

[0046] The processor is connected to the memory;

[0047] The memory is used to store executable program code;

[0048] The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the edge computing privacy task offloading method based on reinforcement learning as mentioned above.

[0049] In a fourth aspect, a computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the aforementioned edge computing privacy task offloading method based on reinforcement learning.

[0050] The beneficial technical effects of the present invention include at least:

[0051] 1. Adopting a reinforcement learning-based edge computing privacy task offloading method, system, and device, the system uses a cost function design with embedded privacy entropy to enable local and offloaded cost calculations to quantify latency, energy consumption, and privacy risks, providing a unified, optimizable goal for the reinforcement learning algorithm. Its deep collaboration with the privacy entropy model, Q-learning's single-agent decision framework, and dynamic reward feedback mechanism creatively addresses the technical issues of the existing technology, such as the difficulty in balancing privacy protection and resource optimization, high algorithm complexity and low efficiency, and insufficient dynamic adaptability. The technical unit of this embodiment, with "extremely simplified modeling-dynamic feedback-real-time optimization" as its core link, transforms the trade-off between privacy protection and resource consumption from a static multi-objective optimization to a dynamic single-objective reinforcement learning task. It adapts to environmental changes through a real-time feedback mechanism, achieving efficient global convergence. By compressing the state and binary action space, the system ensures the feasibility of the algorithm in large-scale scenarios, achieving a breakthrough of "1+1>2" in algorithm efficiency, edge computing scenario adaptability, and privacy protection strength.

[0052] 2. Traditional methods usually require independent optimization of privacy and resource indicators through complex multi-objective optimization algorithms, resulting in a large solution set and the need for complex multi-objective post-processing. At the same time, existing technologies rely on the energy state of the prey to control the algorithm stage switching, but the energy model is static attenuation and cannot respond to network changes (such as channel fluctuations) in real time. The weight g of privacy entropy and resource consumption needs to be manually preset and cannot be adaptively adjusted according to dynamic task requirements (such as sudden high-privacy tasks). As a result, privacy protection may be sacrificed in delay-sensitive scenarios. To this end, this application dynamically balances latency and energy consumption between local execution and offloading costs through normalized weights to adapt to the needs of different scenarios, and directly embeds privacy entropy as the optimization target of offloading decisions into the cost function, transforming the multi-objective problem into a single-objective reinforcement learning task, greatly simplifying the decision logic, and ensuring that privacy protection becomes the core constraint of offloading decisions, achieving adaptive joint optimization of privacy protection and resource efficiency without manual intervention in weight allocation;

[0053] 3. The swarm intelligence algorithms commonly used in the prior art (such as the improved OHHO algorithm) are essentially dependent on the initial population distribution and parameter sensitivity. They explore the solution space through multi-agent collaboration and are prone to falling into local optimality. At the same time, the prior art relies on the energy state of the prey to control the algorithm stage switching, but the energy model is static attenuation and cannot respond to network changes (such as channel fluctuations) in real time. In addition, it is necessary to maintain the population and complex mutation operations in the high-dimensional decision space. The computational overhead increases dramatically with the number of nodes, making it difficult to apply to large-scale node scenarios. To this end, this application models the optimization objective as a Markov decision process (MDP), dynamically adjusts the strategy through the real-time feedback mechanism of the reward function, and greatly improves the adaptability to dynamic environments. The single-agent decision mechanism of Q-learning is adopted. Only the Q table (state-action pair) needs to be maintained. The optimal action strategy can be directly learned through dynamic Q-value iteration to achieve model-free optimization, avoiding the limitations of traditional heuristic algorithms that rely on precise mathematical modeling. There is no need to maintain the population or rely on initial parameters, avoiding the local optimal problem of the swarm intelligence algorithm caused by the initial population distribution, and through the binary action space and state s=C all Compressing high-dimensional network parameters into a single metric significantly reduces computational complexity and accelerates convergence. This design is particularly suitable for edge computing scenarios with large-scale nodes, high dynamics, and strong privacy constraints.

[0054] 4. Through the random exploration strategy, we jump out of the current local high point of Q value. In the early stage, the high exploration rate is used to cover the global action space. In the later stage, the low exploration rate is used to focus on the optimal strategy. Taking into account the convergence speed and stability, efficient global exploration is achieved, local traps are avoided, and the computational complexity is significantly lower than high-dimensional continuous optimization, further overcoming the technical problems of local optimality and low algorithm convergence efficiency in existing technologies.

[0055] Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The present invention will be further described below with reference to the accompanying drawings:

[0057] Figure 1 Schematic diagram of the offloading task execution environment according to an embodiment of the present invention.

[0058] Figure 2 This is a flow chart of the edge computing privacy task offloading method based on reinforcement learning in an embodiment of the present invention.

[0059] Figure 3 This is a structural diagram of the edge computing privacy task offloading system based on reinforcement learning in an embodiment of the present invention.

[0060] Figure 4 The figure is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following is an explanation and description of the technical solutions of the embodiments of the present invention in conjunction with the drawings of the embodiments of the present invention. However, the following embodiments are only preferred embodiments of the present invention and are not exhaustive. Based on the embodiments in the implementation manner, other embodiments obtained by those skilled in the art without creative work are all within the scope of protection of the present invention.

[0062] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, they should not be understood as limiting the present invention.

[0063] Before describing the multiple embodiments of this specification, the uninstallation task execution environment of the embodiment of this specification is first introduced. Figure 1 As shown, the offload task execution environment of the embodiment of this specification is: in an edge computing network, an edge server (MEC) and n terminal devices (such as smart terminals and IoT devices) are deployed. Each terminal device has data processing capabilities and generates independent computing tasks. When the task is offloaded to the MEC server for processing, the terminal will transmit the task load to the MEC server in the form of wireless transmission through the nearest roadside unit RSU. The server then processes the task and transmits the processing results back to the terminal device through the roadside unit.

[0064] Please see the attached Figure 2 , Figure 2A flow chart of an edge computing privacy task offloading method based on reinforcement learning provided in one embodiment of this specification is shown.

[0065] like Figure 2 As shown, the edge computing privacy task offloading method based on reinforcement learning may include at least the following steps:

[0066] Step 101: Obtain computing task information generated by each terminal device and task offloading selection, where the task offloading selection includes selecting to execute the task locally or selecting to offload the task to an edge server.

[0067] The computing task information generated by each terminal device includes the workload of each task (data size, unit: bits), the number of CPU cycles required for the task (unit: cycles), and the X types of data contained in each task. Each terminal device can choose to execute the task locally or offload the task load to the MEC server for processing.

[0068] Step 102 : Based on the computing task information generated by each terminal device and the task offloading selection, the local execution cost or offloading cost of each task is calculated, and the total cost of all tasks is obtained by statistics.

[0069] Furthermore, in this embodiment, based on the computing task information and the task offloading selection, the local execution cost or offloading cost of each task is calculated, including:

[0070] When the task offloading option is local execution, the local execution delay and local execution energy consumption are calculated based on the information of each computing task;

[0071] Specifically, the local execution latency can be expressed as Among them, D i Indicates the number of CPU cycles required for the task (unit: cycles), f i l Indicates the local CPU frequency of terminal device i, that is, the number of CPU cycles per second (unit: Hz, i.e. cycles / s), local execution delay It represents the time required for the task to complete the calculation on the terminal device i, which is determined by the CPU processing power (frequency). The local execution energy consumption can be expressed as Among them, z i Indicates the CPU single cycle energy consumption of terminal device i, that is, the energy consumption of each CPU cycle (unit: J / cycle), which is set to z in this embodiment. i =10 -27 (f i l ) 2(Based on actual experimental fitting), local execution energy consumption refers to the energy consumed by local computing of the task, which is set to be proportional to the square of the CPU frequency, reflecting the energy consumption cost brought by high computing power.

[0072] Perform weight normalization on the local execution latency and local execution energy consumption to obtain the local execution cost;

[0073] Specifically, the local execution cost obtained after weight normalization can be expressed as in, represents the normalized weight of local execution latency, represents the normalized weight of local execution energy consumption, and

[0074] When the task offloading selection is to offload the task to the edge server, based on the information of each computing task, the transmission delay, server processing delay, transmission energy consumption and idle energy consumption are calculated;

[0075] The total offloading delay is obtained by summing the transmission delay and the server processing delay.

[0076] The total offloading energy consumption is obtained by summing the transmission energy consumption and the idle energy consumption;

[0077] It is understandable that when a task is offloaded to the MEC server for processing, the terminal transmits the task load to the MEC server via the nearest roadside unit (RSU) in the form of wireless transmission. The server then processes the task and transmits the processing results back to the terminal device via the roadside unit. Therefore, offloading the task to the MEC server for processing is divided into three parts: data load upload, server processing, and result return. Because the server has sufficient computing power and the amount of processing result data is small enough, the result return delay and energy consumption are negligible in this embodiment.

[0078] Specifically, the data transmission delay from the task offloading from the terminal device to the server can be expressed as Among them, B i Indicates the amount of data to be uploaded for the task (unit: bit), r i It represents the wireless transmission rate from terminal device i to MEC server (unit: bit / s). The transmission energy consumption can be expressed as in, represents the transmission power of terminal device i (unit: W). The server processing delay can be expressed as Among them, f s Indicates the CPU frequency of the server (unit: Hz). The energy consumed by the terminal device while waiting for the server to complete the processing is the idle energy consumption, which can be expressed as Among them, Pi,I Indicates the power (unit: W) of terminal device i when it is in idle state (waiting for server processing).

[0079] By summing the transmission delay and server processing delay, the total offloading delay can be obtained as By summing the transmission energy consumption and idle energy consumption, the total unloading energy consumption can be obtained as

[0080] The total unloading delay and total unloading energy consumption are weighted and normalized to obtain the unloading cost.

[0081] Specifically, the unloading cost obtained after weight normalization can be expressed as in, represents the normalized weight of the total offloading delay, represents the normalized weight of the total energy consumption of unloading, and

[0082] It is understandable that this embodiment can normalize the weights and Unify latency (seconds) and energy consumption (joules) into dimensionless costs, allowing users to adjust priorities as needed by adjusting the normalized weights and Flexible adaptation to the latency and energy consumption requirements of different scenarios. For example, for latency-sensitive tasks (such as autonomous driving), Prioritize reducing latency, and set it for energy-sensitive tasks (such as IoT devices) Extend battery life.

[0083] Step 103: Establish an initial global optimization target based on the total cost of all tasks.

[0084] Specifically, in this embodiment, an initial global optimization objective is established based on the total cost of all tasks, which is expressed as:

[0085]

[0086] Where n represents the total number of terminal devices, represents the local execution cost of terminal device i, represents the offloading cost of terminal device i, a i ∈{0,1}, where 0 represents local execution and 1 represents offloading to the edge server for processing.

[0087] Step 104: Calculate the privacy entropy of the task and embed the privacy entropy of the task into the initial global optimization objective as the additional cost of offloading to obtain an updated global optimization objective.

[0088] Among them, the implementation method of the privacy entropy of the calculation task in this embodiment is similar to the implementation method of the privacy entropy of the calculation task in the prior art, and this embodiment will not be repeated here.

[0089] Specifically, in this embodiment, the privacy entropy of the task is embedded into the initial global optimization objective as the additional cost of offloading, and the updated global optimization objective is obtained, which is expressed as:

[0090]

[0091] Where n represents the total number of terminal devices, represents the local execution cost of terminal device i, represents the offloading cost of terminal device i, a i ∈{0,1}, where 0 represents local execution and 1 represents offloading to the edge server for processing. i Represents the privacy entropy of the task.

[0092] It is understandable that privacy entropy can quantify the privacy risk of a task (the more evenly the data type is distributed, the higher the privacy entropy). When a task is executed locally, no task data is transmitted, so the privacy entropy is 0. However, offloading tasks requires additional privacy leakage costs. Therefore, this embodiment directly embeds privacy entropy as an additional cost of offloading, which to a certain extent can encourage the agent to give priority to local execution in privacy-sensitive tasks. For example, in high-privacy tasks (such as medical data), the privacy entropy is large. Even if the offloading resource cost is low, the total cost may be higher than local execution, triggering a i = 0; in low-privacy tasks (such as environmental sensor data), privacy entropy is small, resource efficiency dominates the decision, and a may be chosen i =1.

[0093] Privacy protection in existing technologies (such as the independent optimization in Chinese patent CN116405975A) usually requires complex multi-objective processing. However, this embodiment creatively embeds privacy entropy as a direct cost item of the offloading decision into the optimization objective rather than an independent objective. That is, privacy protection does not need to be independently modeled but is endogenous to the optimization objective, thereby transforming the complex multi-objective problem into a single-objective reinforcement learning task, ensuring that privacy protection becomes the core constraint of the offloading decision, and realizing adaptive joint optimization of privacy protection and resource optimization without manual intervention in weight allocation, while significantly reducing the complexity of the algorithm.

[0094] Step 105: Use a single-agent reinforcement learning algorithm to find the optimal strategy for the updated global optimization objective.

[0095] Specifically, in this embodiment, a single-agent reinforcement learning algorithm is used to find the optimal strategy for the updated global optimization goal, including:

[0096] Define the state space as s = C all ;

[0097] Design the reward function so that the reward value is proportional to the cost reduction;

[0098] The update mechanism of the Q-learning algorithm is used to iteratively update the Q value until the preset convergence condition is reached, and the optimal strategy for the updated global optimization objective is obtained.

[0099] It can be understood that in this embodiment, by defining the state as the total cost of the current system (s=C all ) as the state input of Q-learning to reflect the comprehensive cost of all current tasks (i.e., global resource consumption and privacy risk), rather than the multi-dimensional position vector in the existing technology (such as X(t)∈R N×K×M ).

[0100] Specifically, the dynamic reward function designed in this embodiment can be expressed as:

[0101]

[0102] Among them, tc l represents the total cost of executing all tasks locally, and tc(s,a) represents the actual total cost of the current state to drive the agent to choose C all The action that minimizes (i.e., the optimal privacy-resource trade-off strategy). The larger the reward value, the better the action.

[0103] Specifically, the Q-value update mechanism of the Q-learning algorithm can be expressed as:

[0104]

[0105] Among them, α∈{0,1} represents the learning rate, which is used to control the rate at which new information covers old knowledge, and γ∈{0,1} represents the discount factor, which is used to weigh immediate rewards and long-term benefits.

[0106] For example, the preset convergence condition in this embodiment may be: when C all If the fluctuation amplitude is less than a threshold value δ (eg, δ=1%), convergence is determined, which is not limited in this embodiment.

[0107] Furthermore, in this embodiment, the single-agent reinforcement learning algorithm is used to find the optimal strategy for the updated global optimization goal, which also includes:

[0108] The action space is defined as a set of binary actions, each of which corresponds to a decision on local execution or offloading of a task.

[0109] Specifically, the action space in this embodiment can be expressed as a=[a1,a2,…,an ], where a i ∈{0,1}, indicating whether the task is executed locally (a i =0) or uninstall (a i =1), the action space size is 2 n (For example, the action space of 10 terminal devices is only 2 10 =1024 types).

[0110] This embodiment compresses the action space by defining an action set a∈{0,1} (binary offloading decision), simplifying the offloading decision to a 0 / 1 choice, avoiding the high computational overhead of the continuous action space. The binary action space is combined with the model-independent characteristics of Q-learning to significantly reduce the computational complexity, making it suitable for large-scale terminal scenarios.

[0111] In summary, this embodiment uses the cost function design with embedded privacy entropy to enable local and offload cost calculations to quantify latency, energy consumption and privacy risks, providing a unified optimizable goal for the reinforcement learning algorithm. Its deep collaboration with the privacy entropy model, Q-learning's single-agent decision framework and dynamic reward feedback mechanism creatively solves the technical problems of privacy protection-resource optimization trade-off difficulties, high algorithm complexity and low efficiency, and insufficient dynamic adaptability in the existing technology. The technical unit of this embodiment takes "extremely simplified modeling-dynamic feedback-real-time optimization" as the core link, transforming the trade-off between privacy protection and resource consumption from static multi-objective optimization to a dynamic single-objective reinforcement learning task, adapting to environmental changes through a real-time feedback mechanism, and achieving efficient global convergence. By compressing the state and binary action space, the feasibility of the algorithm in large-scale scenarios is ensured, achieving a breakthrough of "1+1>2" in algorithm efficiency, edge computing scenario adaptability and privacy protection strength.

[0112] Specifically, on the one hand, traditional methods usually need to independently optimize privacy and resource indicators through complex multi-objective optimization algorithms, resulting in a huge solution set and the need for complex multi-objective post-processing. At the same time, the existing technology relies on the energy state of the prey to control the algorithm stage switching, but the energy model is static attenuation and cannot respond to network changes (such as channel fluctuations) in real time. The weight g of privacy entropy and resource consumption needs to be manually preset and cannot be adaptively adjusted according to dynamic task requirements (such as sudden high-privacy tasks), resulting in the sacrifice of privacy protection in delay-sensitive scenarios. To this end, this embodiment dynamically balances latency and energy consumption between local execution and offloading costs through normalized weights to adapt to the needs of different scenarios, and directly embeds privacy entropy as the optimization target of offloading decisions into the cost function, transforming the multi-objective problem into a single-objective reinforcement learning task, greatly simplifying the decision logic, and ensuring that privacy protection becomes the core constraint of the offloading decision, realizing adaptive joint optimization of privacy protection and resource efficiency without manual intervention in weight allocation; on the other hand, the swarm intelligence algorithms commonly used in the existing technology (such as the improved OH The essence of the HO algorithm depends on the initial population distribution and parameter sensitivity. It explores the solution space through multi-agent collaboration and is prone to falling into local optimality. At the same time, the existing technology relies on the energy state of the prey to control the algorithm stage switching, but the energy model is static attenuation and cannot respond to network changes (such as channel fluctuations) in real time. In addition, it is necessary to maintain the population and complex mutation operations in the high-dimensional decision space. The computational overhead increases dramatically with the number of nodes, making it difficult to apply to large-scale node scenarios. To this end, this embodiment models the optimization target as a Markov decision process (MDP), dynamically adjusts the strategy through the real-time feedback mechanism of the reward function, and greatly improves the adaptability to dynamic environments. The single-agent decision mechanism of Q-learning is adopted. It only needs to maintain the Q table (state-action pair) and can directly learn the optimal action strategy through dynamic Q value iteration to achieve model-free optimization, avoiding the limitations of traditional heuristic algorithms that rely on precise mathematical modeling. There is no need to maintain the population or rely on initial parameters, avoiding the local optimal problem of the swarm intelligence algorithm caused by the initial population distribution, and through the binary action space and state s=C all Compressing high-dimensional network parameters into a single indicator significantly reduces computational complexity and accelerates convergence. This design is particularly suitable for edge computing scenarios with large-scale nodes, high dynamics, and strong privacy constraints.

[0113] In another embodiment of the present disclosure, the following design is added to further avoid local optimality. Specifically, the Q-value is iteratively updated using the update mechanism of the Q-learning algorithm until a preset convergence condition is reached, thereby obtaining the optimal strategy for the updated global optimization objective, including:

[0114] Step 201: Define a linear decay function of the exploration probability ∈(t), which can be expressed as:

[0115] ∈(t)=max(∈min ,∈0-ηt)

[0116] Among them, η represents the decay rate (for example, it decreases by 0.01 every 100 iterations), which is used to gradually reduce the exploration ratio, ∈ min Indicates the preset minimum exploration probability, which can cope with dynamic changes in the environment by retaining the minimum exploration rate.

[0117] Step 202: Generate a random number θ. If θ < ∈(t), randomly select action a. rand (Uniform sampling), if θ>∈(t), then select the action a with the maximum current Q value opt , which can be expressed as:

[0118] a opt =arg max a Q(s,a)

[0119] It can be understood that this embodiment either randomly selects an action (exploring unknown strategies) based on the exploration probability ∈, and the random action selection only requires O(1) complexity, which is significantly lower than the probability distribution sampling of the policy gradient method (such as the O(n) complexity of Softmax), or selects the action with the maximum current Q value (using the known optimal strategy), and the discrete action selection only requires O(n) comparison operations, which is significantly lower than the traditional continuous action space (such as DDPG) that requires calculating the policy gradient (O(n 2 ) complexity).

[0120] Step 203: Execute the selected action a, calculate the reward value R(s,a), and use the Q-learning algorithm to update the Q value corresponding to (s,a);

[0121] Step 204: Update the new exploration probability ∈(t) according to the linear decay function of the exploration probability.

[0122] Specifically, after each iteration, ∈ is dynamically adjusted according to ∈(t), and the exploration ratio is gradually reduced to converge to a stable strategy.

[0123] Step 205, repeating steps 202 to 204 until a preset convergence condition is reached, and obtaining an updated optimal strategy for the global optimization objective.

[0124] This embodiment uses a random exploration strategy to jump out of the current local high point of the Q value, with a high exploration rate in the early stage to cover the global action space, and a low exploration rate in the later stage to focus on the optimal strategy, taking into account both convergence speed and stability, achieving efficient global exploration, avoiding local traps, and the computational complexity is significantly lower than high-dimensional continuous optimization, further overcoming the technical problems of local optimality and low algorithm convergence efficiency in the existing technology.

[0125] In summary, this embodiment combines a dynamic reward feedback mechanism with a random exploration strategy, and Q-learning achieves an efficient balance between global exploration and local utilization in a discrete action space. Specifically, dynamic rewards reflect the impact of actions on the multi-objective system in real time, avoiding the deviation of manually designed reward functions. The random exploration strategy uses controllable randomness to escape from local optimality while maintaining low computational complexity. The decision dimension is reduced through the binary action space, avoiding gradient calculation and dimensionality disasters in high-dimensional continuous optimization. This significantly improves the algorithm convergence speed and the real-time response capability of the edge computing system while ensuring the privacy-resource joint optimization goal.

[0126] Please see the attached Figure 3 , Figure 3 A structural diagram of an edge computing privacy task offloading system based on reinforcement learning is provided as another embodiment of this specification.

[0127] like Figure 3 As shown, the edge computing privacy task offloading system based on reinforcement learning may include at least a data acquisition module 1, a cost calculation module 2, an optimization target establishment module 3, and an optimal strategy output module 4, wherein:

[0128] Data acquisition module 1, used to obtain computing task information generated by each terminal device and task offloading selection, the task offloading selection including task selection local execution or task selection offloading to edge server;

[0129] Cost calculation module 2, used to calculate the local execution cost or offloading cost of each task based on the computing task information generated by each terminal device and the task offloading selection, and to obtain the total cost of all tasks;

[0130] Optimization target establishment module 3 is used to establish the initial global optimization target based on the total cost of all tasks, calculate the privacy entropy of the task, and embed the privacy entropy of the task as the additional cost of offloading into the initial global optimization target to obtain the updated global optimization target;

[0131] The optimal strategy output module 4 is used to use the single-agent reinforcement learning algorithm to find the optimal strategy for the updated global optimization target.

[0132] It can be understood that the technical concept of the edge computing privacy task offloading system based on reinforcement learning provided in this embodiment is similar to the technical concept of the aforementioned edge computing privacy task offloading method based on reinforcement learning, and this embodiment will not be repeated here.

[0133] Please see the attached Figure 4 , Figure 4 This is a schematic diagram of an electronic device structure provided in another embodiment of this specification. Figure 4As shown, the electronic device 400 may include: at least one processor 401 , at least one network interface 404 , a user interface 403 , a memory 405 and at least one communication bus 402 .

[0134] The communication bus 402 may be used to implement connection and communication among the above components.

[0135] The user interface 403 may include buttons, and the optional user interface may also include a standard wired interface or a wireless interface.

[0136] The network interface 404 may include, but is not limited to, a Bluetooth module, an NFC module, a Wi-Fi module, and the like.

[0137] Among them, the processor 401 may include one or more processing cores. The processor 401 uses various interfaces and lines to connect the various parts of the entire electronic device 400, and executes various functions of the electronic device 400 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 405, and calling data stored in the memory 405. Optionally, the processor 401 can be implemented in at least one hardware form of DSP, FPGA, PLA. The processor 401 can integrate one or a combination of CPU, GPU and modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to handle wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 401, but may be implemented separately through a chip.

[0138] Among them, the memory 405 may include RAM or ROM. Optionally, the memory 405 includes a non-transitory computer-readable medium. The memory 405 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 405 may also be at least one storage device located away from the aforementioned processor 401. The memory 405 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application. The processor 401 can be used to call the application stored in the memory 405 and execute the method in one or more of the above-mentioned embodiments.

[0139] Another embodiment of the present disclosure provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of the aforementioned embodiments. If the components of the aforementioned electronic device are implemented as software functional units and used as independent downstream task predictions or tasks, they can be stored in the computer-readable storage medium.

[0140] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of this specification is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. Available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, digital versatile discs (DVDs)), or semiconductor media (eg, solid state disks (SSDs)).

[0141] The above description is merely an illustration of the preferred embodiments disclosed in this application and the technical principles employed. Those skilled in the art should understand that the scope of protection provided by this disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0142] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

Claims

1. The edge computing privacy task offloading method based on reinforcement learning is characterized by: The following steps are involved: Obtaining computing task information generated by each terminal device and task offloading options, wherein the task offloading options include selecting to execute the task locally or selecting to offload the task to an edge server; Based on the computing task information generated by each terminal device and the task offloading selection, the local execution cost or offloading cost of each task is calculated, and the total cost of all tasks is calculated; Establish an initial global optimization goal based on the total cost of all tasks; Calculate the privacy entropy of the task and embed the privacy entropy of the task as the additional cost of offloading into the initial global optimization objective to obtain the updated global optimization objective; A single-agent reinforcement learning algorithm is used to find the optimal strategy for the updated global optimization objective.

2. The edge computing privacy task offloading method based on reinforcement learning according to claim 1, characterized in that: Based on the computing task information and task offloading selection, the local execution cost or offloading cost of each task is calculated, including: When the task offloading option is local execution, the local execution delay and local execution energy consumption are calculated based on the information of each computing task; Perform weight normalization on the local execution latency and local execution energy consumption to obtain the local execution cost; When the task offloading selection is to offload the task to the edge server, based on the information of each computing task, the transmission delay, server processing delay, transmission energy consumption and idle energy consumption are calculated; The total offloading delay is obtained by summing the transmission delay and the server processing delay. The total offloading energy consumption is obtained by summing the transmission energy consumption and the idle energy consumption; The total unloading delay and total unloading energy consumption are weighted and normalized to obtain the unloading cost.

3. The edge computing privacy task offloading method based on reinforcement learning according to claim 1, characterized in that: Based on the total cost of all tasks, the initial global optimization objective is established, which is expressed as: Where n represents the total number of terminal devices, represents the local execution cost of terminal device i, represents the offloading cost of terminal device i, a i ∈{0,1}, where 0 represents local execution and 1 represents offloading to the edge server for processing.

4. The edge computing privacy task offloading method based on reinforcement learning according to claim 1, characterized in that: The privacy entropy of the task is embedded into the initial global optimization objective as the additional cost of offloading, and the updated global optimization objective is obtained, which is expressed as: Where n represents the total number of terminal devices, represents the local execution cost of terminal device i, represents the offloading cost of terminal device i, a i ={0,1}, where 0 represents local execution and 1 represents offloading to the edge server for processing. i Represents the privacy entropy of the task.

5. The edge computing privacy task offloading method based on reinforcement learning according to claim 4 is characterized in that: Use a single-agent reinforcement learning algorithm to find the optimal strategy for the updated global optimization objective, including: Define the state space as s = C all ; Design the reward function so that the reward value is proportional to the cost reduction; The update mechanism of the Q-learning algorithm is used to iteratively update the Q value until the preset convergence condition is reached, and the optimal strategy for the updated global optimization objective is obtained.

6. The edge computing privacy task offloading method based on reinforcement learning according to claim 5 is characterized in that: Use a single-agent reinforcement learning algorithm to find the optimal strategy for the updated global optimization objective, including: The action space is defined as a set of binary actions, each of which corresponds to a decision on local execution or offloading of a task.

7. The edge computing privacy task offloading method based on reinforcement learning according to claim 5, characterized in that: The Q-learning algorithm’s update mechanism is used to iteratively update the Q value until the preset convergence condition is reached, and the optimal strategy for the updated global optimization objective is obtained, including: Step 201, define a linear decay function of exploration probability ∈(t); Step 202: Generate a random number θ. If θ < ∈(t), randomly select action a. rand , if θ>∈(t), then select the action with the maximum current Q value; Step 203: Execute the selected action a, calculate the reward value R(s,a), and use the Q-learning algorithm to update the Q value corresponding to (s,a); Step 204: Update the new exploration probability ∈(t) according to the linear decay function of the exploration probability; Step 205, repeating steps 202 to 204 until a preset convergence condition is reached, and obtaining an updated optimal strategy for the global optimization objective.

8. The edge computing privacy task offloading system based on reinforcement learning is characterized by: include: A data acquisition module is used to obtain computing task information generated by each terminal device and task offloading options, wherein the task offloading options include selecting to execute the task locally or selecting to offload the task to an edge server; The cost calculation module is used to calculate the local execution cost or offload cost of each task based on the computing task information generated by each terminal device and the task offload selection, and to obtain the total cost of all tasks; The optimization target establishment module is used to establish the initial global optimization target based on the total cost of all tasks, calculate the privacy entropy of the task, and embed the privacy entropy of the task as the additional cost of offloading into the initial global optimization target to obtain the updated global optimization target; The optimal strategy output module is used to find the optimal strategy for the updated global optimization objective using a single-agent reinforcement learning algorithm.

9. An electronic device, characterized in that: including a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the edge computing privacy task offloading method based on reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the edge computing privacy task offloading method based on reinforcement learning as described in any one of claims 1 to 7 is implemented.

Images