Edge computing task unloading method and system based on federal deep reinforcement learning

Abstract

The invention discloses an edge computing task unloading method based on federal deep reinforcement learning, and the method comprises the steps: firstly, modeling the time delay, energy consumption and other information of a task generated by each user through formalized description; secondly, establishing a problem model for minimizing energy consumption and optimizing unloading uniformity under the energy consumption constraint by utilizing a knapsack problem theory; then, a deep reinforcement learning algorithm is used for solving the problem; and finally, through a federated learning mode, on the premise of ensuring the minimum energy consumption and the optimal unloading uniformity under the energy consumption constraint, protecting the privacy of the user. Based on the characteristics of limited and distributed resources of the edge server, the resource interaction process is modeled by using the knapsack problem model, and the resource utilization rate of the edge server is improved on the basis of reducing the expenditure of time delay, energy consumption and the like of a user.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing technology, and more specifically, relates to an edge computing task offloading method and system based on federated deep reinforcement learning. Background Art

[0002] With the rapid development of 5G technology, a large number of mobile smart devices have emerged, and the mobile applications deployed on them are becoming increasingly diversified, such as face recognition, telemedicine, unmanned driving, health monitoring, etc. However, due to the physical limitations of mobile devices in terms of mass, volume, and battery capacity, they often cannot meet the high requirements of these complex applications for computing power, storage space, and communication resources. In order to solve this challenge, edge computing came into being. By deploying computing and storage resources on edge servers near users, it realizes the localization of business processing and significantly reduces the amount of data that needs to be transmitted to remote cloud centers. This approach not only improves the user experience quality, but also greatly reduces network operating costs. Mobile edge computing has the advantages of low latency, high security, and low user energy consumption. Therefore, mobile edge computing technology has become one of the key technologies of 5G networks and has received widespread attention from academia and industry.

[0003] Existing edge computing task offloading methods include: first, static heuristic-based methods, which decide whether to offload tasks and to which edge node according to predefined rules (such as task size, computing requirements, etc.); second, dynamic optimization-based methods, which adjust offloading decisions based on real-time data and system status, such as predicting the best location for task offloading through machine learning models; third, game theory-based methods, which optimize task offloading by simulating competition and cooperation between user devices and edge servers. In this framework, each user device tries to minimize its own cost, while the edge server tries to maximize its resource utilization.

[0004] However, the current edge computing task offloading methods have some non-negligible defects:

[0005] First, existing static heuristic-based methods usually do not consider real-time changing network conditions and node states, which can lead to inflexible and poorly adaptable decisions, thus affecting system performance;

[0006] Second, although existing methods based on dynamic optimization can better adapt to environmental changes, they may require high computational overhead for continuous prediction and decision-making, increasing the energy consumption and latency of the system; mobile devices need to consume energy for data transmission when offloading tasks, and edge servers also consume energy when processing tasks. In addition, the processing capacity of edge servers is not unlimited. If the task offloading strategy is unreasonable, it will waste system resources and increase waiting time, resulting in increased energy consumption and latency for mobile devices and edge servers, affecting the energy efficiency and response speed of the system;

[0007] Third, existing methods based on dynamic optimization have high computational complexity in the task offloading decision process;

[0008] Fourth, existing game theory-based methods usually pay little attention to the uniformity of task offloading, and they themselves may require a lot of communication and coordination, which increases the complexity of the system and is not conducive to the robustness of the system: from the perspective of task offloading uniformity, due to the uneven distribution of users and task requirements, some edge servers may be overloaded while other servers are idle. This unbalanced resource utilization will affect the overall performance and user experience of the system, and when the system faces sudden traffic or server failures, a single server may be overloaded due to processing too many tasks, and it is impossible to disperse risks well, thus damaging the robustness of the system;

[0009] Fifth, existing static heuristic-based methods and game theory-based methods usually pay less attention to user privacy issues, which may lead to threats to user privacy. Summary of the invention

[0010] In response to the above defects or improvement needs of the prior art, the present invention provides an edge computing task offloading strategy based on federated deep reinforcement learning, which aims to solve the technical problem that the decision-making method based on static heuristics has poor adaptability in a dynamic environment, and the technical problem that the existing dynamic optimization-based method will increase the energy consumption and latency of mobile devices and edge servers, thereby affecting the energy efficiency ratio and response speed of the system, as well as the technical problem of poor system robustness of the existing game theory-based method, and the technical problem of high computational complexity in the task offloading decision-making process of the existing dynamic optimization-based method.

[0011] To achieve the above object, according to one aspect of the present invention, a method for offloading edge computing tasks based on federated deep reinforcement learning is provided, comprising the following steps:

[0012] (1) The mth edge server receives the parameters of the task to be processed and the terminal user parameters from the nth terminal user, and obtains the local processing delay of the nth terminal user based on the parameters and the terminal user parameters. Task uplink delay Processing delay of the mth edge server Local processing energy consumption Task uplink energy consumption And the processing energy consumption of the mth edge server Where n represents the serial number of the terminal user with pending tasks, and n∈[1, the total number of terminal users with pending tasks in the edge computing system N], m represents the serial number of the edge server, and m∈[1, the total number of edge servers in the edge computing system M].

[0013] (2) The local processing delay of the nth terminal user obtained by the mth edge server according to step (1) Task uplink delay Processing delay of the mth edge server Local processing energy consumption Task uplink energy consumption The processing energy consumption of the mth edge server Get the optimization target of the nth terminal user.

[0014] (3) The edge server with the lowest computing resource load among all edge servers will use the pre-trained global deep reinforcement learning model w I Sent to the nth end user.

[0015] (4) The nth terminal user inputs the task delay, task energy consumption, and edge server status obtained in step (2) into the pre-trained global model w obtained in step (3) I , to obtain the decision corresponding to the pending task of the terminal user (i.e., whether to uninstall the pending task and the edge server to which the pending task is to be uninstalled), and according to the decision corresponding to the pending task, transmit the pending task to the corresponding edge server terminal user for execution.

[0016] Preferably, the parameters of the task to be processed by the nth terminal user include the task size λ of the task to be processed by the nth terminal user. n , and the number of CPU cycles C required for the nth terminal user to process a 1-bit task; the terminal user parameters of the nth terminal user include the local CPU frequency f of the nth terminal user n ;

[0017] Preferably, the local processing delay of the nth terminal user is It is calculated according to the following formula

[0018]

[0019] The task uplink delay of the nth terminal user is calculated according to the following formula:

[0020]

[0021] where r n is the wireless communication rate at which the nth terminal user sends its pending tasks to the mth edge server, which can be expressed by the Shannon principle as Among them B i is the wireless channel bandwidth of the nth terminal user, P n is the transmission power of the nth terminal user; H n is the channel gain of the nth terminal user, which describes the attenuation and fading characteristics of the channel itself; q n is the white noise power.

[0022] The processing delay of the task of the nth terminal user at the mth edge server It is calculated according to the following formula:

[0023]

[0024] where f m The computing resources of the task assigned to the nth end user by the mth edge server.

[0025] Preferably, the local processing energy consumption of the nth terminal user is It is calculated using the following formula

[0026]

[0027] in K The effective switched capacitance of the chip for the nth end user.

[0028] The uplink energy consumption of the nth terminal user's task It is calculated using the following formula:

[0029]

[0030] Where P m It is the energy consumed per bit of data transmitted.

[0031] The task of the nth end user is processed on the mth edge server. It is calculated according to the following formula:

[0032]

[0033] Where Q m is the energy power consumed by executing the computation on the mth edge server.

[0034] Preferably, step (2) comprises the following sub-steps:

[0035] (2-1) The local processing delay of the nth terminal user obtained in step (1) Task uplink delay And the processing delay of the mth edge server Get the final delay cost T of the nth terminal user n , and according to the local processing energy consumption of the nth terminal user obtained in step (1) Task uplink energy consumption The processing energy consumption of the mth edge server Get the final energy consumption E of the nth end user n ; The final delay cost of the end user Final energy consumption

[0036] (2-2) The final delay overhead T of the nth terminal user obtained according to step (2-1) n and final energy consumption E n Get the optimization target P1 of the terminal user:

[0037]

[0038] The constraint C1 indicates that the final energy consumption of all end users cannot exceed the energy budget E budget Constraint C2 indicates that the computing offload of all end users cannot exceed the total computing resources of all edge servers, where E budget represents the energy budget, Represents the total computing resources of all edge servers;

[0039] (2-3) The final delay overhead T of the nth terminal user obtained according to step (2-1) n and final energy consumption E n Get the optimization target P2 of the mth edge server:

[0040]

[0041] Where L m is the total computing resource load of the mth edge server in the edge computing system, It is the average computing resource load of all edge servers in the edge computing system.

[0042] Preferably, the global deep reinforcement learning model w IIt is a global deep reinforcement learning model based on the Q-Transformer architecture, which is trained according to the following steps:

[0043] (3-1) Set iteration counter i = 0;

[0044] (3-2) Determine whether i is equal to the preset model iteration number I, if yes, the process ends, otherwise proceed to step (3-4);

[0045] (3-3) The edge server sends the global deep reinforcement learning model w to all end users i

[0046] (3-4) The edge server sends the status of all edge servers in the edge computing system to all end users; the global deep reinforcement learning model w i With a reward function R, we have:

[0047] R=-(α1T n +α2E n +α3L), R∈[-1,0]

[0048] Among them, the value range of α1, α2, and α3 is [0, 1]. L is the uniformity of task offloading in the edge computing system, and Where L m is the total computing resource load of the mth edge server in the edge computing system, It is the average computing resource load of all edge servers in the edge computing system.

[0049] (3-5) Each end user sends the received status of all edge servers and pending tasks to the global deep reinforcement learning model w i .

[0050] (3-6) Global deep reinforcement learning model w i The states of all edge servers and pending tasks from end users are processed to obtain a global deep reinforcement learning model and task offloading decision after parameter update.

[0051] (3-7) The terminal user performs an offloading operation on the pending task based on the task offloading decision obtained in step (3-6), and saves the global deep reinforcement learning model with updated parameters obtained in step (3-6) as the local model v corresponding to the terminal user. n .

[0052] (3-8) Determine whether the result of i%e is 0, where e is the preset number of model collection rounds. If so, proceed to step (3-9), otherwise proceed to step (3-10).

[0053] (3-9) The edge server collects the local models corresponding to all end users and aggregates all local models to obtain the final global deep reinforcement learning model w i , and save the global deep reinforcement learning model w i , the process ends.

[0054] (3-10) Set the iteration counter i=i+1 and return to step (3-2).

[0055] Preferably, the global deep reinforcement learning model w in step (3-6) i The states of all edge servers and pending tasks from the end users are processed to obtain the global deep reinforcement learning model with updated parameters. The specific process is as follows: first, the global deep reinforcement learning model w i The states of all edge servers from the end user and the tasks to be processed are normalized to obtain the feature tensor. Then, the Q value calculation method processes the feature tensor to obtain the predicted Q value. The feature tensor is processed by the exponential moving average EMA method to obtain the Q value of the next state. Then, the global deep reinforcement learning model w i The reward function R and the Q value of the next state are substituted into the Bellman equation for calculation to obtain the target Q value. Finally, the loss of the predicted Q value and the target Q value is calculated based on the temporal difference TD error method to update the weights of the Q-Transformer network, thereby obtaining a global deep reinforcement learning model after parameter update.

[0056] Preferably, the global deep reinforcement learning model w in step (3-6) i The state of all edge servers and pending tasks from end users are processed to obtain task offloading decisions. The specific process is as follows: first, the global deep reinforcement learning model w i The states of all edge servers from end users and pending tasks are feature normalized to obtain a feature tensor. Then, the feature tensor is processed based on its reward function R and the Q-value calculation method is used to obtain the Q-value. Finally, the task offloading decision is obtained based on the Q-value and the ε-greedy strategy.

[0057] According to another aspect of the present invention, there is provided an edge computing task offloading system based on federated deep reinforcement learning, comprising:

[0058] The first module is set at the mth edge server and is used to receive the parameters of the task to be processed and the terminal user parameters from the nth terminal user, and obtain the local processing delay of the nth terminal user according to the parameters and the terminal user parameters. Task uplink delay Processing delay of the mth edge server Local processing energy consumption Task uplink energy consumption And the processing energy consumption of the mth edge server Where n represents the serial number of the terminal user with pending tasks, and n∈[1, the total number of terminal users with pending tasks in the edge computing system N], m represents the serial number of the edge server, and m∈[1, the total number of edge servers in the edge computing system M].

[0059] The second module is set on the mth edge server and is used to calculate the local processing delay of the nth terminal user obtained by the first module. Task uplink delay Processing delay of the mth edge server Local processing energy consumption Task uplink energy consumption The processing energy consumption of the mth edge server Get the optimization target of the nth terminal user.

[0060] The third module is set to an edge server with the lowest computing resource load among all edge servers, and is used to convert the pre-trained global deep reinforcement learning model w I Sent to the nth end user.

[0061] The fourth module is set at the nth terminal user, and is used to input the task delay, task energy consumption and edge server status of the terminal user obtained by the second module into the pre-trained global model w obtained by the third module. I , to obtain the decision corresponding to the pending task of the terminal user (i.e., whether to uninstall the pending task and the edge server to which the pending task is to be uninstalled), and according to the decision corresponding to the pending task, transmit the pending task to the corresponding edge server terminal user for execution.

[0062] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0063] (1) Since the present invention adopts step (1), it fully considers the resources of the terminal device and the edge server, and updates the status according to the real-time situation, so it can solve the technical problem that the decision-making method based on the static heuristic method has poor adaptability in a dynamic environment;

[0064] (2) Since the present invention adopts step (2), the interaction process between the terminal user and the edge server is modeled as a knapsack problem, and an optimization target that takes into account latency, energy consumption, and unloading uniformity is obtained. Therefore, the technical problem that the existing dynamic optimization-based method increases the energy consumption and latency of the mobile device and the edge server, thereby affecting the energy efficiency ratio and response speed of the system can be solved;

[0065] (3) Since the present invention adopts step (3), it periodically collects the model parameters of the terminal users, aggregates and updates the global model parameters, and distributes the updated model to all terminal users. It also takes into account the uniformity of task offloading and utilizes the parameter interaction between the local device and the edge node to realize iterative training of the model until convergence. Therefore, it can solve the technical problem of poor system robustness existing in the existing game theory-based methods.

[0066] (4) Since the present invention adopts step (3) and step (4), through the distribution of the deep reinforcement learning model and related information, the terminal user can quickly analyze the state of the edge server, and thus quickly make a decision on whether to offload the task and to which edge server to offload. Therefore, it can solve the technical problem of poor user privacy existing in the existing static heuristic-based methods and game theory-based methods, and the technical problem of high computational complexity in the task offloading decision process existing in the existing dynamic optimization-based methods;

[0067] (5) The present invention has wide applicability and can effectively handle both dependent and non-dependent task scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of the edge computing task offloading method based on federated deep reinforcement learning of the present invention;

[0069] Figure 2 This is an environmental architecture diagram of the edge computing task offloading method based on federated deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0071] The main technical concept of the present invention is to jointly optimize resource allocation and computational offloading for computationally intensive applications in edge computing. There are multiple terminal users at the edge of the network to offload tasks. Each task can be divided into two cases and calculated at the local terminal or edge server at the same time. In order to stimulate the edge server to minimize latency under energy constraints and ensure the average degree of task offloading, based on the limited and distributed characteristics of edge server resources, the knapsack problem model is used to model the resource interaction process. On the basis of reducing user latency, energy consumption and other overheads, the utilization rate of edge server resources is improved. The environmental architecture diagram of the present invention is shown in the figure. Figure 2 shown.

[0072] The basic idea of ​​the present invention is, firstly, to model the information such as the delay and energy consumption of each user's task through formal description; secondly, to establish a problem model of minimizing energy consumption and optimizing unloading uniformity under energy consumption constraints by using the knapsack problem theory; then, to solve the problem by using a deep reinforcement learning algorithm; finally, to protect user privacy by means of federated learning while minimizing energy consumption and optimizing unloading uniformity under energy consumption constraints.

[0073] like Figure 1 As shown, the present invention provides an edge computing task offloading method based on federated deep reinforcement learning, the method comprising the following steps:

[0074] (1) The mth edge server receives the parameters of the task to be processed and the terminal user parameters from the nth terminal user, and obtains the local processing delay of the nth terminal user based on the parameters and the terminal user parameters. Task uplink delay Processing delay of the mth edge server Local processing energy consumption Task uplink energy consumption And the processing energy consumption of the mth edge server Where n represents the serial number of the terminal user with pending tasks, and n∈[1, the total number of terminal users with pending tasks in the edge computing system N], m represents the serial number of the edge server, and m∈[1, the total number of edge servers in the edge computing system M].

[0075] Specifically, the parameters of the task to be processed by the nth terminal user include the task size λ of the task to be processed by the nth terminal user. n , and the number of CPU cycles C required for the nth terminal user to process a 1-bit task; the terminal user parameters of the nth terminal user include the local CPU frequency f of the nth terminal user n ;

[0076] In other words, in this step, the local processing delay of the nth terminal user is It is calculated according to the following formula:

[0077]

[0078] The task uplink delay of the nth terminal user is calculated according to the following formula:

[0079]

[0080] where r n is the wireless communication rate at which the nth terminal user sends its pending tasks to the mth edge server, which can be expressed by the Shannon principle as Among them B i is the wireless channel bandwidth of the nth terminal user, P n is the transmission power of the nth terminal user; H n is the channel gain of the nth terminal user, which describes the attenuation and fading characteristics of the channel itself; q n is the white noise power.

[0081] The processing delay of the task of the nth terminal user at the mth edge server It is calculated according to the following formula:

[0082]

[0083] where f m The computing resources of the task assigned to the nth end user by the mth edge server.

[0084] Local processing energy consumption of the nth end user It is calculated using the following formula:

[0085]

[0086] Where κ is the effective switch capacitance, which depends on the chip structure of the nth end user.

[0087] The uplink energy consumption of the nth terminal user's task It is calculated using the following formula:

[0088]

[0089] Where P m It is the energy consumed per bit of data transmitted.

[0090] The task of the nth end user is processed on the mth edge server. It is calculated according to the following formula:

[0091]

[0092] Where Q m is the energy power consumed by executing the computation on the mth edge server.

[0093] The advantage of this step is that it defines the necessary task decision information, thereby efficiently providing a basis for model decisions in subsequent steps.

[0094] (2) The local processing delay of the nth terminal user obtained by the mth edge server according to step (1) Task uplink delay Processing delay of the mth edge server Local processing energy consumption Task uplink energy consumption The processing energy consumption of the mth edge server Get the optimization target of the nth terminal user.

[0095] This step includes the following sub-steps:

[0096] (2-1) The local processing delay of the nth terminal user obtained in step (1) Task uplink delay And the processing delay of the mth edge server Get the final delay cost T of the nth terminal user n , and according to the local processing energy consumption of the nth terminal user obtained in step (1) Task uplink energy consumption The processing energy consumption of the mth edge server Get the final energy consumption E of the nth end user n ;

[0097] Specifically, the final delay cost of the end user Final energy consumption

[0098] (2-2) The final delay overhead T of the nth terminal user obtained according to step (2-1) n and final energy consumption E n Get the optimization target P1 of the terminal user:

[0099]

[0100] The constraint C1 indicates that the final energy consumption of all end users cannot exceed the energy budget E budget Constraint C2 indicates that the computing offload of all end users cannot exceed the total computing resources of all edge servers, where E budget represents the energy budget, Represents the total computing resources of all edge servers;

[0101] (2-3) The final delay overhead T of the nth terminal user obtained according to step (2-1) n and final energy consumption E n Get the optimization target P2 of the mth edge server:

[0102]

[0103] Where L m is the total computing resource load of the mth edge server in the edge computing system, It is the average computing resource load of all edge servers in the edge computing system.

[0104] The advantage of this step is that it defines the optimization objectives for energy consumption, latency, and task offloading uniformity, thus providing a basis for the subsequent design of the reward function.

[0105] (3) The edge server with the lowest computing resource load among all edge servers will use the pre-trained global deep reinforcement learning model w I Sent to the nth end user.

[0106] The global deep reinforcement learning model w in this step I It is a global deep reinforcement learning model based on the Q-Transformer architecture, which is trained according to the following steps:

[0107] (3-1) Set iteration counter i = 0;

[0108] (3-2) Determine whether i is equal to the preset model iteration number I, if yes, the process ends, otherwise proceed to step (3-4);

[0109] Specifically, the preset number of model iterations I ranges from 1 to 20, preferably 10.

[0110] (3-3) The edge server sends the global deep reinforcement learning model w to all end users i

[0111] (3-4) The edge server sends the status of all edge servers in the edge computing system to all end users; the global deep reinforcement learning model w i With a reward function R.

[0112] R=-(α1T n +α2E n +α3L), R∈[-1,0]

[0113] Among them, the value range of α1, α2, and α3 is [0, 1]. L is the uniformity of task offloading in the edge computing system, and Where L m is the total computing resource load of the mth edge server in the edge computing system, It is the average computing resource load of all edge servers in the edge computing system.

[0114] (3-5) Each end user sends the received status of all edge servers and pending tasks to the global deep reinforcement learning model w i .

[0115] (3-6) Global deep reinforcement learning model w i The states of all edge servers and pending tasks from end users are processed to obtain a global deep reinforcement learning model and task offloading decision after parameter update.

[0116] In this step, the global deep reinforcement learning model w i The states of all edge servers and pending tasks from the end users are processed to obtain the global deep reinforcement learning model with updated parameters. The specific process is as follows: first, the global deep reinforcement learning model w i The states of all edge servers from the end user and the tasks to be processed are normalized to obtain a feature tensor. Then, the Q value calculation method processes the feature tensor to obtain a predicted Q value. The feature tensor is processed by the Exponential Moving Average (EMA) method to obtain the Q value of the next state. Then, the global deep reinforcement learning model w i The reward function R and the Q value of the next state are substituted into the Bellman equation for calculation to obtain the target Q value. Finally, the loss of the predicted Q value and the target Q value is calculated based on the temporal difference (TD) error method to update the weights of the Q-Transformer network, thereby obtaining a global deep reinforcement learning model after parameter update.

[0117] In this step, the global deep reinforcement learning model w i The state of all edge servers and pending tasks from end users are processed to obtain task offloading decisions. The specific process is as follows: first, the global deep reinforcement learning model w iThe states of all edge servers from end users and pending tasks are feature normalized to obtain a feature tensor. Then, the feature tensor is processed based on its reward function R and the Q-value calculation method is used to obtain the Q-value. Finally, the task offloading decision is obtained based on the Q-value and the ε-greedy strategy.

[0118] The advantages of the above steps (3-6) are that the exponential moving average (EMA) method can prevent the Q-Transformer network weights from being updated too quickly and ensure the stability of the Q value, the temporal difference error (TD) method can effectively guide the Transformer network weight update, and the ε-greedy strategy can enable the model to maintain a moderate exploration ability when making unloading decisions, so as to adapt to the changing and complex environment. These methods have further improved the stability, flexibility and robustness of the deep reinforcement learning model.

[0119] (3-7) The terminal user performs an offloading operation on the pending task based on the task offloading decision obtained in step (3-6), and saves the global deep reinforcement learning model with updated parameters obtained in step (3-6) as the local model v corresponding to the terminal user. n .

[0120] (3-8) Determine whether the result of i%e is 0, where e is the preset number of model collection rounds. If so, proceed to step (3-9), otherwise proceed to step (3-10).

[0121] Specifically, the preset model collection round number e ranges from 1 to 5, preferably 2.

[0122] (3-9) The edge server collects the local models corresponding to all terminal users, aggregates all local models to obtain the final global deep reinforcement learning model wi, and saves the global deep reinforcement learning model w i , the process ends.

[0123] (3-10) Set the iteration counter i=i+1 and return to step (3-2).

[0124] The advantage of this step is that by regularly collecting and aggregating local models of end users, it not only takes into account the characteristics of different users, but also enhances the generalization ability of the global model, which helps the model capture the key features of system operation and make more accurate decisions. More importantly, the training process of the model does not involve the transmission of the actual content of user data, which effectively protects the privacy of user data.

[0125] (4) The nth terminal user inputs the task delay, task energy consumption and edge server status of the terminal user obtained in step (2) into the pre-trained global model wI obtained in step (3) to obtain the decision corresponding to the pending task of the terminal user (i.e., whether to offload the pending task and the edge server to which the pending task is to be offloaded), and according to the decision corresponding to the pending task, transmits the pending task to the corresponding edge server for execution or executes it locally on the terminal user.

[0126] The advantage of this step is that it uses a pre-trained global model to make decisions. This method can make fast and accurate decisions based on real-time data. The system can respond to user needs more efficiently, improving overall performance and user experience.

[0127] In summary, the method of the present invention first combines the adaptive ability of deep reinforcement learning. The strategy can dynamically optimize task offloading decisions based on real-time network status and device resource conditions. This not only reduces latency, but also reduces energy consumption, making resource utilization more efficient.

[0128] Secondly, the method of the present invention balances the load of each edge node through an intelligent scheduling algorithm, avoids overload and idle resources, and improves the uniformity of task offloading. This method improves the stability and robustness of the system, and can maintain efficient operation as much as possible even in the event of sudden traffic or node failure.

[0129] In addition, the method of the present invention utilizes the distributed characteristics of federated learning to train the model on a local device and transmit only model parameters instead of data, thereby ensuring data security.

[0130] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for offloading edge computing tasks based on federated deep reinforcement learning, characterized in that: The following steps are involved: (1) The mth edge server receives the parameters of the task to be processed and the terminal user parameters from the nth terminal user, and obtains the local processing delay of the nth terminal user based on the parameters and the terminal user parameters. Task uplink delay Processing delay of the mth edge server Local processing energy consumption Task uplink energy consumption And the processing energy consumption of the mth edge server Where n represents the serial number of the terminal user with pending tasks, and n∈[1, the total number of terminal users with pending tasks in the edge computing system N], m represents the serial number of the edge server, and m∈[1, the total number of edge servers in the edge computing system M]. (2) The local processing delay of the nth terminal user obtained by the mth edge server according to step (1) Task uplink delay Processing delay of the mth edge server Local processing energy consumption Task uplink energy consumption The processing energy consumption of the mth edge server Get the optimization target of the nth terminal user. (3) The edge server with the lowest computing resource load among all edge servers will use the pre-trained global deep reinforcement learning model w I Sent to the nth end user. (4) The nth terminal user inputs the task delay, task energy consumption, and edge server status obtained in step (2) into the pre-trained global model w obtained in step (3) I , to obtain the decision corresponding to the pending task of the terminal user (i.e., whether to uninstall the pending task and the edge server to which the pending task is to be uninstalled), and according to the decision corresponding to the pending task, transmit the pending task to the corresponding edge server terminal user for execution.

2. The edge computing task offloading method based on federated deep reinforcement learning according to claim 1 is characterized in that: The parameters of the task to be processed by the nth terminal user include the task size λ of the task to be processed by the nth terminal user n , and the number of CPU cycles C required for the nth terminal user to process a 1-bit task; the terminal user parameters of the nth terminal user include the local CPU frequency f of the nth terminal user n。 3. The edge computing task offloading method based on federated deep reinforcement learning according to claim 1 or 2, characterized in that: Local processing delay of the nth end user It is calculated according to the following formula: The task uplink delay of the nth terminal user is calculated according to the following formula: where r n is the wireless communication rate at which the nth terminal user sends its pending tasks to the mth edge server, which can be expressed by the Shannon principle as Among them B i is the wireless channel bandwidth of the nth terminal user, P n is the transmission power of the nth terminal user; H n is the channel gain of the nth terminal user, which describes the attenuation and fading characteristics of the channel itself; q n is the white noise power. The processing delay of the task of the nth terminal user at the mth edge server It is calculated according to the following formula: where f m The computing resources of the task assigned to the nth end user by the mth edge server.

4. The method for offloading edge computing tasks based on federated deep reinforcement learning according to any one of claims 1 to 3, characterized in that: Local processing energy consumption of the nth end user It is calculated using the following formula: Where κ is the effective switching capacitance of the chip used by the nth end user. The uplink energy consumption of the nth terminal user's task It is calculated using the following formula: Where P m It is the energy consumed per bit of data transmitted. The task of the nth end user is processed on the mth edge server. It is calculated according to the following formula: Where Q m is the energy power consumed by executing the computation on the mth edge server.

5. The method for offloading edge computing tasks based on federated deep reinforcement learning according to claim 4 is characterized in that: Step (2) includes the following sub-steps: (2-1) The local processing delay of the nth terminal user obtained in step (1) Task uplink delay And the processing delay of the mth edge server Get the final delay cost T of the nth terminal user n , and according to the local processing energy consumption of the nth terminal user obtained in step (1) Task uplink energy consumption The processing energy consumption of the mth edge server Get the final energy consumption E of the nth end user n ; The final delay cost of the end user Final energy consumption (2-2) The final delay overhead T of the nth terminal user obtained according to step (2-1) n and final energy consumption E n Get the optimization target P1 of the terminal user: The constraint C1 indicates that the final energy consumption of all end users cannot exceed the energy budget E budget Constraint C2 indicates that the computing offload of all end users cannot exceed the total computing resources of all edge servers, where E budget represents the energy budget, Represents the total computing resources of all edge servers; (2-3) The final delay overhead T of the nth terminal user obtained according to step (2-1) n and final energy consumption E n Get the optimization target P2 of the mth edge server: Where L m is the total computing resource load of the mth edge server in the edge computing system, It is the average computing resource load of all edge servers in the edge computing system.

6. The method for offloading edge computing tasks based on federated deep reinforcement learning according to claim 5 is characterized in that: Global deep reinforcement learning model w I It is a global deep reinforcement learning model based on the Q-Transformer architecture, which is trained according to the following steps: (3-1) Set iteration counter i = 0; (3-2) Determine whether i is equal to the preset model iteration number I, if yes, the process ends, otherwise proceed to step (3-4); (3-3) The edge server sends the global deep reinforcement learning model w to all end users i (3-4) The edge server sends the status of all edge servers in the edge computing system to all end users; The global deep reinforcement learning model w i With a reward function R, we have: R=―(α1T n +α2E n +α3L),R∈[―1,0] Among them, the value range of α1, α2, and α3 is [0, 1]. L is the uniformity of task offloading in the edge computing system, and Where L m is the total computing resource load of the mth edge server in the edge computing system, It is the average computing resource load of all edge servers in the edge computing system. (3-5) Each end user sends the received status of all edge servers and pending tasks to the global deep reinforcement learning model w i . (3-6) Global deep reinforcement learning model w i The states of all edge servers and pending tasks from end users are processed to obtain a global deep reinforcement learning model and task offloading decision after parameter update. (3-7) The terminal user performs an offloading operation on the pending task based on the task offloading decision obtained in step (3-6), and saves the global deep reinforcement learning model with updated parameters obtained in step (3-6) as the local model v corresponding to the terminal user. n . (3-8) Determine whether the result of i%e is 0, where e is the preset number of model collection rounds. If so, proceed to step (3-9), otherwise proceed to step (3-10). (3-9) The edge server collects the local models corresponding to all end users and aggregates all local models to obtain the final global deep reinforcement learning model w i , and save the global deep reinforcement learning model w i , the process ends. (3-10) Set the iteration counter i=i+1 and return to step (3-2).

7. The method for offloading edge computing tasks based on federated deep reinforcement learning according to claim 6, characterized in that: The global deep reinforcement learning model w in steps (3-6) i The states of all edge servers and pending tasks from the end users are processed to obtain the global deep reinforcement learning model with updated parameters. The specific process is as follows: first, the global deep reinforcement learning model w i The states of all edge servers from the end user and the tasks to be processed are normalized to obtain the feature tensor. Then, the Q value calculation method processes the feature tensor to obtain the predicted Q value. The feature tensor is processed by the exponential moving average EMA method to obtain the Q value of the next state. Then, the global deep reinforcement learning model w i The reward function R and the Q value of the next state are substituted into the Bellman equation for calculation to obtain the target Q value. Finally, the loss of the predicted Q value and the target Q value is calculated based on the temporal difference TD error method to update the weights of the Q-Transformer network, thereby obtaining a global deep reinforcement learning model after parameter update.

8. The method for offloading edge computing tasks based on federated deep reinforcement learning according to claim 7 is characterized in that: The global deep reinforcement learning model w in steps (3-6) i The state of all edge servers and pending tasks from end users are processed to obtain task offloading decisions. The specific process is as follows: first, the global deep reinforcement learning model w i The states of all edge servers from end users and pending tasks are feature normalized to obtain a feature tensor. Then, the feature tensor is processed based on its reward function R and the Q-value calculation method is used to obtain the Q-value. Finally, the task offloading decision is obtained based on the Q-value and the ε-greedy strategy.

9. An edge computing task offloading system based on federated deep reinforcement learning, characterized in that: include: The first module is set at the mth edge server and is used to receive the parameters of the task to be processed and the terminal user parameters from the nth terminal user, and obtain the local processing delay of the nth terminal user according to the parameters and the terminal user parameters. Task uplink delay Processing delay of the mth edge server Local processing energy consumption Task uplink energy consumption And the processing energy consumption of the mth edge server Where n represents the serial number of the terminal user with pending tasks, and n∈[1, the total number of terminal users with pending tasks in the edge computing system N], m represents the serial number of the edge server, and m∈[1, the total number of edge servers in the edge computing system M]. The second module is set on the mth edge server and is used to calculate the local processing delay of the nth terminal user obtained by the first module. Task uplink delay Processing delay of the mth edge server Local processing energy consumption Task uplink energy consumption The processing energy consumption of the mth edge server Get the optimization target of the nth terminal user. The third module is set to an edge server with the lowest computing resource load among all edge servers, and is used to convert the pre-trained global deep reinforcement learning model w I Sent to the nth end user. The fourth module is set at the nth terminal user, and is used to input the task delay, task energy consumption and edge server status of the terminal user obtained by the second module into the pre-trained global model w obtained by the third module. I , to obtain the decision corresponding to the pending task of the terminal user (i.e., whether to uninstall the pending task and the edge server to which the pending task is to be uninstalled), and according to the decision corresponding to the pending task, transmit the pending task to the corresponding edge server terminal user for execution.

Images