Edge computing task unloading and resource allocation method and device

Abstract

The invention provides a deep learning reasoning method based on time delay estimation. The method comprises the following steps: acquiring task data of terminal equipment and an edge server in an edge-terminal task scene; the task parameter comprises a task unloading proportion of the terminal equipment; based on the task data, task unloading parameters of the edge-terminal task scene and a target function and constraint conditions of an unloading task are calculated; and inputting the task unloading parameter, the objective function and the conditional constraint into a near-end strategy optimization model for training, and outputting a task unloading strategy and a resource allocation strategy of the edge-terminal task scene by the near-end strategy optimization model after the near-end strategy optimization model completes strategy convergence or reaches a preset training step number. According to the method, the calculation unloading and resource allocation strategy in the end-edge collaborative scene is realized through the PPO improved model, and the problems that the existing calculation unloading has insufficient attention to the load balancing level of the edge node and does not pay attention to the time sequence characteristics of the tasks in the actual scene are solved.

Description

Technical Field

[0001] The present invention belongs to the field of multi-access edge computing technology, and in particular relates to an edge computing offloading strategy and resource allocation method and device based on an improved Proximal Policy Optimization (PPO) algorithm. Background Art

[0002] With the rapid development of wireless communication technology and the sharp increase in the number of mobile devices, the amount of data generated at the edge of the network has shown an explosive growth trend. Various computationally intensive and latency-sensitive tasks continue to emerge, bringing unprecedented challenges to the computing power and battery capacity of mobile devices. Edge computing deploys edge servers with strong computing power near mobile devices to provide computing resources for mobile devices while significantly shortening data transmission delays and improving the quality of service (QoS). Computation offloading, as one of the core technologies of MEC, expands the computing power of mobile devices by transferring computing tasks to edge servers close to users for execution. This not only meets the application's requirements for low latency and low energy consumption, but also optimizes the user experience. In the process of computation offloading, it is necessary not only to consider the location and proportion of task offloading, but also to fully consider environmental factors such as bandwidth, channels, and server computing resources.

[0003] Although edge computing can expand the computing power of mobile devices through computational offloading technology, it still faces the problems of slow response speed, high device energy consumption, and relatively single adaptation scenario. Existing research mainly solves single goals through heuristic algorithms or traditional deep reinforcement learning algorithms, pays insufficient attention to resource allocation in computational offloading scenarios, and does not pay attention to the load balancing of edge servers. Summary of the invention

[0004] In view of the above problems, the present invention designs an improved deep reinforcement learning algorithm suitable for computing offloading and resource allocation scenarios, and realizes single-objective and multi-objective optimization by setting different delays and energy consumption coefficients to solve the shortcomings of existing research. The edge computing task offloading and resource allocation method of the present invention includes: an initialization step, obtaining the task data of the terminal device and the edge server in the edge-terminal task scenario; the task parameters include the task offloading ratio of the terminal device; a data acquisition step, based on the task data, calculating the task offloading parameters of the edge-terminal task scenario, and the objective function and constraints of the offloading task; a strategy generation step, inputting the task offloading parameters, the objective function and the conditional constraints into the proximal policy optimization model for training, and when the proximal policy optimization model completes the strategy convergence or reaches the predetermined number of training steps, the proximal policy optimization model outputs the task offloading strategy and resource allocation strategy of the edge-terminal task scenario.

[0005] Furthermore, the proximal strategy optimization model includes an Actor action network and a Critic evaluation network; the Actor action network is a deep learning network with a multi-layer architecture, which includes an input layer, a long short-term memory network layer, a fully connected layer consisting of a Tanh activation layer, and an output layer.

[0006] Furthermore, the Critic evaluation network limits the probability ratio r(θ) of the new and old strategies during the training process through the Clip function;

[0007]

[0008] Among them, ε is the cutoff constant, v is the optimization constant, π θ (·) indicates a new operation strategy, Indicates the old operation strategy, a t Indicates running action, s t Indicates the state of the environment.

[0009] Furthermore, the objective function and its constraints satisfy:

[0010]

[0011] Among them; ω1 is the delay weight, ω2 is the energy consumption weight, ω3 is the load balancing weight, and the total delay of the task is the local execution delay of the task on the nth terminal device, is the edge delay of the task output to the edge server for execution, and the total energy consumption of the task is the energy consumption of the task processed locally on the nth terminal device, is the energy consumption of the task transmitted from the nth terminal device to the edge server, LB is the load balancing degree of the edge server, L m is the CPU usage of the mth edge server, L avg Indicates the average CPU utilization of edge servers. M is the number of edge servers in the edge-terminal task scenario, and constraint C1 represents the task offloading ratio α. n Constraint C2 means satisfying the bandwidth allocation ratio b n , constraint C3 means that the computing resource allocation ratio k is satisfied n, constraint C4 means that the total bandwidth W is limited to the bandwidth ratio allocated to the terminal device, constraint C5 means that the computing power of the edge server is limited to the computing resource ratio allocated to the terminal device, and constraint C6 means that the total delay of each task is satisfied. Subject to the maximum acceptable delay τ of the task n restrictions.

[0012] The present invention also proposes an edge computing task offloading and resource allocation device, including: an initialization module, used to obtain task data of terminal devices and edge servers in an edge-terminal task scenario; the task parameters include the offloading ratio of the generated tasks of the terminal device; a data acquisition module, used to calculate the task offloading parameters of the edge-terminal task scenario based on the task data, and the objective function and constraints of the offloading task; a strategy generation module, used to input the task offloading parameters, the objective function and the conditional constraints into a proximal policy optimization model for training, and when the proximal policy optimization model completes policy convergence or reaches a predetermined number of training steps, the proximal policy optimization model outputs the task offloading strategy and resource allocation strategy of the edge-terminal task scenario.

[0013] Furthermore, the proximal strategy optimization model includes an Actor action network and a Critic evaluation network; the Actor action network is a deep learning network with a multi-layer architecture, which includes an input layer, a long short-term memory network layer, a fully connected layer consisting of a Tanh activation layer, and an output layer.

[0014] Furthermore, the Critic evaluation network limits the probability ratio r(θ) of the new and old strategies during training through the Clip function:

[0015]

[0016] Among them, ε is the cutoff constant, v is the optimization constant, π θ (·) indicates a new operation strategy, Indicates the old operation strategy, a t Indicates running action, s t Indicates the state of the environment.

[0017] Furthermore, the objective function and its constraints satisfy:

[0018]

[0019]

[0020] Among them; ω1 is the delay weight, ω2 is the energy consumption weight, ω3 is the load balancing weight, and the total delay of the task is the local execution delay of the task on the nth terminal device, is the edge delay of the task output to the edge server for execution, and the total energy consumption of the task is the energy consumption of the task processed locally on the nth terminal device, is the energy consumption of the task transmitted from the nth terminal device to the edge server, LB is the load balancing degree of the edge server, L m is the CPU usage of the mth edge server, L avg Indicates the average CPU utilization of edge servers. M is the number of edge servers in the edge-terminal task scenario, and constraint C1 represents the task offloading ratio α. n , constraint C2 means satisfying bandwidth allocation ratio b n , constraint C3 means that the computing resource allocation ratio k is satisfied n , constraint C4 means that the total bandwidth W is limited to the bandwidth ratio allocated to the terminal device, constraint C5 means that the computing power of the edge server is limited to the computing resource ratio allocated to the terminal device, and constraint C6 means that the total delay of each task is satisfied. Subject to the maximum acceptable delay τ of the task n restrictions.

[0021] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that when the computer-executable instructions are executed, the edge computing task offloading and resource allocation method as described above is implemented.

[0022] The present invention also proposes an electronic device, comprising the edge computing task offloading and resource allocation device as described above.

[0023] The improved model based on the PPO algorithm of the present invention realizes the computing offloading and resource allocation method in the end-edge collaboration scenario, which solves the problem that the existing computing offloading does not pay enough attention to the load balancing level of edge nodes and does not pay attention to the time series characteristics of tasks in actual scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the Actor network architecture of the present invention that introduces the LSTM network.

[0025] Figure 2 It is a schematic diagram of the IPPO algorithm framework of the present invention.

[0026] Figure 3 It is a schematic diagram of the system model of the task offloading scenario under the end-edge collaborative architecture of the present invention.

[0027] Figure 4 It is a flow chart of calculation offloading of the present invention.

[0028] Figure 5 It is a data calculation flow chart of the present invention.

[0029] Figure 6 It is the flow chart of IPPO algorithm of the present invention.

[0030] Figure 7 It is a trend chart of the cumulative reward value under different learning rate combinations of the IPPO algorithm of the present invention.

[0031] Figure 8 It is a line graph of the QoS value of each algorithm under different numbers of terminal devices.

[0032] Fig. 9 It is a bar chart of the total delay of each algorithm under different numbers of terminal devices.

[0033] Fig.10 It is a bar chart of the total energy consumption of each algorithm under different numbers of terminal devices.

[0034] Fig.11 It is a schematic diagram of the edge computing task offloading and resource allocation device of the present invention.

[0035] Fig.12 It is a schematic diagram of an electronic device of the present invention.

[0036] Fig.13 It is a schematic diagram of the hardware structure of an electronic device of the present invention.

[0037] Wherein, the accompanying drawings are marked as follows:

[0038] 100: electronic device 10: pre-processing module

[0039] 20: Data processing module 30: Strategy generation module

[0040] S1, S2, S3, S4, S2.1, S2.2, S2.3, S2.4, S2.5, S4.1, S4.2, S4.3, S4., S45, S4.6, S4.7, S4.8, S4.9, S4.10: Steps DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation method described herein is only used to explain the present invention and is not used to limit the present invention.

[0042] The existing research on computational offloading strategies in resource-constrained scenarios pays little attention to the load balancing of edge nodes and is not comprehensive, which leads to unbalanced task load distribution of some edge nodes and affects the overall stability of the system. At the same time, the existing computational offloading strategies for deep reinforcement learning are not optimized for the problems existing in the offloading scenarios, resulting in the execution speed and stability of the algorithm being difficult to meet the scenarios of a large number of mobile devices.

[0043] In view of the fact that the existing computing offloading does not pay enough attention to the load balancing level of edge nodes and does not attach importance to the time series characteristics of tasks in actual scenarios, the present invention proposes an improved model based on the PPO algorithm to realize computing offloading and resource allocation method in the end-edge collaboration scenario.

[0044] The computing offloading and resource allocation method of the present invention takes the load balancing level of the edge server as the optimization target of the offloading strategy. The existing computing offloading strategy mainly focuses on latency and energy consumption, and pays less attention to the load status level of the edge server. This may cause a certain edge server to receive multiple computing requests from mobile devices at the same time, while the server with relatively less computing resources is often idle. This situation is not conducive to collaborative offloading between edge layers. The present invention analyzes the load balancing of the computing resources of the edge server through the CPU occupancy rate of the edge server to improve the stability of the offloading system. By introducing the load balancing level of the edge server into the optimization target, an objective function including latency, energy consumption and load balancing is constructed to improve the scalability and fault tolerance of the offloading system. Compared with a single optimization target, the system's resource utilization and adaptability to dynamic load changes are significantly improved. The problem that the existing computing offloading algorithm does not pay enough attention to the load of hardware devices is solved.

[0045] The LSTM network is also used in the policy network to enhance the adaptability to tasks with time series characteristics and dynamically changing states. In a multi-edge server system, especially in scenarios involving a large number of terminal devices, the state of the environment is always changing dynamically. This dynamic nature makes it extremely difficult to accurately obtain this information, especially considering that these data usually exhibit certain time series characteristics. LSTM can deeply mine the historical information in time series data and more accurately estimate the current state by capturing the dependencies in the sequence. In this way, by introducing LSTM, we can not only make full use of the advantages of LSTM in processing time series data to improve the performance of the PPO algorithm in a dynamic environment, but also effectively deal with some observable data, solve the problems caused by incomplete data, and improve the stability and performance of the strategy.

[0046] In addition, the computational offloading and resource allocation method of the present invention expands the probability ratio of the algorithm clipping function (Clip) under the same hyperparameters. The original PPO algorithm has the problem of slow learning speed and instability during the update process, of which the limitation of the Clip function is one of the main reasons. This paper improves the Clip function of the algorithm by adjusting the update method of the probability ratio of the new and old strategies under the same truncation constant ε setting, expanding its range, enhancing the learning of the truncation critical range, and limiting the probability ratio of the new and old strategies to The above improvements not only improve the learning rate of the PPO algorithm, but also enhance its learning stability. And on the premise of ensuring the correctness of the strategy, faster convergence and better performance are achieved. The improved algorithm has a larger learning range under the same hyperparameters and accelerates the learning rate of the algorithm. The problems of slow update and instability of the original algorithm are solved.

[0047] In a first embodiment of the present invention, an edge computing task offloading and resource allocation architecture based on an improved PPO algorithm is provided, including:

[0048] 1. Build an edge-to-end task offloading scenario involving multiple terminal devices and multiple edge servers, and obtain the data and offloading model of tasks generated by the terminal devices;

[0049] (1) Constructing a system model with multiple edge servers:

[0050] The system includes M edge servers, represented by the set M = {1, 2, 3, ..., M}; N terminal devices are represented by the set N = {1, 2, 3 ..., N}; the computing resources of the mth edge server are represented by Indicates that the computing resources of the nth terminal device are used express.

[0051] (2) Data and offloading model of tasks generated by terminal devices:

[0052] Each terminal device generates a computation task at the beginning of a time slot. Use a triple to represent {D n ,C n ,τ n}. n Indicates the data size of the task on the mobile terminal device n; C n is the number of CPU cycles required for the task, τ n is the maximum delay requirement of the task.

[0053] The offloading ratio of tasks generated by terminal device n is α n =[0,1], n∈N means that the remaining proportion of tasks are executed locally on the device.

[0054] 2. Calculate data such as time delay, energy consumption and system load balance;

[0055] (1) Constructing a time model for tasks

[0056] According to Shannon's theorem, the transmission speed between mobile terminal devices and edge servers can be expressed as:

[0057]

[0058] Among them, b n Indicates the ratio of bandwidth allocated to terminal devices to total bandwidth, b n ∈[0,1], and satisfies W represents the total bandwidth of the uplink; P n Power for mobile terminal transmission task; g n is the channel gain between the edge and terminal devices; N0 is the noise power spectral density.

[0059] The time it takes for the terminal device to transmit task data to the edge server can be expressed as:

[0060]

[0061] Among them, α n is the ratio of offloading to edge servers.

[0062] The total latency of local execution on the terminal device and execution on the edge node is:

[0063]

[0064] Among them, 1-α n Indicates the proportion of local execution, Indicates the CPU frequency of mobile terminal device n, C n is the total number of CPU cycles for the task, k n The server CPU frequency assigned to the task. Total calculation frequency for the server.

[0065] The total latency of a task is the maximum of the local execution latency and the edge latency, which can be expressed as:

[0066]

[0067] in, Indicates the execution time of the task part, Indicates the time that the task is processed at the edge (2) Constructs the task energy consumption model

[0068] The energy consumption of task processing locally on the mobile device and the energy consumption of transmitting to the edge node are:

[0069]

[0070] Where μ is a constant related to the CPU structure of the edge device and represents the effective switching capacitance, P n It represents the power of mobile device n when transmitting data.

[0071] The total energy consumption of the task is the sum of the energy consumption generated by the transmission process and the local execution energy consumption, which can be expressed as

[0072]

[0073] in, is the computing energy consumption of mobile devices, is the energy consumption of task transmission to edge nodes.

[0074] (3) Building a system load balancing model

[0075] Because the system contains multiple edge servers, in order to effectively describe the load balancing of all devices in the edge layer, this model uses the standard deviation of the CPU occupancy of all edge servers to represent the load balancing degree of the system. The calculation method is as follows:

[0076]

[0077] Among them, the CPU occupancy rate of the mth edge server can be expressed as L m , L avg It indicates the average CPU utilization of edge servers, and LB indicates the load balancing level of the system.

[0078] 3. Describe the partial task offloading model in the scenario of multiple edge servers in each time slot, and set the objective function and conditional constraints of the system;

[0079] The system optimization goal is to minimize the weighted sum of total delay, total energy consumption and edge layer load balancing value. The mathematical expression of this problem is as follows:

[0080]

[0081]

[0082] Among them, ω1, ω2 and ω3 are the weights of delay, energy consumption and load balancing respectively. Considering that the load balancing value is relatively small, in order to balance the impact of load balancing in the total system cost, it is proved through experiments that setting its parameter ω3 between 1 and 2 can more effectively reflect the role of load balancing value on the total system cost. Therefore, the load balancing weight parameter ω3 is set to (1,2), and the sum of the weights of delay and energy consumption is still 1. Constraints C1, C2 and C3 are the unloading ratio, bandwidth allocation ratio and computing resource allocation ratio of terminal devices respectively, and they should all be in the range of 0 to 1. At the same time, C4 indicates that the bandwidth ratio allocated by the system to the terminal device should not exceed the limit of its total bandwidth W to ensure the reasonable use of bandwidth resources. C5 indicates that the computing resource ratio allocated by the server to the terminal device cannot exceed its own computing power to avoid overload operation. C6 indicates that the total delay of each task cannot be greater than the maximum acceptable delay of the task. These constraints jointly ensure the stability and efficiency of the system operation.

[0083] 4. An improved algorithm based on the proximal policy optimization algorithm, namely the IPPO algorithm, is proposed to minimize the weighted sum of latency, energy consumption and load balancing level, and save the model parameters of the algorithm; including:

[0084] (1) Markov decision process

[0085] The Markov decision process is represented by a triple model {S, A, R} consisting of three core elements: state space (S), action space (A) and reward function (R).

[0086] The state space can be represented as s t ={D n (t),C n (t),τ n (t),r n (t),L n (t)}, respectively D n (t) represents the data volume of the task generated by device n; C n (t) represents the computational effort of the task generated by device n; τ n (t) represents the maximum tolerable delay of the task; r n (t) represents the transmission speed; L n (t) is the CPU utilization of the edge server.

[0087] The action space includes: the offloading decision of the mobile terminal device, and the computing and bandwidth resources allocated by the edge server to the terminal device. It can be expressed as A = {α1, α2, ..., α N ; G1, G2, ..., G N ; b1, b2, ..., b N ; k1, k2, ..., kN}. αx represents the uninstall ratio of the nth terminal device; G n Indicates the edge server object to which the task is to be offloaded; b n k is the bandwidth ratio allocated to the terminal device by the edge server; n Indicates the proportion of computing resources allocated to terminal devices.

[0088] The reward function is to minimize the weighted sum of system latency, energy consumption, and edge server load balancing level.

[0089] It can be expressed as

[0090] (2) Improved proximal strategy optimization IPPO algorithm

[0091] The conventional PPO algorithm uses the probability ratio to quantify the difference between the new strategy and the old strategy in each iteration based on the samples collected by the current strategy, and updates the strategy parameters accordingly. By finely controlling the update amplitude, the PPO algorithm ensures the stability of the training process and avoids drastic fluctuations in the strategy update, thereby improving the convergence speed and performance of the algorithm. r(θ)=π θ (a t |s t ) / π θ' (a t |s t ) indicates that the new and old strategies take action a t The ratio of probabilities reflects the degree of change in action selection of the new strategy relative to the old strategy.

[0092] The core of this type of algorithm is to generate the action of the current state through the Actor action network, save a batch of samples to the experience replay pool, and then calculate the Q value of the current action through the Critic evaluation network. The two modules use reverse gradient propagation to continuously update and optimize network parameters.

[0093] Improvement 1: Introduce the long short-term memory network (LSTM) in the Actor network to make full use of its ability to process time series data. LSTM can deeply mine the historical information in the time series data and more accurately estimate the current state by capturing the dependencies in the sequence. The state vector of time slot t is passed as input to the LSTM network, and then the state vector processed by the LSTM network is further input into the network of the PPO algorithm. The Actor action network is a deep learning network composed of a 5-layer architecture, including an input layer, an LSTM layer, a fully connected layer consisting of 2 layers of Tanh activation layers, and a final output layer. Its network structure is as follows Figure 1 As shown:

[0094] Improvement 2: The original PPO algorithm has the problem of slow learning speed and instability during the update process, and the limitation of the Clip function is one of the main reasons. The Clip function of the algorithm is improved by adjusting the update method of the probability ratio of the new and old strategies under the same truncation constant ε setting to expand its range, thereby accelerating the learning rate of the algorithm. Specifically, the probability ratio of the new and old strategies is limited to the interval of 1 / (1+ε), 1 / (1-ε)]. Secondly, in order to ensure that the strategy can be optimized in the right direction and enhance the learning of the truncation critical range, a constant v is introduced. When the probability ratio of the new and old strategies exceeds the predetermined clipping range, it is re-limited to a suitable interval. The calculation process of the improved clip function is as follows:

[0095]

[0096] Among them, v∈(0,1), ε∈(0,1).

[0097] The improved IPPO algorithm framework is as follows Figure 2 As shown, including:

[0098] 4.1. Policy network π in the initialization algorithm θ Parameter θ, old policy network parameter θ old and value network parameters φ, setting hyperparameters including learning rate, discount factor, and truncation coefficient;

[0099] 4.2. Input the environment state space into the algorithm, and capture the system resource state with time series relationship and dynamic changes through the long short-term memory network (LSTM) learning in the policy network, and output the action probability π θ (a t |s t );

[0100] 4.3. Run strategy π in the environment θ Generate a batch of trajectories. Use the generated actions and environment parameters to train the algorithm and get the reward value r at that moment t and the environmental state s at the next moment t+1 , and (s t ,a t ,r t ,s t+1 ,π θ (a t |s t ))Deposit into the experience pool;

[0101] 4.4. Estimating the advantage function using generalized advantage

[0102] 4.5. Calculate the probability ratio of new and old strategies in the experience pool: The final probability ratio is determined by the improved clipping function, and the hyperparameter v is used to control the probability ratio to

[0103] 4.6. Calculate the objective function value J(θ)

[0104]

[0105] 4.7. Update the policy network parameters θ by gradient descent method to maximize the objective function J(θ);

[0106] 4.8. Calculate the loss value of the value function: loss = (V(φ)-v trace ) 2 , update the value network parameter φ through the gradient descent method to minimize the loss value of the value function;

[0107] 4.9. Repeat the process of 4.2-4.8 until the strategy converges or reaches the predetermined number of training steps;

[0108] 4.10. Output the final offloading decision and resource allocation plan.

[0109] In the second embodiment of the present invention, Figure 3 As shown, an application scenario of the present invention is demonstrated, which is mainly used in the task offloading scenario under the end-edge collaborative architecture. Since the computing performance of mobile devices is weak and is limited by battery capacity, they cannot process computing-intensive tasks. Edge nodes close to mobile devices have stronger computing resources and shorter data transmission time than cloud nodes. Therefore, the computing offloading technology can expand the computing power of mobile devices, reduce task processing delays and mobile device energy consumption, and thus improve user service quality. The system includes N mobile devices, M edge servers and a scheduler located at the base station. Each terminal device generates a triplet {D n ,C n ,τn } represents the task, where D n (t) represents the amount of data generated by the mobile device at time t, C n (t) represents the number of CPU cycles required to process the task; τ n (t) represents the maximum tolerable delay of the task. The offloading ratio of the task generated by terminal device n is α n =[0,1], and the remaining proportion of tasks are executed locally on the device. These two parts can be executed in parallel, thereby reducing the overall latency and improving the service response time.

[0110] Based on the above system model architecture, in the third embodiment of the present invention, an edge computing task offloading and resource allocation method is proposed, such as Figure 4As shown, the overall flow chart of the calculation offloading strategy of the present invention is shown. Specifically, it includes:

[0111] Step S1: construct an edge-to-end task offloading scenario including multiple terminal devices and multiple edge servers, and obtain data and offloading models of tasks generated by the terminal devices;

[0112] In the system, the set M = {1,2,3,...,M} is used to represent M edge servers, and the set N = {1,2,3,...,N} is used to represent N terminal devices. Each terminal device generates a computing task at the beginning of the time slot. Use a triple to represent {D n ,C n ,τ n}. n Indicates the data size of the task on the mobile terminal device n; C n is the number of CPU cycles required for the task, τ n is the maximum latency requirement of the task. In this model, each task can be split into two parts and executed locally and at the edge simultaneously.

[0113] The offloading ratio of tasks generated by terminal device n is α n =[0,1], n∈N indicates that the remaining proportion of tasks are executed locally on the device. These two parts can be executed in parallel, thereby reducing the overall latency and improving the service response time.

[0114] Step S2, calculating the time delay, energy consumption and system load balance data; Figure 5 As shown, the specific steps are as follows:

[0115] Step S2.1: Calculate the time of task transfer Divide the task data volume by the transmission speed, and the calculation formula is:

[0116]

[0117] Among them, α n is the ratio of offloading to edge servers, D n Represents the total amount of data for this task, r n The data transmission speed;

[0118] Step S2.2: Split the task according to the offloading ratio, calculate the processing time of the task on the local mobile device and the edge node, and the local delay and edge delay The calculation formula is:

[0119]

[0120] Among them, 1-α nIndicates the proportion of local execution, Indicates the CPU frequency of mobile terminal device n, C n is the total number of CPU cycles for the task, k n The server CPU frequency assigned to the task. Total calculation frequency for the server.

[0121] Step S2.3: Calculate the energy consumption of processing part of the task locally on the mobile device and energy consumption during task transmission The specific calculation formulas are:

[0122]

[0123] Where μ is a constant related to the CPU structure of the edge device and represents the effective switching capacitance, and its value is 10 -26 , P n It represents the power of mobile device n when transmitting data.

[0124] Step S2.4: Get the CPU usage of each edge server and calculate its standard deviation as the load balancing degree LB of the system:

[0125]

[0126] Where M is the number of edge servers, L m represents the CPU resource usage of the mth edge node, L avg Indicates the average utilization of all CPUs.

[0127] Step S2.5: Calculate the total delay and total energy consumption of the task offloading process. is the maximum value of local and edge delay, and the total energy consumption It is the sum of the energy consumption of local execution and task transmission. Its calculation formulas are:

[0128]

[0129] Step S3: describe the partial task offloading model in each time slot multi-edge server scenario, and set the objective function and condition constraints of the system;

[0130] The objective function and its constraints are as follows:

[0131]

[0132] Among them, the constraints C1, C2 and C3 are the unloading ratio, bandwidth allocation ratio and computing resource allocation ratio of the terminal device, respectively, and they should all be in the range of 0 to 1. At the same time, C4 indicates that the bandwidth ratio allocated by the system to the terminal device should not exceed the limit of its total bandwidth W to ensure the rational use of bandwidth resources. C5 indicates that the computing resource ratio allocated by the server to the terminal device cannot exceed its own computing power to avoid overload operation. C6 indicates that the total delay of each task cannot be greater than the maximum acceptable delay of the task. These constraints jointly ensure the stability and efficiency of the system operation.

[0133] Step S4: Input the data calculated in the above process into the IPPO algorithm, and obtain the algorithm parameter model and decision action through iterative training. Figure 6 As shown, the algorithm model specifically includes the following steps:

[0134] Step S4.1: Initialize the policy network πθ of the PPO algorithm 和 The value network parameter φ, setting hyperparameters such as learning rate, discount factor and truncation constant, setting hyperparameters such as learning rate, discount factor and truncation constant; the parameter settings of the IPPO algorithm are shown in Table 1:

[0135] parameter Numeric Network Type LSTM Number of neurons in the hidden layer of the Actor network 256,128 Number of neurons in the hidden layer of the Critic network 256,256,128 Actor Network Activation Function LeakyRelu, Tanh Critic network activation function LeakyRelu Actor Network Update Frequency 10 <![CDATA[Actor network learning rate lr a > <![CDATA[10 -4 ]]> <![CDATA[Critic network learning rate lr c > <![CDATA[10 -3 ]]> Discount rate γ 0.6 The constant ε of the clip function 0.2 The constant ν of the clip function 0.05 Batch size 128

[0136] Table 1

[0137] Step S4.2: Input the state in the system environment into the strategy network of the algorithm and start iterative training as the first state. After learning the strategy network including LSTM, output the action probability π θ (a t |s t );

[0138] Step S4.3: Run strategy π in the environment θ Generate a batch of trajectories. Get the action a at this moment through the action probability t , reward value r t and the environmental state s at the next moment t+1 , and s t ,a t ,r t ,s t+1 ,π θ (a t |s t ))Deposit into the experience pool;

[0139] Step S4.4: Calculate the generalized advantage function Used to evaluate the quality of an action;

[0140] Step S4.5, calculate the probability ratio of the new and old strategies in the experience pool: The final probability ratio is determined by the improved clipping function, and the hyperparameter v is used to control the probability ratio to

[0141] Step S4.6, calculate the objective function value:

[0142]

[0143] Step S4.7, update the policy network parameter θ by the gradient descent method to maximize the target parameter J(θ);

[0144] Step S4.8, calculate the loss value of the value function: loss = (V(φ)-v trace ) 2 , update the value network parameter φ through the gradient descent method to minimize the loss value of the value function;

[0145] Step S4.9, repeat S4.2-S4.8 until the strategy converges or reaches the predetermined number of training steps;

[0146] Step S4.10: Output the final offloading decision and resource allocation plan.

[0147] The following two experiments describe in detail the technical effects obtained by the computation offloading and resource allocation method of the present invention.

[0148] Experiment 1: IPPO algorithm learning rate combination experiment

[0149] This experiment selects four different learning rate combinations to analyze the convergence and stability of the IPPO algorithm, and the number of training rounds is set to 1000. Figure 7 The figure shows the change of the reward value of the IPPO algorithm under different learning rate combinations. It can be observed that when the learning rate lr of the Actor network a And the learning rate lr of the Critic network c Both are 10 -4 When the learning rate is 10 -3 When the learning rate is set to lr a =10 -3 ,lr c =10 -4 When lr is less than 1, the algorithm cannot converge. This is because the Actor network updates parameters based on the unstable Q value, which causes the algorithm to be unable to learn effective information, thus affecting the convergence of the algorithm. a =10 -4 , lr c =10 -3When , the algorithm converges in the 600th round of training, indicating that when lr c Greater than lr a When lr is , the algorithm can achieve better convergence. This is because the Critic network can learn the Q value faster and reach stability, which makes it easier for the Actor network to quickly update parameters in the right direction. However, when lr is a =10 -3 ,lr c =10 -2 When , the algorithm still cannot converge and is in a fluctuating state. Therefore, in order to ensure the convergence and performance of the IPPO algorithm, the subsequent experiments will lr a and lr c Set to 10 respectively -4 and 10 -3 .

[0150] Experiment 2: Experiment with different numbers of terminal devices

[0151] In order to comprehensively evaluate the effect of the IPPO algorithm proposed in this chapter, the following algorithms are selected for comparison:

[0152] ① All Local Execution (AL): All tasks generated by terminal devices are processed completely locally.

[0153] ② Random Offloading (RO): The tasks of each mobile terminal device are evenly distributed to all edge servers according to a random ratio, and the computing resources are also evenly distributed to the terminal devices.

[0154] ③ Original PPO algorithm unloading (PPO): The original PPO algorithm that has not been improved is used to evaluate the effectiveness of the optimization strategy proposed in this chapter.

[0155] The evaluation indicators mainly include total system delay, total system energy consumption and system service quality value.

[0156] The total system latency is the total time consumed to process all terminal device tasks. The calculation formula is as follows:

[0157]

[0158] The total energy consumption of the system refers to the total energy consumed by all terminal devices during local execution and data transmission to the edge server during the offloading process. The calculation formula is as follows:

[0159]

[0160] QoS is used to indicate the optimization effect of each algorithm compared to the task being executed completely locally.

[0161]

[0162] Among them, ω1 and ω2 are delay weight and energy consumption weight respectively. and They represent the latency and energy consumption when the task of device n is completely executed locally.

[0163] When the number of edge servers is fixed to 4 and the computing power is 4Ghz, the number of terminal devices is set to {10, 15, 20, 25, 30, 35, 40} respectively. The changes in the total task processing delay, total device energy consumption and QoS value of each algorithm under different numbers of terminal devices are analyzed.

[0164] Figure 8 The QoS values ​​of each algorithm under different numbers of devices are shown. Figure 8 As can be seen from the figure, the QoS value of the AL algorithm as the baseline algorithm always remains at 0. As the number of terminal devices increases, the QoS value of the RO algorithm using the random offloading strategy fluctuates to a certain extent. This may be because the random offloading strategy cannot effectively manage and allocate computing resources when facing a large number of devices. In contrast, other algorithms show more stable performance in terms of QoS. In particular, the IPPO algorithm, whose QoS value always remains at the highest level, has a maximum optimization of 6.62% compared to PPO. The reason for the small optimization range is that the computing resources are still sufficient under the current number of devices, so the QoS value has not been significantly improved.

[0165] Fig. 9 The figure shows the change of total system delay under different numbers of terminal devices. Fig. 9 It can be found that the delay growth rate of the AL and RO algorithms is significantly higher than that of the other two algorithms, showing a linear growth trend. This shows that with the increasing number of terminal devices, reasonable offloading decisions can significantly optimize the delay. After the number of terminal devices reaches 20, the optimization effect of the IPPO algorithm proposed in this paper is significantly enhanced compared with the PPO algorithm, and an average delay optimization of 9.23% can be achieved. This shows that as the number of terminal devices increases, the IPPO algorithm can better optimize the task processing delay and improve system performance by making more reasonable task offloading decisions.

[0166] Fig.10 The effect of different numbers of terminal devices on the total energy consumption of the system is presented. It can be observed that the IPPO algorithm always maintains the lowest energy consumption level and achieves significant improvement in energy consumption optimization compared to the PPO algorithm, achieving an average optimization of 16.6%. This further proves that the IPPO algorithm can effectively reduce device energy consumption during task offloading, thereby improving the battery capacity utilization efficiency of terminal devices.

[0167] It should be noted that, in various embodiments of the present invention, the sequence numbers of the above steps do not mean the order of execution. The order of execution of the steps should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0168] The following is a device embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0169] In a fourth embodiment of the present invention, an edge computing task offloading and resource allocation device is proposed, such as Fig.11 As shown, including:

[0170] Initialization module 10 is used to obtain task data of terminal devices and edge servers in edge-terminal task scenarios; the task data includes the offloading ratio α of the generated task of the nth terminal device n and the load balancing degree LB of the mth edge server;

[0171] A data acquisition module 12 is used to calculate the task offloading parameters of the edge-terminal task scenario, and the objective function and constraint conditions of the offloading task based on the task data;

[0172] The strategy generation module 13 is used to input the task offloading parameters, objective functions and conditional constraints into the proximal policy optimization model IPPO for training. When IPPO completes strategy convergence or reaches a predetermined number of training steps, IPPO outputs the task offloading strategy and resource allocation strategy for the edge-terminal task scenario.

[0173] The IPPO model includes an Actor action network and a Critic evaluation network; the Actor action network is a multi-layer deep learning network, which includes an input layer, a long short-term memory network layer, a fully connected layer consisting of two Tanh activation layers, and an output layer; the Critic evaluation network limits the probability ratio r(θ) of the new and old strategies during the training process through the Clip function:

[0174]

[0175] Among them, ε is the cutoff constant, v is the optimization constant, π θ (·) indicates a new operation strategy, Indicates the old operation strategy, a t Indicates running action, s t Indicates the state of the environment.

[0176] In the fifth embodiment of the present invention, a computer-readable storage medium is proposed. The edge computing task offloading and resource allocation device of the present invention, if its function is implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a computer-readable storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. Therefore, in the fifth embodiment of the present invention, a computer-readable storage medium is provided for storing a computer program for performing an edge computing task offloading and resource allocation method. It should be understood that the computer-readable storage medium in the embodiment of the present invention may be a volatile memory and / or a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct rambus RAM (DRRAM).

[0177] Fig.12 Schematic diagram of an electronic device of the present invention. Fig.12As shown, in the sixth embodiment of the present invention, an electronic device 100 is proposed, including the edge computing task offloading and resource allocation device as described above. A person of ordinary skill in the art can understand that all or part of the steps in the above method can be completed by instructing related hardware (such as a processor, FPGA, ASIC, etc.) through a program. All or part of the steps of the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiment can be implemented in the form of hardware, such as implementing its corresponding functions through an integrated circuit, or in the form of a software functional module, such as implementing its corresponding functions through a processor executing a program / instruction stored in a memory. The embodiments of the present invention are not limited to any particular form of combination of hardware and software.

[0178] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0179] The electronic device of the present invention may be any device with data processing capability, and the device with data processing capability may be a device or apparatus such as a computer. The device embodiment may be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the processor of any device with data processing capability reads the corresponding computer program instructions in the non-volatile memory into the memory and runs the instructions. Fig.13 This is a schematic diagram of the hardware structure of an electronic device of the present invention. Fig.13 As shown, from the hardware level, it is a hardware structure diagram of any device with data processing capability where the edge computing task offloading and resource allocation device of the present invention is located, except Fig.13 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0180] When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0181] The existing research work on computational offloading strategies in resource-constrained scenarios pays little attention to the load balancing of edge nodes and is not comprehensive, which leads to unbalanced load distribution of tasks in some edge nodes and affects the overall stability of the system. At the same time, the existing computational offloading strategies for deep reinforcement learning are not optimized for the problems existing in the offloading scenarios, resulting in the execution speed and stability of the algorithm being difficult to meet the scenarios of a large number of mobile devices. In view of the above two problems, the present invention proposes to use the load balancing degree of the edge server as the optimization target from the perspective of offloading algorithm optimization, and introduces a long short-term memory network (LSTM) model to learn the changes in the historical environment according to the characteristics of the offloading scenario, thereby achieving an accurate response to the state change, and optimizing the clipping function of the strategy update to speed up the learning speed of the algorithm and improve its stability. On the one hand, the present invention constructs an objective function with delay, energy consumption and load balancing as the optimization targets, and realizes the degree of importance attached to a certain target by setting different weight coefficients, so that the offloading strategy has a more comprehensive adaptation scenario. On the other hand, by optimizing the PPO algorithm to adapt to the scenarios of multiple edge servers and multiple mobile devices, the system can generate offloading decisions faster and more stably, and minimize the delay and instability caused by the operation of the algorithm to the offloading system.

[0182] The above implementation modes are only used to illustrate the present invention, but not to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention. The patent protection scope of the present invention should be defined by the claims.

Claims

1. A method for edge computing task offloading and resource allocation, characterized in that: include: Initialization step, obtaining task data of terminal devices and edge servers in edge-terminal task scenarios; The task parameters include the task offloading ratio of the terminal device; A data acquisition step, based on the task data, calculating the task offloading parameters of the edge-terminal task scenario, and the objective function and constraint conditions of the offloading task; In the strategy generation step, the task offloading parameters, the objective function and the conditional constraints are input into the proximal strategy optimization model for training. When the proximal strategy optimization model completes strategy convergence or reaches a predetermined number of training steps, the proximal strategy optimization model outputs the task offloading strategy and resource allocation strategy for the edge-terminal task scenario.

2. The edge computing task offloading and resource allocation method according to claim 1, characterized in that: The proximal strategy optimization model includes an Actor action network and a Critic evaluation network; The Actor action network is a deep learning network with a multi-layer architecture, which includes an input layer, a long short-term memory network layer, a fully connected layer consisting of a Tanh activation layer, and an output layer.

3. The edge computing task offloading and resource allocation method as claimed in claim 2, characterized in that: The Critic evaluation network limits the probability ratio of new and old strategies during training through the Clip function r(θ) Among them, ε is the cutoff constant, v is the optimization constant, π θ (·) indicates a new operation strategy, Indicates the old operation strategy, a t Indicates running action, s t Indicates the state of the environment.

4. The edge computing task offloading and resource allocation method according to claim 1, characterized in that: The objective function and its constraints satisfy: in; ω1 is the delay weight, ω2 is the energy consumption weight, ω3 is the load balancing weight, and the total task delay is the local execution delay of the task on the nth terminal device, is the edge delay of the task output to the edge server for execution, and the total energy consumption of the task is the energy consumption of the task processed locally on the nth terminal device, is the energy consumption of the task transmitted from the nth terminal device to the edge server, LB is the load balancing degree of the edge server, L m is the CPU usage of the mth edge server, L avg Indicates the average CPU utilization of edge servers. M is the number of edge servers in the edge-terminal task scenario, and constraint C1 represents the task offloading ratio α. n Constraint C2 means satisfying the bandwidth allocation ratio b n , constraint C3 means that the computing resource allocation ratio k is satisfied n , constraint C4 means that the total bandwidth W is limited to the bandwidth ratio allocated to the terminal device, constraint C5 means that the computing power of the edge server is limited to the computing resource ratio allocated to the terminal device, and constraint C6 means that the total delay of each task is satisfied. Subject to the maximum acceptable delay τ of the task n restrictions.

5. An edge computing task offloading and resource allocation device, characterized in that: include: An initialization module is used to obtain task data of terminal devices and edge servers in edge-terminal task scenarios; The task parameters include the offloading ratio of the generated task of the terminal device; A data acquisition module, used to calculate the task offloading parameters of the edge-terminal task scenario, and the objective function and constraint conditions of the offloading task based on the task data; The strategy generation module is used to input the task offloading parameters, the objective function and the conditional constraints into the proximal strategy optimization model for training. When the proximal strategy optimization model completes strategy convergence or reaches a predetermined number of training steps, the proximal strategy optimization model outputs the task offloading strategy and resource allocation strategy of the edge-terminal task scenario.

6. The edge computing task offloading and resource allocation device according to claim 5, characterized in that: The proximal strategy optimization model includes an Actor action network and a Critic evaluation network; The Actor action network is a deep learning network with a multi-layer architecture, which includes an input layer, a long short-term memory network layer, a fully connected layer consisting of a Tanh activation layer, and an output layer.

7. The edge computing task offloading and resource allocation device according to claim 6, characterized in that: The Critic evaluation network limits the probability ratio of new and old strategies during training through the Clip function r(θ) Among them, ε is the cutoff constant, v is the optimization constant, π θ (·) indicates a new operation strategy, Indicates the old operation strategy, a t Indicates running action, s t Indicates the state of the environment.

8. The edge computing task offloading and resource allocation device as claimed in claim 5, characterized in that: The objective function and its constraints satisfy: in; ω1 is the delay weight, ω2 is the energy consumption weight, ω3 is the load balancing weight, and the total task delay is the local execution delay of the task on the nth terminal device, is the edge delay of the task output to the edge server for execution, and the total energy consumption of the task is the energy consumption of the task processed locally on the nth terminal device, is the energy consumption of the task transmitted from the nth terminal device to the edge server, LB is the load balancing degree of the edge server, L m is the CPU usage of the mth edge server, L avg Indicates the average CPU utilization of edge servers. M is the number of edge servers in the edge-terminal task scenario, and constraint C1 represents the task offloading ratio α. n Constraint C2 means satisfying the bandwidth allocation ratio b n , constraint C3 means that the computing resource allocation ratio k is satisfied n , constraint C4 means that the total bandwidth W is limited to the bandwidth ratio allocated to the terminal device, constraint C5 means that the computing power of the edge server is limited to the computing resource ratio allocated to the terminal device, and constraint C6 means that the total delay of each task is satisfied. Subject to the maximum acceptable delay τ of the task n restrictions.

9. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instruction is executed, the edge computing task offloading and resource allocation method as described in any one of claims 1 to 4 is implemented.

10. An electronic device, comprising the edge computing task offloading and resource allocation device as described in any one of claims 5-8.

Images