Edge service caching and task unloading joint optimization method based on hierarchical deep reinforcement learning

Abstract

The invention belongs to the technical field of mobile edge computing, and particularly relates to an edge service caching and task unloading joint optimization method based on hierarchical deep reinforcement learning. The method comprises the following steps: constructing a communication system model and calculating the total time delay of system task processing; constructing an optimization objective function according to the total time delay of system task processing, and splitting the objective function into a first sub-problem and a second sub-problem; constructing respective Markov decision-making processes according to the two sub-problems; the step of constructing the Markov decision process comprises defining a state space, an action space and a reward function; solving a Markov decision process of the first sub-problem by using a DQN algorithm to obtain a service cache decision scheme; solving a Markov decision process of the second sub-problem by using a DDPG algorithm to obtain a task unloading and computing resource allocation scheme; according to the invention, the network service can be further optimized, and the average task processing delay is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile edge computing, and particularly relates to a joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning. Background Art

[0002] With the rapid development of Internet of Things technology, the demand for extremely low latency and high reliability in emerging mobile applications is increasing day by day. In the MEC framework, the computing tasks generated by users can be offloaded to edge auxiliary devices for processing, and this process often requires relying on service caching. Taking the face recognition application as an example, the object database and visual recognition model required for processing the input data of user devices can be cached in the edge server, so as to provide a better experience for users than routing requests to the cloud server. However, given the limited caching capacity of each edge server, only a small number of services can be cached, and it is crucial to correctly select the cached services, which will directly affect the offloading strategy of user devices. In addition, the competition among numerous users for limited computing and caching resources also brings additional technical challenges. However, traditional caching strategies are difficult to adapt to the rapid changes in service popularity, resulting in increased latency. Therefore, there is an urgent need for a new joint optimization method for edge caching and task offloading to solve this problem. Summary of the Invention

[0003] Aiming at the deficiencies of the existing technology, the present invention proposes a joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning. The method includes:

[0004] S1: Construct a communication system model and calculate the total system task processing latency;

[0005] S2: Construct an optimization objective function according to the total system task processing latency, and split the objective function into a first sub-problem and a second sub-problem;

[0006] S3: Construct Markov decision processes for the two sub-problems respectively; constructing a Markov decision process includes defining a state space, an action space, and a reward function;

[0007] S4: Use the DQN algorithm to solve the Markov decision process of the first sub-problem to obtain a service caching decision scheme;

[0008] S5: Use the DDPG algorithm to solve the Markov decision process of the second sub-problem to obtain a task offloading and computing resource allocation scheme.

[0009] Preferably, the communication system model includes: a cloud server, user devices, a base station, and edge servers; define the set of edge servers I is the number of edge servers; edge server i has total computing resources F i and total storage resources Ci , the total memory resource M i ; Define the time set as T is the total number of time slots; The system contains a total of K different types of computing services, and the service set is defined as Each service exhibits heterogeneous characteristics, expressed as a quadruple {d k , m k , c k , τ}; where d k is the task size of service k, m k is the memory space required by service k, c k represents the storage space size required by service k, and τ is the maximum processing delay that service k can accept.

[0010] Preferably, the process of calculating the average task processing delay includes:

[0011] Calculate the task communication transmission delay of the user according to the communication system model;

[0012] Calculate the task computing delay according to the communication system model;

[0013] Calculate the total task processing delay of the task computing system according to the task communication transmission delay.

[0014] Furthermore, the formula for calculating the total task processing delay of the system is:

[0015]

[0016] where D(t) represents the total task processing delay of the system at time slot t, Δx i,k (t) represents whether the edge server caches the service, represents the delay of task k being offloaded to the associated edge server, and the service model of the corresponding task is cached, y i,k (t) represents the task offloading decision, represents the number of tasks k arriving at edge server i at time slot t, represents the computing delay. When i = o, it represents the cloud server computing delay. When , it represents the edge server computing delay; represents the delay of task k being offloaded to the cooperative edge server, j represents the offloading decision target, represents the edge server set, represents the service set, and o represents the cloud server.

[0017] Preferably, the optimization objective function is:

[0018]

[0019] Among them, D(t) represents the total task processing delay of the system in the t-th time slot, T represents the total number of time slots, ΔX represents the service cache update decision set of the edge server, represents the task offloading decision set, represents the resource allocation decision set, x i,k (t) represents the service cache status of edge server i, c k represents the storage space size required for service k, C i represents the total storage resource of edge server i, represents the set of edge servers, represents the set of time, K represents the set of services, m k represents the memory space required for service k, M i represents the total memory resource of edge server i, Δx i,k (t) represents whether the edge server caches the service, y i,k (t) represents the task offloading decision, represents the set of services, d k represents the task size of service k, I represents the number of edge servers, represents the number of tasks of service k arriving at edge server i in the t-th time slot, ρ i,k (t) represents the ratio of the computing resources allocated by edge server i to service k in the t-th time slot to the total computing resources, and o represents the cloud server.

[0020] Preferably, the first sub-problem is expressed as:

[0021]

[0022] Among them, ΔX represents the service cache update decision set of the edge server, represents the optimal task offloading decision set, represents the optimal resource allocation decision set, T represents the total number of time slots, D(t) represents the total task processing delay of the system in the t-th time slot, x i,k (t) represents the service cache status of edge server i, c k represents the storage space size required for service k, C i represents the total storage resource of edge server i, represents the set of edge servers, represents the set of time, K represents the set of services, m k represents the memory space required for service k, M i represents the total memory resource of edge server i, Δx i,k (t) represents whether the edge server caches the service, represents the set of services.

[0023] Preferably, the second sub - problem is expressed as:

[0024]

[0025] where D(t) represents the total task - processing delay of the system at time slot t, T represents the total number of time slots, ΔX * represents the optimal service - caching decision set, represents the task - offloading decision set, represents the resource - allocation decision set, C i represents the total storage resource of edge server i, represents the set of edge servers, represents the set of time, K represents the set of services, y i,k (t) represents the task - offloading decision, represents the set of services, d k represents the task size of service k, I represents the number of edge servers, represents the number of tasks k arriving at edge server i at time slot t, ρ i,k (t) represents the ratio of the computing resources allocated by edge server i to service k at time slot t to the total computing resources, and o represents the cloud server.

[0026] Preferably, the state space of the Markov decision process of the first sub - problem is expressed as:

[0027]

[0028] where, represents the state space of edge server i at time slot t, (C i , M i ) represents the caching space of edge server i, C i represents the total storage resource of edge server i, M i represents the total memory resource of edge server i, X i (t) represents the caching state of edge server i at time slot t, V i (t) represents the workload of edge server i at time slot t;

[0029] The action space is expressed as:

[0030]

[0031] where, represents the action space of edge node i, a i,k (t) represents the service - caching decision, represents the set of services;

[0032] The reward function is expressed as:

[0033]

[0034] Among them, represents the reward of edge node i, and D(t) represents the total task processing delay at system time slot t.

[0035] Preferably, the state space of the Markov decision process of the second sub-problem is represented as:

[0036]

[0037] Among them, represents the state space of the cloud server at time slot t, C i represents the total storage resource of edge server i, F i represents the total computing resource owned by edge server i, X i (t) represents the cache state of edge node i at time slot t, V i (t) represents the workload of edge node i at time slot t, ΔX i (t) represents the cache decision action of edge server i at time slot t, represents the set of edge servers;

[0038] The action space is represented as:

[0039]

[0040] Among them, represents the action space of the cloud server at time slot t, Δy i,k (t) represents the continuous action space for task offloading, ρ i,k (t) represents the continuous action space for resource allocation, represents the set of edge servers, represents the set of services;

[0041] The reward function is represented as:

[0042]

[0043] Among them, represents the reward of edge node i, and D(t) represents the total task processing delay at system time slot t.

[0044] The beneficial effects of the present invention are as follows: Aiming at the problems in the prior art that traditional caching strategies are difficult to adapt to the rapid changes in service popularity, the resources of edge servers are limited, and the coupling between service caching and task offloading is strong, the present invention proposes a joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning. By splitting the optimization problem into two sub-problems: service caching update (P1) and task offloading and resource allocation (P2), and solving them using DQN and DDPG algorithms respectively, and modeling the dynamic environment using Markov decision process (MDP), the present invention realizes the joint optimization of service caching and task offloading.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] Stronger dynamic adaptability: Through hierarchical deep reinforcement learning, the present invention can dynamically adapt to the changes in service popularity and task load, overcoming the disadvantage of poor adaptability of traditional caching strategies (such as LRU, FIFO) to dynamic environments. The simulation results show that when the number of service types increases, the present invention has the smallest increase in delay (from 5.5 seconds to 9 seconds), which is better than the comparison scheme RCO (from 7 seconds to 17 seconds).

[0047] More efficient joint optimization: The hierarchical framework reduces the computational complexity of joint optimization while ensuring the effect of global optimization, avoiding the sub-optimal solution problem caused by independent optimization in traditional methods.

[0048] Higher resource utilization: By optimizing the calculation resource allocation using the DDPG algorithm, the present invention significantly improves the resource utilization efficiency of edge servers. The simulation results show that the average task processing delay of the present invention at 8 edge nodes is only 10.2 seconds, which is better than other schemes (such as DDQN, PPO).

[0049] Better delay performance: Aiming at minimizing the task processing delay, the present invention dynamically adjusts the strategy through reinforcement learning, significantly reducing the average task processing delay, and is applicable to low-delay and high-reliability scenarios. Brief Description of the Drawings

[0050] Figure 1 It is the structural diagram of the communication system model of the present invention;

[0051] Figure 2 It is the delay performance comparison diagram between the present invention and the comparison scheme under different numbers of edge servers;

[0052] Figure 3 It is the delay performance comparison diagram between the present invention and the comparison scheme under different numbers of service types. Detailed Embodiment

[0053] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0054] The present invention proposes a joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning, and the method includes the following contents:

[0055] S1: Construct a communication system model and calculate the total system task processing delay.

[0056] As Figure 1 shown, the system model includes: a cloud server, user equipment, a base station, and an edge server. Define the edge server set I as the number of edge servers; the edge server i has a total computing resource F i , a total storage resource C i , a total memory resource M i ; define the time set as T as the total number of time slots.

[0057] In a multi-edge server network, the computing tasks of users are usually executed on the local server with the required service. To maximize the network capacity, the tasks of each node are allowed to be routed to adjacent nodes through the LAN. This usually occurs when the required service is not on the local server but is cached by a nearby edge server. In other words, the successful execution of a computing task depends on the matching with the corresponding service cache; if the matching fails, the task cannot be executed locally. On the premise that all tasks can only be routed once, the tasks that do not meet the local execution conditions will be routed to the cloud service center for execution. As Figure 1 shown, the service type (identified by the same color square) corresponding to the task represented by the left red circle has been cached at the local node, so it can be executed locally. The task represented by the green circle is offloaded to the adjacent base station for execution, while the task represented by the purple circle is offloaded to the remote cloud service center for execution.

[0058] Assume that there are a total of K different types of computing services in this network, and define the service set as Each service exhibits heterogeneous characteristics, and K different services correspond to K different tasks; the service can be represented as a quadruple {d k , m k , c k , τ}, where d k is the task size of service k (in MB), m k is the memory space required for service k, ck denotes the storage space size required for service k, and τ is the maximum processing delay that service k can accept. Assume that the number of tasks k arriving at the edge server follows a Poisson distribution, and use to denote the number of tasks k arriving at edge server i at time slot t.

[0059] In this model, service caching and task offloading will inevitably cause communication delays, while the allocation of computing resources affects the computing delays of tasks. Therefore, the communication and computing delay models will be introduced below:

[0060] (1) Communication delay: Communication delay mainly occurs when tasks cannot be processed by local edge servers and need to be transmitted through the local area network to nearby edge nodes or cloud service centers for processing. Use W i and W o to denote the average transmission rates of the edge server and the cloud service center respectively. Compared with the edge server, since the cloud service center is geographically farther away, it should satisfy W i > W o . Therefore, the communication transmission delay of task k on edge server i can be expressed as:

[0061]

[0062] where j represents the target of the offloading decision. When j = i, it means that task k is computed on the associated edge server and no communication delay is generated; when , it means that task k is offloaded and executed on the cooperative edge server instead of the cloud service center; when j = o, it means that task k is offloaded and executed on the cloud service center.

[0063] (2) Computing delay: After service caching and task offloading are completed, task k is allocated specific computing resources to support the execution of multiple tasks. Assume that the edge server can process multiple tasks simultaneously, and the computing resources are related to the operating frequency of the CPU. The processing delay of the task is linearly related to the number of bits of the task and the CPU operating frequency. Use where ρ i,k ∈[0,1] represents the ratio of the computing resources allocated by the edge server to task type k to the total computing resources. Then the corresponding task computing delay is expressed as:

[0064]

[0065] where F o represents the average computing resource quantity of the cloud service center, and F i denotes the total computing resources owned by edge server i. At the same time, the computing resources owned by the edge node are limited, and the constraint condition is expressed as:

[0066]

[0067] Among them, y i,k (t) represents the task offloading decision.

[0068] According to the task communication transmission delay and the total task processing delay of the task computing system, the total task processing delay of the system within time slot t is:

[0069]

[0070] Among them, D(t) represents the total task processing delay of the system within time slot t, and Δx i,k (t) represents whether the edge server caches the service, the delay of offloading task k to the associated edge server, represents the number of task k arriving at edge server i within time slot t, represents the computing delay. When i = o, it represents the computing delay of the cloud server. When it represents the computing delay of the edge server. represents the delay of offloading task k to the collaborative edge server, and j represents the offloading decision target.

[0071] S2: Construct an optimization objective function based on the total task processing delay of the system, and split the objective function into a first sub-problem and a second sub-problem.

[0072] Construct an optimization objective function based on the total task processing delay of the system, expressed as:

[0073]

[0074] Among them, represents the service cache update decision set of the edge server, represents the task offloading decision set, represents the resource allocation decision set, and x i,k (t) represents the service cache status of edge server i, and ρ i,k (t) represents the ratio of the computing resources allocated by edge server i to service k to the total computing resources within time slot t. Constraint (a) and constraint (b) respectively state that the cached services shall not exceed the storage capacity and memory space limits of the edge node; Constraint (c) stipulates that the update of the service cache shall not conflict with its current cache status; Constraint (d) ensures that within one time slot, the service cache is only allowed to be updated once; Constraint (e) emphasizes that task offloading can only be executed once, and repeated offloading is prohibited; Constraint (f) indicates that the space occupied by task offloading shall not exceed the storage upper limit of the edge node to avoid service overload; Constraint (g) stipulates that the resource allocation amount on the edge node shall not exceed its computing capacity; (h) to (k) clarify the legal value ranges of each variable.

[0075] The objective function is split into a first sub - problem P1 and a second sub - problem P2. The first sub - problem P1 is the service cache update problem, which is expressed as:

[0076]

[0077] s.t. (a)~(d)

[0078]

[0079] where, ΔX represents the service cache update decision set of the edge server, represents the optimal task offloading decision set, represents the optimal resource allocation decision set. First, fix the optimal task offloading decision set and the optimal resource allocation set Then solve the service cache update decision problem. The update of the service cache is mainly affected by the task load and memory resources of each edge node. To reduce the complexity of the centralized scheme, a reinforcement learning method based on Deep Q - Network (DQN) is subsequently independently adopted in each edge node to update the service cache policy.

[0080] The second sub - problem P2 is the task offloading and computing resource allocation problem, which is expressed as:

[0081]

[0082] s.t. (e)~(g)

[0083]

[0084] where, when the optimal service cache variable ΔX * is solved, the cache update actions of each edge server will be sent to the cloud service center. Subsequently, the cloud service center will perform cache updates and formulate computing task offloading and computing resource allocation policies. Subsequently, a reinforcement learning algorithm based on Deep Deterministic Policy Gradient (DDPG) is used for task offloading and computing resource allocation.

[0085] S3: Construct the respective Markov decision processes according to the two sub - problems; constructing the Markov decision process includes defining the state space, action space, and reward function.

[0086] The Markov decision process of the first sub - problem:

[0087] The service status involves cache space, cached services, etc., while the request status involves factors such as workload and service demand. Based on these factors, the state space of the edge server is expressed as:

[0088]

[0089] Among them, represents the state space of edge node i, and (C i , M i ) represents the cache space of the edge node, represents the cache state of the edge node, represents the workload of the edge node, that is, the number of tasks.

[0090] The action space refers to the set of decisions made by the agent based on the current state. In each time slot, the edge server needs to decide whether to cache the service. Therefore, the action space of the edge node is expressed as:

[0091]

[0092] Among them, represents the action space of edge node i, and a i,k (t) is the service caching decision, which is equivalent to Δx i,k (t).

[0093] The reward function can provide feedback on the actions taken in the current state. Based on the reward mechanism, the system can optimize actions to obtain higher utility in the future. The present invention aims to reduce the long-term delay of task processing. For this reason, the reward function is defined as:

[0094]

[0095] Among them, represents the reward of edge node i.

[0096] The Markov decision process of the second sub-problem:

[0097] At the beginning of each time slot, the cloud center server will obtain the global state of the system. This mainly includes the following aspects: the resource usage, cache status information, workload of each edge node, and the current service caching decision status:

[0098]

[0099] Among them, represents the state space of the cloud server at time slot t, and ΔX i (t) represents the caching decision action of edge server i at time slot t.

[0100] As a learning agent, is the set of operations that the central cloud agent can choose to consider the current state, expressed as:

[0101]

[0102] where ρ i,k (t) represents the continuous action space of resource allocation, that is, the ratio of the computing resources allocated to task k by edge server i in time slot t to the total computing resources; Δy i,k (t) ∈ [0, 1] represents the continuous action space of task offloading. This invention considers a binary offloading strategy, and thus designs a quantizer to convert continuous variables into binary variables. The expression is as follows:

[0103]

[0104] The reward function is:

[0105]

[0106] This invention adopts a hierarchical reinforcement learning strategy (HD2N) to solve the service caching decision scheme, task offloading, and computing resource allocation scheme. Specifically: load the pre-trained DQN and DDPG models, which are designed to predict the optimal actions in different states, so as to reduce service latency and improve resource utilization efficiency. Next, by resetting the mobile edge cloud computing network environment, ensure that each decision-making cycle starts from a clear state, avoiding the interference of the decisions in the previous cycle on the current cycle. Subsequently, the algorithm enters an iterative loop. In each loop, the algorithm extracts actions from the environmental information and converts them into specific service caching and computing offloading decisions. These decisions are applied to the environment, and the strategy is adjusted according to the feedback of the environment. After the decisions are executed, the environmental state will be updated and a new reward signal will be generated, which will be fed back into the DRL (Deep Reinforcement Learning) model to guide more accurate decisions in future iterations. The specific process of the algorithm solution includes:

[0107] S4: Use the DQN algorithm to solve the Markov decision process of the first sub-problem to obtain the service caching decision scheme.

[0108] DQN is a technology that combines deep learning and reinforcement learning, and is usually used in dynamic systems without prior knowledge. Specifically, DQN is a reinforcement learning algorithm that uses DNN to approximate the Q function, that is, the function of the expected return of taking a certain action in a given state. For a given state action and reward respectively store them into the experience replay pool and construct a target network and an evaluation network, and define the parameters as and The DQN aims to learn the optimal policy to generate state-based actions. To minimize the gap between the output of the evaluation network and the Q-value of the target network, the present invention uses the following loss function for training:

[0109]

[0110] wherein, describes the process of estimating the target Q-value in j steps, and γ ∈ [0, 1] represents the discount reward factor. To train the deep neural network, the gradient descent algorithm is used to update the network parameters of the above loss function. Its expression is as follows:

[0111]

[0112] The evaluation network and the target network adopt the same parameter values during initialization. The parameters of the evaluation network are updated at each step but the parameters of the target evaluation network are updated at every fixed time step

[0113] In DQN, to achieve a balance between exploration and exploitation, the ε-greedy strategy is adopted. The core idea of the ε-greedy strategy is that there is a probability ε to choose a random action (exploration) during decision-making, and a probability of 1 - ε to choose the currently known best action (exploitation). Here, ε is a parameter between 0 and 1, representing the probability of random exploration. Over time, ε usually gradually decreases, which means that the algorithm will increasingly rely on the knowledge already learned and reduce random exploration. The mathematical expression is:

[0114]

[0115] In the present invention, the exploration probability ε of the ε-greedy strategy is dynamically adjusted within the range of 0.1 to 0.9 to optimize the balance between exploration and exploitation.

[0116] Through the above process, the Markov decision process of the first sub-problem is solved to obtain the service cache decision scheme.

[0117] S5: Use the DDPG algorithm to solve the Markov decision process of the second sub-problem to obtain the task offloading and computing resource allocation scheme.

[0118] For the DDPG algorithm, its core consists of two aspects: the policy network (Actor) and the value network (Critic). The initial state of the environment is determined at the beginning. Subsequently, the environment generates new states and rewards based on the actions taken. Once the reward is received, the Critic network will be updated. Then, according to the guidance provided by the Critic network, the Actor network will also be updated. This process will be iterated until the set maximum number of iterations is reached, thus training an efficient Actor network. In each iteration, the Critic network updates the parameters w of its network through interaction with the environment. At the same time, it also needs to estimate the Q value of the current state and action to guide the update of the Actor network.

[0119] Update the parameters ω of the Critic deep neural network by minimizing the loss function:

[0120]

[0121] Among them, represents randomly sampling the minimum batch of m groups of data from the experience replay set and calculating the current network target value y j :

[0122] Update the parameters θ of the Actor deep neural network using policy gradients:

[0123]

[0124] Update the target network parameters through soft update:

[0125] θ′ = τθ + (1 - τ)θ′

[0126] ω′ = τω + (1 - τ)ω′

[0127] To achieve hierarchical reinforcement learning, each edge server DNN consists of an input layer, two hidden layers, an LSTM layer, and an output layer. Similarly, the DNN structure of the central cloud server includes an input layer, three hidden layers, and an output layer. The parameter configuration of the neural network is as follows: The network is constructed using 128 neurons, and trained using the Adam optimizer and a learning rate of 0.0001. The algorithm sets a batch size of 64 for the edge agent and a batch size of 128 for the central agent. In addition, the key parameters ε and γ that control the learning process are both set to 0.9 to balance exploration and exploitation, as well as to evaluate the importance of future rewards.

[0128] Solve the Markov decision process of the second sub-problem through the above process to obtain the task offloading and computing resource allocation scheme.

[0129] The system executes a service caching decision scheme, a task offloading and computing resource allocation scheme to minimize the average task processing delay of the system, thereby optimizing network services.

[0130] Evaluate the present invention:

[0131] Simulate the present invention. The simulation considers a multi-edge cloud computing network within a rectangular area of 4.0 square kilometers, including 8 edge nodes with heterogeneous resources and 9 services with heterogeneous characteristics, both of which are configured according to a uniform distribution. The simulation data in this chapter is a synthetic data set: the characteristics of tasks are determined by random distribution, and the number of tasks arriving at each node is determined according to a Poisson distribution with λ = 5.

[0132] To compare the performance of this solution, four different comparison algorithms are described in detail below. These algorithms each adopt different strategies to handle caching, offloading, and computing resource problems. As follows:

[0133] (1) Random Caching and Offloading (RCOF): The RCO algorithm processes caching, offloading, and computing resource allocation through random decisions. This method may lead to uneven resource utilization in computing resource allocation.

[0134] (2) Heuristic Caching and Offloading (HCOF): The caching decision is determined according to the number of task types; the offloading decision is based on more edge executions. The HCOF algorithm makes decisions based on heuristic rules according to task types and edge execution capabilities, and can allocate computing resources more precisely.

[0135] (3) Double Deep Q-Network (DDQN): The DDQN dynamically adjusts resource allocation by learning the optimal strategy, enabling the allocation of computing resources to adapt to environmental changes and task requirements. This method continuously optimizes the resource allocation strategy through continuous learning to achieve higher resource utilization efficiency and task execution efficiency.

[0136] (4) Proximal Policy Optimization (PPO): PPO is an optimization algorithm based on policy gradients, aiming to solve the policy update problem in reinforcement learning. It avoids excessive fluctuations during the update process by restricting the step size of policy updates, thereby improving the stability of learning.

[0137] Figure 2It shows the comparison of the processing delay performance of five solutions under different numbers of edge nodes, i.e., edge servers, when dealing with specific tasks. In the figure, the vertical axis is the average task processing delay of the task, and the horizontal axis is the change in the number of edge nodes. It can be seen from the figure that as the number of edge nodes increases, the average service delay of all algorithms also increases. The reason for this phenomenon is that more edge nodes mean more task assignments, which in turn leads to an increase in service delay. In particular, the hierarchical reinforcement learning strategy (HD2N), i.e., the solution of the present invention, has the lowest average service delay in all cases, demonstrating its efficiency in dealing with dynamic tasks, heterogeneous resources, and multi-dimensional decision-making problems. When the number of nodes is small, HD2N shows the best performance and achieves the shortest completion time. In contrast, the DDQN strategy is slightly insufficient when the number of nodes is small, but as the number of nodes increases, its performance shows a steady increase and a lower delay. The performance of the PPO strategy remains at a medium level, and the growth trend of its average completion time is also relatively stable. On the contrary, the HCO strategy performs poorly when the number of nodes is small, and the task processing delay of the RCO strategy increases significantly as the number of nodes increases. In particular, when the number of edge nodes reaches 8, the average service delay of HD2N is only about 10.2 seconds, significantly lower than other algorithms, which reflects the advantage of HD2N in making intelligent decisions.

[0138] Figure 3 It shows the impact of the increase in the number of service types on the average task processing delay of each solution. The results show that as the number of service types increases, the average task processing delay of all solutions increases. Significantly, the increase in the delay of the HD2N solution is the smallest, increasing from 5.5 seconds for 5 service types to 9 seconds for 9 service types, while the delay of the RCO algorithm increases sharply from 7 seconds to 17 seconds. This indicates that when the number of service types increases, the performance of HD2N is less affected, demonstrating its strong adaptability to dynamic environmental changes. In contrast, the impact of the number of service types on the performance of other algorithms is more significant. For example, the average task processing delay of the DDQN algorithm increases by about 7 seconds between 5 and 9 service types, which reflects the impact of service diversity on the edge proxy caching decision. Compared with DDQN, PPO performs better when the number of services is large, while HCO and RCO perform weakly, especially RCO, which has the longest delay in all tests. This may be because RCO does not effectively consider the dynamics of tasks and the heterogeneity of resources, resulting in sub-optimal task offloading decisions. Overall, the experimental results reveal the impact of different service numbers on the performance of each algorithm, and HD2N is particularly outstanding in terms of stability and reliability, adapting to application scenarios with diverse services.

[0139] The simulation experiment results show that compared with multiple existing algorithms, the present solution can significantly reduce the average service delay in different scenarios.

[0140] In summary, in view of the problems in the prior art that traditional caching strategies are difficult to adapt to the rapid changes in service popularity, the resources of edge servers are limited, and the coupling between service caching and task offloading is strong, the present invention proposes a joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning. When considering the task processing delay, the present invention uses hierarchical deep reinforcement learning to train a model, and by finding a better strategy, it aims to minimize the task processing delay. The present invention establishes an average task processing delay model according to system modeling, constructs the average task processing delay into an optimization objective function, and splits the objective function into two sub-problems. Then, the MDP is used to model this problem, and a quadruple is defined. For sub-problem P1, the DQN algorithm is used for solution. The system state includes the cache space of the edge server, the cache state, and the number of tasks. The action function is the cache decision of the edge server, and the reward function refers to minimizing the task processing delay. For sub-problem P2, the DDPG algorithm is used for solution. The system state includes the cache space of the edge server, the cache state, the number of tasks, and the cache decision of the edge server. The action function is the offloading decision and the resource allocation decision, and the reward function is still minimizing the task processing delay. Finally, the hierarchical deep reinforcement learning algorithm is used for solution to maximize the cumulative reward of the system, which can further optimize network services and reduce the average task processing delay.

[0141] The above-mentioned embodiments further elaborate on the purpose, technical solutions, and advantages of the present invention. It should be understood that the above-mentioned embodiments are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made to the present invention within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning, characterized in that, Including: S1: Construct a communication system model and calculate the total system task processing delay; S2: Construct an optimization objective function based on the total system task processing delay, and split the objective function into a first sub-problem and a second sub-problem; S3: Construct respective Markov decision processes according to the two sub-problems; constructing a Markov decision process includes defining a state space, an action space, and a reward function; S4: Use the DQN algorithm to solve the Markov decision process of the first sub-problem to obtain a service caching decision scheme; S5: Use the DDPG algorithm to solve the Markov decision process of the second sub-problem to obtain a task offloading and computing resource allocation scheme.

2. The joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning according to claim 1, characterized in that The described communication system model includes: a cloud server, user equipment, a base station, and an edge server; define the set of edge servers Let I be the number of edge servers; edge server i has a total computing resource F i , a total storage resource C i , and a total memory resource M i ; define the time set as Let T be the total number of time slots; the system contains a total of K different types of computing services, and define the service set as Each service exhibits heterogeneous characteristics, represented as a quadruple {d k , m k , c k , τ}; where d k is the task size of service k, m k is the memory space required for service k, c k represents the storage space size required for service k, and τ is the maximum processing delay that service k can accept.

3. The joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning according to claim 1, characterized in that, The process of calculating the average task processing delay includes: Calculate the task communication transmission delay of the user according to the communication system model; Calculate the task computing delay according to the communication system model; Based on the task communication transmission delay and the task computing system task processing total delay.

4. The joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning according to claim 3, characterized in that The formula for calculating the total system task processing delay is: Among them, D(t) represents the total task processing delay of the system in time slot t, and Δx i,k (t) represents whether the edge server caches the service, represents the delay of task k being offloaded to the associated edge server, and the service model of the corresponding task is cached, y i,k (t) represents the task offloading decision, represents the number of task k arriving at edge server i in time slot t, represents the computing delay. When i = o, it represents the computing delay of the cloud server. When it represents the computing delay of the edge server; represents the delay of task k being offloaded to the collaborative edge server, and j represents the offloading decision target, represents the set of edge servers, represents the set of services, and o represents the cloud server.

5. The joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning according to claim 1, wherein The optimization objective function is: Among them, D(t) represents the total task processing delay of the system in the t-th time slot, T represents the total number of time slots, ΔX represents the service cache update decision set of the edge server, represents the task offloading decision set, represents the resource allocation decision set, x i,k (t) represents the service cache status of edge server i, c k represents the storage space size required for service k, C i represents the total storage resource of edge server i, represents the set of edge servers, represents the set of time, K represents the set of services, m k represents the memory space required for service k, M i represents the total memory resource of edge server i, Δx i,k (t) represents whether the edge server caches the service, y i,k (t) represents the task offloading decision, represents the set of services, d k represents the task size of service k, I represents the number of edge servers, represents the number of tasks of service k arriving at edge server i in the t-th time slot, ρ i,k (t) represents the ratio of the computing resources allocated by edge server i to service k in the t-th time slot to the total computing resources, and o represents the cloud server.

6. The joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning according to claim 1, characterized in that The first sub-problem is expressed as: Among them, ΔX represents the service cache update decision set of the edge server, represents the optimal task offloading decision set, represents the optimal resource allocation decision set, T represents the total number of time slots, D(t) represents the total task processing delay of the system at time slot t, x i,k (t) represents the service cache status of edge server i, c k represents the storage space size required for service k, C i represents the total storage resource of edge server i, represents the set of edge servers, represents the set of time, K represents the set of services, m k represents the memory space required for service k, M i represents the total memory resource of edge server i, Δx i,k (t) represents whether the edge server caches the service, represents the set of services.

7. A joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning according to claim 1, characterized in that The second sub-problem is expressed as: Among them, D(t) represents the total task processing delay of the system in the t-th time slot, T represents the total number of time slots, and ΔX / represents the set of optimal service cache decisions, represents the set of task offloading decisions, represents the set of resource allocation decisions, C i represents the total storage resource of edge server i, represents the set of edge servers, represents the set of time, K represents the set of services, y i,k (t) represents the task offloading decision, represents the set of services, d k represents the task size of service k, I represents the number of edge servers, represents the number of tasks of service k arriving at edge server i in the t-th time slot, ρ i,k (t) represents the ratio of the computing resources allocated by edge server i to service k in the t-th time slot to the total computing resources, and o represents the cloud server.

8. A joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning according to claim 1, characterized in that The state space of the Markov decision process of the first sub-problem is expressed as: S i (t) = {(C i , M i ), X i (t), V i (t)} Among them, represents the state space of edge server i at time slot t, (C i , M i ) represents the cache space of edge server i, C i represents the total storage resource of edge server i, M i represents the total memory resource of edge server i, X i (t) represents the cache state of edge server i at time slot t, V i (t) represents the workload of edge server i at time slot t; The action space is expressed as: Among them, represents the action space of edge node i, and a i,k (t) represents the service caching decision, represents the service set; The reward function is expressed as: Among them, represents the reward of edge node i, and D(t) represents the total task processing delay at system time slot t.

9. A joint optimization method for edge service caching and task offloading based on hierarchical deep reinforcement learning according to claim 1, characterized in that, The state space of the Markov decision process of the second sub-problem is expressed as: Among them, represents the state space of the cloud server at time slot t, C i represents the total storage resources of edge server i, F i represents the total computing resources owned by edge server i, X i (t) represents the caching state of edge node i at time slot t, V i (t) represents the workload of edge node i at time slot t, ΔX i (t) represents the caching decision action of edge server i at time slot t, represents the set of edge servers; The action space is expressed as: Among them, represents the action space of the cloud server at time slot t, Δy i,k (t) represents the continuous action space for task offloading, ρ i,k (t) represents the continuous action space for resource allocation, represents the set of edge servers, represents the set of services; The reward function is expressed as: Among them, represents the reward of edge node i, and D(t) represents the total task processing delay at system time slot t.

Images