Calculation network resource scheduling optimization method based on multi-agent deep reinforcement learning

Abstract

The invention discloses a computing network resource scheduling optimization method based on multi-agent deep reinforcement learning, and the method comprises the steps: building a distributed service caching, task scheduling and computing network resource distribution model through comprehensively considering the multi-party computing power of a cloud side end, the system time delay, the system energy consumption and the load balance of heterogeneous nodes of an edge layer in a computing network scene; taking an edge server as an intelligent agent, considering the time dependence of a task request and the influence of the task request on service cache, and using an Attention-LSTM algorithm to make a service cache decision; based on a multi-agent depth deterministic policy gradient algorithm, constructing an MA-LDDPG algorithm to perform task scheduling and resource allocation decision; in consideration of the influence of service caching on task scheduling and resource allocation, a service caching strategy obtained by an intelligent agent is used as an additional observation state of task scheduling and resource allocation decision to participate in training, so that joint optimization of system time delay and edge layer heterogeneous node load balancing is achieved.

Description

Technical Field

[0001] The present invention relates to the field of resource scheduling technology, in particular to deep reinforcement learning and computing network resource scheduling optimization technology, and specifically to a computing network resource scheduling optimization method based on multi-agent deep reinforcement learning. Background Art

[0002] Data centers, edge computing nodes, and various end-user devices form a large and complex distributed computing resource network. In this network, relying solely on a single cloud, edge, or terminal computing power is difficult to meet the diverse, complex, and ever-changing computing needs of service tasks.

[0003] Service cache affects the task processing efficiency of edge servers, thereby affecting system latency. How to consider the time dependency of task requests and their impact on service cache, and make better task scheduling and resource allocation decisions based on improving the service cache hit rate is currently a major challenge.

[0004] Moreover, with the massive growth of devices and data, the single-agent control method will cause the action space of the agent to increase dramatically, resulting in low learning efficiency and inability to make optimal decisions. The centralized strategy learning scheme of deep reinforcement learning also faces the problem of significantly reduced performance and sharply increased complexity. In multi-agent problems, traditional reinforcement learning algorithms such as DQN have obstacles. The most important reason is that during the training process, for each agent, other agents are part of the environment, so changes in other agents will make the environment unstable, making it difficult for traditional reinforcement learning algorithms such as DQN to converge in a multi-agent environment. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a computing network resource scheduling optimization method based on multi-agent deep reinforcement learning, which is used to solve the technical problem that the resource scheduling algorithm based on traditional reinforcement learning has not considered the time dependence of task requests and is not easy to converge in a multi-agent environment.

[0006] To achieve the above object, the present invention provides a computing network resource scheduling optimization method based on multi-agent deep reinforcement learning, comprising the following steps: S1: Construct resource evaluation models for cloud data centers, edge servers, local devices, service types, and task requests in computing network scenarios; S2: To minimize system latency and maximize the load balancing level of heterogeneous nodes at the edge layer, a multi-objective joint optimization problem of distributed service caching, task scheduling, and computing network resource allocation is constructed under the constraints of storage space, computing performance, and bandwidth. S3: Further model the optimization problem as a Markov game, and design the corresponding system environment state, agent action and reward function of the Markov game; S4: Each edge server is regarded as an intelligent agent. For each intelligent agent, a service cache algorithm based on Attention-LSTM is preset to make service cache decisions; S5: For each agent, based on LSTM and MADDPG multi-agent deep deterministic policy gradient algorithms, the MA-LDDPG algorithm is preset to perform task scheduling and resource allocation decisions in a centralized training and decentralized execution manner. The service cache strategy obtained by the agent is used as an additional observation state for task scheduling and resource allocation decisions to participate in training; S6: Deploy the trained multi-agent strategy to the actual computing network system to solve the S3 optimization problem. The edge server acts as an intelligent agent to learn service caching, task scheduling, and resource allocation decisions, thereby achieving joint optimization of system latency and load balancing of heterogeneous nodes in the edge layer.

[0007] Further: S1 includes: ; The cloud data center set CF is modeled as, where ; The edge server set EF is modeled as ,in, ,and ,in, , and They represent the storage space size, CPU frequency and bandwidth of the edge server EFn respectively; there are i MD mobile devices under EFn, which are modeled as a set ,in, ; The set of service types is modeled as , where service type , and the storage space required to cache this type of task is ; Defining discrete-time sets ; for , the task generated in time slot t is modeled as: ; in, , , and They respectively represent the data size of the task, the maximum tolerable delay, the number of CPU cycles required to execute the task per unit bit, and the service type of the task.

[0008] Further: In S2, the service cache decision model is constructed as follows: For tasks of K service types, only edge servers that have cached the relevant service types can process tasks of this type. is the cache decision variable of service k in EFn, when , it means that EFn caches the service of type k in time slot t, and can process the related tasks. Otherwise, it is not cached, and EFn cannot process the task of this type. Then the cache decision of EFn for all services in time slot t is: .

[0009] Further: In S2, the task scheduling decision and system delay model are constructed as follows: For the task generated by MDi in time slot t , which has four scenarios of treatment; First, Represented as the task scheduling decision of MDi0, in, and are the decision variables for processing tasks locally, offloading tasks to the corresponding edge server, migrating tasks to nearby edge servers, and offloading tasks to the cloud data center. ; and ; Scenario 1: , that is, the task is processed locally; make represents the local CPU frequency of MDi, then the total latency of the task being processed locally is: ; Scenario 2: , that is, the task is offloaded to the corresponding edge server for processing; If the edge server EFn to which MDi belongs caches the task If the task type is k, the task can be offloaded to EFn for processing; according to Shannon's formula, the task The transmission delay from offloading to EFn is: ; in, Indicates that EFn is assigned to the task in time slot t The bandwidth allocation coefficient is is the transmit power of MDi, is the channel gain between MDi and EFn, is the Gaussian noise probability of the device in time slot t; The processing time of the task on the edge server EFn is: ; in, Indicates that EFn gives the task in time slot t The CPU allocation factor; Therefore, the total delay of offloading the task to the corresponding edge server for processing is: ; Scenario 3: , that is, the task is migrated to the nearby edge server for processing; If the edge server EFn to which MDi belongs does not cache the task If the task type k is k, and the nearby edge server EFn' has cached the task type k, the task can be migrated to the nearby edge server for processing through the edge server to which it belongs; According to Shannon's formula, the forwarding time of a task from its edge server to a nearby edge server is: ; in, is the transmission power of EFn, EFn and The channel gain between Tasks on edge servers The processing time on is: ; Therefore, the total delay of migrating tasks to nearby edge servers for processing is: ; Scenario 4: , that is, the task is offloaded to the cloud data center for processing; The transmission time of the task forwarded from the edge server EFn to the cloud data center CFm is: ; in, represents the transmission rate of the task from EFn to CFm in time slot t; Therefore, the total latency of offloading the task to the cloud data center for processing is: .

[0010] Further: In S2, the edge computing layer load balancing model is constructed as follows: In order to allocate resources reasonably, the load level of each edge server in the edge computing layer in time slot t is defined as: ; in, They represent the bandwidth, CPU and memory space resource utilization of EFn in time slot t, They are Their respective weight coefficients, and: ; ; ; The standard deviation is used to define the load balancing level of all servers in the edge computing layer in time slot t: ; in, Represents the average load level of all servers in the edge computing layer in time slot t: .

[0011] Further: In S2, the optimization problem is constructed as follows: ; ; ; ; ; ; ; ; in, The weight coefficients represent the system latency and the load balancing degree of the edge computing layer respectively; and They are the decision variables for processing tasks locally, offloading tasks to the corresponding edge server, migrating tasks to nearby edge servers, and offloading tasks to the cloud data center. is the cache decision variable of service k in EFn; Indicates the storage space size of the edge server EFn; Indicates that EFn is assigned to the task in time slot t The bandwidth allocation coefficient of Indicates that EFn gives the task in time slot t The CPU allocation factor; They are The respective weight coefficients.

[0012] Further: in said S3; The system environment status is: ; in , there are N edge servers in the system, that is, N intelligent agents, represents the local observation state of agent n at time slot t, which is defined as: ; The agent actions are: ; in, It represents the action taken by agent n in time slot t, which consists of two parts, one of which is the service cache action: The other part is the task scheduling and resource allocation actions based on the service cache action: ; therefore: ; The reward function is: ; Among them, the first part is the reward for system latency, and the second part is the reward for the load balancing level of the edge computing layer.

[0013] Further: in said S4, the number of agents is equal to the number of edge servers, each edge server is controlled by an agent, and each agent is provided with a service cache decision algorithm based on Attention-LSTM and a task scheduling and resource allocation decision algorithm based on MA-LDDPG algorithm; The steps of the service cache decision algorithm based on Attention-LSTM are as follows: Step 1: Input the task request data in the agent area and extract the service type information; Step 2: The forget gate uses the sigmoid function to decide which information to forget and retain. At each time slot t, the forget gate is defined as: ; in, is the weight matrix of the forget gate, is the bias term, is the hidden state of the previous time slot, is the input of the current time slot, , represents the retention ratio of historical information; Step 3: Update the cell state through the input gate. The input gate consists of two parts. One part is the sigmoid layer, which determines which state information needs to be updated; the other part is the tanh layer, which updates the cell state with the new state information. The input gate is defined as: ; in, , , and Represent the weights of the sigmoid layer and the tanh layer respectively, and They represent the corresponding bias respectively; Step 4: Based on the cell state, the prediction result is obtained and output through the output gate. The output gate also contains two layers: sigmoid layer and tanh layer; First, the hidden state of the previous time slot and the current input are passed to the sigmoid layer to determine the state information for the output. Then the newly obtained cell state is passed to the tanh layer to convert it into the output result. Finally, the output of the sigmoid layer is multiplied by the output of the tanh layer. The output gate is defined as: ; in, , and Represent the corresponding weights and biases respectively; Step 5: Calculate the attention distribution weight and obtain the final output result through weighted summation. The calculation formula is: ; in, represents the implicit representation of the output of the LSTM layer, and are the corresponding weights and biases, represents the probability of initializing attention distribution, represents the attention distribution weight obtained by the softmax function, Represents the weighted output result.

[0014] Further: in S5, the cache strategy obtained by the service cache decision algorithm will participate in the training as an additional observation state of the agent in the task scheduling and resource allocation decision; the constructed MA-LDDPG algorithm introduces the LSTM network into the actor network and critic network of the MADDPG algorithm so that the agent can obtain the network state information related to the time series, which helps the agent to better understand the future environment state, take better actions, and obtain higher rewards; In addition, centralized training and decentralized execution means that in the training state, the critic network of each agent not only knows its own behavior, but also knows the action strategies of other agents. Therefore, the Q function is expressed as: ; In the execution state, the actor network only needs to know its own local information, that is, each agent can learn from the experience of other agents through the shared experience pool during the training process to better cooperate and adjust local strategies accordingly.

[0015] The beneficial effects of the present invention are: 1: In the computing network scenario, the computing power of multiple parties in the cloud, edge and terminal is comprehensively considered, and the system latency, system energy consumption and load balancing of heterogeneous nodes in the edge layer are considered to build a distributed service caching, task scheduling and computing network resource allocation model.

[0016] 2: Considering the time dependency of task requests and their impact on service cache, a long short-term memory network (Attention-LSTM) combined with an attention mechanism is used to capture time-related task information and make service cache decisions, thereby improving the cache hit rate of the agent.

[0017] 3: Based on LSTM and multi-agent deep deterministic policy gradient algorithm (MADDPG), the present invention constructs the MA-LDDPG algorithm, and performs task scheduling and resource allocation decisions in a centralized training and decentralized execution manner; the algorithm takes into account the network status information related to timing, and introduces the LSTM network in the actor network and critic network of the MADDPG algorithm, so that the agent can obtain the network status information related to timing, which helps the agent to better understand the future environment status and make better task scheduling and resource allocation decisions. In addition, each agent can learn from the experience of other agents by sharing the experience pool during the training process, so as to better cooperate and adjust the local strategy to achieve the global optimum. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention and the technical solutions in the prior art, the drawings required for use in the embodiments and the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some implementation methods recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is a flow chart of the computing network resource scheduling method in the present invention; Figure 2 This is a schematic diagram of the computing network scenario system framework in the present invention; Figure 3 This is the architecture diagram of the Attention-LSTM algorithm in the present invention; Figure 4 This is the architecture diagram of the MA-LDDPG algorithm in the present invention. DETAILED DESCRIPTION

[0020] The specific embodiments of the present invention are described in detail below, and the process of the specific embodiments is shown in the accompanying drawings, wherein the same reference numerals represent the same parameters from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0021] Furthermore, in the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure; however, it is apparent that one or more embodiments may be practiced without these specific details.

[0022] See also Figures 1 to 4 , the present invention provides a computing network resource scheduling optimization method based on multi-agent deep reinforcement learning: See also Figure 1 : The flow chart of the computing network resource scheduling method provided by the present invention includes the following steps: Step 1: Build a resource evaluation model for cloud data centers, edge servers, local devices, service types, and task requests in computing network scenarios; Specifically: The schematic diagram of the computing network scenario system model framework is as follows: Figure 2 As shown in the figure: the computing network scenario includes several cloud data centers and edge servers. There are several mobile devices within the range of each edge server. The tasks generated by the mobile devices can be offloaded to the edge servers associated with them through the public network or dedicated lines. The edge servers can migrate the tasks to nearby edge servers or cloud data centers through wireless links or backhaul links. The cloud data center set CF is modeled as ; The edge server set EF is modeled as ,in, , and ,in, , and They represent the storage space size, CPU frequency and bandwidth of the edge server EFn respectively; there are i MD mobile devices under EFn, which are modeled as a set ,in, ; The set of service types is modeled as , where service type , and the storage space required to cache this type of task is ; Defining discrete-time sets ; for , the task generated in time slot t is modeled as: ; in, , , and They respectively represent the data size of the task, the maximum tolerable delay, the number of CPU cycles required to execute the task per unit bit, and the service type of the task.

[0023] Step 2: With the goal of minimizing system latency and maximizing the load balancing level of heterogeneous nodes at the edge layer, a multi-objective joint optimization problem of distributed service caching, task scheduling, and computing network resource allocation is constructed under the constraints of storage space, computing performance, and bandwidth. Specifically: The service cache decision model is built based on the following principles: For tasks of K service types, only edge servers that have cached the relevant service types can process tasks of this type. is the cache decision variable for service k in EFn, when , it means that EFn caches the service of type k in time slot t, and can process the related tasks. Otherwise, it is not cached, and EFn cannot process the task of this type. Then the cache decision of EFn for all services in time slot t is: .

[0024] Specifically: The task scheduling decision and system delay model are constructed as follows: For the task generated by MDi in time slot t , which has four scenarios of treatment; First, Represented as the task scheduling decision of MDi0, in, and are the decision variables for processing tasks locally, offloading tasks to the corresponding edge servers, migrating tasks to nearby edge servers, and offloading tasks to cloud data centers. ; and ; Scenario 1: , that is, the task is processed locally; make represents the local CPU frequency of MDi, then the total latency of the task being processed locally is: ; Scenario 2: , that is, the task is offloaded to the corresponding edge server for processing; If the edge server EFn to which MDi belongs caches the task If the task type is k, the task can be offloaded to EFn for processing; according to Shannon's formula, the task The transmission delay from offloading to EFn is: ; in, Indicates that EFn is assigned to the task in time slot t The bandwidth allocation coefficient is is the transmit power of MDi, is the channel gain between MDi and EFn, is the Gaussian noise probability of the device in time slot t; The processing time of the task on the edge server EFn is: ; in, Indicates that EFn gives the task in time slot t The CPU allocation factor; Therefore, the total delay of offloading the task to the corresponding edge server for processing is: ; Scenario 3: , that is, the task is migrated to the nearby edge server for processing; If the edge server EFn to which MDi belongs does not cache the task If the task type k is k, and the nearby edge server EFn' has cached the task type k, the task can be migrated to the nearby edge server for processing through the edge server to which it belongs; According to Shannon's formula, the forwarding time of a task from its edge server to a nearby edge server is: ; in, is the transmission power of EFn, EFn and The channel gain between Tasks on edge servers The processing time on is: ; Therefore, the total delay of migrating tasks to nearby edge servers for processing is: ; Scenario 4: , that is, the task is offloaded to the cloud data center for processing; The transmission time of the task forwarded from the edge server EFn to the cloud data center CFm is: ; in, represents the transmission rate of the task from EFn to CFm in time slot t; Therefore, the total latency of offloading the task to the cloud data center for processing is: .

[0025] Specifically: In order to allocate resources reasonably, the load level of each edge server in the edge computing layer in time slot t is defined as: ; in, They represent the bandwidth, CPU and memory space resource utilization of EFn in time slot t, They are Their respective weight coefficients, and: ; ; ; The standard deviation is used to define the load balancing level of all servers in the edge computing layer in time slot t: ; in, Represents the average load level of all servers in the edge computing layer in time slot t: .

[0026] Further: The optimization problem is constructed as follows: ; ; ; ; ; ; ; ; in, The weight coefficients represent the system latency and the load balancing degree of the edge computing layer respectively; and They are the decision variables for processing tasks locally, offloading tasks to the corresponding edge server, migrating tasks to nearby edge servers, and offloading tasks to the cloud data center. is the cache decision variable of service k in EFn; Indicates the storage space size of the edge server EFn; Indicates that EFn is assigned to the task in time slot t The bandwidth allocation coefficient of Indicates that EFn gives the task in time slot t The CPU allocation factor; They are The respective weight coefficients.

[0027] Step 3: Further model the optimization problem as a Markov game, and design the system environment state, agent action, and reward function corresponding to the Markov game; Specifically: The system environment status is: ; Where n∈N, there are N edge servers in the system, that is, N intelligent agents. represents the local observation state of agent n at time slot t, which is defined as: ; The agent actions are: ; in, It represents the action taken by agent n in time slot t, which consists of two parts, one of which is the service cache action: The other part is the task scheduling and resource allocation actions based on the service cache action: ; therefore: ; The reward function is: ; Among them, the first part is the reward for system latency, and the second part is the reward for the load balancing level of the edge computing layer.

[0028] Step 4: Consider each edge server as an intelligent agent. For each intelligent agent, a service cache algorithm based on Attention-LSTM is preset to make service cache decisions. The steps of the service cache decision algorithm based on Attention-LSTM are as follows: Step 1: Input the task request data in the agent area and extract the service type information; Step 2: The forget gate uses the sigmoid function to decide which information to forget and retain. At each time slot t, the forget gate is defined as: ; in, is the weight matrix of the forget gate, is the bias term, is the hidden state of the previous time slot, is the input of the current time slot, , represents the retention ratio of historical information; Step 3: Update the cell state through the input gate. The input gate consists of two parts. One part is the sigmoid layer, which determines which state information needs to be updated; the other part is the tanh layer, which updates the cell state with the new state information. The input gate is defined as: ; in, , , and Represent the weights of the sigmoid layer and the tanh layer respectively, and They represent the corresponding bias respectively; Step 4: Based on the cell state, the prediction result is obtained and output through the output gate. The output gate also contains two layers: sigmoid layer and tanh layer; First, the hidden state of the previous time slot and the current input are passed to the sigmoid layer to determine the state information for the output. Then the newly obtained cell state is passed to the tanh layer to convert it into the output result. Finally, the output of the sigmoid layer is multiplied by the output of the tanh layer. The output gate is defined as: ; in, , and Represent the corresponding weights and biases respectively; Step 5: Calculate the attention distribution weight and obtain the final output result through weighted summation. The calculation formula is: ; in, represents the implicit representation of the output of the LSTM layer, and are the corresponding weights and biases, represents the probability of initializing attention distribution, represents the attention distribution weight obtained by the softmax function, Represents the weighted output result; Step 5: For each agent, based on LSTM and multi-agent deep deterministic policy gradient algorithm (MADDPG), the MA-LDDPG algorithm is preset to perform task scheduling and resource allocation decisions in a centralized training and decentralized execution manner. The service cache strategy obtained by the agent is used as an additional observation state for task scheduling and resource allocation decisions to participate in training; Specifically: The MA-LDDPG algorithm architecture is as follows: Figure 4 As shown; Among them: The Q function of the critic network is expressed as: ; In the execution state, the actor network only needs to know its own local information, that is, each agent can learn from the experience of other agents through the shared experience pool during the training process to better cooperate and adjust local strategies accordingly; Further: The specific steps of the MA-LDDPG algorithm are as follows: Input: number of episodes, training step size, network learning rate, discount factor ,Soft update rate and , Cache Action ; Initialization: network parameters of actor network and critic network, experience buffer pool, random noise ; Output: Task scheduling and resource allocation actions ; 1: ; 2: Command , observe the local observation state of the agent ; 3: ; 4: Get service cache action from service cache decision algorithm ; 5: Agent n based on local observation state , service cache action , timing-related network status information in the actor network And its own task scheduling and resource allocation strategy , select the Task Scheduling and Resource Allocation action: ,in, represents random noise; 6: Agent n gets corresponding rewards based on actions And the next moment observation state ; 7: Stored in the experience cache pool, where Represents the timing-related network status information in the actor network and critic network respectively; 8: Update relevant information: ; 9: for Agent ; 10: Randomly extract several samples from the experience buffer pool to participate in training ; 11: Update the critic network parameters by minimizing the loss function : ; in, ; represents the network parameters of the critic network in agent n, is the discount factor, represents the expected return of agent n in the next time slot; 12: Update actor network parameters by minimizing policy gradients : ; in, It is the task scheduling and resource allocation strategy of agent n; 13: end for; 14: Update the target network parameters of the actor network and critic network in agent n: ;in They represent the network parameters of the actor network and critic network in agent n at the next moment, and denote the soft update rate of the target network in the actor network and the critic network respectively, and , ; 15: end for ends.

[0029] Step 6: Deploy the trained multi-agent strategy to the actual computing network system to solve the problem. The edge server acts as an intelligent agent to learn service caching, task scheduling, and resource allocation decisions, thereby achieving joint optimization of system latency and load balancing of heterogeneous nodes in the edge layer.

[0030] The present invention comprehensively considers the multi-party computing power of the cloud edge in the computing network scenario, considers the system delay, system energy consumption and load balancing of heterogeneous nodes in the edge layer, and constructs a distributed service cache, task scheduling and computing network resource allocation model; and considers the time dependency of task requests and its impact on service cache, uses a long short-term memory network (Attention-LSTM) combined with an attention mechanism to capture time-related task information and make service cache decisions, thereby improving the cache hit rate of the intelligent agent; it also considers the impact of service cache on task scheduling and computing network resource allocation, and uses the service cache strategy obtained by the intelligent agent as an additional observation state for task scheduling and resource allocation decisions to participate in training, and obtains the service cache strategy when the intelligent agent obtains the service cache strategy Based on LSTM and multi-agent deep deterministic policy gradient algorithm (MADDPG), the MA-LDDPG algorithm is constructed to perform task scheduling and resource allocation decisions in a centralized training and decentralized execution manner; the algorithm takes into account the network state information related to timing, introduces the LSTM network in the actor network and critic network of the MADDPG algorithm, so that the agent can obtain the network state information related to timing, helps the agent to better understand the future environment state, and make better task scheduling and resource allocation decisions. In addition, each agent can learn from the experience of other agents by sharing the experience pool during the training process, so as to better cooperate and adjust the local strategy to achieve the global optimum.

[0031] The above is a very detailed description of one or more embodiments of the present invention, but the content is only a specific example of the present invention and cannot be considered to limit the scope of application of the present invention. Any other methods and changes proposed based on the content of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A computing network resource scheduling optimization method based on multi-agent deep reinforcement learning, characterized in that: The steps include: S1: Construct resource evaluation models for cloud data centers, edge servers, local devices, service types, and task requests in computing network scenarios; S2: To minimize system latency and maximize the load balancing level of heterogeneous nodes at the edge layer, a multi-objective joint optimization problem of distributed service caching, task scheduling, and computing network resource allocation is constructed under the constraints of storage space, computing performance, and bandwidth. S3: Further model the optimization problem as a Markov game, and design the corresponding system environment state, agent action and reward function of the Markov game; S4: Each edge server is regarded as an intelligent agent. For each intelligent agent, a service cache algorithm based on Attention-LSTM is preset to make service cache decisions; S5: For each agent, based on LSTM and MADDPG multi-agent deep deterministic policy gradient algorithms, the MA-LDDPG algorithm is preset to perform task scheduling and resource allocation decisions in a centralized training and decentralized execution manner. The service cache strategy obtained by the agent is used as an additional observation state for task scheduling and resource allocation decisions to participate in training; S6: Deploy the trained multi-agent strategy to the actual computing network system to solve the S3 optimization problem. The edge server acts as an intelligent agent to learn service caching, task scheduling, and resource allocation decisions, thereby achieving joint optimization of system latency and load balancing of heterogeneous nodes in the edge layer.

2. According to claim 1, a computing network resource scheduling optimization method based on multi-agent deep reinforcement learning is characterized in that: The S1 includes: ; The cloud data center set CF is modeled as, where ; The edge server set EF is modeled as ,in, ,and ,in, , and They represent the storage space size, CPU frequency and bandwidth of the edge server EFn respectively; there are i MD mobile devices under EFn, which are modeled as a set ,in, ; The set of service types is modeled as , where service type , and the storage space required to cache this type of task is ; Defining discrete-time sets ; for , the task generated in time slot t is modeled as: ； in, , , and They respectively represent the data size of the task, the maximum tolerable delay, the number of CPU cycles required to execute the task per unit bit, and the service type of the task.

3. According to claim 1, a computing network resource scheduling optimization method based on multi-agent deep reinforcement learning is characterized in that: In S2, the service cache decision model is constructed as follows: For tasks of K service types, only edge servers that have cached the relevant service types can process tasks of this type. is the cache decision variable of service k in EFn, when , it means that EFn caches the service of type k in time slot t, and can process the related tasks. Otherwise, it is not cached, and EFn cannot process the task of this type. Then the cache decision of EFn for all services in time slot t is: 。 4. According to claim 1, a computing network resource scheduling optimization method based on multi-agent deep reinforcement learning is characterized in that: In S2, the task scheduling decision and system delay model are constructed as follows: For the task generated by MDi in time slot t , which has four scenarios of treatment; First, The task scheduling decision represented by MDi0, in, and are the decision variables for processing tasks locally, offloading tasks to the corresponding edge server, migrating tasks to nearby edge servers, and offloading tasks to the cloud data center. ;and ; Scenario 1: , that is, the task is processed locally; make represents the local CPU frequency of MDi, then the total latency of the task being processed locally is: ； Scenario 2: , that is, the task is offloaded to the corresponding edge server for processing; If the edge server EFn to which MDi belongs caches the task If the task type is k, the task can be offloaded to EFn for processing; according to Shannon's formula, the task The transmission delay from offloading to EFn is: ； in, Indicates that EFn is assigned to the task in time slot t The bandwidth allocation coefficient is is the transmit power of MDi, is the channel gain between MDi and EFn, is the Gaussian noise probability of the device in time slot t; The processing time of the task on the edge server EFn is: ； in, Indicates that EFn gives the task in time slot t The CPU allocation coefficient; Therefore, the total delay of offloading the task to the corresponding edge server for processing is: ； Scenario 3: , that is, the task is migrated to the nearby edge server for processing; If the edge server EFn to which MDi belongs does not cache the task If the task type k is k, and the nearby edge server EFn' has cached the task type k, the task can be migrated to the nearby edge server for processing through the edge server to which it belongs; According to Shannon's formula, the forwarding time of a task from its edge server to a nearby edge server is: ； in, is the transmission power of EFn, EFn and The channel gain between Tasks on edge servers The processing time on is: ； Therefore, the total delay of migrating tasks to nearby edge servers for processing is: ； Scenario 4: , that is, the task is offloaded to the cloud data center for processing; The transmission time of the task forwarded from the edge server EFn to the cloud data center CFm is: ； in, represents the transmission rate of the task from EFn to CFm in time slot t; Therefore, the total latency of offloading the task to the cloud data center for processing is: 。 5. According to claim 1, a computing network resource scheduling optimization method based on multi-agent deep reinforcement learning is characterized in that: In S2, the edge computing layer load balancing model is constructed as follows: In order to allocate resources reasonably, the load level of each edge server in the edge computing layer in time slot t is defined as: ； in, They represent the bandwidth, CPU and memory space resource utilization of EFn in time slot t, They are Their respective weight coefficients, and: ； ； ； The standard deviation is used to define the load balancing level of all servers in the edge computing layer in time slot t: ； in, Represents the average load level of all servers in the edge computing layer in time slot t: 。 6. According to claim 1, a computing network resource scheduling optimization method based on multi-agent deep reinforcement learning is characterized in that: In S2, the optimization problem is constructed as follows: ； ； ； ； ； ； ； ； in, The weight coefficients represent the system latency and the load balancing degree of the edge computing layer respectively; and They are the decision variables for processing tasks locally, offloading tasks to the corresponding edge server, migrating tasks to nearby edge servers, and offloading tasks to the cloud data center. is the cache decision variable of service k in EFn; Indicates the storage space size of the edge server EFn; Indicates that EFn is assigned to the task in time slot t The bandwidth allocation coefficient of Indicates that EFn gives the task in time slot t The CPU allocation coefficient; They are The respective weight coefficients.

7. According to claim 1, a computing network resource scheduling optimization method based on multi-agent deep reinforcement learning is characterized in that: In said S3; The system environment status is: ； Where n∈N, there are N edge servers in the system, that is, N intelligent agents. represents the local observation state of agent n at time slot t, which is defined as: ； The agent actions are: ； in, It represents the action taken by agent n in time slot t, which consists of two parts, one of which is the service cache action: The other part is the task scheduling and resource allocation actions based on the service cache action: ； therefore: ; The reward function is: ； Among them, the first part is the reward for system latency, and the second part is the reward for the load balancing level of the edge computing layer.

8. According to claim 1, a computing network resource scheduling optimization method based on multi-agent deep reinforcement learning is characterized in that: In S4, the number of agents is equal to the number of edge servers, each edge server is controlled by an agent, and each agent is provided with a service cache decision algorithm based on Attention-LSTM and a task scheduling and resource allocation decision algorithm based on MA-LDDPG algorithm; The steps of the service cache decision algorithm based on Attention-LSTM are as follows: Step 1: Input the task request data in the agent area and extract the service type information; Step 2: The forget gate uses the sigmoid function to decide which information to forget and retain. At each time slot t, the forget gate is defined as: ； in, is the weight matrix of the forget gate, is the bias term, is the hidden state of the previous time slot, is the input of the current time slot, , represents the retention ratio of historical information; Step 3: Update the cell state through the input gate. The input gate consists of two parts. One part is the sigmoid layer, which determines which state information needs to be updated; the other part is the tanh layer, which updates the cell state with the new state information. The input gate is defined as: ； in, , , and Represent the weights of the sigmoid layer and the tanh layer respectively, and They represent the corresponding bias respectively; Step 4: Based on the cell state, the prediction result is obtained and output through the output gate. The output gate also contains two layers: sigmoid layer and tanh layer; First, the hidden state of the previous time slot and the current input are passed to the sigmoid layer to determine the state information for output. Then the newly obtained cell state is passed to the tanh layer to convert it into the output result. Finally, the output of the sigmoid layer is multiplied by the output of the tanh layer. The output gate is defined as: ； in, , and Represent the corresponding weights and biases respectively; Step 5: Calculate the attention distribution weight and obtain the final output result through weighted summation. The calculation formula is: ； in, represents the implicit representation of the output of the LSTM layer, and are the corresponding weights and biases, represents the probability of initializing attention distribution, represents the attention distribution weight obtained by the softmax function, Represents the weighted output result.

9. The method for optimizing network resource scheduling based on multi-agent deep reinforcement learning according to claim 1, characterized in that: In S5, the cache strategy obtained by the service cache decision algorithm will participate in the training as an additional observation state of the agent in the task scheduling and resource allocation decision; the constructed MA-LDDPG algorithm introduces the LSTM network into the actor network and critic network of the MADDPG algorithm so that the agent can obtain time-related network state information, helping the agent to better understand the future environment state, take better actions, and obtain higher rewards; In addition, centralized training and decentralized execution means that in the training state, the critic network of each agent not only knows its own behavior, but also knows the action strategies of other agents. Therefore, the Q function is expressed as: ； In the execution state, the actor network only needs to know its own local information, that is, each agent can learn from the experience of other agents through the shared experience pool during the training process to better cooperate and adjust local strategies accordingly.

Images