An online resource scheduling method with guaranteed service quality

Abstract

The present invention relates to an online resource scheduling method for ensuring service quality, and belongs to the field of communication technology. The method studies edge server resource coordination and dynamic task scheduling, with the aim of achieving higher user QoS by relying on existing network resources. A user QoS evaluation model is constructed by combining network throughput and long-term average response delay. In order to optimize QoS, taking into account the temporal continuity of task scheduling, an online task scheduling algorithm based on a multi-agent bidirectional coordination network is adopted to perform long-term reward learning of scheduling decisions, so as to achieve long-term global optimal resource coordination. Compared with centralized learning, the multi-agent bidirectional coordination network adds a bidirectional recurrent neural network as a communication layer between agents for information exchange and temporary storage, which effectively reduces complexity.

Description

Technical Field

[0001] The invention belongs to the technical field of communications and relates to an online resource scheduling method for ensuring service quality. Background Art

[0002] With the rapid development of the Internet of Things and the increasing popularity of various smart devices, various emerging industries such as industrial control, entertainment industry, smart cities and smart medical care have generated a large amount of computing data, which relies on real-time and efficient applications for processing and analysis, such as virtual reality (VR), autonomous driving, wearable devices, etc. Cloud computing has sufficient computing resources and service operators have deployed related applications in the cloud, so it can effectively cope with intensive requests for various tasks. However, since the remote cloud is far away from the user, the communication delay generated cannot guarantee the user's quality of service (QoS). To meet this challenge, Mobile Edge Computing (MEC) provides services to users by setting up edge clouds close to user terminals, effectively reducing communication delays and improving user QoS. At the same time, in order to better utilize the resources of the edge cloud and enhance the flexibility of task processing, service providers provide users with lightweight services by decomposing applications into a series of loosely coupled microservice modules. Once the user generates a request task, the microservices deployed on the edge cloud can process the task.

[0003] However, since the number of requests for various services changes dynamically in time and space, ES has an uneven load problem. This leads to redundant resources on some servers and overload on others. In order to achieve resource coordination in the network and improve the resource utilization of ES, it is necessary to schedule tasks to neighboring ES for collaborative processing. Collaborative scheduling will generate additional latency overhead, further affecting user QoS. Therefore, how to perform collaborative task scheduling to achieve efficient resource coordination while ensuring QoS is an urgent problem to be solved. Summary of the invention

[0004] In view of this, an object of the present invention is to provide an online resource scheduling method with service quality assurance, which performs collaborative task scheduling to achieve efficient resource coordination while ensuring QoS.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] An online resource scheduling method for ensuring service quality, specifically comprising the following steps:

[0007] S1: Global QoS evaluation: The task scheduling matrix is ​​set based on the task request and microservice deployment, and the response delay is analyzed in a fine-grained manner to build a long-term average response delay model for different applications within the time range T. The network throughput is obtained based on the real-time requirements of the task, and a QoS evaluation model is established by combining the long-term average response delay and throughput.

[0008] S2: Long-term collaborative scheduling: A collaborative task scheduling model is established based on the task scheduling mode of ES and network QoS. The MDP model is used to analyze the long-term continuous scheduling process. At the same time, each ES is regarded as an intelligent agent, and the intelligent agents use a cooperative game model to make collaborative decisions.

[0009] S3: Multi-agent collaborative learning: A bidirectional coordination network BiCNet is used to build a cooperative communication layer, and an Actor-Critic network is used to interact between agents and the environment. The global QoS is used as a reward for each agent, and the global optimal solution is obtained through iterative learning of strategies and values. Finally, a collaborative scheduling bidirectional coordination network algorithm is constructed to realize the overall task scheduling process.

[0010] Further, the step S1 specifically includes the following steps:

[0011] S11: Assume that the request set generated by the user in time slot t is is the number of request types; request k is represented by a four-tuple in, Indicates the type of microservice required by the request, Indicates the requested task data size, k Indicates the latency tolerance threshold of the request. represents the size of the result data after processing; the number of requests k issued by users in ESi in time slot t is

[0012] Define the scheduling scheme for time slot t as e t =[e k,i,j ] K×V×V , where e t is a three-dimensional matrix of K×V×V, and the element e in the matrix k,i,j represents the number of requests k scheduled from ESi to ESj, which is a natural number greater than or equal to zero, satisfying the constraint

[0013] S12: According to the actual task scheduling process, the user first uploads the request to the nearest ES node. After each ES receives all the requests in the current time slot, it performs collaborative task scheduling between ESs. After the scheduling is completed, each ES sends the corresponding task to the corresponding ES according to the scheduling result; for tasks that cannot be processed, they are sent to the remote cloud for processing by the remote cloud; tasks are processed in parallel at the microservice; after the task processing is completed, the result data is transmitted back to the original ES node and sent back to the user;

[0014] S13: The goal of collaborative task scheduling is to maximize the number of tasks that meet the delay tolerance threshold from a long-term perspective, and further minimize the ATRL of the tasks;

[0015] For request k, if its ATRL exceeds the delay tolerance threshold ζ k , the task will be interrupted; otherwise, the task will be considered completed and the network throughput will increase; then the network throughput in time slot t is expressed as:

[0016]

[0017]

[0018] In the entire time range T, the average throughput per time slot in the network is expressed as:

[0019]

[0020] S14: The main goal is to maximize the throughput within the entire time range, and minimize ATRL on this basis. The QoS evaluation index constructed in this way is expressed as:

[0021] θ=σ·P ave -(1-σ)·T ave

[0022] In the formula, σ is the weight factor; let e = [e 1 ,e 2 ,...,e t ,...,e T ] is the scheduling decision on the entire time series, then the collaborative task scheduling problem (Collaborative Task Scheduling Problem, CTSP) is expressed as:

[0023]

[0024]

[0025]

[0026] In the formula, Represents the node set consisting of the remote cloud and all ES nodes. Represents the set of natural numbers.

[0027] Further, in step S12, the delay of each stage is defined as follows:

[0028] The task upload delay is described as the ratio of the upload task data size to the data transmission rate. The wireless transmission rate between the user and the ES is calculated by the Shannon formula; w i represents the wireless channel bandwidth of ES i, SNR i represents the signal-to-noise ratio, then the task upload delay of request k in ESi is:

[0029]

[0030] After the task is uploaded to ES, ES transfers the task according to the scheduling result. If j = 0, it means that the task is transferred to the remote cloud for processing. i,j represents the link transmission bandwidth between ESi and ESj, then the scheduling delay is:

[0031]

[0032] After the scheduling is completed, the task needs to be processed; define μ k is the task processing rate of the microservice that processes request k, h k,j and k,j are the remaining resources and resource usage of microservice k in the current time slot on ESj; y k,j Satisfy the constraints:

[0033] 0<y k,j ≤h k,j

[0034] If the required microservice is not available in the neighboring ES, the task will be scheduled to the remote cloud for processing, i.e., j = 0; V represents the number of ES in the network, and in the parallel processing process, the task processing delay is expressed as:

[0035]

[0036] After the task is processed, the processing result of the task is transmitted back to the original ES node. The transmission delay is:

[0037]

[0038] Finally, the processing result of the user downloading the ES node generates a download delay expressed as:

[0039]

[0040] Then, in time slot t, the total delay of co-processing of request k scheduled from ESi to ESj is expressed as:

[0041]

[0042] If the required microservice is not available on ES, the task needs to be scheduled to the remote cloud for processing, that is, j = 0, and the above process remains unchanged; if all requests are considered from a long-term perspective, the long-term average task response latency (Average Tsk Response Latency, ATRL) of K types of requests within the time range T is expressed as:

[0043]

[0044] Further, the step S2 specifically includes the following steps:

[0045] S21: Take a tuple Describe the game model of collaborative task scheduling; represents the state space of the current game, which is shared by all agents; the network state at time slot t is expressed as in, is the network request status at the beginning of time slot t, h t =[h 1,1 ,h 1,2 ,...,h k,i ,...,h K,V ] is the remaining microservice resources of all ES at the beginning of time slot t; the action space of ES i decision is expressed as is the action of ESi in time slot t; represents the state transition function; represents the reward function of ESi;

[0046] S22: For microservice k on ESj, define the resources that are being occupied by the dth group as a binary express, Indicates the amount of resources occupied. represents the remaining occupied time; let ψ ooc represents the set of occupied resources, then at the beginning of the current time slot, the resource usage y k,j Will be placed in the occupied resources collection Update the remaining time of all occupied resources at the same time:

[0047] Then the remaining resources of microservice k on ESj at the beginning of the next time slot are expressed as: S23: Each ES makes its own scheduling decision by observing the current state of the environment and obtains a reward value. The global reward of the network is defined by the average of the reward values ​​of all ESs, expressed as: In the formula, r t Indicates that in state s t In this case, all agents take action a t The reward value obtained by the global network is the reward obtained by each agent i The average value is used to describe it; V represents the number of agents, which is equal to the number of ES; the purpose of ES global optimization is to maximize the discounted reward expectation over the entire time range through learning Among them, 0≤δ<1 is the discount factor; therefore, the goal of the network at each moment is to maximize the state s t Next, take action a t The action value function Q * , expressed as: The global reward function reflects the overall result of the cooperative game of all ESs, which needs to be jointly determined by the reward value of each ES. At the same time, each ES determines how to coordinate scheduling based on its own reward value.

[0048] S24: According to step S11, the reward value of ESi is calculated as: In the formula, Indicates that in state s t Next, the global network adopts action a t The reward value obtained by ESi is determined by the average throughput and ATRL; then the action value function corresponding to V ES is expressed as: The ES reward value is used as the action feedback of the ES for training. After a certain period of learning and training, each ES forms its own decision-making model. At each moment, the ES finds the scheduling decision that is most conducive to the global optimality in the decision-making model.

[0049] Further, the step S3 specifically includes the following steps:

[0050] S31: First, the original network is expanded into a network consisting of V child nodes, and time back propagation is used to calculate the reverse gradient to learn the parameters of the bidirectional coordination network BiCNet; the reward gradients of all agents are passed to the BiCNet network to guide the agent's strategy, and the BiCNet network parameters θ are further updated through the back propagation of the communication layer to train the BiCNet network; the goal of each agent is to maximize the reward r i The expected sum of in, Represented by the state transfer function F state The action obtained is a θThe state distribution of the multi-agent is:

[0051] According to the deterministic policy gradient theorem, the policy gradient corresponding to the parameter θ is expressed as:

[0052] A deterministic non-strategy Actor-Critic algorithm is used for modeling; a Critic function is used to estimate the action value of non-strategy exploration The square sum loss function is used to train the Critic network, and the parameterized Critic gradient is Then the gradient corresponding to the parameter ξ is expressed as:

[0053]

[0054] Finally, according to the policy gradient corresponding to the parameter θ and the gradient corresponding to the parameter ξ, the stochastic gradient descent method is used to iteratively optimize the Actor-Critic network;

[0055] S32: Based on the above analysis, a cooperative scheduling bidirectionally coordinated network algorithm (CS-BiCNet) is constructed. The specific steps are as follows:

[0056] Initialize the model: First, randomly select values ​​for the Actor online network parameter θ and the Critic online network parameter ξ within the valid range, and determine the initial values ​​of the target network parameters based on θ and ξ, i.e. θ′←θ and ξ′←ξ; initialize the experience replay pool Initialize the random process ε to determine the action value;

[0057] Execution strategy: Record the remaining resource status and network request status of each ES at the beginning of training, and merge them into the initial state of the MEC network s t ; According to the initial state of the network s t and the random process ε generates a policy action a=a(s t )+ε, after executing all ES actions, update the state s t+1 =s(a t ), and calculate the reward value according to the QoS evaluation index described in step S14, and transfer the state Stored as experience data in the experience replay pool middle;

[0058] Computing Gradient Estimates: From the Experience Replay Pool Sample M data in small and medium batches and calculate the target value of the i-th ES in the m-th data It is expressed as:

[0059] At the same time, the policy gradient of the Actor network is calculated according to the policy gradient corresponding to the parameter θ, which is expressed as:

[0060] The value gradient of the Critic network is calculated according to the gradient corresponding to the parameter ξ, which is expressed as:

[0061] Update the target network: get the gradient and After that, the Adam optimizer is used to update the online network parameters θ and ξ according to the gradient descent method, and the target network parameters are updated according to the online network parameters, which is expressed as:

[0062] θ′=β·θ+(1-β)·θ′, ξ′=β·ξ+(1-β)·ξ′ where β∈(0,1) is the learning factor.

[0063] The beneficial effects of the present invention are as follows: the present invention focuses on QoS optimization, and performs dynamic task scheduling from the perspective of the global and long-term benefits of the network to achieve efficient resource coordination. Specifically, for the request tasks issued by users to different applications, the long-term average task response delay is taken as one of the goals, and ATRL and network throughput are combined to describe the global QoS of the network. The MDP model is used for analysis of the temporal continuity of task scheduling. In order to promote collaboration at the edge, a bidirectional coordination network is used as the communication layer, and the interaction between the edge and the environment is realized through the Actor-Critic network. By iteratively learning the dynamic task scheduling strategy, it is possible to ensure that task scheduling decisions that are conducive to global optimization are made.

[0064] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0066] Figure 1 This is a network system architecture diagram of the present invention;

[0067] Figure 2 This is a flow chart of the dynamic task scheduling of the present invention. DETAILED DESCRIPTION

[0068] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0069] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0070] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0071] See also Figure 1-2 The online resource scheduling method for ensuring service quality of the present invention specifically comprises the following steps:

[0072] Step 1: Global QoS evaluation: Set the task scheduling matrix based on the task request and microservice deployment, and conduct fine-grained analysis of the response delay based on it, and build a long-term average response delay model for different applications within the time range T. Combined with the real-time requirements of the task, obtain the network throughput, and establish a QoS evaluation model by combining the long-term average response delay and throughput;

[0073] Step 1.1: Assume that the request set generated by the user in time slot t is is the number of request types. Request k is represented by a four-tuple as in, Indicates the type of microservice required by the request, Indicates the requested task data size, k Indicates the latency tolerance threshold of the request. represents the size of the result data after processing. The number of requests k issued by users in ESi in time slot t is

[0074] Define the scheduling scheme for time slot t as e t =[e k,i,j ] K×V×V , where e t is a three-dimensional matrix of K×V×V, and the elements e in the matrix k,i,j represents the number of requests k scheduled from ESi to ESj, which is a natural number greater than or equal to zero and needs to satisfy the constraint

[0075] Step 1.2: According to the actual task scheduling process, the user first uploads the request to the nearest ES node. After each ES receives all the requests in the current time slot, it performs collaborative task scheduling between ES. After the scheduling is completed, each ES sends the corresponding task to the corresponding ES according to the scheduling result; for tasks that cannot be processed, they are sent to the remote cloud for processing by the remote cloud. Tasks are processed in parallel at the microservice. After the task processing is completed, the result data is transmitted back to the original ES node and sent back to the user. Based on the above analysis, the delay of each stage is defined as follows:

[0076] The task upload delay is described as the ratio of the uploaded task data size to the data transmission rate. The wireless transmission rate between the user and the ES is calculated by the Shannon formula. i represents the wireless channel bandwidth of ES i, SNR i represents the signal-to-noise ratio, then the task upload delay of request k in ESi is:

[0077]

[0078] After the task is uploaded to ES, ES transfers the task according to the scheduling result. Note that if j = 0, it means that the task is transferred to the remote cloud for processing. i,j represents the link transmission bandwidth between ESi and ESj, then the scheduling delay is:

[0079]

[0080] After the scheduling is completed, the tasks need to be processed. Define μ k is the task processing rate of the microservice that processes request k, h k,j and k,j are the remaining resources and resource usage of microservice k in the current time slot on ESj. Note that y k,j The constraints must be satisfied:

[0081] 0<y k,j ≤hk,j

[0082] If the required microservice is not available in the neighboring ES, the task needs to be scheduled to the remote cloud for processing, i.e. j = 0. Let V represent the number of ES in the network. In the parallel processing process, the task processing delay is expressed as:

[0083]

[0084] After the task is processed, the processing result of the task is transmitted back to the original ES node. The transmission delay is:

[0085]

[0086] Finally, the processing result of the user downloading the ES node generates a download delay expressed as:

[0087]

[0088] Then, in time slot t, the total delay of co-processing of request k scheduled from ESi to ESj is expressed as:

[0089]

[0090] If the required microservice is not available on ES, the task needs to be scheduled to the remote cloud for processing, that is, j = 0, and the above process remains unchanged. If all requests are considered from a long-term perspective, the long-term average task response latency (Average Tsk Response Latency, ATRL) of K types of requests within the time range T is expressed as:

[0091]

[0092] Step 1.3: In the process of network responding to user requests, ES expects to consume less resources to respond to more task requests while enhancing the user's QoS experience. The goal of collaborative task scheduling is to maximize the number of tasks that meet the delay tolerance threshold from a long-term perspective and further minimize the ATRL of the tasks.

[0093] For request k, if its ATRL exceeds the delay tolerance threshold ζ k , the task will be interrupted; otherwise, the task will be considered completed and the network throughput will increase. The network throughput in time slot t is expressed as:

[0094]

[0095]

[0096] Then in the entire time range T, the average throughput per time slot in the network is expressed as:

[0097]

[0098] Step 1.4: Due to the resource constraints of ES, in order to improve the QoS of users, it is necessary to maximize resource utilization and make the task as close as possible to ζ k is completed within the time range, so the main goal is to maximize the throughput in the entire time range, and minimize ATRL on this basis. The QoS evaluation index constructed in this way is expressed as:

[0099] θ=σ·P ave -(1-σ)·T ave

[0100] In the formula, σ is the weight factor. Let e=[e 1 ,e 2 ,...,e t ,...,e T ] is the scheduling decision on the entire time series, then the collaborative task scheduling problem (Collaborative Task Scheduling Problem, CTSP) is expressed as:

[0101]

[0102]

[0103]

[0104] In the formula, Represents the node set consisting of the remote cloud and all ES nodes. Represents the set of natural numbers.

[0105] Step 2: Long-term collaborative scheduling: A collaborative task scheduling model is established based on the task scheduling mode of the ES and the network QoS. Since the task scheduling process has time continuity, the MDP model is used to analyze the long-term continuous scheduling process. At the same time, each ES is regarded as an intelligent agent. In order to effectively achieve the global optimal, the intelligent agents use a cooperative game model to make collaborative decisions;

[0106] Step 2.1: Take a tuple Describe the game model of collaborative task scheduling. represents the state space of the current game, which is shared by all agents. The network state at time slot t is expressed as in, is the network request status at the beginning of time slot t, h t =[h 1,1 ,h 1,2 ,...,h k,i ,...,h K,V] is the remaining microservice resources of all ES at the beginning of time slot t. The action space of ES i decision is expressed as is the action of ESi in time slot t. Represents the state transition function. Denotes the reward function of ESi.

[0107] Step 2.2: For microservice k on ESj, define the resources that are being occupied by the dth group as a tuple express, Indicates the amount of resources occupied. represents the remaining occupied time. Let ψ ooc represents the set of occupied resources, then at the beginning of the current time slot, the resource usage y k,j Will be placed in the occupied resources collection Update the remaining time of all occupied resources at the same time:

[0108] Then the remaining resources of microservice k on ESj at the beginning of the next time slot are expressed as:

[0109] Step 2.3: Each ES makes its own scheduling decision by observing the current state of the environment and obtains a reward value. The global reward of the network is defined by the average of the reward values ​​of all ESs, expressed as:

[0110] In the formula, r t Indicates that in state s t In this case, all agents take action a t The reward value obtained by the global network is the reward r obtained by each agent i i t (s t ,a t ) is described by the average value; V represents the number of agents, which is equal to the number of ES. The purpose of ES global optimization is to maximize the discounted reward expectation over the entire time range through learning Among them, 0≤δ<1 is the discount factor. Therefore, the goal of the network at each moment is to maximize the state s t Next, take action a t The action value function Q * , expressed as:

[0111] The global reward function can reflect the overall result of the cooperative game of all ESs, which needs to be jointly determined by the reward value of each ES. At the same time, each ES decides how to coordinate scheduling based on its own reward value.

[0112] Step 2.4: According to step S11, the reward value of ESi can be calculated as:

[0113] In the formula, r i t (s t ,a t ) means in state s t Next, the global network adopts action a t The reward value obtained by ESi is determined by the average throughput and ATRL. Then the action value function corresponding to V ES is expressed as:

[0114] The reward value of ES can be used as the action feedback of ES for training. After a certain amount of learning and training, each ES forms its own decision-making model. Therefore, at each moment, ES can find the most favorable scheduling decision in the decision-making model. In order to simplify the expression, the header symbol of the time slot t in the formula is omitted from the following.

[0115] Step 3: Multi-agent collaborative learning: In order to promote cooperation between ESs, a two-way coordination network is used to build a cooperative communication layer, and the Actor-Critic network is used for interaction between agents and the environment. The global QoS is used as the reward for each agent, and the global optimal solution is obtained through iterative learning of strategies and values. Finally, a collaborative scheduling two-way coordination network algorithm is constructed to realize the overall task scheduling process.

[0116] Step 3.1: To learn the parameters of the BiCNet, the original network is first expanded into a network consisting of V child nodes, and the back-propagation through time is used to calculate the reverse gradient. To train the policy network, the reward gradients of all agents are passed to the policy network to guide the agent's strategy, and the policy network parameters θ are further updated through the back-propagation of the communication layer. Then the goal of each agent is to maximize the reward r i The expected sum of in, Represented by the state transfer function F state The action obtained is a θ The state distribution of . Then the goal of the multi-agent is:

[0117] According to the deterministic policy gradient theorem, the policy gradient corresponding to the parameter θ is expressed as:

[0118] In order to reduce variance in the calculation process, a deterministic non-strategy Actor-Critic algorithm is used for modeling. A Critic function is used to estimate the action value of non-strategy exploration. The square sum loss function is used to train the Critic network, and the parameterized Critic gradient is Then the gradient corresponding to the parameter ξ is expressed as:

[0119]

[0120] Finally, according to the policy gradient corresponding to the parameter θ and the gradient corresponding to the parameter ξ, the stochastic gradient descent method is used to iteratively optimize the Actor-Critic network.

[0121] Step 3.2: Based on the above analysis, the specific steps of the Cooperative Scheduling Bidirectionally-Coordinated Nets (CS-BiCNet) algorithm are as follows:

[0122] Initialize the model: First, randomly select values ​​for the Actor online network parameter θ and the Critic online network parameter ξ within the valid range, and determine the initial values ​​of the target network parameters based on θ and ξ, i.e. θ′←θ and ξ′←ξ; initialize the experience replay pool Initialize the random process ε to determine the action value.

[0123] Execution strategy: Record the remaining resource status and network request status of each ES at the beginning of training, and merge them into the initial state of the MEC network s t According to the initial state of the network s t and the random process ε generates a policy action a=a(s t )+ε, after executing all ES actions, update the state s t+1 =s(a t ), and calculate the reward value according to the QoS evaluation index described in step 1.4, and transfer the state Stored as experience data in the experience replay pool middle.

[0124] Computing Gradient Estimates: From the Experience Replay Pool Sample M data in small and medium batches and calculate the target value of the i-th ES in the m-th data It is expressed as:

[0125] At the same time, the policy gradient of the Actor network is calculated according to the policy gradient corresponding to the parameter θ, which is expressed as:

[0126] The value gradient of the Critic network is calculated according to the gradient corresponding to the parameter ξ, which is expressed as:

[0127] Update the target network: get the gradient and After that, the Adam optimizer is used to update the online network parameters θ and ξ according to the gradient descent method, and the target network parameters are updated according to the online network parameters, which is expressed as:

[0128] θ′=β·θ+(1-β)·θ′, ξ′=β·ξ+(1-β)·ξ′ where β∈(0,1) is the learning factor.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An online resource scheduling method with service quality assurance, characterized in that: The specific steps include: S1: Global QoS evaluation: The task scheduling matrix is ​​set based on the task request and microservice deployment, and the response delay is analyzed in a fine-grained manner to build a long-term average response delay model for different applications within the time range T. The network throughput is obtained based on the real-time requirements of the task, and a QoS evaluation model is established by combining the long-term average response delay and throughput. S2: Long-term collaborative scheduling: A collaborative task scheduling model is established based on the task scheduling mode of ES and network QoS. The MDP model is used to analyze the long-term continuous scheduling process. At the same time, each ES is regarded as an intelligent agent, and the intelligent agents use a cooperative game model to make collaborative decisions. S3: Multi-agent collaborative learning: A bidirectional coordination network BiCNet is used to build a cooperative communication layer, and an Actor-Critic network is used to interact between agents and the environment. The global QoS is used as a reward for each agent, and the global optimal solution is obtained through iterative learning of strategies and values. Finally, a collaborative scheduling bidirectional coordination network algorithm is constructed to realize the overall task scheduling process.

2. The online resource scheduling method with service quality assurance according to claim 1, characterized in that: The step S1 specifically includes the following steps: S11: Assume that the request set generated by the user in time slot t is is the number of request types; request k is represented by a four-tuple in, Indicates the type of microservice required by the request, Indicates the requested task data size, k Indicates the latency tolerance threshold of the request. represents the size of the result data after processing; the number of requests k issued by users in ESi in time slot t is Define the scheduling scheme for time slot t as e t =[e k,i,j ] K×V×V , where e t is a three-dimensional matrix of K×V×V, and the elements e in the matrix k,i,j represents the number of requests k scheduled from ESi to ESj, which is a natural number greater than or equal to zero, satisfying the constraint S12: According to the actual task scheduling process, the user first uploads the request to the nearest ES node. After each ES receives all the requests in the current time slot, it performs collaborative task scheduling between ESs. After the scheduling is completed, each ES sends the corresponding task to the corresponding ES according to the scheduling result; for tasks that cannot be processed, they are sent to the remote cloud for processing by the remote cloud; tasks are processed in parallel at the microservice; after the task processing is completed, the result data is transmitted back to the original ES node and sent back to the user; S13: The goal of collaborative task scheduling is to maximize the number of tasks that meet the delay tolerance threshold from a long-term perspective, and further minimize the ATRL of the tasks; For request k, if its ATRL exceeds the delay tolerance threshold ζ k , the task will be interrupted; otherwise, the task will be considered completed and the network throughput will increase; then the network throughput in time slot t is expressed as: In the entire time range T, the average throughput per time slot in the network is expressed as: S14: The main goal is to maximize the throughput within the entire time range, and minimize ATRL on this basis. The QoS evaluation index constructed in this way is expressed as: Where σ is the weight factor; let e=[e1,e2,...,e t ,...,e T ] is the scheduling decision on the entire time series, then the collaborative task scheduling problem CTSP is expressed as: s.t.C1: C2: In the formula, Represents the node set consisting of the remote cloud and all ES nodes. Represents the set of natural numbers.

3. The online resource scheduling method with service quality assurance according to claim 2, characterized in that: In step S12, the delay of each stage is defined as follows: The task upload delay is described as the ratio of the upload task data size to the data transmission rate. The wireless transmission rate between the user and the ES is calculated by the Shannon formula; w i represents the wireless channel bandwidth of ESi, SNR i represents the signal-to-noise ratio, then the task upload delay of request k in ESi is: After the task is uploaded to ES, ES transfers the task according to the scheduling result. If j = 0, it means that the task is transferred to the remote cloud for processing. i,j represents the link transmission bandwidth between ESi and ESj, then the scheduling delay is: After the scheduling is completed, the task needs to be processed; define μ k is the task processing rate of the microservice that processes request k, h k,j and k,j are the remaining resources and resource usage of microservice k in the current time slot on ESj; y k,j Satisfy the constraints: 0<y k,j ≤h k,j If the required microservice is not available in the neighboring ES, the task will be scheduled to the remote cloud for processing, i.e., j = 0; V represents the number of ES in the network, and in the parallel processing process, the task processing delay is expressed as: After the task is processed, the processing result of the task is transmitted back to the original ES node. The transmission delay is: Finally, the processing result of the user downloading the ES node generates a download delay expressed as: Then, in time slot t, the total delay of co-processing of request k scheduled from ESi to ESj is expressed as: If the required microservice is not available on ES, the task needs to be scheduled to the remote cloud for processing, that is, j = 0, and the above process remains unchanged; if all requests are considered from a long-term perspective, the long-term average task response delay ATRL of K types of requests within the time range T is expressed as:

4. The online resource scheduling method with service quality assurance according to claim 1, characterized in that: The step S2 specifically includes the following steps: S21: Take a tuple Describe the game model of collaborative task scheduling; represents the state space of the current game, which is shared by all agents; the network state at time slot t is expressed as in, is the network request status at the beginning of time slot t, h t =[h 1,1 ,h 1,2 ,...,h k,i ,...,h K,V ] is the remaining microservice resources of all ES at the beginning of time slot t; the action space of ESi decision is expressed as is the action of ESi in time slot t; represents the state transition function; represents the reward function of ESi; S22: For microservice k on ESj, define the resources that are being occupied by the dth group as a binary express, Indicates the amount of resources occupied. represents the remaining occupied time; let ψ ooc represents the set of occupied resources, then at the beginning of the current time slot, the resource usage y k,j Will be placed in the occupied resources collection Update the remaining time of all occupied resources at the same time: Then the remaining resources of microservice k on ESj at the beginning of the next time slot are expressed as: S23: Each ES makes its own scheduling decision by observing the current state of the environment and obtains a reward value. The global reward of the network is defined by the average of the reward values ​​of all ESs, expressed as: In the formula, r t Indicates that in state s t In this case, all agents take action a t The reward value obtained by the global network is the reward r obtained by each agent i i t (s t ,a t ) is described by the average value; V represents the number of agents, which is equal to the number of ES; the purpose of ES global optimization is to maximize the discounted reward expectation over the entire time range through learning Among them, 0≤δ<1 is the discount factor; therefore, the goal of the network at each moment is to maximize the state s t Next, take action a t The action value function Q * , expressed as: The global reward function reflects the overall result of the cooperative game of all ESs, which needs to be jointly determined by the reward value of each ES. At the same time, each ES determines how to coordinate scheduling based on its own reward value. S24: According to step S11, the reward value of ESi is calculated as: In the formula, r i t (s t ,a t ) means in state s t Next, the global network adopts action a t The reward value obtained by ESi is determined by the average throughput and ATRL; then the action value function corresponding to V ES is expressed as: The ES reward value is used as the action feedback of the ES for training. After a certain period of learning and training, each ES forms its own decision-making model. At each moment, the ES finds the scheduling decision that is most conducive to the global optimality in the decision-making model.

5. The online resource scheduling method with service quality assurance according to claim 1, characterized in that: The step S3 specifically comprises the following steps: S31: First, the original network is expanded into a network consisting of V child nodes, and time back propagation is used to calculate the reverse gradient to learn the parameters of the bidirectional coordination network BiCNet; the reward gradients of all agents are passed to the BiCNet network to guide the agent's strategy, and the BiCNet network parameters θ are further updated through the back propagation of the communication layer to train the BiCNet network; the goal of each agent is to maximize the reward r i The expected sum of in, Represented by the state transfer function F state The action obtained is a θ The state distribution of the multi-agent is: According to the deterministic policy gradient theorem, the policy gradient corresponding to the parameter θ is expressed as: A deterministic non-strategy Actor-Critic algorithm is used for modeling; a Critic function is used to estimate the action value of non-strategy exploration The square sum loss function is used to train the Critic network, and the parameterized Critic gradient is Then the gradient corresponding to the parameter ξ is expressed as: Finally, according to the policy gradient corresponding to the parameter θ and the gradient corresponding to the parameter ξ, the stochastic gradient descent method is used to iteratively optimize the Actor-Critic network; S32: Based on the above analysis, a cooperative scheduling bidirectionally coordinated network algorithm (CS-BiCNet) is constructed. The specific steps are as follows: Initialize the model: First, randomly select values ​​for the Actor online network parameter θ and the Critic online network parameter ξ within the valid range, and determine the initial values ​​of the target network parameters based on θ and ξ, i.e. θ′←θ and ξ′←ξ; initialize the experience replay pool Initialize the random process ε to determine the action value; Execution strategy: Record the remaining resource status and network request status of each ES at the beginning of training, and merge them into the initial state of the MEC network s t ; According to the initial state of the network s t and the random process ε generates a policy action a=a(s t )+ε, after executing all ES actions, update the state s t+1 =s(a t ), and calculate the reward value according to the QoS evaluation index described in step S14, and transfer the state Stored as experience data in the experience replay pool middle; Computing Gradient Estimates: From the Experience Replay Pool Sample M data in small and medium batches and calculate the target value of the i-th ES in the m-th data It is expressed as: At the same time, the policy gradient of the Actor network is calculated according to the policy gradient corresponding to the parameter θ, which is expressed as: The value gradient of the Critic network is calculated according to the gradient corresponding to the parameter ξ, which is expressed as: Update the target network: After obtaining the gradients ▽θ and ▽ξ, the Adam optimizer is used to update the online network parameters θ and ξ according to the gradient descent method, and the target network parameters are updated according to the online network parameters, which is expressed as: θ′=β·θ+(1-β)·θ′, ξ′=β·ξ+(1-β)·ξ′ where β∈(0,1) is the learning factor.

Images