Service network resource scheduling method based on soft reinforcement learning

Abstract

The embodiment of the invention provides a service network resource scheduling method based on soft reinforcement learning, and the method comprises the steps: obtaining a plurality of resource packets operating in a network topology structure, importing the plurality of resource packets into a trained target recognition network, generating an action decision of each resource packet at a target moment, and network resource scheduling is carried out according to the action decision at the target moment, so that reinforcement learning, an entropy regularization item and a deep neural network are combined, the expansibility of an algorithm is improved, and the robustness of a large-scale scheduling strategy is improved on the premise of ensuring the precision. Besides, the method can be widely applied to scenes such as cloud computing, edge computing and distributed systems, the resource utilization rate is remarkably increased, energy consumption is reduced, service quality is improved, and the method has important economic and social values.

Description

Technical Field

[0001] The embodiments of this specification relate to the fields of computer technology and artificial intelligence technology, and in particular to a service network resource scheduling method based on soft reinforcement learning. Background Art

[0002] With the rapid development of information technology, cloud computing, edge computing, and distributed systems have become the core infrastructure supporting modern computing services. In these systems, the scheduling of service resources directly impacts system performance, efficiency, and user experience. Traditional resource scheduling methods are primarily based on static rules, heuristic algorithms, or simple optimization models. These methods often exhibit the following limitations when faced with dynamic, complex, and changing workloads: First, traditional methods struggle to adapt to dynamically changing workloads and resource states, resulting in low resource utilization or degraded service quality. Furthermore, the algorithms' global optimization capabilities are limited, making it difficult for scheduling strategies based on local rules to achieve optimal global resource allocation. More critically, traditional resource scheduling methods lack learning capabilities: they cannot learn from historical data and improve scheduling strategies, making them difficult to handle in complex scenarios.

[0003] In recent years, reinforcement learning (RL), an artificial intelligence technology that can learn optimal strategies through interaction with the environment, has demonstrated great potential in the field of resource scheduling. However, traditional RL methods still face challenges in practical applications. Classic RL algorithms, such as Q-learners or actor-critic algorithms, often struggle to balance exploring new strategies with leveraging known ones. This can lead to slow convergence or trapping in local optima, posing a challenge for large-scale resource scheduling. Furthermore, traditional deep learning methods often have high dimensionality in their state and action spaces when processing high-dimensional processing spaces, making it difficult and inefficient to effectively search for optimal actions. They are also sensitive to noise and uncertainty, which can lead to scheduling instability. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a service network resource scheduling method.

[0005] According to an embodiment of this specification, a service network resource scheduling method is provided, including: Acquire multiple resource packages operating in a network topology, wherein the network topology is obtained by complex service network modeling, the network topology has multiple service sites, each resource package is set at one of the service sites, each service site has a site information vector, the site information vector includes state information, site constraint information, and neighboring site information, and each resource package has a resource state vector, the resource state vector includes attribute information, transportation information, site location information, and resource constraint information; Import multiple resource packages into a trained target recognition network to generate action decisions for each resource package at the target time. The recognition network is a recognition model based on reinforcement learning, combined with entropy regularization and neural networks. Network resources are scheduled based on the action decision at the target time.

[0006] In some embodiments, the object recognition network training step includes: Constructing an original recognition network, wherein the original recognition network has a policy network, a current state value network, a target state value network, a current action value network, and a target action value network; Build experience pools based on multiple resource packs; Perform multiple rounds of iterative training based on the experience pool until the preset requirements are met to obtain the target recognition network. In each round of training, the target state vector of the target state value network, the target action vector of the target action value network, and the policy parameters of the policy network are updated through the back propagation neural network. The constraint function of the policy network includes:

[0007]

[0008] in, Indicates policy parameters In state The entropy under L represents the sum entropy, represents the action-state reward function, Represents the temperature coefficient, which is used to balance the proportion of entropy and reward in the loss function value. The bigger it is, the greater the proportion of exploration is. represents the trajectory generated by the policy parameters π and the environment dynamics Expectations on t Represents the state information at time t, a t represents the action information at time t, Indicates policy parameters The state-action or state access distribution under represents the discount factor, .

[0009] In some embodiments, an experience pool is constructed based on multiple resource packages, including: Obtain multiple resource packages at a first moment, and a target scheduling task for each resource package at a second moment, wherein the second moment is a moment after the first moment; Import multiple resource packages at the first moment into the policy network to generate multiple experience packages and obtain an experience pool. Each experience package includes the state information, action information, and completion information at the first moment, as well as the state information and score reward at the second moment. The score reward is the feedback score of any resource package for other resource packages based on the state reward function.

[0010] In some embodiments, the step of generating an experience package based on any resource package at the first moment includes: Import any resource package at the first moment into the policy network to generate multiple action probabilities for all actions; Sampling is performed based on the probability of each action to generate the decision action at the first moment; After executing the environment interaction according to the decision action, the state information of any resource package at the second moment and the score reward at the second moment are obtained; According to whether any resource package completes the target scheduling task at the second moment, the completion information at the first moment is determined, and the resource package corresponding to any resource package at the first moment is obtained.

[0011] In some embodiments, the step of updating the target motion vector includes: Get experience packs from any resource pack from the experience pool; Update the target action vector of the target action value network based on the preset first calculation formula, the experience package, the state information, action information, and completion information at the first moment in the experience package, and the state information and score reward at the second moment. The first calculation formula includes:

[0012]

[0013] Among them, L Q represents the expected target value of the target action vector, θ represents the shared parameter of the target action vector and the target state vector, IE represents the expectation, represents the target action vector, s t Represents the state information at time t, a t represents the action information at time t, r t+1 represents the score reward at the next moment after time t, d t Indicates the completion information at time t, represents the target state vector, Indicates the preset discount factor; represents the expected target value of the shared parameter, Indicates the preset weight coefficient, Represents policy parameters.

[0014] In some embodiments, the step of updating the policy parameters includes: Update the policy parameters based on the preset loss function, which includes:

[0015] in, represents the expected loss. Represents policy parameters.

[0016] At least one embodiment of the embodiments of this specification obtains multiple resource packages operating in a network topology structure, imports the multiple resource packages into a trained target recognition network, generates action decisions for each resource package at a target time, and schedules network resources based on the action decisions at the target time, thereby combining reinforcement learning, entropy regularization terms and deep neural networks to improve the scalability of the algorithm and enhance the robustness of large-scale scheduling strategies while ensuring accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flowchart of some embodiments of a service network resource scheduling method provided by some embodiments of this specification. DETAILED DESCRIPTION

[0018] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0019] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The modifications of "one" and "a plurality" mentioned in this disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0020] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0021] Reinforcement learning (RL), an artificial intelligence technology that can learn optimal strategies through interaction with the environment, has shown great potential in the field of resource scheduling. However, traditional RL methods still face challenges in practical applications. Classic RL algorithms, such as Q-learners or actor-critic algorithms, typically struggle to balance exploring new strategies with leveraging known ones. This can lead to slow convergence or trapping in local optima, posing a challenge for large-scale resource scheduling. Furthermore, traditional deep learning methods often have high dimensionality in their state and action spaces when processing high-dimensional processing spaces, making it difficult and inefficient to effectively search for optimal actions. They are also sensitive to noise and uncertainty, which can lead to instability in scheduling.

[0022] To address these issues, this paper proposes a service resource scheduling method based on soft reinforcement learning. This method combines the adaptive capabilities of reinforcement learning with a soft decision-making mechanism to achieve efficient and stable resource scheduling in dynamic and complex environments.

[0023] See also Figure 1 , Figure 1 A flowchart of a service network resource scheduling method provided according to some embodiments of this specification is shown, which specifically includes the following steps.

[0024] Step 101: Acquire multiple resource packages running in a network topology structure.

[0025] In some embodiments, the execution subject of the service network resource scheduling method (such as a preset computing device) can connect to the target device via a wired connection or a wireless connection, and then obtain multiple resource packages operating in a network topology structure, wherein the network topology structure is obtained by complex service network modeling and processing, and the network topology structure has multiple service sites, each resource package is set at one of the service sites, each service site has a site information vector, the site information vector includes status information, site constraint information and neighboring site information, and each resource package has a resource status vector, the resource status vector includes attribute information, transportation information, site location information and resource constraint information. The above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G / 6G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultrawideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0026] When dealing with complex resource networks with complex topologies, they can be viewed as a graph structure. Each resource package in resource scheduling can be considered an intelligent agent with both perception and decision-making capabilities. Furthermore, this complex network exhibits a typical complex graph network structure. Therefore, from the perspective of serving network resources, the resource package scheduling problem on this complex resource network transforms into a complex interaction problem involving multiple agents in a complex graph topology. The movement of resource packages on the network is considered an intelligent agent decision. Each intelligent agent's movement and decision is considered a time step.

[0027] A network topology structure can refer to a mathematical model that describes the connectivity between service nodes and is an abstract representation of a complex service network. A resource package can refer to a data unit transmitted within a network, containing complete descriptive information such as attribute status and scheduling constraints. A complex service network can refer to a distributed system composed of heterogeneous service nodes. A service site can refer to a functional node in a complex service network, or an abstract representation of a functional node in a network topology, with independent information processing and resource carrying capabilities. A resource state vector can refer to a standardized data structure that represents the comprehensive characteristics of a resource package. State information can refer to a set of dynamic parameters that reflect the real-time operating status of a service site. Site constraint information can refer to boundary condition parameters that limit the resource processing capacity of a service site. Neighboring site information can refer to topological relationship data that records the directly connected nodes of a target site. A site information vector can refer to a standardized data structure that represents the comprehensive characteristics of a service site. Attribute information can refer to a set of metadata that defines the inherent characteristics of a resource package. Transportation information can refer to navigation data that records the transmission path and timeliness requirements of a resource package. Site information can refer to topological coordinate parameters that identify the current location of a resource package. Resource constraint information can refer to boundary conditions that limit the feasible domain for resource package scheduling.

[0028] Step 102: Import multiple resource packages into the trained target recognition network to generate action decisions for each resource package at the target moment, wherein the recognition network is a recognition model based on reinforcement learning combined with entropy regularization term and neural network.

[0029] Step 103: Perform network resource scheduling according to the action decision at the target time.

[0030] The target recognition network refers to a trained recognition model based on deep reinforcement learning combined with entropy regularization and a neural network. By integrating entropy regularization with a neural network architecture, dynamic scheduling strategies for multiple resource packages can be generated. The action decision refers to the scheduling instructions assigned to a resource package at a specific moment, calculated by the target recognition network based on the environment state and constraints.

[0031] In some optional implementations, the training steps of the target recognition network include: constructing an original recognition network, wherein the original recognition network includes a policy network, a current state value network, a target state value network, a current action value network, and a target action value network; constructing an experience pool based on multiple resource packages; performing multiple rounds of iterative training based on the experience pool until the target recognition network meets preset requirements, thereby obtaining the target recognition network, wherein in each round of training, a target state vector of the target state value network, a target action vector of the target action value network, and policy parameters of the policy network are updated through a backpropagation neural network, wherein the constraint function of the policy network includes:

[0032]

[0033] in, Indicates policy parameters In state The entropy under L represents the sum entropy, represents the action-state reward function, Represents the temperature coefficient, which is used to balance the proportion of entropy and reward in the loss function value. The bigger it is, the greater the proportion of exploration is. represents the trajectory generated by the policy parameters π and the environment dynamics The expectation on time is, st represents the state information at time t, and at represents the action information at time t. Indicates policy parameters The state-action or state access distribution under represents the discount factor, .

[0034] The original recognition network refers to an initially untrained recognition model based on deep reinforcement learning combined with an entropy regularizer and a neural network. The policy network refers to the computational network responsible for generating action probability distributions, mapping environment states to decision-making policies for executable actions. The current state value network refers to a predictive model that assesses the potential benefits of environment states in real time, guiding the short-term optimization direction of the policy network. The target state value network refers to a state value assessment model used for stabilization training, providing a reliable value benchmark through delayed updates. The current action value network refers to an estimator that calculates the immediate value of a specific action, supporting fine-grained action selection in the policy network. The target state vector refers to the stabilized state feature representation output by the target state value network, providing a delayed update benchmark for policy evaluation. The target action vector refers to the normalized action value distribution generated by the target action value network, used to balance immediate decision-making with long-term benefits. The policy parameters refer to the set of trainable parameters in the policy network that determine the mapping from state to action and are continuously optimized via gradient descent.

[0035] In some optional implementations, an experience pool is constructed based on multiple resource packages, including: obtaining multiple resource packages at a first moment, and scheduling a target task for each resource package at a second moment, wherein the second moment is the next moment after the first moment; importing multiple resource packages at the first moment into a strategy network, generating multiple experience packages, and obtaining an experience pool, wherein each experience package includes state information, action information, and completion information at the first moment, as well as state information and score rewards at the second moment, and the score rewards are the feedback scores of any resource package for other resource packages based on the state reward function. The first moment can refer to a pre-set time point, and the second moment can refer to a time point after the first moment. The time step between each moment can be a smaller preset value.

[0036] In some optional implementations, the step of generating an experience package based on any resource package at the first moment includes: importing any resource package at the first moment into the strategy network to generate multiple action probabilities for all actions; sampling according to each action probability to generate a decision action at the first moment; after interacting with the execution environment according to the decision action, obtaining the status information of any resource package at the second moment and the score reward at the second moment; determining the completion information at the first moment based on whether any resource package completes the target scheduling task at the second moment, and obtaining the resource package corresponding to any resource package at the first moment.

[0037] In some optional implementations, the step of updating the target action vector includes: obtaining an experience package of any resource package from the experience pool; and updating the target action vector of the target action value network according to a preset first calculation formula, the experience package, and the state information, action information, and completion information at the first moment in the experience package, as well as the state information and score reward at the second moment, wherein the first calculation formula includes:

[0038]

[0039] Among them, L Q represents the expected target value of the target action vector, θ represents the shared parameter of the target action vector and the target state vector, IE represents the expectation, represents the target action vector, s t Represents the state information at time t, a t represents the action information at time t, r t+1 represents the score reward at the next moment after time t, d t Indicates the completion information at time t, represents the target state vector, Indicates the preset discount factor; represents the expected target value of the shared parameter, Indicates the preset weight coefficient, Represents policy parameters.

[0040] In some optional implementations, the step of updating the policy parameters includes updating the policy parameters based on a preset loss function, where the loss function includes:

[0041] in, represents the expected loss. Represents policy parameters.

[0042] The beneficial effects of one of the embodiments of this specification include at least: obtaining multiple resource packages operating in a network topology, importing the multiple resource packages into a trained target recognition network, generating action decisions for each resource package at a target time, and scheduling network resources based on the action decisions at the target time, thereby combining reinforcement learning, entropy regularization terms, and deep neural networks to improve the scalability of the algorithm and enhance the robustness of large-scale scheduling strategies while ensuring accuracy. In addition, the present invention can be widely used in scenarios such as cloud computing, edge computing, and distributed systems, significantly improving resource utilization, reducing energy consumption, and improving service quality, and has important economic and social value.

Claims

1. A service network resource scheduling method, characterized in that: include: Acquire multiple resource packages operating in a network topology, wherein the network topology is obtained by complex service network modeling, the network topology has multiple service sites, each resource package is set at one of the service sites, each service site has a site information vector, the site information vector includes state information, site constraint information, and neighboring site information, and each resource package has a resource state vector, the resource state vector includes attribute information, transportation information, site location information, and resource constraint information; Importing the multiple resource packages into a trained target recognition network to generate an action decision for each resource package at a target time, wherein the recognition network is a recognition model based on reinforcement learning combined with an entropy regularization term and a neural network; Network resources are scheduled according to the action decision at the target time.

2. The method according to claim 1, characterized in that The training steps of the target recognition network include: Constructing an original recognition network, wherein the original recognition network is provided with a policy network, a current state value network, a target state value network, a current action value network, and a target action value network; Building an experience pool based on the multiple resource packs; Perform multiple rounds of iterative training based on the experience pool until the preset requirements are met to obtain the target recognition network, wherein the target state vector of the target state value network, the target action vector of the target action value network, and the policy parameters of the policy network are updated through the back propagation neural network in each round of training, wherein the constraint function of the policy network includes: in, Indicates policy parameters In state The entropy under L represents the sum entropy, represents the action-state reward function, Represents the temperature coefficient, which is used to balance the proportion of entropy and reward in the loss function value. The bigger it is, the greater the proportion of exploration is. represents the trajectory generated by the policy parameters π and the environment dynamics Expectations on t Represents the state information at time t, a t represents the action information at time t, Indicates policy parameters The state-action or state access distribution under represents the discount factor, .

3. The method according to claim 2, characterized in that Building an experience pool based on the multiple resource packs, including: Acquire multiple resource packages at a first moment, and a target scheduling task for each resource package at a second moment, wherein the second moment is a moment next to the first moment; Import multiple resource packages at a first moment into the policy network to generate multiple experience packages to obtain the experience pool, wherein each experience package includes state information, action information, and completion information at the first moment, as well as state information and score rewards at the second moment, and the score rewards are the feedback scores of any resource package for other resource packages based on the state reward function.

4. The method according to claim 3, characterized in that The step of generating an experience pack according to any resource pack at the first moment includes: Importing any resource package at the first moment into the policy network to generate multiple action probabilities for all actions; Sampling is performed based on the probability of each action to generate the decision action at the first moment; After executing the environment interaction according to the decision action, obtaining the state information of the second moment of any resource package and the score reward at the second moment; According to whether any resource package completes the target scheduling task at the second moment, the completion information at the first moment is determined, and a resource package corresponding to any resource package at the first moment is obtained.

5. The method according to claim 2, characterized in that The step of updating the target motion vector includes: Obtain an experience pack of any resource pack from the experience pool; The target action vector of the target action value network is updated according to a preset first calculation formula, the experience package, the state information, action information, and completion information at the first moment in the experience package, and the state information and score reward at the second moment, wherein the first calculation formula includes: Among them, L Q represents the expected target value of the target action vector, θ represents the shared parameter of the target action vector and the target state vector, IE represents the expectation, represents the target action vector, s t Represents the state information at time t, a t represents the action information at time t, r t+1 represents the score reward at the next moment after time t, d t Indicates the completion information at time t, represents the target state vector, Indicates the preset discount factor; represents the expected target value of the shared parameter, Indicates the preset weight coefficient, Represents the policy parameters.

6. The method according to claim 2, characterized in that The steps of updating the policy parameters include: The strategy parameters are updated based on a preset loss function, wherein the loss function includes: in, represents the expected loss. Represents policy parameters.

Images