Offline-to-online deep reinforcement learning method in resource scheduling scene

Abstract

The invention discloses an offline-to-online deep reinforcement learning method in a resource scheduling scene, and the method comprises the following steps: S1, obtaining resource scheduling data, and carrying out the data preprocessing; s2, performing offline training of the strategy model in combination with expert imitation learning and a multi-head attention mechanism; s3, using an SAC reinforcement learning algorithm to iteratively optimize the strategy model, refining the strategy model through dynamic interaction with a real-time environment, learning response resource changes, and carrying out online training of the strategy model; s4, a dynamic strategy updating module is used, optimistic and pessimistic strategy updating is combined, strategy model parameters are adjusted according to online feedback, and the trained strategy model network is returned; according to the method, the SAC reinforcement learning algorithm of online learning and the expert imitation learning technology of offline learning are fused, the priority experience playback and multi-head attention mechanism are introduced, and the efficiency and accuracy of resource scheduling can be effectively improved by dynamically updating the proxy strategy model.

Description

Technical Field

[0001] The present invention belongs to the technical field of resource scheduling, and in particular relates to an offline-to-online deep reinforcement learning method in a resource scheduling scenario. Background Art

[0002] In the field of resource scheduling, the uncertainty of demand and the dynamic changes of resources pose challenges to traditional scheduling methods. Deep reinforcement learning (DRL) technology has gradually been applied to solve complex resource scheduling problems due to its autonomous learning ability and adaptability to dynamic environments. Deep reinforcement learning learns the optimal policy model through interaction with the environment, which is independent of the traditional physical model and enables it to adapt to the dynamic characteristics of available resources. Existing studies have shown the successful application of deep reinforcement learning in resource scheduling, proving its potential in solving the complexity and dynamics of resource scheduling. However, despite its broad application prospects in practical applications, deep reinforcement learning still faces two major challenges: first, a large number of interaction samples are required in the online learning stage, resulting in high learning cost and time consumption; second, the random exploration in the initial training process becomes unstable due to noisy data or environmental changes, and even leads to unsafe operations. Especially in practical systems, this uncertainty may cause equipment damage or waste of resources. Summary of the invention

[0003] To solve the above problems, the present invention proposes an offline-to-online deep reinforcement learning method in a resource scheduling scenario. The method integrates the SAC reinforcement learning algorithm of online learning and the expert imitation learning technology of offline learning, and introduces the priority experience replay and multi-head attention mechanism. By dynamically updating the agent strategy model, the efficiency and accuracy of resource scheduling can be effectively improved.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] An offline-to-online deep reinforcement learning method in a resource scheduling scenario for optimizing resource scheduling includes the following steps:

[0006] S1. Obtain resource scheduling data and perform data preprocessing;

[0007] S2. Combine expert imitation learning and multi-head attention mechanism to perform offline training of the strategy model;

[0008] The specific process of step S2 is:

[0009] S21. Use expert imitation learning to initialize the strategy model. Expert imitation learning trains the strategy model by imitating the behavior of experts. The calculation formula is:

[0010]

[0011] Among them, L GAIL is the loss function of the GAIL algorithm; is the expected operation; θ (a|s) is the policy model; a is the action or decision performed by the agent in state s; s is the current state of the environment;

[0012] S22. Add a multi-head attention mechanism to the policy model network, and use the multi-head attention mechanism to process high-dimensional state space to focus on the most relevant environmental features at the moment;

[0013] S3, use the SAC reinforcement learning algorithm to iteratively optimize the policy model, refine the policy model through dynamic interaction with the real-time environment, learn to respond to resource changes, and perform online training of the policy model;

[0014] The specific process of step S3 is:

[0015] S31. Use the SAC reinforcement learning algorithm to optimize the strategy model by maximizing the return of entropy enhancement. The calculation formula is:

[0016]

[0017] Among them, L SAC is the strategy loss function of the SAC algorithm; is the expected operation; α is the entropy weight; π θ (a|s) is the strategy model; π θ A strategy for selecting actions for the agent; is the value function of the state-action pair; is the advantage function; a is the action or decision performed by the agent in state s; s is the current state of the environment;

[0018] S32. Update the value network, and the calculation formula is:

[0019]

[0020] Among them, L V is the loss function of the value network; For the expected operation; is the output of the value network; r is the immediate reward; γ is the discount factor; Predict the value of the next state s';

[0021] S4. Use the dynamic policy model update module, combine optimistic and pessimistic policy model updates, adjust the policy model parameters according to online feedback, and return the trained policy model network.

[0022] Preferably, in step S1, the specific process of data preprocessing is:

[0023] S11. Use priority experience playback to select the most valuable samples for learning from historical data, and weight the samples according to the time difference error of experience. The calculation formula of time difference error is:

[0024] δ t =R t+1 +γV(S t+1 )-V(S t ),

[0025] Among them, δ t is the time difference error; R t+1 is the reward obtained at time step t+1; γ is the discount factor; V(S t+1 ) is state S t+1 The estimated value of S t+1 is the next state; V(S t ) is state S t The estimated value of S t is the current state;

[0026] S12. Set the priority to the absolute value of the time difference error. The specific calculation formula is:

[0027] p=|δ t |+ε,

[0028] Among them, p is the priority; δ t is the time difference error; ε is a constant;

[0029] S13. Use importance sampling weights to correct the deviation caused by non-uniform sampling. The calculation formula of importance sampling weights is:

[0030]

[0031] Among them, w i is the importance sampling weight; N is the total number of samples in the experience replay pool; p i is the priority of sample i; β is the exponential bias term of importance sampling.

[0032] Preferably, the calculation formula for combining the optimistic and pessimistic strategy model updates in step S4 is:

[0033]

[0034] Among them, θ new is the updated model parameter; θ old is the model parameter before updating; η is the learning rate; is the inverse of the objective function J with respect to the parameter θ; online (θ old ) is the objective function based on online data; Joffline (θ old ) is the objective function based on offline data.

[0035] After adopting the above technical scheme, the present invention has the following beneficial effects: In the offline learning stage, the present invention adopts a method combining Prioritized Experience Replay (PER), Expert Imitation Learning (GAIL) and multi-head attention mechanism. This method can quickly initialize an efficient and robust policy model model by using high-quality samples in historical data, that is, by using high-quality samples and multi-head attention mechanism, the policy model model can learn and converge faster, which can lay a solid foundation for the online learning stage and provide an efficient starting point. In the online learning stage, the present invention uses the SAC reinforcement learning algorithm to iteratively optimize the policy model, refines the policy model through dynamic interaction with the real-time environment, learns to respond to resource changes, and continuously improves decision-making efficiency and stability. In the initial stage of the offline-to-online algorithm, the present invention adopts a pessimistic update policy model, gives priority to the high-quality characteristics of expert data, provides a stable and reliable foundation for model learning, and achieves rapid convergence; as the training proceeds, the agent gradually transitions to online learning, explores new data and experience through real-time interaction with the online environment, and makes full use of the exploration ability of the SAC reinforcement learning algorithm. Therefore, the present invention integrates the SAC reinforcement learning algorithm of online learning and the expert imitation learning technology of offline learning, and introduces the priority experience replay and multi-head attention mechanism. By dynamically updating the agent strategy model, the efficiency and accuracy of resource scheduling can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the process of the present invention;

[0037] Figure 2 It is a structural schematic diagram of offline training and online training of the present invention;

[0038] Figure 3 This is an experimental comparison chart of the present invention and other methods under cost indicators;

[0039] Figure 4 This is an experimental comparison diagram of the present invention and other methods under the average reward index;

[0040] Figure 5 This is an experimental comparison diagram of the present invention and other methods under load imbalance indicators. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0042] like Figures 1 to 5 As shown, an offline-to-online deep reinforcement learning method in a resource scheduling scenario is used to optimize resource scheduling, including the following steps:

[0043] S1. Obtain resource scheduling data and perform data preprocessing;

[0044] In step S1, the specific process of data preprocessing is:

[0045] S11. Use priority experience playback to select the most valuable samples for learning from historical data, and weight the samples according to the time difference error of experience. The calculation formula of time difference error is:

[0046] δ t =R t+1 +γV(S t+1 )-V(S t ),

[0047] Among them, δ t is the time difference error; R t+1 is the reward obtained at time step t+1; γ is the discount factor; V(S t+1 ) is state S t+1 The estimated value of S t+1 is the next state; V(S t ) is state S t The estimated value of S t is the current state;

[0048] S12. Set the priority to the absolute value of the time difference error. The specific calculation formula is:

[0049] p=|δ|+ε,

[0050] Among them, p is the priority; δ t is the time difference error; ε is a constant;

[0051] S13. Use importance sampling weights to correct the deviation caused by non-uniform sampling. The calculation formula of importance sampling weights is:

[0052]

[0053] Among them, w i is the importance sampling weight; N is the total number of samples in the experience replay pool; p i is the priority of sample i; β is the exponential bias term of importance sampling;

[0054] S2. Combine expert imitation learning and multi-head attention mechanism to perform offline training of the strategy model;

[0055] The specific process of step S2 is:

[0056] S21. Use expert imitation learning to initialize the strategy model. Expert imitation learning trains the strategy model by imitating the behavior of experts. The calculation formula is:

[0057]

[0058] Among them, L GAIL is the loss function of the GAIL algorithm; is the expected operation; θ (a|s) is the policy model; a is the action or decision performed by the agent in state s; s is the current state of the environment;

[0059] S22. Add a multi-head attention mechanism to the policy model network, and use the multi-head attention mechanism to process high-dimensional state space to focus on the most relevant environmental features at the moment;

[0060] S3, use the SAC reinforcement learning algorithm to iteratively optimize the policy model, refine the policy model through dynamic interaction with the real-time environment, learn to respond to resource changes, and perform online training of the policy model;

[0061] The specific process of step S3 is:

[0062] S31. Use the SAC reinforcement learning algorithm to optimize the strategy model by maximizing the return of entropy enhancement. The calculation formula is:

[0063]

[0064] Among them, L SAC is the strategy loss function of the SAC algorithm; is the expected operation; α is the entropy weight; π θ (a|s) is the strategy model; π θ A strategy for selecting actions for the agent; is the value function of the state-action pair; is the advantage function; a is the action or decision performed by the agent in state s; s is the current state of the environment;

[0065] S32. Update the value network, and the calculation formula is:

[0066]

[0067] Among them, L V is the loss function of the value network; For the expected operation; is the output of the value network; r is the immediate reward; γ is the discount factor; Predict the value of the next state s';

[0068] S4. Use the dynamic policy update module, combine optimistic and pessimistic policy updates, adjust the policy model parameters according to online feedback, and return the trained policy model network;

[0069] The calculation formula for combining the optimistic and pessimistic strategy model updates in step S4 is:

[0070]

[0071] Among them, θ new is the updated model parameter; θ old is the model parameter before updating; η is the learning rate; is the inverse of the objective function J with respect to the parameter θ; online (θ old ) is the objective function based on online data; J offline (θ old ) is the objective function based on offline data.

[0072] Performance Test:

[0073] The performance of the offline-to-online deep reinforcement learning method (abbreviated as O2O-DRL) in a resource scheduling scenario of the present invention is tested against five traditional online reinforcement learning algorithms: expert imitation learning (abbreviated as GAIL), deep deterministic policy gradient (abbreviated as DDPG), soft actor-critic reinforcement learning (abbreviated as SAC), double-delayed deep deterministic policy gradient (abbreviated as TD3) and proportional policy optimization (abbreviated as PPO). The test results are as follows: Figures 3 to 5 The experimental results show that the offline-to-online deep reinforcement learning method (abbreviated as O2O-DRL) in the resource scheduling scenario of the present invention can significantly reduce costs, obtain better benefits and a relatively stable convergence effect. Compared with traditional online reinforcement learning algorithms such as DDPG, SAC, TD3 and PPO, it has a faster convergence speed and better performance. Compared with the single offline reinforcement learning algorithm GAIL, the performance improvement is more obvious.

[0074] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An offline-to-online deep reinforcement learning method in a resource scheduling scenario, used to optimize resource scheduling, characterized in that: The following steps are involved: S1. Obtain resource scheduling data and perform data preprocessing; S2. Combine expert imitation learning and multi-head attention mechanism to perform offline training of the strategy model; The specific process of step S2 is: S21. Use expert imitation learning to initialize the strategy model. Expert imitation learning trains the strategy model by imitating the behavior of experts. The calculation formula is: Among them, L GAIL is the loss function of the GAIL algorithm; is the expected operation; θ (a|s) is the policy model; a is the action or decision performed by the agent in state s; s is the current state of the environment; S22. Add a multi-head attention mechanism to the policy model network, and use the multi-head attention mechanism to process high-dimensional state space to focus on the most relevant environmental features at the moment; S3, use the SAC reinforcement learning algorithm to iteratively optimize the policy model, refine the policy model through dynamic interaction with the real-time environment, learn to respond to resource changes, and perform online training of the policy model; The specific process of step S3 is: S31. Use the SAC reinforcement learning algorithm to optimize the strategy model by maximizing the return of entropy enhancement. The calculation formula is: Among them, L SAC is the strategy loss function of the SAC algorithm; is the expected operation; α is the entropy weight; π θ (a|s) is the strategy model; π θ A strategy for selecting actions for the agent; is the value function of the state-action pair; is the advantage function; a is the action or decision performed by the agent in state s; s is the current state of the environment; S32. Update the value network, and the calculation formula is: Among them, L V is the loss function of the value network; For the expected operation; is the output of the value network; r is the immediate reward; γ is the discount factor; Predict the value of the next state s'; S4. Use the dynamic policy update module, combine optimistic and pessimistic policy updates, adjust the policy model parameters according to online feedback, and return the trained policy model network.

2. The offline-to-online deep reinforcement learning method in a resource scheduling scenario according to claim 1, characterized in that: In step S1, the specific process of data preprocessing is: S11. Use priority experience playback to select the most valuable samples for learning from historical data, and weight the samples according to the time difference error of experience. The calculation formula of time difference error is: δ t =R t+1 +γV(S t+1 )-V(S t ), Among them, δ t is the time difference error; R t+1 is the reward obtained at time step t+1; γ is the discount factor; V(S t+1 ) is state S t+1 The estimated value of S t+1 is the next state; V(S t ) is state S t The estimated value of S t is the current state; S12. Set the priority to the absolute value of the time difference error. The specific calculation formula is: p=|δ t |+e, Among them, p is the priority; δ t is the time difference error; ε is a constant; S13. Use importance sampling weights to correct the deviation caused by non-uniform sampling. The calculation formula of importance sampling weights is: Among them, w i is the importance sampling weight; N is the total number of samples in the experience replay pool; p i is the priority of sample i; β is the exponential bias term of importance sampling.

3. The offline-to-online deep reinforcement learning method in a resource scheduling scenario according to claim 1, characterized in that: The calculation formula for combining optimistic and pessimistic strategy updates in step S4 is: Among them, θ new is the updated model parameter; θ old is the model parameter before updating; η is the learning rate; is the inverse of the objective function J with respect to the parameter θ; online (θ old ) is the objective function based on online data; J offline (θ old ) is the objective function based on offline data.

Images