An Efficient Deep Reinforcement Learning Algorithm for Combinatorial Optimization in Continuous Decision Spaces

Abstract

An efficient deep reinforcement learning algorithm for continuous decision space combinatorial optimization, comprising the following steps. Step 1: Model the problem as a sequential decision problem and set the definition of the deep reinforcement learning framework elements required to obtain the continuous decision combination optimization problem. Step 2: Through the definition of the deep reinforcement learning framework elements in Step 1, model the continuous decision space combinatorial optimization problem in the temporal task as a Markov decision process and obtain the training objective of the deep reinforcement learning. Step 3: Calculate the expected value of the training objective G(t) by using the probabilistic dynamic programming algorithm. Step 4: Obtain the optimal solution to the continuous decision combination optimization problem. The present invention can enable the agent to reduce the environmental interaction cost. Through the effective optimal action space search in the deep reinforcement learning of the present invention and the probabilistic dynamic programming to calculate the reward expectation, the work of the continuous decision space combinatorial optimization problem in the temporal task is solved. Based on the above, the present invention has good application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of deep reinforcement learning, and in particular to an efficient deep reinforcement learning algorithm for combinatorial optimization in a continuous decision space. Background Art

[0002] In the process of deep reinforcement learning, an agent (such as an intelligent robot) interacts with the environment all the time. After the agent obtains a certain state in the environment, it will use this state to output an action, which is also called a decision. Then this action will be executed in the environment, and the environment will output the next state and the reward brought by the current action according to the action taken by the agent. The purpose of the agent is to obtain as much reward from the environment as possible. Deep reinforcement learning combines the powerful representation ability of neural networks to fit the policy model and value model of the agent, and the ability to solve complex problems has been greatly improved. In recent years, it has made great progress in various intelligent decision-making problems and has become a rapidly developing branch in the field of artificial intelligence. Most of the combinatorial optimization problems involved in deep reinforcement learning involve the decision-making order, that is, the sequential decision-making problem. For example, for the TSP problem (Traveling Salesman Problem), it is to decide in what order to visit each city, and for the Job shop problem (Job Shop Scheduling Problem), it is to decide in what order to process workpieces on the machines. And the recurrent neural network in the deep neural network can just complete the mapping problem from one sequence to another sequence. Therefore, borrowing the recurrent neural network to directly solve the combinatorial optimization problem is completely a feasible solution. Another technical solution is to use reinforcement learning. Reinforcement learning is inherently used for sequential decision-making. Then the sequential decision-making problem in the combinatorial optimization problem can also be directly solved by reinforcement learning. The technical difficulty is how to define the state and the reward.

[0003] Although deep learning has strong perception capabilities, it lacks certain decision-making capabilities; while reinforcement learning has decision-making capabilities, it is helpless in dealing with perception problems. Therefore, combining the two and complementing each other's advantages can provide ideas for solving the perception and decision-making problems of complex systems; thus, the very popular and trendy deep reinforcement learning has emerged. Deep reinforcement learning combines the perception capabilities of deep learning and the decision-making capabilities of reinforcement learning, and is an artificial intelligence method that is closer to the human thinking mode. Early reinforcement learning developed from trial-and-error learning, simulating the learning behaviors of humans and animals; such algorithms are only suitable for dealing with simple decision-making problems, such as maze-solving, tic-tac-toe, etc. Modern RL (reinforcement learning) is a class of algorithms for solving optimal control problems, especially for sequential decision-making tasks in stochastic environments. In principle, it requires that the problem has two properties: one is the Markov (chain) property, and the other is that the reward and state are separable. This is the key to applying the Bellman (algorithm) principle. If the deep reinforcement learning method is used to solve the sequential decision-making combinatorial optimization problem in a stochastic environment, it usually requires a large number of interactions between the neural network model and the environment, which is extremely time-consuming and full of noise.

[0004] Currently, traditional deep reinforcement learning algorithms are difficult to quickly solve complex long-time series task problems. For example, the DQN algorithm is used to solve Atari2600 games. During the game-playing process, it can be found that the observations obtained by the agent are not independently and identically distributed. There is actually a very strong continuity between the previous frame and the next frame, and the data obtained is correlated time series data, which does not meet the independent and identically distributed requirement; the task scenario of this game is relatively simple, the decision-making time series is short, the decision-making space is small, and the complexity of the problems existing in the game is not high. If the DQN is applied to more complex time series tasks with a significant increase in decision-making difficulty, the effect will be significantly reduced. Summary of the Invention

[0005] In order to overcome the drawbacks that existing deep reinforcement learning algorithms, due to technical limitations, cannot effectively solve the combinatorial optimization problem of continuous decision-making spaces in time series tasks and the low efficiency of continuous decision-making in time series tasks, the present invention provides an efficient deep reinforcement learning algorithm for combinatorial optimization of continuous decision-making spaces, which models time series tasks by using Markov decision processes, separates state information into environmental states and decision states, and at the same time designs an effective reward expectation calculation method through probabilistic dynamic programming. The agent can more effectively collect environmental feedback in the learning paradigm by avoiding techniques based on trajectory sampling, which provides favorable technical support for the development of deep reinforcement learning algorithm technology.

[0006] The technical solution adopted by the present invention to solve its technical problems is:

[0007] An efficient deep reinforcement learning algorithm for combinatorial optimization in continuous decision spaces, characterized in that the algorithm process framework is as follows: A: The DRL agent takes limited time series data and combinatorial weight states as inputs and generates operations for weight adjustment, so that the reward return of the combinatorial optimization task changes accordingly; B: After the environment receives the corresponding actions, it generates the rewards corresponding to each policy. Then, using the rewards of multiple action trajectories, the parameters of the policy network are updated through gradient descent optimization. At the same time, the environment also feeds back the combinatorial state to the DRL agent for subsequent training processing; C: The DRL agent learns from the evaluations of multiple interactions with the environment so that the policy network can generate favorable behaviors, thereby obtaining the maximum reward return of the combinatorial optimization task. Specifically, it includes the following steps: Step 1: For the modeling of time series tasks in deep reinforcement learning, the problem is modeled as a sequential decision problem. To ensure the analysis of a large number of interactions between the neural network model and the environment, the definition of the elements of the deep reinforcement learning framework for the continuous decision combinatorial optimization problem needs to be set. The specific element definition data includes states, actions, and rewards; Step 2: Through the definition of the elements of the deep reinforcement learning framework in Step 1, the combinatorial optimization problem in the continuous decision space of the time series task is modeled as a Markov decision process, and the training objective of deep reinforcement learning is obtained; Step 3: Calculate the expected value of the training objective G(t) using the probabilistic dynamic programming algorithm; Step 4: Optimize the network parameters through gradient updates, so as to gradually obtain the optimal expected return, and finally obtain the optimal solution to the continuous decision combinatorial optimization problem.

[0008] Further, in A above, the policy network of the agent consists of a recurrent neural network layer and a fully connected layer. At the same time, a long short-term memory network is used as the deep feature representation learning module of the policy network.

[0009] Further, in Step 1 above, the state space formula is: The formula for the combinatorial optimization weight state involved in the state space is: The formula for the combinatorial weights at the same moment is:

[0010]

[0011] Further, in Step 1 above, in the action space, the next state of the combinatorial weight involved is expressed as the following formula:

[0012]

[0013] Further, in Step 1 above, the reward function is expressed as the following formula: The reward function for the task combination of the set N involved is expressed as the following formula:

[0014]

[0015] Further, in the second step, the training objective of the deep reinforcement learning is defined as follows:

[0016]

[0017] The value function V(t) involved is expressed as:

[0018]

[0019] Further, in the third step, it involves calculating the The formula for the expectation J(θ) is as follows:

[0020]

[0021] Further, in the fourth step, the derived expected gradient estimator is involved, and the summation formula at time t is as follows:

[0022]

[0023] The beneficial effects of the present invention are as follows: By directly inferring the policy weights in the action space oriented by the maximum cumulative reward, the present invention uses the Markov decision process to model the sequential tasks. At the same time, by separating the state information into the environmental state and the decision state, the agent can reduce the environmental interaction cost. An effective reward expectation calculation method is also designed through probabilistic dynamic programming, enabling the agent to more effectively collect environmental feedback in the learning paradigm proposed by the present invention by avoiding techniques based on trajectory sampling. Through the effective optimal action space search in the deep reinforcement learning of the present invention and the calculation of the reward expectation by probabilistic dynamic programming, the work of solving the continuous decision space combinatorial optimization problem in sequential tasks is completed. Based on the above, the present invention has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below in conjunction with the drawings and embodiments.

[0025] Figure 1 is a block diagram of the architecture of the present invention.

[0026] Figure 2 is a flowchart of an efficient deep reinforcement learning algorithm for continuous decision space combinatorial optimization of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0027] Figure 1 、 2As shown, an efficient deep reinforcement learning algorithm for combinatorial optimization in a continuous decision space. The algorithm flow framework is as follows: A: The DRL agent (agent of deep reinforcement learning) takes limited time series data and combinatorial weight states as inputs and generates operations for weight adjustment, thereby causing corresponding changes in the reward return of the combinatorial optimization task. Specifically, the policy network of the agent consists of a recurrent neural network layer and a fully connected layer. At the same time, the present invention uses a long short-term memory network (LSTM) as the deep feature representation learning module of the policy network. B: After receiving the corresponding action, the environment generates the reward corresponding to each policy. Then, using the rewards of multiple action trajectories, the parameters of the policy network are updated through gradient descent optimization; at the same time, the environment also feeds back the combinatorial state to the DRL agent for subsequent training processing. C: The DRL agent learns from the evaluations of multiple interactions with the environment so that the policy network can generate favorable behaviors, thereby obtaining the maximum reward return for the combinatorial optimization task.

[0028] Figure 1 , 2As shown in [figure reference], an efficient deep reinforcement learning algorithm for continuous decision space combinatorial optimization specifically includes four steps. Step 1: In practical applications, by modeling time-series tasks in deep reinforcement learning, the problem usually needs to be modeled as a sequential decision-making problem. At the same time, the neural network model needs to analyze a large number of interactions with the environment. Therefore, first, it is necessary to obtain the element definitions of the deep reinforcement learning framework for the continuous decision-making combinatorial optimization problem (the element definitions include states, actions, and rewards). The role of Step 1 is to lay the foundation for Step 2. Since this invention uses deep reinforcement learning, it is necessary to analyze and summarize the problem to abstract the element definitions corresponding to the problem, that is, the state space, action space, and reward function. At the same time, it is necessary to ensure the rationality and correctness of these elements (in deep reinforcement learning, the policy network needs to randomly sample an action in the action space according to the state corresponding to the state space at the current moment, interact with the environment through this action, and at the same time, the environment evaluates this action according to the defined reward function and returns the corresponding reward value reward. The policy network receives the evaluation reward of the environment for this action, which is the interaction process. At the same time, the state is updated at the next moment, and then the above operations are repeated, which realizes a process of continuous decision-making and continuous interaction). In this step, considering the problem combination of N sub-tasks within the T time period, these problems are all optimization problems of time-series tasks. There are N such identical problems within the T time period. The core problem of this invention is to combine these N sub-tasks together and optimize them through combinatorial optimization. It is necessary to assign a dynamic weight to each sub-task (this weight will change dynamically, that is, the proportion of this sub-task in the whole combination, and the sum of the proportions is 1. The detailed concept will be explained accordingly in the following state space). Each sub-task will generate a reward return during the optimization process of deep reinforcement learning. The task objective of combinatorial optimization is to maximize the total reward return.

[0029] Figure 1 , 2 As shown in [figure reference], in Step 1, the state space (the state information in deep reinforcement learning represents the environmental information perceived by the agent and the changes brought about by its own actions; the state information is the basis for the agent to make decisions and evaluate its long-term benefits, and the quality of the state design directly determines whether the deep reinforcement learning algorithm can converge, the convergence speed, and the final performance. The state space is the set that contains all the designed state information) is represented by the following formula: In the formula describes the agent's perception of the environment, including the characteristics of time-series task data. The weight state of the above data combination optimization is denoted as: In the formula, w(t) is the combination weight in the sequential task combination optimization problem. The combination weight is initialized to be equal at the beginning of time stamp t; at the same time, the sum of the combination weight is always 1, that is, R N The meaning is to express Is an N-dimensional numerical vector; in the environmental modeling of the present invention, it is assumed that Does not change according to different operations, but The agent's strategic actions will change accordingly.

[0030] Figure 1 、 2 As shown in , in step 1, the action space (The action space of the continuous decision-making combination optimization problem of the technical solution of the present invention includes the set of all actions that can be sampled in the task). In this step, the action a(t) corresponds to the combined weight state determined at the beginning and end of time t. arrive changes due to And w(t) satisfies the constraints in the above state space, so this step uses a softmax function to map as the last layer of the policy network. After passing the softmax function, it is ensured that the sum of the weight ratios of all subtasks at each moment is 1, while satisfying the constraints of the LSTM in the policy network. Then subtract the mapping value E(h L (t)) is taken as the action a(t) at time t, so for task i, the next state of the combined weights can be expressed as follows:

[0031]

[0032] Figure 1 、 2 As shown in , in step 1, the reward function is marked as r(t) (at each moment, the agent selects an action in the current state according to its policy, and the environment responds to these actions, moves to the new state, and generates a reward signal (reward), which is usually a numerical value, and is usually calculated through the reward function): The reward function is marked with the following formula The reward function is an incentive and guide for the agent to learn favorable strategies. The environment uses the reward function to evaluate the action a(t) obtained by the above strategy network and generates the reward value corresponding to the action. Then, the reward function of the task combination of set N can be obtained as: Among them, the cumulative sum of the reward discounts is called the return, which is the goal that the agent wants to maximize during the action selection process.

[0033] Figure 1 、 2 As shown in 2 , in step 2, through the definition of the element actions and rewards in the deep reinforcement learning framework in step 1, the combined optimization problem of the continuous decision-making space in the time-series task is modeled as a Markov decision process, and the training objective of deep reinforcement learning is obtained. Specifically, it is defined as the following formula:

[0034] ,

[0035] The above formula is a variant of the loop reward function of the Markov decision process (MDP), where γ is the discount factor; for simplicity, this step uses G(t) to represent uses R(t) to represent Then the above formula can be reformulated as the reward function of the Markov decision process, and the value function V(t) can be expressed as the following formula:

[0036]

[0037] 。In this step, the reason for saying that the problem in this paper needs to be modeled as a Markov decision process is that in reinforcement learning, during the interaction between the agent and the environment, after the agent obtains the state of the environment, it will take an action and return the taken action to the environment; after the environment obtains the action of the agent, it will enter the next state, and at the same time generate a reward and send the next state back to the agent. This interaction process can be represented by the Markov decision process, so the Markov decision process is a basic framework in reinforcement learning. Therefore, without this step, it will be impossible to use deep reinforcement learning technology to solve the optimization problem.

[0038] Figure 1 、 2 As shown in 2 , in step 3, the expected value of the training objective G(t) is calculated by using the probabilistic dynamic programming algorithm. Specifically, given that the state at time t starts as In this step, the action policy is defined as This is the output value of the policy neural network, where θ is the parameter of the policy neural network, and then the probabilistic dynamic programming algorithm is used to calculate the expectation J(θ) in the above second step, and the formula is as follows:

[0039] The above formula is a variant of the Bellman expectation backup function, where the state transition probability P(s) is one, through the formula As a basis for transferring to In step 3, an effective method for calculating the expected reward is designed through probabilistic dynamic programming, enabling the agent to more effectively collect environmental feedback in the learning paradigm proposed in the present invention by avoiding techniques based on trajectory sampling. In traditional methods, calculating the expected reward requires continuous trajectory sampling to fit the expected reward value, which is very time-consuming and greatly reduces the algorithm efficiency. Since the present invention calculates the expected reward value through probabilistic dynamic programming, it demonstrates the high efficiency of the technical solution of the present invention and greatly improves the efficiency of the deep reinforcement learning algorithm.

[0040] Figure 1 , 2 As shown in In step 3, the network parameters are optimized through gradient update to gradually obtain the optimal expected return, and finally the optimal solution of the continuous decision-making combination optimization problem is obtained. The main objective of this step is to train a policy network that can select the optimal parameter θ to achieve an optimal expected return E((G(t)|π, θ), that is, the expected value of G(t) mentioned above. Then, the gradient can be updated through a function to update the parameters, so that the expectation of G(t) becomes larger and further improves the overall return G(t). η in the following formula is the learning rate: Since the above action space transformation is an indicator function, an expected gradient estimator can be derived, and at the same time, the summation formula at time t:

[0041] In the formula,

[0042] is the reward observation value, is the return value of the action trajectory, and this value may have a large variance. Therefore, in this step, the baseline is subtracted from the policy gradient to reduce the variance. The role of step 4 is that the technical solution of the present invention can derive an expected gradient estimator and optimize the network parameters of the policy neural network through the gradient update optimization algorithm. Finally, the optimal expected value, that is, the optimal solution of this problem, is obtained through continuous optimization.

[0043] Figure 1 , 2As shown, the technical solution of the present invention uses the widely used LSTM network as the policy network (Policy Network). In particular, the LSTM is provided with a chronological state, and all weights are trained using the same structure in the LSTM layer and share weights during the parameter optimization process. Generally speaking, the technical solution proposed by the present invention separates the environment into a "static" environmental state and a "dynamic" weight state, which is different from the traditional method (weight assignment after prediction). At the same time, by designing a probabilistic dynamic programming method that does not require trajectory sampling, the most inefficient process in deep reinforcement learning, that is, calculating the expected reward of agent interaction, is overcome, thus greatly improving the efficiency of the deep reinforcement learning algorithm and achieving a good optimization effect.

[0044] Those skilled in the art should understand that although this specification is described according to embodiments, not every embodiment only contains an independent technical solution. This narrative way of the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in the embodiments can also be appropriately combined to form other embodiments that can be understood by those skilled in the art. Therefore, the protection scope of the present invention is defined by the claims.

Claims

1. An efficient deep reinforcement learning algorithm for combinatorial optimization in a continuous decision space, characterized in that, The algorithm flow framework is as follows. A: The DRL agent takes a finite time series data and a combined weight state as inputs and generates an operation for weight adjustment, so that the reward return of the combinatorial optimization task changes accordingly. B: After the environment receives the corresponding action, it generates the reward for each policy. Then, using the rewards of multiple action trajectories, it updates the parameters of the policy network through gradient descent optimization. At the same time, the environment also feeds back the combined state to the DRL agent for subsequent training processing. C: The DRL agent learns from the evaluations of multiple interactions with the environment so that the policy network can generate favorable behaviors to obtain the maximum reward return of the combinatorial optimization task. It specifically includes the following steps. Step 1: For the modeling of time series tasks in deep reinforcement learning, the problem is modeled as a sequential decision-making problem. To ensure the analysis of a large number of interactions between the neural network model and the environment, the definition of the elements of the deep reinforcement learning framework for the continuous decision-making combinatorial optimization problem needs to be set. The specific element definition data includes state, action, and reward. Step 2: Through the definition of the elements of the deep reinforcement learning framework in Step 1, the continuous decision-making space combinatorial optimization problem in the time series task is modeled as a Markov decision process, and the training objective of deep reinforcement learning is obtained. Step 3: Calculate the expected value of the training objective G(t) using the probabilistic dynamic programming algorithm. Step 4: Optimize the network parameters through gradient updates to gradually obtain the optimal expected return, and finally obtain the optimal solution to the continuous decision-making combinatorial optimization problem.

2. The efficient deep reinforcement learning algorithm for continuous decision space combinatorial optimization according to claim 1, characterized in that In A, the policy network of the agent consists of a recurrent neural network layer and a fully connected layer. At the same time, a long short-term memory network is used as the deep feature representation learning module of the policy network.

3. An efficient deep reinforcement learning algorithm for continuous decision space combinatorial optimization according to claim 1, characterized in that, In Step 1, the state space formula is: The combinatorial optimization weight state formula related to the state space is: The combinatorial weight formula at the same moment involved is:

4. An efficient deep reinforcement learning algorithm for continuous decision space combinatorial optimization according to claim 1, characterized in that In Step 1, in the action space, the next state of the combined weight involved is expressed as the following formula:

5. An efficient deep reinforcement learning algorithm for combinatorial optimization in a continuous decision space according to claim 1, characterized in that, In step one, the reward function is expressed as the following formula: The reward function for the task combination of the set N involved is expressed as the following formula:

6. An efficient deep reinforcement learning algorithm for continuous decision space combinatorial optimization according to claim 1, characterized in that, In Step 2, the training objective of deep reinforcement learning is defined as follows: The value function V(t) involved is expressed as:

7. An efficient deep reinforcement learning algorithm for continuous decision space combinatorial optimization according to claim 1, characterized in that, Step 3 involves calculating the expectation J(θ) formula in Step 2 as follows: 。 8. An efficient deep reinforcement learning algorithm for continuous decision space combinatorial optimization according to claim 1, characterized in that, In Step 4, the derived expected gradient estimator is involved, and the summation formula at time t is as follows:

Images